Using address to authenticate

I'm currently using monero addresses as the sole authentication method for a custodial service, similar to how mullvad VPN has a single account number to authenticate. My understanding is that these are unique, and impossible to guess. For a custodial service, this makes withdrawing user funds trivial as well.

Can anyone tell me why this is a bad idea?