There's a lot of historical and cultural baggage with that term in the US. Specifically concerning its use in reference to house slaves and then freed slaves during and after our reconstruction.
I'm from the US and I've never heard of it being super problematic, though I'd consider it a bit dehumanizing to refer to anyone by their job rather than by their name.