I have an Egyptian coworker (immigration laws, yay) and I'd not call his skin color brown (as in "brown bear" brown). More like a rather well tanned local person. I'd have a hard time calling him non-white, unless in comparison with an Irish person or something…