if by westerners you mean english speakers, then yes, it's known to be one of the more difficult ones. it's ultimately subjective, but what people find hard about mandarin is 1.the writing system 2.tones
what also doesn't help is definitely lack of exposure, chinese popular media isn't very popular in the west