I don't think that's totally accurate. I think it's more that they don't know how a mask works, and they think that you're deliberately being fake to fit in.
If masking wasn't a thing, it would probably be decent advice. Problem is 99% of us have no idea we're masking until way later, if ever.