What they're really saying is "But you don't look regarded."

I have no idea why anyone would ever think "hey you don't match my negative stereotypes about x" would be an ok and sensible thing to say, really with regards to anything, not just autism, sexuality etc.