Probably falls into the same or similar category as calling them dumb, ones open a time an acceptable word to call them, but not really today. You know unless you are looking to insult them for being mute.
It derives from Latin infans where "in-" is a negation prefix and "fans" is the present participle form of "for", which translates to "to speak".
So an infant is a non-speaker (too small to speak).
But my opener was of course a joke, where I purpously misunderstood what "fant" is derived of, by claiming that "fant" must be the opposite of a child, thus an adult.
There are tons of Latin words in the English language and many of them only survived in English in their compounded form (e.g. "in-fant", where no other version of the actual verb in there survived, except the negated form).
Often the parts of these Latin root words have no meaning at all anymore in English, so that people don't notice that they are actually using compound words and also the original meaning of the word is forgotten.
Not a lot of people would associate "infant" with "hearing".