In a first, researchers have demonstrated that different variants of SARS-CoV-2, the virus behind Covid-19, have the ability to infect the central nervous system.
...and it describes a mode of viral spread which does not depend on cell surface receptors, and is used by multiple viruses from different viral genera. The mechanism is forming a cytoplasm tunnel to physically reach the adjascent cell.
To infect cells on a different host, or elsewhere in the body with a ready-made viral particle, COVID needs the target cell to express the ACE2 receptor... but to infect the direct neighbour of an infected cell with unencapsulated viral RNA, it does not require the target to express ACE2.
As far as I can reason one o'clock at night:
it is infecting and replicating in neurons
since patients often lose sense of smell or taste, it definitely kills some neurons
since the senses typically recover (although altered), it does not kill all neurons
As for why most people are OK - I think because neurons are not an environment which COVID has adapted to "work with". This mode of infection may be slow (the immune system catches up and deploys antibodies). It may be unreliable (cells may stop forming cytoplasm projections when they sense that they are compromised, or other cells may start rejecting such projections) and there may be defenses against it (stress / death signals from one cell may trigger universal antiviral defense mechanisms in adjascent cells).