That's a combination of too simple/short in your sentences, mixed with too specific jargon with no clarification. It's dumb as hell that people don't know stuff like what a server is, but if they don't you have to abstract it more.
My go to is some form of:
I'm in IT, I do systems administration. I help keep all the things behind the scenes working so that everyone's stuff works at my workplace. Less of making your email work, more of making everyone's email work.
Obviously I work with a hell of a lot more than just email. I'm mostly scripting out custom automation jobs to bridge gaps in the integrations between different systems. But like you said, keep it simple.