Voldy was more nuanced than that. The movies don't really cover it that well, but his big motivator was to be special. He used control of the ministry and domination of the muggles as a carrot for his supporters. But at a personal level, he didn't care about power and control as end goals.
He wanted to be the most renown, feared and admired person in the world. He wanted to be so special that everyone just did whatever he wanted. Not that he wanted to rule the world - but whoever did rule it answered to his every whim.
It's a very petty and vain motivator. But, I can actually believe it. Actually ruling the world sounds like too much work.