I mean, most idioms are silly if you take them at face value.
To me, "easier said than done" is a response to an unrealistic idea that carries the meaning of "I don't think that'll work"/"that's too much"/"etc.
If I told a friend "hey let's make spaghetti for dinner" and they replied "easier said than done" I'd be like "...okay, you're technically correct, but I want spaghetti so I'm gonna do that anyways"
But if I told a friend "hey let's solve world hunger today" and they replied "easier said than done" I'd be like "yeah, I know...maybe we can go help out at a soup kitchen?"