The biggest reason to knock off working on vacation or after hours is that it creates a false expectation on the the workload. If you can't get it done during regular office hours, than that means your company needs more people or a process improvement.
If you are working these extra untracked hours, you are the problem. If you get rewarded for doing so, your company is toxic and will only expect more as you move up the ladder.
I'll put my hand up and say that I will wish I worked harder. My job is simple and i work remote. If I was willing to work harder, I could either move up in the company or move to a competitor. That would get me more money. More money would help me to pay rent on a nicer place to live. And then with the new nice place, I could get the rest of my head in order. So I will absolutely go to my doom wishing I worked harder, put in more hours, and showed a high degree of dedication.
That's what everybody thinks before they are on their deathbed, not so much when they're actually there... The peace you're looking for will most likely always be 2 steps ahead of you