It's justified because this is how professional pilots are expected to operate. Passengers are not supposed to deal with the logistics of their transportation.
For that matter, it's also how car-sharing is expected to operate. Imagine you took an Uber to your hotel, and the driver said "Hey, my next fare is two blocks from your hotel. So how about I drop you off there, it's just a five minute walk the rest of the way. Plus, it would save me from driving with no passenger for a few minutes, thus wasting fuel."
I think the vast majority of us would consider this unprofessional and respond, "No thanks, drive me all the way to my hotel". Which is why Uber drivers don't actually discuss their next fare with their current passenger.
I'm sure they have limos. Their limo driver is told, "Meet A at B street and bring them to XXX airport by Y pm."
It would be very unprofessional for the pilot to call the driver at the last minute and say, "Change of plans, now you bring them to ZZZ airport, it's a little further away which means you'll have to leave T minutes sooner". Depending on their schedule, A might not even be able to leave T minutes sooner.
We live in a society where staying on schedule is valued way more than reducing carbon emissions. Another thought experiment: you are 30 minutes late to work. Your boss demands an explanation. You say, "I have a very good explanation, today I decided to bike to work instead of drive". That's probably not going to be good enough. You are expected to emit CO2 whenever it's necessary to stay on schedule.
It's not just the rich. It's everyone who takes a car to work instead of bike, it's everyone who takes an Uber or even a bus instead of walking a mile, it's everyone who demands that stores open on time and packages are delivered on time.
It's hypocritical to hold Musk to a standard that nobody else meets.