Anyone drive? How do you find it? I am absolutely terrified after my first lesson

Sorry in advance for the wall of text! Some background...

Enter me, someone who usually gets around by bicycle or public transport. I'm about 200 miles away from anyone close (besides my partner) and the trains, while fast and convenient, are expensive and quite limited at some times of the day.

So, as a solution, I decided I'll pick up some driving lessons so I can drive to friends with a rental or my own vehicle, on much more flexible terms. Since I had some existing experience in various driving simulators (almost 200h combined), I decided why not?

Now, about the lesson.

The instructor was absolutely amazing, got me up to speed with all kinds of things I wasn't familiar with, like adjusting the mirrors, wheel and stuff.

The car is a stick shift/manual, as that's the norm here. To be honest, changing gears was the easiest part - it felt really familiar because of the simulators. However I really struggled with how much information you need to take in from around you during the actual driving, literally had to try so hard to not make my mind wander for even a second, because I'd lose track of the environment and stuff. It was dark too so that made things a little challenging.

I'd say a major stress point too is the fact that i'm operating a 2000kg SUV, not an agile 20kg bicycle.

On one hand I'm hoping things improve with time, on the other I really wish we had good, affordable public transport to begin with.

What are your thoughts?