Wait, I thought downhill skiing was the risky one. What are the risks of cross country skiing? I would have expected that to be pretty safe, aside from exhaustion and tree wells.
There are still small hills that you can wipe out on and the bindings don't release in CC so your legs have the potential of getting pretty messed up. Same with hitting a tree (and helmets aren't really a thing in CC either).
Um, the risk is like quadrupled from a groomed slope. It depends on what you do, but I know lots of cross country skiers that hike up and ski downhill. Avalanches, exposure, wildlife, what you said etc.
There are really tame routes that ski resorts make, but you're still driving out there, parking, etc.
What you're describing sounds more like AT skiing to me? Granted I have gone up to where people were AT skiing before with my cross-country skis, but I have back-country ones and skins.