State of Wayland gaming with Nvidia RTX? Is it good, or stick with Xorg?

As an AMD user on Wayland who primarily uses Steam/Proton, I haven't had any Wayland-related issues with gaming at all. This isn't surprising, since the compositor used by Steam Deck (Gamescope) is also Wayland-based.

FYI: If you ever do have compositor-related issues (X11 OR Wayland), you can almost always fix the issue by running the game in a nested Gamescope instance by editing the game's launch command.

545 will have better wayland support. For now X11 is a better choice i guess. (i don't own an nvidia gpu)

Last I looked it still wasn't working well enough, so I decided to stick with Xorg. But that was a few months ago.

It works fine (runs with minor graphical gliches) for me but i have heard that its a complete mess for others. I'd recommend checking how well your particular system works.

If it works well then it might be a good idea to switch since wayland does have some pros comared to X11.

Depending on your desktop environment, it might actually be very easy and safe to test by yourself. I am using KDE Plasma, and after installing wayland (I installed the plasma-wayland-session package on Arch), I am able to select wayland instead of X11 just before logging to my session. In my case it's completely broken though, I have a lot of freezes, glitches, etc. that make wayland completely unusable and I have the same graphic card as you. However, because it is as easy to switch back to X11 as to select X11 in the dropbox before logging to my session, it is not a big deal.

One thing to note with sway and other wlroots based compositors - tearing is not a merged feature in wlroots, so you get pretty much always-on vsync, which means increased input lag which can matter in games like Osu.

I would personally stick with X11 for now. Last time I tried Wayland it worked but all sorts of just strange issues started popping up everywhere. Nothing big but annoying and frustrating to fix. This was using KDE Plasma.

With recent drivers and recent enough DEs it is usable to a certain extent. There are some known issues documented in the release notes of each version. Here's for 535. TL; DR the major blockers are: (1) Variable Refresh Rate doesn't work for some older cards; (2) GAMMA_LUT is not implemented (no night light) and (3) Nested X11 clients have synchronisation issues that might result in some flickering or dupicate frames, it's more noticeable on things that refresh slowly, although much better recently. Also (4) NvFBC capture doesn't work for X11 applications within a Wayland session which might or might not be important for some people.

I'm using it with Plasma; it's OK, no major concerns but my setup is pretty basic.

This blocks nVidia on Wayland until we get enough applications away from X11. There are some people who think of this as no big deal, or there are a lot of use cases where people don't run into this, but I've seen enough applications stutter, drop frames, or mess up entirely under Xwayland because of the lack of implicit sync.

The 535 series gets us much farther in a lot of ways on Wayland, but we still have so far yet to go due to this issue. We're still not there in regards to VRR, either, as nVidia has stated VRR (gsync) isn't reported by the nvidia-drm module, but there has been a public statement that they are trying to get it into the 545 release.

As far as VRR itself is concerned, Mutter itself under version 44 needs a rebase to support it, but that's a Mutter issue. Plasma users are going to have a better experience than Gnome users at this point, which I find very odd to say, but that's more of a testament to the Plasma devs being absolutely awesome on another level lately, and is not to detract from the good work the Gnome project has been doing.

TL:DR We're not quite there, even still, for a very small list of reasons now as opposed to a large list just a year ago.

Nvidia or not, stick with X11 until you have a reason to use Wayland.

I need Wayland for my packaging work, and one of my devices requires Wayland so I use it for that and it's a decent experience, but Wayland still has drawbacks (weaker automation API, some programs still don't work) so I wouldn't recommend switching off X11 unless you have a reason.