I'll go against what most comments said and recommend DirectX 11. Yes, DXVK will translate it to Vulkan anyway, but Larian's own Vulkan implementation is definitely less stable compared to DX11.
I've experienced multiple crashes during simple things like opening the character sheets using the tab key, or crafting alchemy potions. I never had a single crash using DX11. I used Fedora 39/40 and openSUSE Tumbleweed, so the kernels were fairly recent. Radeon 7800 XT GPU.
I had the same experience under Windows 10 (before I switched to Linux), Vulkan has smoother frametimes but DX11 is more stable.
YMMV, this is just my experience from almost 400 hours played so far.
They borked the Vulkan Renderer somewhere around Patch...3 I think? It used to be so performant, but now it runs only at 40-60fps on my Nvidia 3090 compared to the DX11 renderer which can render at 80-120 T_T
DirectX is a Windows thing, so you'll just have those calls translated to Vulkan under the hood (DXVK). You'll probably get better performance from just setting it to Vulkan directly.
Edit: As some others say, for BG3 specifically, DXVK does a really good job. My personal experience is that "the best" option is very patch dependent. At launch Vulkan was best, then after a few patches did DXVK ran better, but personally I'm back to straight Vulkan, for no other reason than wanting to be a +1 in the statistics.
OpenGL is a bit like Vulkan, but discontinued since... 2014, with a single update since then. It was actually stopped because Vulkan seemed better, and both API's were maintained by the same organisation.
In general it's more likely to work on older devices, but would be less performant than Vulkan.
To expand - DirectX is a proprietary Windows solution. Any time you pick it on Linux, it will run through a translation layer
OpenGL/Vulkan are cross-platform
OpenGL is to DirectX 11 as Vulkan is to DirectX 12
Microsoft kept the same branding, but also followed in Vulkans/Metals footsteps of using lower level calls to the hardware.
This makes the graphics drivers simpler, and can be way more performant because the CPU doesn’t have to do as much
Dxvk is usually better, but using vulkan is the better strategic move, you'll increase their vulkan stats and provide QA. Good native vulkan support will beat dxvk every time.
Up until a few months ago, Vulkan was very unstable on BG3. It’s been fine for a while though. I haven’t made performance or smoothness comparisons though, I just default to Vulkan and it’s been fine.
Directx 11 in this case, played bg3 on Linux and that was the only option that worked, and it did work quite well.
As for when to use one or the other, just check protondb. People usually leave what they played on, they even leave some useful launch commands or solutions to issues that could possibly arise, so it's always worth a look.