I was thinking Ryzen 7 3000 series. The Ryzen 9's are a bit too high of a price point for me, and most of my workloads are single core dependent. The Performance gulf between the 5000 ryzen 7 and 3000 is not too much, so I can cheap out if I wanted to keep this system going.
That said, if I want a proper upgrade for my workload, I need to go Intel with an AMD GPU. But I defiantly don't have the budget for that.
For single threaded workloads, you might find the ryzen 5000 series to be more of a jump than it initially seems since the IPC was improved (20-30% in some cases).
In Handbrake, what do you have set for the video encoder? I have an AMD GPU and I've found that setting the encoder to one of the AMD options eased some of the load on the CPU.