Is there a way to keep Linux responsive when at ~100% CPU usage?
One big difference that I've noticed between Windows and Linux is that Windows does a much better job ensuring that the system stays responsive even under heavy load.
For instance, I often need to compile Rust code. Anyone who writes Rust knows that the Rust compiler is very good at using all your cores and all the CPU time it can get its hands on (which is good, you want it to compile as fast as possible after all). But that means that for a time while my Rust code is compiling, I will be maxing out all my CPU cores at 100% usage.
When this happens on Windows, I've never really noticed. I can use my web browser or my code editor just fine while the code compiles, so I've never really thought about it.
However, on Linux when all my cores reach 100%, I start to notice it. It seems like every window I have open starts to lag and I get stuttering as the programs struggle to get a little bit of CPU that's left. My web browser starts lagging with whole seconds of no response and my editor behaves the same. Even my KDE Plasma desktop environment starts lagging.
I suppose Windows must be doing something clever to somehow prioritize user-facing GUI applications even in the face of extreme CPU starvation, while Linux doesn't seem to do a similar thing (or doesn't do it as well).
Is this an inherent problem of Linux at the moment or can I do something to improve this? I'm on Kubuntu 24.04 if it matters. Also, I don't believe it is a memory or I/O problem as my memory is sitting at around 60% usage when it happens with 0% swap usage, while my CPU sits at basically 100% on all cores. I've also tried disabling swap and it doesn't seem to make a difference.
EDIT: Tried nice -n +19, still lags my other programs.
EDIT 2: Tried installing the Liquorix kernel, which is supposedly better for this kinda thing. I dunno if it's placebo but stuff feels a bit snappier now? My mouse feels more responsive. Again, dunno if it's placebo. But anyways, I tried compiling again and it still lags my other stuff.
The Linux kernel uses the CPU default scheduler, CFS, a mode that tries to be fair to all processes at the same time - both foreground and background - for high throughput. Abstractly think "they never know what you intend to do" so it's sort of middle of the road as a default - every CPU cycle of every process gets a fair tick of work unless they've been intentionally nice'd or whatnot. People who need realtime work (classic use is for audio engineers who need near-zero latency in their hardware inputs like a MIDI sequencer, but also embedded hardware uses realtime a lot) reconfigure their system(s) to that to that need; for desktop-priority users there are ways to alter the CFS scheduler to help maintain desktop responsiveness.
Have a look to Github projects such as this one to learn how and what to tweak - not that you need to necessarily use this but it's a good point to start understanding how the mojo works and what you can do even on your own with a few sysctl tweaks to get a better desktop experience while your rust code is compiling in the background. https://github.com/igo95862/cfs-zen-tweaks (in this project you're looking at the set-cfs-zen-tweaks.sh file and what it's tweaking in /proc so you can get hints on where you research goals should lead - most of these can be set with a sysctl)
There's a lot to learn about this so I hope this gets you started down the right path on searches for more information to get the exact solution/recipe which works for you.
I would agree, and would bring awareness of ionice into the conversation for the readers - it can help control I/O priority to your block devices in the case of write-heavy workloads, possibly compiler artifacts etc.
I feel like if Linux wants to be a serious desktop OS contender, this needs to "just work" without having to look into all these custom solutions. If there is a desktop environment with windows and such, that obviously is intended to always stay responsive. Assuming no intentions makes more sense for a server environment.
One of my biggest frustrations with Linux. You are right. If I have something that works out of the box on windows but requires hours of research on Linux to get working correctly, it's not an incentive to learn the complexities of Linux, it's an incentive to ditch it. I'm a hobbyist when it comes to Linux but I also have work to do. I can't be constantly ducking around with the OS when I have things to build.
What do you even mean as serious contender? I've been using Linux for almost 15 years without an issue on CPU, and I've used it almost only on very modest machines. I feel we're not getting your whole story here.
On the other hand whenever I had to do something IO intensive on windows it would always crawl in these machines
I have the 6.6 on my desktop, and I guess the compilations don't freeze my media anymore, though I have little experience with it as of now, need more testing.
The Linux kernel uses the CPU default scheduler, CFS,
Linux 6.6 (which recently landed on Debian) changed the scheduled to EEVDF, which is pretty widely criticized for poor tuning.
100% busy which means the scheduler is doing good job. If the CPU was idle and compilation was slow, than we would look into task scheduling and scheduling of blocking operations.
Responsiveness for typical everyday usage is one of the main scenarios kernels like Zen/Liquorix and their out of the box scheduler configurations are meant to improve, and in my experience they help a lot. Maybe give them a go sometime!
Edit: For added context, I remember Zen significantly improving responsiveness under heavy loads such as the one OP is experiencing back when I was experimenting with some particularly computationally intensive tasks
It's more of a workaround than a solution. I don't want to have to do this for every intensive program I run. The desktop should just be responsive without any configuration.
The System76 scheduler helps to tune for better desktop responsiveness under high load: https://github.com/pop-os/system76-scheduler
I think if you use Pop!OS this may be set up out-of-the-box.
I distro hop occasionally but always find myself coming back to popos. There are so many quality of life improvements that seem small but make all the difference.
Lots of bad answers here. Obviously the kernel should schedule the UI to be responsive even under high load. That’s doable; just prioritise running those over batch jobs. That’s a perfectly valid demand to have on your system.
This is one of the cases where Linux shows its history as a large shared unix system and its focus as a server OS; if the desktop is just a program like any other, who’s to say it should have more priority than Rust?
I’ve also run into this problem. I never found a solution for this, but I think one of those fancy new schedulers might work, or at least is worth a shot. I’d appreciate hearing about it if it does work for you!
Hopefully in a while there are separate desktop-oriented schedulers for the desktop distros (and ideally also better OOM handlers), but that seems to be a few years away maybe.
In the short term you may have some success in adjusting the priority of Rust with nice, an incomprehensibly named tool to adjust the priority of your processes. High numbers = low priority (the task is “nicer” to the system). You run it like this: nice -n5 cargo build.
Obviously the kernel should schedule the UI to be responsive even under high load.
Obviously... to you.
This is one of the cases where Linux shows its history as a large shared unix system and its focus as a server OS; if the desktop is just a program like any other,
No. I'm sorry but if you are logged in with a desktop environment, obviously the UI of that desktop needs to stay responsive at all times, also under heavy load. If you don't care about such a basic requirement, you could run the system without a desktop or you could tweak it yourself. But the default should be that a desktop is prioritized and input from users is responded to as quickly as possible.
This whole "Linux shouldn't assume anything"-attitude is not helpful. It harms Linux's potential as a replacement for Windows and macOS and also just harms its UX. Linux cannot ever truly replace Windows and macOS if it doesn't start thinking about these basic UX guarantees, like a responsive desktop.
This is one of the cases where Linux shows its history as a large shared unix system and its focus as a server OS; if the desktop is just a program like any other,
Exactly.
You say that like it's a good thing; it is not. The desktop is not a program like any other, it is much more important that the desktop keeps being responsive than most other programs in the general case. Of course, you should have the ability to customize that but for the default and the general case, desktop responsiveness needs to be prioritized.
I meant, obviously in the sense that Windows and macOS both apparently already do this and that it’s a desirable property to have, not that it’s technically easy.
"The kernel runs out of time to solve the NP-complete scheduling problem in time."
More responsiveness requires more context-switching, which then subtracts from the available total CPU bandwidth.
There is a point where the task scheduler and CPUs get so overloaded that a non-RT kernel can no longer guarantee timed events.
So, web browsing is basically poison for the task scheduler under high load. Unless you reserve some CPU bandwidth (with cgroups, etc.) beforehand for the foreground task.
Since SMT threads also aren't real cores (about ~0.4 - 0.7 of an actual core), putting 16 tasks on a 16/8 machine is only going to slow down the execution of all other tasks on the shared cores.
I usually leave one CPU thread for "housekeeping" if I need to do something else. If I don't, some random task is going to be very pleased by not having to share a core. That "spare" CPU thread will be running literally everything else, so it may get saturated by the kernel tasks alone.
nice +5 is more of a suggestion to "please run this task with a worse latency on a contended CPU.".
(I think I should benchmark make -j15 vs. make -j16 to see what the difference is)
That's all fine, but as I said, Windows seems to handle this situation without a hitch. Why can Windows do it when Linux can't?
Also, it sounds like you suggest there is a tradeoff between bandwidth and responsiveness. That sounds reasonable. But shouldn't Linux then allow me to easily decide where I want that tradeoff to lie? Currently I only have workarounds. Why isn't there some setting somewhere to say "Yes, please prioritise responsiveness even if it reduces bandwidth a little bit". And that probably ought to be the default setting. I don't think a responsive UI should be questioned - that should just be a given.
You're right of course. I think the issue is that Linux doesn't care about the UI. As far as it is concerned GUI is just another program. That's the same reason you don't have things like ctrl-alt-del on Linux.
Windows lies to you. The only way they don't get this problem is that they are reserving some CPU bandwidth for the UI beforehand. Which explains the 1-2% y-cruncher worse results on windows.
I agree that UI should always take priority. I shouldn't have to do anything to guarantee this.
I have HZ_1000, tickless kernel with nohz_full set up. This all has a throughput/bandwidth cost (about 2%) in exchange for better responsiveness by default.
But this is not enough, because the short burst UI tasks need near-zero wake-up latency... By the time the task scheduler has done its re-balancing the UI task is already sleeping/halted again, and this cycle repeats. So the nice/priorities don't work very well for UI tasks. Only way a UI task can run immediately is if it can preempt something or if the system has a somewhat idle CPU to put it on.
The kernel doesn't know any better which tasks are like this. The on-going EEVDF, sched_ext scheduler projects attempt to improve the situation. (EEVDF should allow specifying the desired latency, while sched_ext will likely allow tuning the latency automatically)
Sounds like Kubuntu's fault to me. If they provide the desktop environment, shouldn't they be the ones making it play nice with the Linux scheduler? Linux is configurable enough to support real-time scheduling.
FWIW I run NixOS and I've never experienced lag while compiling Rust code.
I have a worrying feeling that if I opened a bug for the KDE desktop about this, they'd just say it's a problem of the scheduler and that's the kernel so it's out of their hands. But maybe I should try?
If you compile on windows server the same problem happens. The server is basically gone. So there seems to be some special scheduler configuration in windows client os.
I wonder if Linux should also provide server and desktop variants like Windows does, with different scheduler settings and such. The use cases are quite different after all, it's kinda weird they use the same settings.
it's typically up to the distribution to configure things like that, and many Linux distributions do come in both server and desktop or workstation variants like Ubuntu desktop vs Ubuntu server, or RHEL server vs RHEL Workstation
I can't say how well they tune these things as I haven't ran them personally, but they do exist.
Is your browser Firefox?
What kind of storage devices do you have? NVMe?
Did you check with tools like iotop to see if something is going on IO wise?
You assumed that the problem is caused by the CPU being utilized at 100%.
This may not be the case.
A lot of us don't run a DE at all. I myself use Awesome WM.
For non-tilers, Openbox with some toolbar would be the ideal setup.
I mention this because we (non-DE users) would have no experience with some funky stuff like a possible KDE indexer running in the background killing IO performance and thrashing buffered/cached memory.
Also, some of us run firefox with eatmydata because we hate fsync 🤨
Neither KDE nor Gnome is peak Desktop Linux experience.
Ubuntu and its flavors is not peak distro experience either.
If you want to try Desktop Linux for real, you will need to dip your toes a little bit deeper.
Yes Firefox, yes NVMe. No, there is no IO happening and again, sitting at relatively low memory usage. I was not running anything else than the compiler, my editor and Firefox. I'm fairly confident the CPU usage is the culprit as memory usage is not severely affected and disk usage by the compiler should be pretty minimal (and I don't see how disk usage would make Firefox slow if there's still plenty of RAM available).
Neither KDE nor Gnome is peak Desktop Linux experience.
Ubuntu and its flavors is not peak distro experience either.
If you want to try Desktop Linux for real, you will need to dip your toes a little bit deeper.
I've heard much of the opposite - KDE is touted as an easy-to-use desktop and Ubuntu is largely a popular "just works" distro. And honestly that has been my primary experience. Mostly everything works, but there are some hiccups here and there like the problem I posted about in this thread.
You say it doesn't top out on memory, so you don't need the -p MemoryLimit=500M parameter. Set your compiler CPUQuota to maybe 80%, or whatever you can work out with trial and error.
I don't really want to limit the Rust compiler. If I leave my computer running while I take a break, I don't want it to artificially throttle the compiler. I just want user input and responsiveness of open windows to take priority over the compiler.
All the comments here are great. One other suggestion I didn't see: use chrt to start the build process with the sched_batch policy. It's lower than sched_other, which most processes will be in, so the compilation processes should be bumped off the CPU for virtually everyone else
If that is the case, Linux will never be a viable desktop OS alternative.
Either that needs to change or distributions targeting desktop needs to do it. Maybe we need desktop and server variants of Linux. It kinda makes sense as these use cases are quite different.
EDIT: I'm curious about the down votes. Do people really believe that it benefits Linux to deprioritise user experience in this way? Do you really think Linux will become an actual commonplace OS if it keeps focusing on "performance" instead of UX?
Linux is already a popular and viable desktop OS - for its target audience.
The downvote comes from you implying people cannot dev in Linux when its the platform of choice for this workload.
Now surely the user experience could be polished, but advanced users are at this point used to the workflow, and basic ones will stick to Windows out of inertia no matter what. Therefore the incentive for improving this kind of things is extremely low.
"Desktop" Linux exists in this state for decades. Who cares? Maybe we won't have consumer desktops as a niche soon. Existing users are fine with that. Don't say you are waiting that Linux will become "a viable desktop OS alternative" in next few years.
It's also not about "desktop and sever variants". Desktop Linux is either conservative or underresourced. Conservatives will told you that you are wrong and there is no issue. And they are major Linux zealots. For the other side someone need to write code and do system design, and there are not many of people for that. So, it's better not to expect a solution anytime soon, if you are not planning to work on it by yourself.
While I ultimately think your solution is to use a different scheduler, and that the most useful responses you've gotten have been about that; and that I agree with your response that Linux distros should really be tuning the scheduler for the UI by default and let developers and server runners take the burden of tuning differently for their workloads... all that said, I can't let this comment on your post go by:
which is good, you want it to compile as fast as possible after all
If fast compile times are your priority, you're using the wrong programming language. One of Go's fundamental principles is fast compile times; even with add-on caching tooling in other languages, Go remains one of the fastest-compiling statically compiled, strongly typed programming languages available. I will not install Haskell programs unless they're precompiled bin packages, that's a hard rule. I will only reluctantly install Rust packages, and will always choose bins if available. But I'll pick a -git Go package without hesitation, because they build crazy fast.
Anyway, I hope you find the scheduler of your dreams and live happily ever after.
There’s no free lunch after all. Go’s quick compilation also means the language is very simple, which means all the complexity shifts to the program’s code.
P.S. Since it wasn't mentioned already, look up cgroups.
Back when I had a humble laptop (pre-Rust), using nice and co. didn't help much. Custom schedulers come with their own stability and worst-case-scenario baggage. cgroups should give you supported and well-tested tunable kernel-level resource usage control.
TLDR you might be interested in the rust-based scheduler one of the Canonical Devs released as a PoC. Seemed to be designed similar to your needs of keeping the system (particularly games) responsive even whilst running heavy tasks like kernel compilations. You can swap out schedulers at run time on Linux iirc?
My work windoz machine clogged up quite much recompiling large projects (GB s of C/C++ code), so I set it to use 19/20 "cores". Worked okayish but was not some snappy experience IMO (64GB RAM & SSD).
Actually, I've experienced the opposite. I find Windows lagging much more often than Linux when compiling something, especially since Linux switched to the EEVDF scheduler. The most important factor that influences lag on both systems seems to be the power profile though. If I set my power profile to save battery, the system lags from time to time but if I set it to performance it basically never happens (on GNOME you can change that in the quick menu, not sure about KDE). It might be that your Windows is simply tuned more towards performance by default at the cost of higher power consumption.
EDIT: Tried nice -n +19, still lags my other programs.
yea, this is wrong way of doing things. You should have better results with CPU-pinning. Increasing priority for YOUR threads that interact all the time with disk io, memory caches and display IO is the wrong end of the stick. You still need to display compilation progress, warnings, access IO.
There's no way of knowing why your system is so slow without profiling it first. Taking any advice from here or elsewhere without telling us first what your machine is doing is missing the point. You need to find out what the problem is and report it at the source.
Hmm, I can't say that I've ever noticed this. I have a 3950x 16-core CPU and I often do video re-encoding with ffmpeg on all cores, and occasionally compile software on all cores too. I don't notice it in the GUI's responsiveness at all.
Are you absolutely sure it's not I/O related? A compile is usually doing a lot of random IO as well. What kind of drive are you running this on? Is it the same drive as your home directory is on?
Way back when I still had a much weaker 4-core CPU I had issues with window and mouse lagging when running certain heavy jobs as well, and it turned out that using ionice helped me a lot more than using nice.
I also remember that fairly recently there was a KDE/plasma stutter bug due to it reading from ~/.cache constantly. Brodie Robertson talked about it: https://www.youtube.com/watch?v=sCoioLCT5_o
Yeah I think the philosophy of Linux is to not assume what you are going to be use it for. Why should Linux know where your priorities are better than you?
Some people want to run their rustc, ffmpeg or whatever intensive program and don't mind getting a coffee while that happens, or it's running on a non-user facing server anyway, to ensure that the process happens as soon as technically possible. Mind you that your case is not an "average usecase" either, not everyone is a developer that does compilation tasks.
So you've got a point that the defaults could be improved for the desktop software developer user or somehow made more easily configurable. As suggested downthread, try the nice command, an optimized scheduler or kernel, or pick a distribution equipped with that kind of kernel by default. The beauty of Linux is that there are many ways to solve a problem, and with varying levels of effort you can get things to pretty much exactly where you want them, rather than some crowdpleasing default.
Why should Linux know where your priorities are better than you?
Because a responsive desktop is basic good UX that should never ever be questioned. That should at least be the default and if you don't want your desktop to have special priority, then you can configure it yourself.
pick a distribution equipped with that kind of kernel by default.
I'm running Kubuntu, an official variant of Ubuntu which is very much a "just works" kind of distribution - yet this doesn't just work.
There's a setting in windows to change the priority management, most people never see it.
By default it's configured for user responsiveness, but you can set it for service responsiveness.
Though this is nothing like the process priority management in Linux, it's one setting, that frankly I've never seen it make any difference. At least with Linux you can configure all sorts of priority management, on the fly no less.
Even with a server, you'd still want the UI to have priority. God knows when you do have to remote in, it's because you gotta fix something, and odds are the server is gonna be misbehavin' already.
Even with a server, you'd still want the UI to have priority. God knows when you do have to remote in, it's because you gotta fix something, and odds are the server is gonna be misbehavin' already.
That's a fair point.
I still contend that regularly using processes that hog every available cpu cycle it can get its hands on was not a common enough desktop use case that warranted changing the defaults. It should be up to the user to configure to their needs. That said, a toggle switch like the hidden windows setting you described would be nice.
It sounds like the issue is that the Rust compiler uses 100% of your CPU capacity. Is there any command line option for it that throttles the amount of cpu it will use? This doesn't sound like an issue that you should be tackling at the OS level. Maybe you could wrap the compiler in a docker container and use resource constraints?
It sounds like the issue is that the Rust compiler uses 100% of your CPU capacity.
No, I definitely want it to use as many resources it can get. I just want the desktop and the windows I interact with to have priority over the compiler, so that the compiler doesn't steal CPU time from those programs.
Observe the cpu usage of nice +5 job: it's ~1/10 of the nice +0 job. End one of the tasks and the remaining jumps back to 100%.
Nice'ing doesn't limit the max allowed cpu bandwidth of a task; it only matters when there is contention for that bandwidth, like running two tasks on the same CPU thread. To me, this sounds exactly what you want: run at full tilt when there is no contention.