I want to learn more about file systems from the practical point of view so I know what to expect, how to approach them and what experience positive or negative you had / have.
I found this wikipedia's comparison but I want your hands-on views.
For now my mental list is
NTFS - for some reason TVs on USB love these and also Windows + Linux can read and write this
Ext4 - solid fs with journaling but Linux specific
Btrfs - some modern fs with snapshot capability, Linux specific
xfs - servers really like these as they are performant, Linux specific
FAT32 - limited but recognizable everywhere
exFAT - like FAT32 but less recognizable and less limited
Ext4 for system disks because it's default in OS installers and it works well. I typically use it on top of LVMRAID (LVM-managed mdraid) for redundancy and expansion flexibility.
ZFS for storage because it's got data integrity verification, trivial setup, flexible redundancy topologies, free snapshots, blazing fast replication, easy expansion, incredible flexibility in separating data and performance tuning within the same filesystem. I'd be looking into setting up ZFS on root for my next machine. Among other things that would enable trivial and blazing fast backup of the system while it's running - as simple as syncoid -r rpool backup-server:machine4-rpool.
I'd be looking into setting up ZFS on root for my next machine
I too was on the path of adventure once but then the kernel module hasn't been built after the upgrade. Also btrfs offers some nice features for root especially that zfs doesn't have.
Every year I buy a couple ~$5 USB drives and plug them into my jbod machine in a software raid1. At this point there's about a hundred in long array of daisy chained USB hubs.
Each drive is formatted with fat32 and added to an LVM. Don't judge my ghetto NAS.
It also has self healing, no "partitions", high performance, compression, smart drive redundancy without RAID holes, encryption, deduplication and an extremery intelligent cache called ARC
ZFS is completely ridiculous. It's like someone actually sat down to design an intelligent filesystem instead of making a slightly improved version of what's already out there.
I like ext4 because it's easy. If anything breaks, ANY live USB can fix it. I use fat32 for my removeable drives, because anything can read it. I don't use journalling for anything manually, but I imagine it's useful when my disk crashes because I let my laptop die
Which RAID?
I need to read about SystemD encrypt hooks because I know nothing.
Also why not LVM? Is btrfs more flexible in partitioning when you want to extend it or shrink it? I heard that you can merge "partitions" on 2 different disks so they are visisble under one mount point.
Btrfs can mostly fo everything you would normaly use LVN or raid for natively.
Btrfs raid0 lets you combine any number of differently sized drives into one (just without the speed boost of traditional raid0 because with flexible drive sizes data is not symmetrical striped). And btrfs raid1 keeps every data duplicated, again with flexible number and sizes of drive (also with metadata on every drive).
The sytemd hooks (instead of the traditional busybox ones) then manage the one other task you use LVM for: unlocking multiple partitons (for example multiple raid partitons and swap) with just one password. Because the systemd encrypt function tries unlooking all luks partitions it finds with the first password provided and only asks for passwords for each partition if that doesn't work.
PS: btrfs subvolumes are already flexible in size and don't need predefined sizes. So the only things that need to be created separately are non-btrfs stuff like the efi system partition or a physical swap (which you can also skip by using a swap file instead of a partition).
ZFS is completely different than XFS. XFS is like a better (different?) ext4. ZFS is an error-checking software raid COW filesystem that does snapshots and can have multiple replicas, both local and remote. It uses zvols and datastores. Think btrfs on steroids and with a working raid subsystem.
It's got a weird semi-closed license because Oracle is involved but it's never been enforced and at this point is in such widespread use in large and small enterprises that it would be impossible to enforce.
Been running BTRFS since 2010. Ext2/3/4 before that.
Using it for CoW, de-duplication, compression. My home file server has had a long-lived array of mis-matched devices. Started at 4x2TB, through 6x4TB and now 2x18+4TB. I just move up a size whenever a disk fails.
Well, snapshots, too. I just consider them to be a special case of de-duplication.
I had an issue when I ran out of space during conversion between RAID profiles a few years back. I didn't lose any data, but I couldn't get the array to mount (and stay) read-write.
I'm sure it's great and all, but the hassle of having a filesystem that's not in the kernel is a no-starter for me. Maybe one of those fancy NAS-distros that are based on some *BSD.
ZFS all the things. On my workstations, I wipe / on every boot except for the files that I specify, and I backup /home to my NAS on ZFS and I backup my NAS snapshots to Backblaze.
If you install and then uninstall something, it will almost certainly leave logs, configurations and other garbage in places you don't expect. Next time you want to use it, it isn't the clean install you expected
BTRFS in FS-managed RAID configuration for automatic self-healing and snapshots for instant automated backups (though I keep a traditional backup too for protection against bugs and user error).
Storage is cheap compared to how much I value my data. BTRFS has very good support on Linux, integration with some backup tools, and I really want to use a FS that has full data checksums to make sure the data stays correct at rest. I like that I don’t have to use equal sized drives and can use whatever I have available, though I would appreciate a better read distribution model rather than the current where it just chooses a random drive to read from when multiple copies are available.
Disadvantages include difficulty accessing from Windows in my experience, less than stellar performance on HDDs, not very space efficient for small files systems because of the bulky metadata, and some uncommon RAID types don’t work correctly and will eat your data. I also don’t recommend it for use over USB because many such devices don’t correctly implement sync, and this is very important to stay on the correct transaction number and prevent file system inconsistencies. If I had to boot from USB, I wouldn’t pick BTRFS.
I don’t think exFAT or FAT32 offer POSIX permissions. I’m not sure if you could have a root file system there.
I like BTRFS and it's features but sadly Debian doesn't have a preset for it in it's installer so the only way to use it is to manually partition and I absolutely suck at that.
Sorry if this is a dumb question, but have you tried using gParted? GUI, new-user friendly, easy to visualize your system, I've used it for over a decade on multiple devices...
Actually, I was using gparted yesterday! I was trying to format a USB drive to exFAT but the option was greyed out, so either gparted doesn't have support for formatting in exFAT or I needed an optional dependency.
nope, it works really well, for more than a year now, this is my work PC using 8h/day, I'm using MX23 AHS version. Directly in the setup you can select encryption and btrfs volume etc. btrfs is pretty stable.
Edit: reasons added in because I can't read the post title
OpenBSD laptop: ffs2, vfat for efi system partition
Why: Contrary to popular belief, OpenBSD does not support zfs. The only other filesystem options are msdos (fat family), and ext2fs (mostly for Linux compatibility as far as I can tell, filesystem is experimental and lacks a bunch of features according to the manpage). Makes ffs2 the only sane option.
Why: I prefer not having journaling on flash memory. This hasn't bitten me in the ass too hard yet, and even when it does I can usually get around system files being lost with integrity tools. Maybe I'll dabble with f2fs some day, but I'll need to read about its features and shortcomings compared to ext2.
Alpine Linux VM: ext4
Why: Would have installed as ext2 as well, but I forgot
I prefer not using journaling filesystems on flash memory, I haven't had any major data integrity issues yet because of it. I would have made the Alpine fs ext2 as well, but I guess I missed it during install. I think you can just disable journaling in ext4 anyways, so if I care enough I'll just do that.
Btrfs, for the compression and CoW. I've been using it since a couple years. It seems stable for my use. I need to fully wrap my head around how snapshots work, though.
You mentioned CoW. I’m really taking advantage of this because I have multiple Wine prefixes that have lots of duplicate data. I want to give every application it’s own prefix, and my underlying file system allows me to duplicate the blocks so the prefixes are basically free where before it’s several hundred megabytes just to make a new prefix.
Honestly I saw btrfs in the arch install guide and read about it because I thought the name sounded funny. I used it until I distro hopped to NixOS couldn't figure out how to install it with btrfs, so I'm back on ext4.
Maybe I'll give it another try next hop, which is likely soon since Qt theming seems impossible on Nix. :/
I am now all-in on bcachefs. I don't like btrfs, cause you still sometimes read about people loosing their data. I know that might happen with bcachefs too since it's early days still but fuck it. I like the risk.
Filesystem level compression and encryption are so nice to have.
I don't like btrfs, cause you still sometimes read about people loosing their data.
That was only on RAID setups. So if you have only a singular disk, as opposed to an array, you're fine. And that issue has been fixed for a while now anyways.
I've been running btrfs on my laptop's root partition for well over a year now and it's fine.
I get the feeling I’ll become a bcachefs fan for those reasons in the future (I tested it on a spare laptop as soon as 6.7 got into Debian Testing), but for now, I use a mix of ext4 and btrfs, as bcachefs-tools isn’t in Testing. It is trivial to apt-pin, but I try not to make FrankenDebian a regular thing.
I have a feeling that they’ll iron it out and Bcachefs will be an option in Trixie by the tome it hits stable, if still with a /boot partition considering the slow state of Grub support.
I think Linux users must have some touch of masochism.
Also this might interest you: https://www.phoronix.com/review/linux-611-filesystems
But me too, I've just installed Cosmic DE PopOS 24.04 for my main laptop.
NTFS support is pretty solid on Linux these days, but just so you know, never use it as a root partition.
I have generally used ext4. There's ways to massage it to mount on Windows, as with btrfs. Ext4 is very likely what you should do if you're installing Linux for the first time, as it has had decades of testing and is rather battle-tested
I recently did my first btrfs install. For now, I've had no issues. Of course, some could happen, but I've generally heard btrfs is fine these days. One of its cool things is native compression support, although I forgot to enable it when I did that install.
I've never used XFS.
FAT32 should be rarely used these days due to file size limits and file name limits. The only place where it should still be used is for your EFI partition.
Now exFAT really isn't that unrecognizable. It's supported by pretty much every operating system these days. It's definitely not for root partitions, but should be your default for flash drives and portable hard drives.
On another note, I recently tried Bcachefs on Debian Testing on a random old Chromebook. It is still in development, and not all distros support it yet, but I liked what I saw from my limited experience. It also supports snapshots, and unlike btrfs, has native encryption. For now, just ignore it, but like many in this post have said, keep an eye out for it.
As for ZFS, I've never tried it. The main caveat is due to licensing incompatibility, it is not in the standard Linux kernel and you have to do some special stuff.
I agree with everythinf but exFAT, some devices expect either FAT32 or NTFS. I had this issue when I wanted to play totally acquired big mkv movie through USB and because of that FAT32 wasn't an option so I went with exFAT. Not visible but apperantly it liked NTFS. It was the LG TV, my parents have 2 and same issue on both.
My pleasure. The LG problem is unfortunate. Most other devices tend to support exFAT, but LG is an exception, albeit a very big one due to its pervasiveness as a brand. I do have an LG TV, but an older one that’s getting annoying to the point it’s tempting to throw a Roku behind it.
Also, do you have a laptop with HDMI? That could also be a solution.
I use BTRFS on my Artix system, Ext4 on my Librem 5, Ext4 on my Devuan laptop and Ext4 on my Pinebook Pro. Basically when given the choice in the installer I choose BTRFS but if the installer doesn't let me pick I don't care enough to manually partition. I have had no negative experiences with any file system luckily so I just roll with whatever.
I use ext4 for all boot drives and root filesystems. Anything really important goes on a ZFS array. And for my Linux isos, I use a drive with ext4 + snapraid. The parity drive has xfs because ext4 has a 16tb file size limit.
Got rid of anything NTFS as it was unreliable and slow on Linux.
ext4 on all hard disks, but my installs are all several years old at this point, and I might choose differently if I were starting over from scratch. The boot partition on the ancient laptop might actually be ext2; I don't remember and it's certainly old enough that that might still have been preferred Gentoo procedure when I first set it up. Removable media might be ext3, ext4, or vfat, depending on compatibility needs and how long ago I formatted it. If I buy an SD card or USB stick that turns out to be preformatted in exFAT, I reformat it before use to ensure everything can read it.
They're all solidly reliable filesystems (well, except for the vfat), but perhaps not the most featureful.
Yeah same here, everything is ext4 'cause it's always worked and has never given me any troubles. But next time I have to reinstall I am tempted to give Btrfs a go.
Btrfs on my laptop with openSUSE, mainly because it's default, but also for its snapshot capabilities.
Whatever file system my default Raspberry Pi installation uses (probably Ext4).
NTFS on my main computer With Windows 10, because... well... I don't really have any other choice, although I know there's some kind of 3rd party Btrfs driver for Windows as well and you can ever have boot partition formatted as Btrfs, but I think it's still experimental.
@Psyhackological
Work stations all run Ext4.
Main server: Ext4 on main partition, ZFS RAIDZ2 on the data.
Secondary server: BTRFS on main, BTRFS RAID1 on data.
If BTRFS could natively encrypt and had stable RAID6, I'd be using it probably on everything.
ExFAT is the LCD filesystem for flash sticks. FAT32 is the filesystem that you have to use for devices designed back when Microsoft was awful about Exfat licensing.
Everywhere else, Btrfs. If Oracle didn't poison-pill ZFS licensing and it was common on Linux, I would be using that instead. Basically, taking it on faith that a drive didn't fuck up your data is crazy. The most basic responsibility for a filesystem should be ensuring that "the files come out exactly the same as when they went in".
ZFS on my file server. I'd use it on my laptops and desktops too (and have done when I was using Xubuntu) but I've switched toFedora which doesn't come with a way to easily install with ZFS and I don't feel like jumping through hoops to get it done. And I can't stand btrfs. I don't know what it is about it, but I just don't like it.
ZFS for nearly everything plus ZFSBootMenu EFI on root data pools. Get a bad upgrade? No problem, boot a previous snapshot (auto created with a pacman hook), which I had to do recently when 6.6.39 LTS kernel had a bug. Snapshots are also great when doing things such as upgrading postgres, hass, Plex, etc.
I've been using ZFS now for a few years for all my data drives/pools but I haven't gotten brave enough to boot from it yet. Snapshotting a system drive would be really handy.
I use bcachefs for my games, I like that it lets me have multiple disks with redundant data copies, plus ssd caching of frequently accessed files, this fs is linux specific for now as far as I know, and is still experimental. I use ext4 for everything else, and FAT32 for flash drives.
In my use it has been pretty stable so far with 7 disks participating (3 caching SSDs, 4 mechanical disks, with 3 copies of metadata and 2 of data), but I'm not using the more experimental features like erasure coding, I will note the on-disk-format has changed twice since it has been in the kernel, and it hasnt been there long, but it has succeeded both on-disk-format upgrades without obvious data loss, and it recently got self healing for some checksum errors, Id say its probably ready for use if the data is backed up, replaceable, or can be gone without (so for me games are all I have that fits this). Otherwise I would use caution if you use it, but I am very optimistic about the future of the FS, as Kent Overstreet (the creator) has taken a lot of care with it.
Btrfs, ZFS and ext4. My servers use ZFS, my client machines mostly use btrfs and I have a sprinkling of ext4 partitions for specific workloads. I'm all in on CoW filesystems for snapshots, send receive, transparent compression and reflinks. I like btrfs on client machines and SBCs because it's easily available (baked into the kernel) and doesn't require maintaining dkms or holding kernel versions until ZFS supports them and because snapshot handling and other filesystem admin tasks are simple and straightforward. I run ZFS wherever data integrity is important, eg: storage servers and backup targets, but largely prefer working with btrfs.
It's all Ext4, but I run SnapRAID on top of that on my data drives. I'm sure lots of people would tell me I should use ZFS/BTRFS instead, but I'm used to SnapRAID, and I like the idea if something goes wrong, I won't lose all my data.
exFAT external "archive" drive (easy to connect to Windows machines if ever I need my backup in someone else's windows machine in an emergency and such)
Idk, exFAT is meant to be compatible with more stuff than NTFS. If I ever decide to hook this HDD up to a console or smartTV to play the videos I have on it, it is guaranteed to work.
We use btrfs for the / partition and xfs for any data partitions. Has served us well, the snapshot feature saves us some valuable time when an update goes awry.
The main distribution we use has it like that by default and our (admittedly rudimentary) benchmarks haven't shown much of a performance difference versus ext4 so we kept to the default.
Yep, got Timeshift hooked up to make a snapshot each time I update my system and I can boot into them via GRUB. Haven't needed that so far, thankfully, but it's there just in case.
I've been basically using btrfs on a lot of my disks because of the features it has.
Before I switched to a borg based system, my backups partition used btrfs for compression.
My main OS disk is btrfs so I can use timeshift snapshots, which are really worth checking out if you tinker with your system a lot.
I have two more btrfs partitions software raid0'd together for my steam library, nix store and other big but loosable things.
And my main home folder uses btrfs because I think the checksumming thing it does is more reliable for error detection, and cow is more fault tollerant on power failure?
... And I now fell like I'm one of those people with an over engineered storage solution. I just never get rid of old ssds or hard disks!
ext4 because I value my data and don't want to lose it. I used to mess about with ZFS for mass storage but it's a university course to learn how to use and have decent performance.
I used to use XFS, but ext4 caught up.
And I used to use XFS... on something other than Linux.
xfs has reflinks. That means you can copy huge wodges of data nearly for free on one filesystem. For backup systems this is a killer feature. Veeam rolling up incremental backups into the last full happens in seconds because pointers to blocks are juggled around rather than the data blocks themselves.
xfs has been around for a very, very long time. I use it for larger filesystems eg Nextcloud, Zoneminder and the like (and Veeam backup repos that are not object storage). I use ext4 by default.
pfSense boxes - zfs because the alternative is ufs.
RPi - OverlayFS (with ext4 and tmpfs) gets you a generally read only filesystem with changes held in RAM. Ideal for kiosks, appliances and keeping memory sticks alive.
Windows - NTFS, it works well and has streams and there aren't many other options (ReFS is a bit new but it does have reflinks)
Every photocopy machine I've come across that accept USB sticks do not support exFAT, so what I would do with my USB stick is to split it into two partitions, one FAT32 and the rest exFAT.
Most of my drives are EXT4, but I started using BTRFS a couple years ago and will be using it on all new installs from now on. I really like being able to make snapshots and compression reduces the install size quite a bit.
ext4, but the btrfs activity visible in the kernel changelog has slowed down recently after a long period of many bug fixes, so maybe I'll give it a try next time.
I use Btrfs for my root partition to be able to rollback if something goes wrong after update.
XFS: in all other cases, since I hate the lost+found directory on ext4. Although I don't think there's any significant difference between ext4 and xfs in performance and reliability.
Basically, I just followed this tutorial for my EndeavourOS installations. It's as easy as choosing an older entry in GRUB. Fedora offers something similar by default, and I think Tumbleweed does too.
Moreover I'm now playing with Arkane Linux (https://arkanelinux.org/), immutable flavour of Arch, it features another magic with btrfs and rollbacks without snapshots and GRUB
Depends. Slower desktop machines XFS.
Standard desktop XFS, if it has a smaller SSD, Btrfs.
Home server ext4/XFS + ZFS. Generic servers at work ext4/XFS, backup/storage servers ZFS.
Database server, experiment with ZFS with compression enabled - ratio 2:1, but encountered problems (probably a bad HBA model), standard ext4/XFS.
Hosts with virtualization, small server - XFS, big server - ZFS (technically a ZVOL).
I don't dual boot, I just have some other Windows machines that I use rarely for Windows-only software that require an external connection, like Odin for Samsung devices.
Well, as far as I know, BTRFS and ZFS are the recommended file systems for NAS's. They have self-healing capabilities so I can be slightly more sure that my data does not get corrupted over time.
I've got Btrfs on my desktop for the OS drive cuz that was what Fedora recommended when I was installing it. It took a bit of effort to get snapshots working properly, but other than that, I've had no issues with it at all over the past year. I've got an exFAT drive and an NTFS drive in there that are kind of leftovers from using Windows. I've been thinking about reformatting the exFAT drive to ext4 or something, since all it really does is store games, and having the ability to symlink to it would be nice.
I've got a TrueNAS machine as well and that uses ZFS for pretty much everything.
I wish I'd actually chosen a file system instead of just letting window's at the time default to NTFS for external drives.
Moving from Windows to Debian; NTFS has been nothing but a headache. I've actually had to setup a windows machine to serve that drive pool via SAMBA as Linux just won't play nicely with it.
I think btrfs was the default the last time I installed Bazzite, but I don't really know anything about it so I switched it to ext4. I understand the snapshot ability is nice with rolling release distros, though.
It'd been ages since I'd used FAT32 for anything until I made a Debian live USB when I was setting up my pi-hole on an old Core2Duo recently. It would only boot on FAT32 for reasons I probably once knew. 😆
NTFS was an improvement over the FATs what with the journaling, security, file streams, etc. I use it wherever I still use Windows (work).
Most of my general purpose USB flash drives use exFAT. I like not having to worry about eject/unmount.
NTFS feels rock solid if you use only Windows and extremely janky if you dual-boot. Linux currently can't really fix NTFS volumes and thus won't mount them if they're inconsistent.
As it happens, they're inconsistent all the time. I've had an NTFS volume become dirty after booting into Windows and then shutting down. Not a problem for Windows but Linux wouldn't touch the volume until I'd booted into Windows at least once.
I finally decided to use a storage upgrade to move most drives to Btrfs save for the Windows system volume and a shared data partition that's now on ExFAT because it's good enough for it.
By default, windows does "Fast Boot" which doesn't make booting any faster, but does have the benefit of leaving the volume in a mounted state when you shut it down.
I mainly started using exFAT on flash drives (even on new ones) since it is interoperable between Windows, Linux, and Intel Mac. To be clear, I never don't unmount the drive properly under normal conditions, but I remember reading around the time it was introduced that the Windows implementation guaranteed the buffers were flushed after every write (meaning no unwritten data remains when the activity indicator on the drive stops blinking) but now I can't find any evidence that was ever the case. Wouldn't be the first time I got bad info from the Internet. 🤷♂️
Are you sure this is the only good FS?
I know it's solid and stable and used for many years as default Linux's FS but I disagree that's the only good one.