Topic
ZFS & Storage
Practical ZFS, filesystems, backups, replication, and storage failure analysis for operators.
Landlock: per-process filesystem restriction without root
Linux had seccomp for syscalls and nothing for filesystem access for years. Landlock landed in 5.13 and fixed that. Here is how to use it without the silent no-op that catches everyone the first time.
bcachefs has been in the kernel for three years. Is it your production filesystem yet?
Kent Overstreet's B-tree copy-on-write filesystem merged into Linux 6.7 in January 2024. Three kernel years of real distributions, real upgrades, real user reports. Here is an honest accounting of where it stands.
pledge(2) and unveil(2): what OpenBSD figured out that Linux is still catching up to
OpenBSD shipped two syscalls in 2016 and 2018 that let any unprivileged daemon sandbox itself down to exactly the filesystem paths and syscall categories it needs. Linux got there eventually, with seccomp-bpf in 2012, Landlock in 2021, and namespaces throughout. Compare and decide which one you'd rather write.
The transaction counter hit two billion
Production PostgreSQL stopped accepting writes on December 21st. Disk fine. CPU fine. Connections fine. The error said something about 'wraparound data loss.' I had seen it once before and I knew immediately how bad the next six hours were going to be.
The load was forty and the box was idle
Load average alert fired at 47.8 on an eight-core box. SSH'd in, ran top, saw 95% idle. Every metric looked healthy except the number that triggered the page. Thirty-two processes in D state, waiting on an NFS mount that had stopped answering three hours earlier.
statx(2): what stat(2) should have been
stat(2) is forty years old and it shows. No birth time, no way to ask for only what you need, no inode attribute flags. Linux 4.11 shipped statx(2). Almost nobody uses it. Here is what it can do.
The new mount API: fsopen(2), fsmount(2), and doing it right
Linux 5.2 shipped a replacement for the forty-year-old mount(2) syscall. The new interface is fd-based, composable, and lets you configure a filesystem before committing the mount. Almost nobody uses it. Here is what it looks like.
fanotify(7): watch and block file opens at the VFS layer
fanotify has been in the kernel since 2.6.36 and is how file integrity monitors and endpoint-detection daemons actually hook into the VFS. Unlike inotify, it hands you an open fd to the accessed file and can block opens before the calling process gets control. Here is the API and a working example.
The disk was forty percent free
Production throwing ENOSPC. df shows 40% free. You run df again because you don't believe it. Same numbers. The filesystem is completely full and completely not full at the same time, and once you understand why, you will never run df without -i again.
OpenZFS 2.3 ships with BLAKE3 stable, faster dRAID rebuilds, and the native encryption fix it needed
OpenZFS 2.3.0 dropped this week. BLAKE3 checksums graduate from experimental to production-stable. dRAID distributed spare rebuild got measurably faster on spinning rust. And someone finally fixed the raw-send re-keying behavior that made native encryption a liability for off-site replication. Here is what changed and who it affects.
The process that wouldn't die
kill -9 is supposed to be final. You send it, the process is gone. Then one morning you send it and nothing happens, and you send it again, and the process is still sitting there in ps, and you have to go rebuild an assumption you've had for twenty years.
The disk that du couldn't find
df said 97% full. I walked every directory with du and came up 40GB short. The filesystem was not corrupted. The math was just wrong in a way that took embarrassingly long to recognize for what it was.
landlock: sandbox your process without root
Since Linux 5.13, landlock lets an unprivileged process restrict its own filesystem and network access using three new syscalls. No SELinux policy, no AppArmor profile, no root. Chrome uses it for renderer isolation. Here is how to wire it up.
memfd_create and file sealing: an anonymous file nobody can modify (including you)
memfd_create() gives you a file descriptor backed by anonymous memory, with no filesystem, no path, no name. Add F_SEAL_WRITE and the kernel refuses all future write access. Pass it across process boundaries via SCM_RIGHTS or pidfd_getfd. It's the right way to share immutable data between processes.
cgroups v2 is just files. Here's how to use them without touching systemd.
Linux cgroups v2 exposes memory, CPU, and I/O limits through a plain filesystem interface under /sys/fs/cgroup/. No systemd, no container runtime, just write to files and read the results. Here's the real workflow for memory.max, cpu.max, freezer, and io.max.
OpenZFS 3.0 ships the RAIDZ expansion and the dedup engine they finally got right
OpenZFS 3.0 landed with RAIDZ expansion, a ground-up rewrite of the dedup engine, and block cloning. Three features that have been "coming soon" for most of the past decade are now in a release build. Worth looking at.
bpftrace gives Linux what FreeBSD had in 2005. The one-liners are worth the wait.
dtrace showed up on Solaris in 2004, shipped in FreeBSD 7 in 2008, and Linux users spent the next decade pretending strace was sufficient. bpftrace is the real answer, doing dynamic kernel tracing, histograms, stack walks, zero overhead when idle.
The storage was fine. We were swapping.
Three weeks, two engineers, one open storage vendor ticket, and roughly forty collective hours chasing read latency on a ZFS pool. The pool was fine the whole time. There was a swapfile nobody remembered adding seven months earlier.
pledge() turns 10. Linux still doesn't have anything half as clean.
OpenBSD's pledge(2) landed in 5.9 in 2016. Ten years later, Linux has Landlock (good) and seccomp-bpf (powerful and painful) and still nothing that lets a process sandbox itself in a single readable line. RodHat on why API simplicity is a security property.
Upgrade like you can undo it, because with bectl you can
A ZFS boot environment is a bootable clone of your root dataset. Make one before every upgrade and a wrecked kernel becomes a reboot, not a recovery-media evening. The bectl walk, and the loader trick that saves you when the new one won't boot.
Your backups are fine and one account suspension deletes all of them
Three copies, two media, one offsite, and all three sitting under a single cloud identity that a fraud algorithm can disable at 4am with no appeal path. The rule was written when a copy meant a physical object somebody had to come and take.
Your shell pipeline succeeded because the last command was polite
A failed producer can disappear behind a successful consumer. RodHat explains pipeline exit status, pipefail, PIPESTATUS, and how to stop backup scripts from lying.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
The temporary NFS mount that ran production for six years
Somebody stood it up in an afternoon to unblock a launch. It had no monitoring, no owner, no backup, and no entry in any diagram. It survived two datacentre moves, a company acquisition, and every engineer who knew it existed.
The six rsync flags worth knowing, and the one that builds you free snapshots
-a doesn't preserve what you think it preserves. --inplace and --partial solve opposite problems and combining them wrong loses data. And --link-dest turns rsync into a deduplicating snapshot system in one flag.
Stop your backup job from taking production down with it
nice does nothing for I/O. ionice only works on one scheduler that most systems no longer use. cgroup v2 io.latency and io.max are the two knobs that actually work, and they work on the thing you already have.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
ZFS boot environments + jails will do 90% of what you're using Docker for, and the other 10% is the part you don't actually need
A real walkthrough of FreeBSD jails on ZFS datasets as a lighter, more honest isolation model than a container runtime for a lot of workloads.
No, ZFS did not eat your RAM. Read arc_summary before you tune anything.
The ARC is designed to consume all the memory you aren't using and give it back under pressure. But it does give it back slowly, and there are two real cases where you should cap it. Here's how to tell which situation you're actually in.
The restore worked perfectly and we still lost eleven days
We tested our backups. Monthly, documented, signed off. The restore ran clean, the checksums matched, the database came up on the first try. And the data in it was eleven days old, because for eleven days the job had been backing up a directory nobody was writing to any more.
The disk is full and du says it isn't. It's an open file descriptor.
df reports 100%, du walks the whole tree and finds nothing. The space is in a file somebody deleted while a process still had it open, and the inode lives until the last fd closes. lsof +L1 finds it in one command.