Topic
Reliability & Observability
Production incidents, failure modes, observability, deployment safety, and resilient operations.
The listen queue had been five since 2014
Nine years of incrementing connection drops, masked by retry logic added in 2017 and promptly forgotten. The counter was in netstat -s. The backlog was in ss -tlnp. Both said the same thing for nine years.
Two years after PKfail, most affected hardware is still booting on a compromised Secure Boot key
In July 2024, Binarly found AMI's development Platform Key in production UEFI firmware across hundreds of devices. The private key is publicly accessible. Secure Boot's trust hierarchy has a root, and on the affected systems, it is cracked open.
bcachefs has been in the kernel for three years. Is it your production filesystem yet?
Kent Overstreet's B-tree copy-on-write filesystem merged into Linux 6.7 in January 2024. Three kernel years of real distributions, real upgrades, real user reports. Here is an honest accounting of where it stands.
DNSSEC validates about 38% of the internet. Here's why the other 62% hasn't moved.
DNSSEC has been production-ready since 2005 and the root zone has been signed since 2010. Two-thirds of DNS queries still aren't validated. The protocol is not the problem.
The conntrack table had twenty-four slots left
The load balancer was healthy. CPU fine. Memory fine. One in thirty connections was timing out and nobody could explain why. The answer was in dmesg, six lines down, logged and completely ignored.
The transaction counter hit two billion
Production PostgreSQL stopped accepting writes on December 21st. Disk fine. CPU fine. Connections fine. The error said something about 'wraparound data loss.' I had seen it once before and I knew immediately how bad the next six hours were going to be.
The load was forty and the box was idle
Load average alert fired at 47.8 on an eight-core box. SSH'd in, ran top, saw 95% idle. Every metric looked healthy except the number that triggered the page. Thirty-two processes in D state, waiting on an NFS mount that had stopped answering three hours earlier.
fanotify(7): watch and block file opens at the VFS layer
fanotify has been in the kernel since 2.6.36 and is how file integrity monitors and endpoint-detection daemons actually hook into the VFS. Unlike inotify, it hands you an open fd to the accessed file and can block opens before the calling process gets control. Here is the API and a working example.
The disk was forty percent free
Production throwing ENOSPC. df shows 40% free. You run df again because you don't believe it. Same numbers. The filesystem is completely full and completely not full at the same time, and once you understand why, you will never run df without -i again.
OpenZFS 2.3 ships with BLAKE3 stable, faster dRAID rebuilds, and the native encryption fix it needed
OpenZFS 2.3.0 dropped this week. BLAKE3 checksums graduate from experimental to production-stable. dRAID distributed spare rebuild got measurably faster on spinning rust. And someone finally fixed the raw-send re-keying behavior that made native encryption a liability for off-site replication. Here is what changed and who it affects.
glibc 2.44 enables FORTIFY_SOURCE=3 by default and your struct tricks are now fatal
glibc 2.44 shipped last week with FORTIFY_SOURCE=3 on by default for GCC 12+ builds. It catches real buffer overflows. It also catches thirty years of creative struct-boundary memcpy that technically worked until it didn't. Here is what changed and what is now crashing.
SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits
Linux 3.9 added SO_REUSEPORT. Bind N sockets to the same address:port, the kernel hashes connections across them. Linux 4.5 added eBPF dispatch so you control the routing. The part nobody explains is what happens when one of those sockets disappears while connections are in flight.
FreeBSD 14.5-RC1 has a sound ioctl UAF and a hardware counter that ignored your credentials
FreeBSD 14.5-RC1 dropped August 29 with three security fixes. The use-after-free in SNDCTL_DSP_SYNCSTART is local privilege escalation. The HWPMC credential-transition bug is subtler and more interesting. Here is what they fixed and why the HWPMC one matters more than it sounds.
The conntrack table was full
Connections to services behind our firewall started failing intermittently. The iptables rules were correct, the NICs were clean, the routing was fine. The problem was a kernel table we had never configured, silently dropping packets when it ran out of room.
The pool was full of dead connections
An intermittent burst of broken-pipe database errors had been logged as "flaky" for two months before anyone looked closely enough to notice they were always the first query on a connection. The connection pool was handing out corpses.
`sudo` wrote its last heap overflow in C. The Rust rewrite shipped.
sudo-rs is production-ready for most deployments. A setuid-root binary sitting at uid 0, accumulating privilege-escalation CVEs for four decades, is exactly the right target for a memory-safe rewrite. This one is hard to argue with.
The OOM killer was doing its job
A slow memory leak ran undetected for five weeks because the kernel's out-of-memory killer, executing its heuristic correctly, kept choosing the monitoring agent over the leaking service. The pager never fired. The monitoring gaps were there in the data the whole time.
ss -i shows you what netstat never could: TCP internals live
Everyone knows ss -tulnp. Almost nobody uses ss -i, which surfaces congestion window size, RTT, retransmit counts, and send/receive buffer fill, directly from the kernel, no tool required.
The number 1024 was not a coincidence
A file descriptor limit from 2003 survived three infrastructure migrations and one complete platform rewrite because nobody ever questioned a number that had always been there. It bit us at 2am on a Tuesday.
The pipeline succeeded. We found out from a customer.
A deployment that broke production ran undetected for forty minutes because the notification step was written to never fail, and the monitoring alert that should have fired went to the same broken endpoint.
The postrotate script worked. We had no logs.
Three months of production logs, all gone. logrotate ran clean on schedule every week. The daemon was healthy and writing the whole time. The postrotate signal was firing correctly, just to a PID that had been dead since deployment day.
Your browser encrypted your DNS queries. It also decided who gets to read them.
DNS-over-HTTPS is a correct answer to a real problem, cleartext DNS has been leaking every hostname you visit since 1983. The way browsers deployed it, pointing users to Cloudflare and Google by default without explanation, is a different story. The encryption is real. The 'privacy' framing is doing a lot of work.
The storage was fine. We were swapping.
Three weeks, two engineers, one open storage vendor ticket, and roughly forty collective hours chasing read latency on a ZFS pool. The pool was fine the whole time. There was a swapfile nobody remembered adding seven months earlier.
The change was correct. Deploying it to every region simultaneously was not.
A one-line config change, reviewed, tested, and correct in every environment we tried it in. It went to all six regions in the same rollout because config isn't code, and config doesn't need a staged rollout. It does now.
The alert had been firing for eight months and it was right the whole time
A rule that pages nightly and gets acknowledged nightly isn't monitoring, it's a ritual. We had trained an entire team to dismiss a specific alert without reading it, and then it started telling us about something new.
Observability priced per gigabyte means you log least when you need it most
Ingest-based pricing creates an incentive that fires exactly backwards: the incident that generates the most telemetry is the one that costs the most to observe. Teams respond by sampling, and then wonder why the postmortem has a gap in the middle.
The observability platform was down during the outage. Good. Now do it with your hands.
A junior engineer froze during a production fire because the SaaS dashboard that watches production was part of the fire. RodHat on the tools that were on every Unix box before the kid was born, the racket that sold competence back to us as a monthly invoice, and why the fire isn't out.
Your restart policy is deleting the crime scene
Automatic restarts keep services available, but an aggressive restart loop can erase the timing, logs, cores, and state needed to understand why a process failed. Preserve evidence before recovery becomes amnesia.
The health check was green because it was checking the wrong damn thing
A service can answer HTTP 200 while its queue is wedged, its database writes are failing, and every useful request is dying. RodHat on health checks that prove process existence instead of service capability.
Stop waiting beside tcpdump like it owes you an incident
Intermittent network failures do not happen while you are watching. Use tcpdump's rotating capture files, size limits, and post-trigger preservation to keep the packets that existed before the pager fired.
Staging had its own everything, except the one thing that mattered
Separate servers, separate config, separate deploy pipeline, a big banner saying STAGING. And a database connection string that pointed, through two layers of indirection, at production. We found out during a load test.
The diff nobody wrote and everybody approved
AI coding tools now generate a meaningful percentage of commits at shops that have adopted them. RodHat on what happens to institutional knowledge when the author of the diff has never been paged.
We wrote the retry logic to survive a blip. It turned a blip into four hours.
One backend got slow for ninety seconds. Every client retried, in lockstep, three times, with no jitter. The retries were larger than the original traffic, the backend never recovered, and every fix we tried made it worse until we did the thing nobody wanted to do.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
The temporary NFS mount that ran production for six years
Somebody stood it up in an afternoon to unblock a launch. It had no monitoring, no owner, no backup, and no entry in any diagram. It survived two datacentre moves, a company acquisition, and every engineer who knew it existed.
Stop your backup job from taking production down with it
nice does nothing for I/O. ionice only works on one scheduler that most systems no longer use. cgroup v2 io.latency and io.max are the two knobs that actually work, and they work on the thing you already have.
The billing job ran twice for six weeks and the totals still balanced
No errors. No alerts. The reconciliation report matched every single day. And a few hundred customers were being double-charged, because the job that generated the charges and the job that verified them were the same code with the same bug.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
Passkeys fix phishing and nobody wants to talk about account recovery
The cryptography is right, the phishing resistance is real, and I've stopped arguing about it. What hasn't been solved is what happens when the device is gone, and every deployed answer to that reintroduces the thing passkeys replaced.
Your AI pair programmer went down for four hours and so did half of Hacker News' composure
A major AI coding assistant had a multi-hour outage this week. RodHat has seen this movie before, just with a different lead actor.
The migration was flawless. The TTL was 86400.
Six weeks of planning, a rehearsed cutover, and a maintenance window we finished forty minutes early. Then a fifth of our traffic kept arriving at a datacentre we'd already started decommissioning, for a full day, and there was nothing whatsoever we could do about it.
Process supervision is a 40-line problem. It has been solved since 1997.
You don't need a service manager with a DNS resolver in it to restart a daemon that died. daemontools, runit and s6 do supervision trees, log rotation and clean shutdown in a handful of files, and the design idea underneath them is worth stealing even if you never install one.
Certificate lifetimes are collapsing to 47 days and your renewal process is a person
The CA/Browser Forum voted to phase TLS certificate validity down to under seven weeks by 2029. If your renewal runbook has a human in it, the schedule just became your problem, and that's the actual point of the change.
The restore worked perfectly and we still lost eleven days
We tested our backups. Monthly, documented, signed off. The restore ran clean, the checksums matched, the database came up on the first try. And the data in it was eleven days old, because for eleven days the job had been backing up a directory nobody was writing to any more.
The Friday deploy that ate my weekend, and whose fault it actually was (mine)
A sysadmin war story about a Friday-afternoon deploy, a silent DNS TTL assumption, and the two-day outage it caused. RodHat owns every part of it.