The disk was forty percent free

The alarm came in at 02:17. Production web tier, all three nodes, throwing ENOSPC: no space left on device while writing session data. The on-call rotation landed on me because I had set up the session store two years earlier, and apparently that makes you responsible for it permanently.

First thing I ran was df -h.

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   18G   29G  38% /
/dev/sdb1       200G   81G  107G  41% /data

Forty-one percent used. Over a hundred gigabytes free on the data partition. ENOSPC made no sense.

I ran it again. Same numbers. The errors kept arriving in the log stream. I called bullshit on the monitoring and went to verify the error directly:

touch /data/sessions/test-$(date +%s)

touch: cannot touch '/data/sessions/test-1757116243': No space left on device

The filesystem had a hundred and seven gigabytes free and would not let me create a new file.

The second resource pool

A filesystem has two things that can run out. The first one everyone knows: blocks. Physical space on disk, measured in bytes, reported by df -h. When blocks run out, writes fail. This is what you think of when you think “disk full.”

The second one: inodes. An inode is the data structure the kernel uses to represent a file. Permissions, ownership, timestamps, pointers to data blocks: all in the inode. One inode per file, regardless of how big the file is. A filesystem has a fixed number of inodes allocated when it was formatted and cannot exceed that count.

When inodes run out, the filesystem cannot create new files. It does not matter how much block space is available. No inode, no file. The write returns ENOSPC.

df -h reports block usage and says nothing about inodes unless you ask for it:

df -i /data

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb1      6553600 6553600       0  100% /data

Six and a half million inodes. All of them used. One hundred and seven gigabytes of block space sitting empty and completely useless.

There it was.

Six million files

Six and a half million files on a two-hundred-gigabyte partition is a serious number. I needed to find what they were before I could do anything useful. du -sh would not help: du reports bytes, not file counts. What I needed was an inode count aggregated by directory:

find /data -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

This prints the parent directory for every file on the filesystem, then sorts and counts. A directory holding ten thousand files shows up as 10000 /path/to/dir. Takes a few minutes on a large filesystem but it gives you the answer.

6109842 /data/sessions
  201344 /data/uploads
   98211 /data/cache

Six million files in /data/sessions.

The session store was writing one file per session. Every web request that hit an unauthenticated endpoint created a new session. Bots, crawlers, API probes, security scanners, health checks without a session cookie: new file, four hundred bytes, disposable. The site had been running for two years. The directory had been growing the whole time. Nobody had noticed because df -h was always healthy.

Six million sessions at four hundred bytes each is about two and a half gigabytes of block space on a two-hundred-gigabyte partition. The blocks were barely touched. The inodes were gone.

Stopping the bleeding

The first job at 02:30 is stopping errors, not root-causing perfectly.

Deleting the oldest session files with find -mtime on six million files takes time you do not have in an active incident. The faster move is to identify what can be deleted without looking at it. Session TTL was four hours. Anything older than two days was garbage by definition:

find /data/sessions -maxdepth 1 -type f -mtime +2 -delete

This ran for about eight minutes and cleared four million files. Inode usage dropped to roughly sixty percent. The errors stopped. The on-call Slack channel went quiet.

That bought time to fix the actual problem. The session store had no business being file-based. It was using the file driver because that was the framework default and nobody had changed it at setup time. The application already had Redis. Redis was three config lines away.

That change took thirty minutes to implement, test in staging, and deploy. After the deploy, no new session files were written to the filesystem.

Then I cleaned the remaining two and a half million stale files at a controlled rate to avoid spiking disk I/O during business hours:

find /data/sessions -maxdepth 1 -type f -delete -print | pv -l -s 2500000 > /dev/null

pv as a rate limiter and a progress indicator. About four hours to finish, done before anyone showed up to the office.

Why the inode count is fixed

The inode table is allocated contiguous on disk at filesystem creation time and cannot be expanded afterward on ext4. mkfs.ext4 estimates the inode count from the partition size and an assumed average file size. The default ratio is one inode per sixteen kilobytes of disk space.

For a two-hundred-gigabyte partition, that math gives you roughly thirteen million inodes, which is more than most workloads will ever need. The /dev/sdb1 partition had ended up with six and a half million because it had been formatted with an older tool using a different ratio, by someone in 2021 who was not thinking about session directories, who had since left the company.

If you know a filesystem is going to hold a large number of small files, set the inode ratio at format time with -i:

mkfs.ext4 -i 4096 /dev/sdb1

One inode per four kilobytes instead of one per sixteen. Four times the inode density. Costs you space in the inode table itself, which the block allocator cannot use for data. The tradeoff makes sense for mailboxes, session stores, package caches, or anything else that writes many tiny files.

You cannot change the inode ratio on ext4 after the fact without reformatting. XFS handles this differently: it can allocate additional inode space dynamically as usage grows. If your workload involves a filesystem full of small files, XFS is the right answer. We were on ext4 because the partition predated anyone thinking carefully about what would run on it.

The alert that was not there

df -h is a lie of omission. It shows block usage and nothing else. Without inode monitoring, the first signal of inode exhaustion is an outage. Block exhaustion degrades gradually: you get warnings at eighty percent, ninety percent, ninety-five. Inode usage can read sixty percent on Tuesday and hit one hundred percent Friday night during a traffic spike.

Add this to your monitoring:

df -i | awk 'NR>1 && $5+0 >= 80 {print $0}'

Any filesystem at eighty percent or more inode usage prints a line. If you run Prometheus, node_exporter exposes node_filesystem_files_free and node_filesystem_files for every mounted filesystem. Alert when the ratio crosses eighty percent.

For quick manual checks when something feels wrong, count files directly:

find /data/sessions -maxdepth 1 -type f | wc -l

If that number is in the millions, you know what you are looking at.

The df -h result and the df -i result are two separate questions to two separate accounting systems and you need both answers. The deleted-file variant of this class of problem gives you a different mismatch: df shows a nearly full disk, du adds up to a fraction of it, because the space is in files that have been unlinked but not released. The diagnosis tool there is lsof +L1. Same category of problem, different layer. The tip on that specific case covers the procedure.

The inode version is worse in one respect: there is no gradual signal. The disk does not look full because it is not full. Blocks are fine. You have to specifically ask about inodes to know they are nearly gone, and most monitoring setups never ask.

Two years

The session directory had been growing since the site launched. Nobody had checked the file count because there was no reason to. df -h looked healthy. The application worked. The directory was just sitting there accumulating files.

The specific ceiling got hit during a traffic spike at 02:17 on a Friday night. That part was bad luck. The inode table running out was not bad luck, it was a slow-motion certainty that had been in motion since day one. File-based sessions on a filesystem with a default inode ratio, no inode monitoring, no session cleanup job, two years of uninterrupted operation. The only question was when.

The fix took forty minutes of actual work once the diagnosis was clear. The diagnosis took about five minutes once I knew to run df -i. Everything before that was staring at a filesystem that looked healthy and trying to understand why it would not let me create a file.

df -h. Then df -i. Both commands. Every time. The disk was forty percent full. The filesystem was completely full. Both of those were true at the same time.