The disk that du couldn't find

The alert said disk full. The disk said 97%. du said I was a liar.

That’s the whole riddle. Everything else is archaeology.

What the numbers said

Application server, mid-sized deployment, Java service doing something enterprise-ish with documents. I got paged at 22:40 because the disk had crossed 90% and the SLA said page at 90%. By the time I logged in it was 97%.

$ df -h /var/app
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       200G  193G  7.3G  97% /var/app

193 gigabytes. I have logs for this application and I know the data directory doesn’t grow that fast. Something went wrong, something accumulated, and I needed to find it before the disk hit 100% and the application started throwing java.io.IOException: No space left on device into every log line.

Standard approach: find the big directories.

$ du -sh /var/app/*
2.1G    /var/app/data
340M    /var/app/logs
18G     /var/app/cache
1.2G    /var/app/tmp
4.8G    /var/app/uploads

That’s about 26 gigabytes. The disk has 193 used. I am 167 gigabytes short.

I went one level deeper on everything. Then two levels. I ran du -sh from the root of the mount. I ran it with --exclude to rule out symlink loops. I added --apparent-size in case something weird was happening with sparse files or compression. The total kept coming back in the 26-30GB range, regardless of how many ways I counted. df was firm: 193GB used.

The filesystem was not corrupted. I checked. fsck on a mounted filesystem tells you almost nothing useful, but dumpe2fs -h on the device showed no error flags and the block counts were consistent. The block group accounting lined up with what df reported. 193 gigabytes of blocks were allocated. The directory tree accounted for 26 of them.

The mechanism

It took me longer than I am going to admit to remember this. I knew the answer before I got there, somewhere in the back of my head, in the part of my brain that stores things I learned from an incident in 2004 and then didn’t need again for years.

On Linux (and any Unix worth the name), deleting a file with rm does not immediately free the disk blocks. What rm does is remove the directory entry: it calls unlink(2), which decrements the link count on the inode. If the link count reaches zero, the kernel is free to reclaim the blocks. But there’s a second condition: the kernel will not reclaim the blocks while any process has the file open. The inode has two reference counts, the link count (directory entries pointing to it) and the open-file-descriptor count. The blocks are not freed until both reach zero.

So: process opens a file. Another process (or a shell command, or a cron job) deletes the file. The file disappears from the directory: ls can’t see it, du can’t count it, find can’t find it. But the first process still has an fd pointing to the inode, and the kernel keeps the inode alive, blocks and all, until that fd is closed.

df talks to the filesystem directly, which counts allocated blocks. du walks the directory tree, which counts what the directory tree can see. The difference between those two numbers is exactly the space consumed by deleted-but-still-open files.

167 gigabytes of deleted-but-still-open files.

lsof +L1

lsof has a flag I want more people to know about: +L1. It means “show open files with a link count less than 1.” A link count of zero means the directory entry is gone but the fd is still held. This is exactly the list I needed.

$ lsof +L1
COMMAND   PID  USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
java     8472  app  1823r  REG    8,17  168742918112     0 3847291 /var/app/logs/app.log.2026-08-15 (deleted)
java     8472  app  1824r  REG    8,17   892471038     0 3847292 /var/app/logs/app.log.2026-08-14 (deleted)
[... eleven more lines ...]

Thirteen deleted log files. The largest was 157 gigabytes.

The application was a Java service using Log4j2 with a RollingFileAppender. Log rotation was handled by logrotate, which ran nightly and compressed the previous day’s log, then renamed it with a datestamp and kicked it out of the main log directory into an archive path. The logrotate job had been running fine. The compressed archives in the archive path were there and correct. The active log file was being written to and looked fine.

The problem was the sequence. Logrotate renames app.log to app.log.2026-08-15, compresses it to app.log.2026-08-15.gz, then deletes app.log.2026-08-15 once compression is done. From logrotate’s point of view this is clean and complete. But the JVM had app.log open by inode, not by name. When logrotate renamed the file, the JVM’s fd still pointed at the same inode, now known to the directory as app.log.2026-08-15 and then, after logrotate deleted it, known to the directory as nothing at all. The JVM kept writing to it. Every byte the JVM wrote after the rotation went into a file that the directory tree had forgotten existed.

Logrotate supports a postrotate script and a copytruncate option for exactly this reason. copytruncate copies the log file to the rotated name and then truncates the original in place, so the JVM’s fd still points at the right inode and the file just gets shorter. The configuration was not using copytruncate. Nobody had configured a postrotate to signal the JVM to reopen its log files. The log rotation was doing exactly what it was told to do, which was not quite what was needed.

This had been happening for thirteen days. Thirteen days of daily log rotations, each one leaving a ~12GB ghost behind. The disk had absorbed it silently until it couldn’t.

Getting the space back

The immediate fix is to close the file descriptors. There are two ways to do that: kill the process (or restart it cleanly), or find the fds in /proc and close them surgically.

The surgical approach first, because we were at 97% and “restart the Java application” meant a maintenance window that nobody had approved yet:

# find the fd numbers from lsof output and close them via /proc/<pid>/fd
# WARNING: closing an fd out from under a running process can cause it to write
# to a different fd with the same number, or explode. Test on your application
# before assuming this is safe.
ls -la /proc/8472/fd | grep deleted

lrwx------ 1 app app 64 Aug 29 22:51 1823 -> /var/app/logs/app.log.2026-08-15 (deleted)
lrwx------ 1 app app 64 Aug 29 22:51 1824 -> /var/app/logs/app.log.2026-08-14 (deleted)

You can close an fd from outside the process using /proc/<pid>/fd/<n> as a target for a gdb attach, or by opening /proc/<pid>/fdinfo and using python to call os.close on the specific fd number after attaching with ctypes or similar. This is surgery with a fork: it works, and it will ruin your evening if you do it wrong.

I did not do the surgery. I got approval for a restart window, restarted the JVM, and watched 167 gigabytes of inode references evaporate instantly. df dropped to 26%. Same number du had been reporting all along.

The thing logrotate does wrong

Logrotate ships with a copytruncate directive that should be the default for any process that holds log file handles open. It is not the default. The default behavior assumes that the application will be signaled to reopen its log files after rotation, which is correct for daemons that handle SIGHUP or have a dedicated log-reopen signal. Log4j2 can be configured to watch for log rotation and reopen appropriately. This JVM was not configured to do so, and nobody had added copytruncate to the logrotate config because the rotation had been working fine, in the sense that it produced correctly named rotated files and the active log continued to be written to.

The silence of the failure is what makes this category of bug obnoxious. The application is logging. The disk appears to be growing at the expected rate. The rotated archives exist. Everything looks fine until df reports 97% and you discover that the “expected rate” you were seeing was the active log file and the ghost of twelve previous log files all growing in parallel, invisible to every metric that walked the directory tree.

Monitoring that only checks the directory tree is monitoring with a blind spot. If your disk usage alerting is based on du, you will not see this coming. If it’s based on df (which it should be, since df is what the kernel actually reports), you’ll see the disk fill up but you’ll spend a confusing amount of time with du telling you there’s nothing there.

The correct posture: df for alerting, lsof +L1 for investigation when du and df disagree. That discrepancy is a diagnostic category with exactly one cause.

find /proc/*/fd as a fallback

If you don’t have lsof on the system (I have encountered containers without it and BSD jails where the port isn’t installed), the same information is available via /proc:

find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'

Slower, noisier, requires parsing, but it works everywhere that has /proc. On FreeBSD and other systems without /proc mounted, fstat(1) covers the same ground, and I like it better anyway.

What we fixed

After the restart cleared the space, we made three changes:

First: added copytruncate to the logrotate config for this application. Not elegant, but immediate and reliable. The JVM doesn’t need to do anything; logrotate handles the copy-and-truncate atomically from its perspective.

Second: added a daily lsof +L1 | awk 'NR>1 {sum += $7} END {print sum}' to the ops metrics script. It reports the total bytes consumed by deleted-but-open files and pushes it to the monitoring system. If this number is more than a few hundred megabytes, something is wrong and we want to know before df pages us at 97%.

Third: added a logrotate configuration audit to the runbook. Every application that writes log files gets reviewed: does it handle log rotation signals, or does it need copytruncate? Two other applications on the same host needed copytruncate and didn’t have it.

The audit took an afternoon. The incident that prompted it cost a late-night maintenance window and thirteen days of accumulated phantom disk usage. The audit was the better use of time. It’s almost always the better use of time.

lsof +L1. Put it in your toolkit. You will need it, and when you need it, you will need it at 22:40 on a disk that du insists is half empty.