The file that root couldn't touch

There are moments in this work where the machine makes you feel stupid. Not because you made a mistake, but because you forgot something you definitely knew twenty years ago. These are the moments I find most useful, once the embarrassment fades. I had one of those moments at eleven in the evening in front of a terminal that was refusing to do something I have done ten thousand times.

The disk was full. I tried to delete a file. Root said no.

The setup

Log directory, application server, legacy deployment. The application in question had been running fine for two years and had never asked for attention it wasn’t owed. The disk alert fired because the log directory had grown to 94% of the allocated volume. The plan was simple: archive the old rotated logs, purge the ones older than 90 days, verify the deletion, go back to whatever I was doing. Ten minutes, door to door.

find /var/log/appname -name "*.log.*" -mtime +90 -delete

Permission denied.

I am root. I am the sysadmin. I have been doing this since before most of these log files’ directory structures existed. This machine does not tell me “permission denied.”

I ran it again, more slowly this time, dropping the -delete to just list the files. The listing worked. The files existed, were owned root:root, had 644 permissions, nothing exotic. The directory itself: root:root, 755. Everything in order.

I added -delete back. Permission denied.

The obvious checks

I am not proud of how long I spent in the obvious-checks phase. But I am going to describe it honestly because anyone who says they went straight to the right answer is compressing the story.

First: whoami. Root. Obviously. I knew I was root. I checked anyway.

Second: id. uid=0(root) gid=0(root) groups=0(root). Yes. Root. Confirmed.

Third: the mount. Maybe the filesystem had been remounted read-only? That happens: kernel panics, I/O errors, the driver decides to protect itself by flipping the mount to RO. mount | grep appname. Read-write. cat /proc/mounts | grep the device. Read-write. tune2fs -l on the underlying device to check the last-mount-as-ro flag. Nothing.

Fourth: disk full. I know df is lying to me a lot of the time, but maybe the partition was 100% full and the system was refusing new writes for that reason? df -h: 94% used, not 100%, and deletion isn’t a write anyway. df -i to check inodes. Plenty of inodes.

Fifth: strace. If you don’t know what’s happening, strace tells you what syscall is failing and with what errno. I ran strace -e trace=unlink,unlinkat rm /var/log/appname/app.log.2026-07-01. Clear as anything: unlinkat(AT_FDCWD, "/var/log/appname/app.log.2026-07-01", 0) = -1 EPERM (Operation not permitted). The kernel was refusing the unlink syscall with EPERM, not EACCES. Not a permissions problem. An operation-not-permitted problem. Those are different things.

That distinction matters. EACCES means the permission check failed: you don’t have the right bits. EPERM means the operation is not permitted for this object, regardless of who’s asking. Even root.

There are a few conditions that produce EPERM on unlink for root. Sticky bit on the directory protecting files owned by other users, but the files were root-owned, so that’s out. The file is a mount point: clearly not. The filesystem is read-only: already checked. The file has an immutable flag set.

I checked the immutable flag approximately thirty seconds after I should have.

lsattr

lsattr /var/log/appname/app.log.2026-07-01
----i---------e--- /var/log/appname/app.log.2026-07-01

There it is. The i in position four. Immutable.

I ran lsattr on the whole directory:

lsattr /var/log/appname/
----i---------e--- /var/log/appname/app.log.2026-07-01
----i---------e--- /var/log/appname/app.log.2026-07-04
----i---------e--- /var/log/appname/app.log.2026-07-08
[... thirty-seven more lines ...]

Every rotated log in the directory. Immutable. All of them.

The running log (app.log, no datestamp, the live file being written to by the running process) did not have the flag. It was writeable. The rotation was working fine; new log files were being created, the live log was being rotated, the rotated files were being renamed. But at some point in the rotation process, after the file was renamed and before or during compression, something was setting the immutable flag on the rotated log. And then it was sitting there, accumulating, unable to be touched, for as long as the system had been running.

I knew immediately who set this flag. I didn’t know yet which part of the machinery was doing it. But I knew why.

The hardening script

Two years ago, this system went through a compliance review. Someone with authority over the environment and good intentions about audit trails ran a CIS benchmark hardening script against it. The script did a lot of things: disabled unnecessary services, tightened SSH configuration, set password complexity rules, adjusted kernel parameters, and (I found it in the script about forty minutes later) applied chattr +i to the log directory for several specified services.

The intent was forensic integrity. If you’re worried about an attacker covering tracks by deleting logs, making the logs immutable prevents that. Even a root-level attacker can’t unlink an immutable file without first removing the immutable flag, which requires CAP_LINUX_IMMUTABLE, which requires either root or an explicit capability grant. If you’ve done a full chattr +i on your logs and someone compromises root and tries to clean up, they have to know about extended attributes and specifically know to run chattr -i before deleting. It’s not cryptographic protection but it raises the bar.

Good intent. Reasonable intent. The problem is that the hardening script ran chattr +i on the rotated log files that existed at the time of the audit and also set up a cron job (I found this later, buried in /etc/cron.d/cis-harden-logs, not commented, not documented, not announced to the team) that re-applied chattr +i to any file in the specified directories with a datestamp older than 24 hours.

So: log is written. Log is rotated. Log is renamed with a datestamp. Twenty-four hours later, the cron job runs, finds the datestamped file, applies immutable. Nobody can touch it anymore. Not root. Not find -delete. Not the log cleanup job that runs monthly. Nobody.

The cron job had been running faithfully, every day at 03:00, for two years. The logs had been accumulating, every day, for two years. The disk had grown slowly, consistently, and unnoticeably, right up until it wasn’t unnoticeable anymore.

The system was doing exactly what it was told to do. The configuration was correct from the hardening script’s point of view. The problem was that nobody who operated the system knew this configuration existed, because nobody documented it when it was applied.

The fix and the argument

The immediate fix is one line per file:

chattr -i /var/log/appname/app.log.2026-07-01
rm /var/log/appname/app.log.2026-07-01

Or in bulk:

find /var/log/appname -name "*.log.*" -mtime +90 \
  -exec chattr -i {} \; \
  -delete

That cleared the disk. Twenty minutes of work, most of it archaeology.

The argument came after: what do we do with the hardening policy? There are three positions you can take.

Position one: remove the cron job, don’t apply immutable to logs. The audit-trail protection is real but limited: an attacker who has root and knows what they’re doing will remove the flag before deleting. Meanwhile, the operational cost of not being able to manage your own logs is concrete and ongoing.

Position two: keep immutable, fix the retention policy. Make the log cleanup job aware that it needs to chattr -i before deleting. Update the cron job to have a hard expiry: apply immutable up to N days, then strip it and delete files older than M days. The protection window is preserved; the accumulation problem is solved.

Position three: use a different mechanism for audit trail integrity. Ship logs to an append-only, write-once remote destination (an S3-compatible store with object lock, a write-once tape, a WORM filesystem) and stop using filesystem-level immutable flags on the live system for this purpose. The protection is stronger (remote store is harder to compromise than a local flag) and there’s no operational conflict with log management.

We went with position two for the short term and opened a project for position three. The cron job got updated, the retention policy got documented in the runbook, and I added lsattr -R /var/log/appname to the monthly operational check checklist because once you’ve been surprised by an immutable file you add it to every checklist.

What lsattr actually shows you

The lsattr output format is a fixed-width string of flags. Most of them you will never see in the wild. A few are worth knowing:

• i: immutable. No write, no rename, no delete, no link. Even root. Even with O_TRUNC. If you see this and you’re surprised, now you know.
• a: append-only. Writes must be appends; truncation and deletion are not permitted. Common on log files that you want to grow but not overwrite: chattr +a on a log file means a compromised process can’t zero it out, only add to it.
• e: extents. This is set automatically on ext4 files that use extent-based allocation. You’ll see it on almost every file on a modern ext4 filesystem. It means nothing interesting for our purposes; don’t confuse it with something you or someone else set.
• u: undeletable. When deleted, the file’s contents are saved, allowing later undeletion. Rarely used; support varies by filesystem.

The flags that matter for incident investigation are i and a. If an rm or a write fails with EPERM on a file you have every other reason to think you should be able to modify, run lsattr before you spend thirty minutes on mount tables.

Setting and clearing these requires CAP_LINUX_IMMUTABLE. On a standard system that means root. On a system with a capability-aware container runtime or with file capabilities set on specific binaries, the picture gets more complicated, but on a conventional server, if the flag is set, root set it, either directly or through something root ran.

The documentation problem

This incident was caused by a documentation failure, not a technical failure. The hardening script worked correctly. The cron job worked correctly. The log rotation worked correctly. Nobody knew about any of it, because the person who applied the hardening didn’t write it down in a place that operators would ever read, and didn’t tell anyone it introduced a cron job, and the cron job had no comments explaining why it existed.

I’ve applied hardening scripts myself. I probably did not document all of them adequately either. This is not a gotcha: it’s a structural problem with how “compliance” activities get applied to running systems. Compliance reviews happen at a point in time, produce a set of changes, and those changes get applied by someone who is focused on satisfying the audit criteria. The operational implications of the changes (like “this cron job will now prevent log deletion forever”) are secondary to the compliance outcome.

The result is a system that accumulates configuration that operators don’t know about. Hidden cron jobs. Extended attributes on files. Kernel parameters in obscure sysctl.d files. Systemd unit overrides in drop-in directories nobody checks. The longer a system runs, the more of this sediment accumulates, and the more likely you are to spend eleven p.m. checking whoami because a file you have every right to delete is refusing to be deleted.

lsattr -R / on a system you’ve inherited is thirty seconds of work and will sometimes tell you things about the system that nobody living knows. I run it. I recommend you run it. You will occasionally find something that surprises you, and you will want to find it before it surprises you at the wrong moment.

The file that root couldn’t touch is rarely the last surprise a well-traveled system has waiting.