The conntrack table had twenty-four slots left

Saturday morning. 03:14. The pager fires on a connection timeout rate of 4.2%. Not zero. Not catastrophic. Not the kind of number that means everything is on fire. The kind of number that means something is wrong and you are going to spend the next few hours finding out what.

The setup was a single Linux box running as a NAT gateway for a medium-sized private network: about 200 application servers routing their egress traffic through this one machine. iptables masquerade rules, nothing exotic. The box had been running fourteen months without incident. CPU around 12%. Memory comfortable. Network throughput well inside the NIC limit.

I logged in and ran through the obvious things.

top: CPU 14%, memory 8GB of 32GB. Fine.

sar -n DEV 1 5: average 1.4Gbps on the primary interface, normal for a Saturday morning.

ss -s:

Total: 18402
TCP:   17841 (estab 11982, closed 842, orphaned 0, timewait 5017)
UDP:   561

17,000 TCP connections is a lot but not surprising for an active NAT box. The 5,017 TIME_WAIT entries caught my eye, not enough to explain the timeout rate on its own. Nothing here explained anything.

The thing dmesg knows that you don’t

Most people stop at this point. Top says fine, sar says fine, ss says fine, the box is probably fine. The problem is upstream. Maybe the destination is slow. Maybe a route is flapping. Maybe it will clear up.

I checked dmesg anyway. Scrolled past the usual kernel noise. Found this, repeated about forty times in the last three minutes:

[2847293.441823] nf_conntrack: nf_conntrack: table full, dropping packet.
[2847293.441991] nf_conntrack: nf_conntrack: table full, dropping packet.
[2847293.442104] nf_conntrack: nf_conntrack: table full, dropping packet.

The kernel logs that message, then rate-limits it to avoid flooding the ring buffer. Forty appearances in three minutes meant the actual event had fired hundreds of times.

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

65512
65536

Twenty-four slots remaining. The conntrack table was at 99.96% capacity.

What conntrack is and why it matters

Linux’s netfilter/iptables does stateful packet inspection. When a TCP connection passes through the firewall or NAT box, the kernel creates an entry in the connection tracking table recording the source IP, destination IP, ports, protocol, and connection state. This entry is how the kernel matches reply packets back to the original connection, handles address translation, and enforces stateful firewall rules.

Every tracked connection takes one slot. The maximum is nf_conntrack_max. Default on most distributions: 65,536. On older kernels this number was set at boot and never reconsidered. On this box, fourteen months old, running a 4.x kernel image from its original provisioning, it was still 65,536.

When the table fills, the kernel cannot allocate a new conntrack entry for an incoming SYN. It drops the packet. The client’s TCP stack waits for the retransmission timer (typically 1 second for the first retry) and tries again. If the table has freed a slot by then, the retry goes through and the connection succeeds. If not, the client keeps retrying until it gives up and the application gets a timeout.

That was the 4.2%. Not every connection failing. Connections whose initial SYN arrived during the brief moments when all 65,536 slots were occupied. Most retransmits got through. Roughly one in twenty-five did not, and those timed out.

Where all the slots went

Each conntrack entry has a timeout. After the timeout expires with no matching traffic, the entry is removed. The defaults are generous: established TCP connections stay tracked for 5 days. TIME_WAIT entries persist for 120 seconds.

On a NAT gateway with sustained traffic, you accumulate entries from three sources: currently active connections, recently closed connections draining through TIME_WAIT, and long-lived persistent connections like database pool members and long-polling HTTP clients.

conntrack -L --output ktimestamp 2>/dev/null | awk '{print $1}' | sort | uniq -c | sort -rn | head -5

Hundreds of entries with ages in the range of hours or days. Long-lived persistent connections from application servers to external APIs, database replicas, internal services. All legitimate. All occupying slots that were never getting freed.

The TIME_WAIT situation compounded it. conntrack -L | grep TIME_WAIT | wc -l came back with 5,204. Each of those was sitting in the table for 120 seconds before expiring. With 200 application servers all making outbound connections through the same NAT box, 5,000 concurrent TIME_WAIT conntrack entries is easy to reach and easy to maintain indefinitely.

Fourteen months of traffic growth had filled a table nobody had ever sized for the actual load.

Fixing it

Three things: raise the ceiling, tune the TIME_WAIT timeout, add monitoring.

sysctl -w net.netfilter.nf_conntrack_max=524288

The ceiling moved from 65,536 to 512,288. The “dropping packet” messages in dmesg stopped within seconds. The application timeout rate dropped to zero within about 90 seconds as TCP retransmits worked through the queue of backed-up SYNs.

The hash table conntrack uses for lookups should scale proportionally. On a 32GB machine, 512K entries is trivial overhead. A conntrack entry on x86-64 is about 352 bytes. Half a million entries is 176MB. On a box with 32GB that is 0.5% of available memory.

echo 131072 > /sys/module/nf_conntrack/parameters/hashsize

The hash table resize takes effect immediately but does not persist across reboots. For the TIME_WAIT timeout:

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Dropping TIME_WAIT tracking from 120 seconds to 30. This is safe on a NAT gateway because the kernel’s TCP state machine tracks the connection independently. The conntrack entry does not need to outlive the actual TIME_WAIT window. On a pure stateful firewall without NAT you would be more careful; on this box, 30 seconds was fine and cut the steady-state TIME_WAIT count to around 1,300.

Persisted to /etc/sysctl.d/50-conntrack.conf:

net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_established = 86400

The established timeout dropped from 5 days to 1 day. A live connection will refresh its conntrack entry with each packet. A connection that has been idle for 24 hours is either dead and burning a slot for no reason, or it can survive the brief period when conntrack ages it out and recreates the entry on the next packet.

The monitoring that was missing

The ratio nf_conntrack_count / nf_conntrack_max had been climbing for weeks before the incident. There was no alert on it because nobody had thought to add one.

If you run node_exporter, conntrack metrics are exposed by default since 1.0:

node_nf_conntrack_entries
node_nf_conntrack_entries_limit

The ratio node_nf_conntrack_entries / node_nf_conntrack_entries_limit is your utilization. Alert at 0.70. Page at 0.85. If you hit 0.99 without a prior page, the monitoring is broken.

Without node_exporter, two reads and some arithmetic:

count=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
max=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
echo "conntrack_utilization $(echo "scale=4; $count / $max" | bc)"

Wire that into whatever you use for custom metrics. The query is four milliseconds. There is no excuse for not having it.

The default that aged poorly

65,536 is an old number. It comes from a time when a “large” server had a few gigabytes of RAM and 65K conntrack entries represented real overhead. At 352 bytes per entry, 65,536 entries is 23MB. On a modern machine that is a rounding error.

Newer kernels have gotten better about this. The auto-scaling formula in 5.15+ bases nf_conntrack_max on available memory. On a 32GB machine you will likely see a default closer to 200,000. That is better than 65,536. It is still not a substitute for setting the value explicitly based on your actual expected connection count.

The 4.x kernel on this box predated the auto-scaling improvements. The default was never changed at provisioning. Fourteen months later, at 03:14 on a Saturday, that became my problem.

I will admit: the kernel developers have put real work into making conntrack self-tune over the years. The newer defaults are reasonable starting points. But “reasonable starting point” and “correct for your workload” are not the same thing, and a reasonable starting point does not tell you what to alert on.

What to check when you provision a NAT box

The engineer who set up this box did what most people do: got the iptables rules working, tested that NAT worked, deployed it. Nothing in the standard provisioning checklist says to verify conntrack capacity for expected traffic.

There should be. For any box doing stateful packet inspection or NAT:

1. Check the current nf_conntrack_max. If it is under 200,000 and you expect sustained traffic, set it higher.
2. Check that the conntrack module parameters persist across reboots via sysctl.d.
3. Add conntrack utilization to monitoring before the box handles production traffic.
4. Run a load test and watch nf_conntrack_count under realistic conditions.

None of this is hard. The failure mode is that all of it is invisible until it stops working. CPU, memory, and bandwidth were all fine on this box. The metric that mattered was three /proc reads away and nobody knew to take them.

It is the same situation as the inode count that crept toward zero while df reported forty percent free and the XID counter that climbed for two years until PostgreSQL stopped accepting writes. The number that puts you in an incident at 3am is never the number on your dashboard. It is always the number you have to know exists before you think to look for it.

How the kernel handles a full table

When the conntrack table is full, the kernel drops the packet silently. No RST. No ICMP unreachable. The SYN arrives, no conntrack entry gets allocated, the packet is discarded, and nothing tells the sender what happened. The client waits for the retransmit timer, tries again, and either gets lucky on a retry or times out.

From the application’s perspective this is indistinguishable from a slow server or a slow network. The server is healthy. The destination is healthy. The network hardware between them is fine. The NAT box dropped the SYN and told nobody.

This is why dmesg is always in the first five things I check on a production networking problem. The kernel sees things the application layer never sees, and it writes them down. nf_conntrack: table full, dropping packet tells you precisely what went wrong. It just does not volunteer that information unless you look.

dmesg -T | grep -i "conntrack\|dropping"

If that comes back empty, this particular failure mode is off the table. If it comes back with a wall of suppressed “table full” messages from forty minutes ago, you have your answer and the rest of the investigation is just cleaning up.

Twenty-four slots in a 65,536-entry table. One sysctl write. Two prometheus metrics and a 0.70 threshold. That is the whole story, start to finish. None of it helps you at 03:14 if you have not done the work in advance.