The ephemeral port range had eight hundred slots left

04:17. The pager fires on a 14% error rate from inventory-api. The error, logged on every failed request, is EADDRNOTAVAIL on outbound database connections. Not all connections. Not a pattern anyone on the preceding shift had identified. One in seven, random, and climbing.

EADDRNOTAVAIL is errno 99 on Linux. “Cannot assign requested address.” When a process opens a TCP connection and the kernel returns that error, it means the kernel tried to pick a local port for the outgoing socket and found none available. The ephemeral port range was full.

The service connects to Postgres. It uses a connection pool. The pool documentation says pooling is enabled by default. That statement is true. It is also part of the problem.

What the first five minutes looked like

SSH in, run the obvious list.

top: CPU 19%. Memory 5.8GB of 32GB. Fine.

ss -s:

Total: 8914
TCP:   8307 (estab 2901, closed 128, orphaned 0, timewait 5278)

5,278 connections in TIME_WAIT. That is the number that mattered. I did not know it yet. I moved on to look for something more interesting. Six minutes spent.

df -h: all filesystems had room. Not an inode problem this time.

Application logs:

dial tcp: address inventory-db.internal:5432: connect: cannot assign requested address
dial tcp: address inventory-db.internal:5432: connect: cannot assign requested address

“Cannot assign requested address.” That is connect() returning EADDRNOTAVAIL, the kernel’s way of saying it went through the full ephemeral range looking for a free port and came up empty. Not a routing problem. Not a DNS problem. Not the database refusing connections. The kernel cannot hand this process a source port.

ss -tan state time-wait | wc -l

27419

27,419 connections in TIME_WAIT. I went back to that ss -s output and did the math.

The ephemeral port range

When a process opens an outbound TCP connection without binding a specific local address, the kernel chooses a source port from the ephemeral range. On Linux:

cat /proc/sys/net/ipv4/ip_local_port_range

32768	60999

32,768 to 60,999. That is 28,232 available ports. It sounds like a lot. For a service that holds connections open in a pool, it is more than enough. For a service that opens and closes a fresh connection on every query, at 480 queries per second across a 32-core thread pool, it is not.

The problem is TIME_WAIT. When a TCP connection closes, the side that sends the final ACK enters TIME_WAIT and stays there for 60 seconds. Linux defines TCP_TIMEWAIT_LEN as 60 seconds in the kernel source, reflecting the 2*MSL convention where Maximum Segment Lifetime is 30 seconds. During those 60 seconds the kernel holds the source port. It cannot be reused for another connection to the same destination on the same protocol. The TIME_WAIT period exists to handle delayed segments. A stale packet from the old connection should not corrupt a new connection that happened to reuse the same four-tuple.

The math is not complicated. 480 new connections per second, each port in TIME_WAIT for 60 seconds: 28,800 ports committed to TIME_WAIT in steady state. The range holds 28,232. The service had been running at slightly-under-critical utilization for weeks. Traffic grew, the range filled, and at 04:17 the race was consistently going the wrong way.

cat /proc/net/sockstat | grep TCP

TCP: inuse 2901 orphan 0 tw 27419 alloc 8307 mem 2819

tw 27419. That number was right there. It does not alert you. It does not cross-reference itself against the ephemeral range and estimate how many minutes remain before the errors start. It just sits in /proc/net/sockstat and waits.

The connection pool that had zero idle slots

This is the part that took a while to accept.

The service was written in Go using the standard database/sql package with a standard Postgres driver. The developer who wrote it knew about connection pooling. The service was not the work of someone who skips the docs. The pool was explicitly configured:

db, err := sql.Open("postgres", connStr)
if err != nil {
    log.Fatal(err)
}
db.SetMaxIdleConns(0)
db.SetMaxOpenConns(50)

SetMaxOpenConns(50) is correct. Fifty concurrent connections maximum. Reasonable for this workload.

SetMaxIdleConns(0) is the problem.

In database/sql, SetMaxIdleConns(0) means “keep zero idle connections in the pool.” The documentation says: “If n <= 0, no idle connections are retained.” After each query completes, if no other goroutine is waiting for a connection at that exact instant, the connection closes. The port enters TIME_WAIT. A fresh connection will be opened for the next query.

The developer who wrote db.SetMaxIdleConns(0) was trying to say “no limit on idle connections.” Zero means unlimited in Unix. SetMaxOpenConns(0) does mean unlimited open connections. SetMaxIdleConns(0) means something different: no idle connections retained at all. Same function, opposite semantic for the zero value.

The Go docs describe this correctly. “If n <= 0, no idle connections are retained” is unambiguous if you read it. The mistake was assuming the two sibling functions used the same zero-means-unlimited convention. They do not.

Under normal load, this configuration is invisible. Queries come in, the pool opens connections, queries complete, connections close. The TIME_WAIT count stays in the low hundreds. Under the load this service saw at peak hours, “queries complete, connections close” became “480 connections per second open and close, each burning a fresh port for 60 seconds,” and the ephemeral range stopped having room.

Nine months the service had been running with this configuration. The prior incidents had been logged as “transient database connectivity, self-resolving.” They had always been this.

The fix

Three changes.

First, fix the pool configuration:

db.SetMaxIdleConns(10)
db.SetMaxOpenConns(50)
db.SetConnMaxLifetime(5 * time.Minute)

Keep ten idle connections. Queries reuse idle connections instead of opening fresh ones. The new-connection rate dropped from ~480/second to ~6/second within the first minute after the restart. TIME_WAIT count fell from 27,419 to around 400 over the following few minutes. The EADDRNOTAVAIL errors stopped.

Second, widen the ephemeral port range as a baseline. The default is adequate for reasonable connection behavior. It is not a substitute for reasonable connection behavior, but it provides headroom for traffic spikes:

sysctl -w net.ipv4.ip_local_port_range="1024 65535"

64,511 available ports instead of 28,232. Persisted in /etc/sysctl.d/50-network.conf. This is not a fix for a broken pool. It is margin that buys time to find the broken pool instead of scrambling through the most confusing part of the incident at 04:17.

Third, enable tcp_tw_reuse:

sysctl -w net.ipv4.tcp_tw_reuse=1

This lets the kernel reuse a TIME_WAIT socket for a new outbound connection when the socket has been in TIME_WAIT for at least one second and the new connection cannot collide with the old connection’s delayed segments. It is safe for outbound connections. It has been safe since the late 2.6 era. Do not confuse it with tcp_tw_recycle, which was removed in kernel 4.12 because it broke connections behind NAT. tcp_tw_reuse and tcp_tw_recycle are two characters apart and have nothing in common operationally. One is safe and useful. The other will eventually destroy production in a way that takes six hours to diagnose.

echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.d/50-network.conf
sysctl -p /etc/sysctl.d/50-network.conf

What database/sql exposes and what this service ignored

database/sql has a Stats() method on the *DB object. It returns a DBStats struct:

stats := db.Stats()
// stats.OpenConnections  - current open connections
// stats.Idle             - current idle connections
// stats.WaitCount        - total goroutines that waited for a connection
// stats.WaitDuration     - total time spent waiting
// stats.MaxIdleClosed    - connections closed because MaxIdleConns was exceeded
// stats.MaxLifetimeClosed - connections closed due to MaxConnLifetime

If MaxIdleClosed is increasing rapidly, connections are being closed because there are no idle slots to park them in. With SetMaxIdleConns(0), MaxIdleClosed would have been incrementing on every single query. The service exported CPU, memory, request rate, and p99 latency. No pool stats. So the pool had been misconfigured for nine months and the only observable symptom was the occasional burst of errors that “self-resolved,” because they did self-resolve, once the TIME_WAIT entries aged out fast enough to free up ports.

If you use database/sql and you do not export pool stats, you do not know what your pool is doing. Exporting stats.Idle and stats.MaxIdleClosed as Prometheus gauges takes about ten lines of code. Add them before the service handles production traffic.

What to check when a service opens database connections

Before a service connecting to a database handles production load:

1. Read the specific pool documentation for the driver in use. Not the framework’s docs, not the ORM’s docs. The driver’s. The zero-value behavior for pool size parameters is not standardized across languages or libraries and it is not always what you expect.
2. Export pool metrics. In Go: db.Stats(). In Python with psycopg2: connection_pool.closed, connection_pool.size. In Java with HikariCP: the metrics registry. Whatever the driver exposes, wire it to your metrics system before launch.
3. Run ss -s at realistic load and look at the timewait count. If it is in the tens of thousands, the service is opening and closing connections at a rate that will eventually exhaust the ephemeral range under higher load.
4. Check net.ipv4.ip_local_port_range. Widen it from the default if the service handles significant outbound connection volume.
5. Enable net.ipv4.tcp_tw_reuse=1. There is no downside for outbound connections and it provides headroom when TIME_WAIT counts spike during traffic peaks.

The pattern holds

tw 27419 was sitting in /proc/net/sockstat for weeks. The right number to watch, readable in one command, free. The kernel tracks these things. It does not cross-correlate them against the port range and tell you how close you are to the edge. That is your job, and the job does not get done unless someone does it before the 04:17 page.

This is the same failure mode as the conntrack table that had twenty-four slots left, the disk at forty percent free while the inodes hit zero, and the transaction counter that ran for two years before it stopped accepting writes. The number that puts you in an incident at 4am is never the number on your dashboard. It is the number you have to know exists before you think to look.

27,419 TIME_WAIT connections. 28,232-port range. 813 slots left, and the service opening 480 new ones per second.

One line: db.SetMaxIdleConns(10). That is the whole fix. The nine months and the 04:17 page are what skipping the pool stats costs.