$RodHat_
Console Tips

SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits

Published by

SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits
Photo: AI-generated — no human photographer / RodHat AI Cover

Before Linux 3.9, a multi-process TCP server meant one socket shared across all workers. You call socket(), bind(), listen(), then fork. Every worker blocks on accept() against the same socket. A connection arrives, the kernel wakes every sleeping worker, one wins the race, the rest go back to sleep. This is the thundering herd problem and it is as old as Unix. The kernel worked around it in 3.x with SOCK_NONBLOCK and edge-triggered epoll on the shared socket, which prevented the useless wakeups. It did not fix the underlying serialization: the accept queue is still protected by a single socket lock, and every accept call contends on it.

The other option was accepting in one dedicated thread and handing connections off to workers. This moves the bottleneck from the kernel lock to your dispatch code, which is only marginally better and now you own the bug.

SO_REUSEPORT takes the other approach: give each worker its own socket. Each has its own accept queue, its own lock, no contention between workers. The kernel hashes incoming connections across the group and each worker calls accept() against only its own fd.

The socket setup

Set SO_REUSEPORT on every socket in the group before bind(). That is the entire API:

#include <sys/socket.h>
#include <netinet/in.h>

int make_reuseport_listener(const struct sockaddr *addr, socklen_t addrlen)
{
    int fd = socket(addr->sa_family, SOCK_STREAM | SOCK_CLOEXEC, 0);
    if (fd < 0)
        return -1;

    int one = 1;
    if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0)
        goto err;

    if (bind(fd, addr, addrlen) < 0 || listen(fd, 1024) < 0)
        goto err;

    return fd;
err:
    close(fd);
    return -1;
}

Call this in each worker after fork, or in each thread before entering the accept loop. The kernel groups sockets by address:port. For TCP, incoming connections are distributed using a 4-tuple hash: source IP, source port, destination IP, destination port. The same 4-tuple always maps to the same socket in the group, so a client’s retransmits or retries reach the same worker, which matters if you are holding per-connection state.

UDP works the same way: datagrams hash to one socket, not broadcast to all of them.

The BPF dispatch

The 4-tuple hash is a fixed kernel policy. Linux 4.5 added SO_ATTACH_REUSEPORT_EBPF to let you replace it.

Attach an eBPF program to any socket in the group. The kernel runs it for every incoming connection instead of the hash. The program type is BPF_PROG_TYPE_SK_REUSEPORT and the context is struct sk_reuseport_md:

struct sk_reuseport_md {
    void    *data;         /* start of packet headers */
    void    *data_end;
    __u32    len;          /* packet length */
    __u32    hash;         /* kernel's own hash, if you want it */
    __u32    reuseport_id; /* stable id for the socket group */
    __u32    is_migrating; /* 1 when handling a migration (Linux 5.14+) */
};

Return a non-negative integer to select the socket at that index in the group, or SK_DROP to drop the connection.

The common use is session or tenant affinity. Parse the first few bytes of packet data, extract an identifier, and map it to a specific worker index. You can also override the kernel hash with something stable across worker restarts, which matters for the migration problem below.

SO_ATTACH_REUSEPORT_CBPF takes classic BPF instead of eBPF, with the same semantics and less capability. Either works for simple dispatch.

BPF_MAP_TYPE_REUSEPORT_SOCKARRAY

Index-based dispatch breaks when the group size changes. If you have four workers at indices [0, 1, 2, 3] and worker 1 exits, the group shrinks to three sockets and the indices shift. Any session affinity keyed to fixed slot numbers now routes some sessions to the wrong worker.

BPF_MAP_TYPE_REUSEPORT_SOCKARRAY (Linux 4.19) solves this by storing actual socket file descriptors in a BPF map. The dispatch program selects by map key instead of by position:

struct {
    __uint(type, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY);
    __uint(max_entries, 256);
    __type(key, __u32);
    __type(value, __u64);
} worker_map SEC(".maps");

SEC("sk_reuseport")
int dispatch(struct sk_reuseport_md *ctx)
{
    __u32 key = bpf_get_hash_recalc(ctx) % 256;
    return bpf_sk_select_reuseport(ctx, &worker_map, &key, 0);
}

When a worker restarts, update the map entry for its slot to point at the new socket before removing the old one. The BPF program always routes by map lookup, so there is no index shift and no window where a session key maps to nowhere.

The migration problem

None of the above touches the hard case: a socket is removed from the reuseport group while connections are in flight.

When a socket closes, connections in SYN_RECV state (three-way handshake not yet complete) get re-hashed to the remaining sockets in the group, so most of them land somewhere and complete. Connections already fully established and sitting in the accept queue waiting for a worker to call accept() are dropped. There is no mechanism for the kernel to hand them to a different socket. The client gets a RST or a timeout.

For rolling restarts, the practical options are:

Drain before closing. Get the new socket listening and added to the BPF map before removing the old socket from it. Stop routing new connections to the old socket by updating the map, wait long enough for any in-flight SYNs to have either completed or timed out (a few hundred milliseconds is conservative on a local network, longer on WAN), then close. The accept queue should be empty by then because you stopped accepting new connections and the worker was draining it. This requires that your BPF dispatch program actually respects the map and that you control the map update timing.

Linux 5.14 added BPF_SK_REUSEPORT_MIGRATE support. When set on SO_ATTACH_REUSEPORT_EBPF, the kernel calls your BPF program for each SYN_RECV connection on a closing socket, with is_migrating == 1. The program selects a replacement socket. You still lose established connections in the accept queue, but you keep the in-progress handshakes. If your traffic is bursty enough that losing accept-queue connections during a deploy is unacceptable, combine this with the drain approach: migration catches the handshaking connections, drain handles the accepted-but-unread ones.

Contention on the socket lock

SO_REUSEPORT eliminates cross-worker contention. It does not eliminate contention within a single worker’s socket. If one worker is accepting connections faster than it processes them, its accept queue lock is still a serialization point. The queue fills, the kernel starts dropping or deferring SYNs. Tune net.core.somaxconn and net.ipv4.tcp_max_syn_backlog appropriately. The Recv-Q column in ss -tlnp shows the current accept queue depth for each listening socket; a queue that is perpetually at or near its limit is the diagnostic, not a guess. The ss internals tip covers what those columns actually mean. If a single worker’s queue is chronically full, you need more workers or faster processing, not more tuning.

FreeBSD

FreeBSD had SO_REUSEPORT since 4.4BSD with different semantics: multiple sockets can bind to the same address:port, but UDP datagrams are replicated to all sockets in the group rather than distributed. This is useful for multicast receivers. It is not useful for multi-worker load distribution.

FreeBSD 12.0 added SO_REUSEPORT_LB, which provides Linux-style distribution: connections and datagrams hash to one socket, not all. Write code that should run on both systems conditionally:

#ifdef __linux__
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one));
#elif defined(__FreeBSD__)
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT_LB, &one, sizeof(one));
#endif

The eBPF dispatch (SO_ATTACH_REUSEPORT_EBPF) is Linux-only. FreeBSD’s BPF layer does not have an equivalent hook at the socket group level.

The SCM_RIGHTS tip covers the complementary pattern: accepting in one place and passing the live fd to another process without any of the reuseport machinery. Different trade-off, worth knowing both.

The kernel implementation of reuseport dispatch is in net/core/sock_reuseport.c. It is short and readable. The accept queue mechanics that underlie all of this are in W. Richard Stevens’ TCP/IP Illustrated Vol. 1 in the TCP connection establishment chapters; understanding what is actually in that queue makes the migration problem obvious instead of surprising.