Eliminating Multi-Core Socket Contention with SO_REUSEPORT and SO_INCOMING_CPU

High-throughput TCP servers on multi-core Linux systems frequently hit a latency wall long before saturating CPU execution pipelines or network link bandwidth. When hundreds of worker threads share a single listening socket, kernel lock contention on the socket accept queue degrades connection setup times. Eliminating this lock bottleneck requires shifting from single-socket multiplexing to per-core listening sockets using SO_REUSEPORT and SO_INCOMING_CPU.

The Single Accept Queue Lock Bottleneck

When multiple worker processes call epoll_wait() and accept() on the same file descriptor, the kernel serializes access through the socket lock inside inet_csk_accept(). Under TCP SYN floods or workloads with thousands of short-lived HTTP connections per second, CPU cores spend substantial cycles waiting on spin_lock_bh() across socket structures.

Historical solutions relied on a single parent process accepting connections and passing file descriptors over UNIX domain sockets using SCM_RIGHTS. That approach moves lock contention from kernel space to user space, adding IPC latency and context switches for every incoming connection.

Splitting Accept Queues with SO_REUSEPORT

Linux kernel 3.9 introduced SO_REUSEPORT for TCP and UDP sockets. When set on multiple socket file descriptors bound to the same IP address and port, the kernel builds an array of listener sockets. Incoming TCP SYN packets are hashed using a 4-tuple (source IP, source port, destination IP, destination port) to distribute connections across listener queues.

Each worker thread binds its own socket, manages its own epoll instance, and accepts connections directly from its dedicated accept queue. Kernel spinlock contention on connection arrival disappears entirely because each accept queue operates independently.

Eliminating Inter-Core Cache Invalidation via SO_INCOMING_CPU

While SO_REUSEPORT divides socket contention, standard 4-tuple hashing ignores CPU affinity. If Receive Side Scaling (RSS) on the network card delivers a packet interrupt to CPU core 1, but the 4-tuple hash selects a socket assigned to worker thread running on CPU 6, the kernel must transfer socket data across CPU cache boundaries. This cross-core traffic introduces cache invalidation penalties on NUMA nodes.

Linux 4.6 added the SO_INCOMING_CPU socket option to align socket selection with CPU interrupt processing. Setting SO_INCOMING_CPU instructs the kernel lookup function to match the socket assigned to the CPU currently handling the NIC queue interrupt.

Binding one SO_REUSEPORT socket per CPU core and pinning network interface IRQs to matching CPUs ensures packet ingress, TCP handshake processing, and application worker execution remain isolated to a single CPU cache hierarchy.

Configuring Sockets in C

Enabling per-core sockets requires setting socket options prior to calling bind() and listen(). Below is the C implementation setup for a worker process pinned to a specific CPU core:

#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
#include <sys/socket.h>
#include <netinet/in.h>
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int create_pinned_socket(int cpu_id, int port) {
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    if (fd < 0) return -1;

    int opt = 1;
    setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt));

    // Pin process execution to targeted CPU core
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(cpu_id, &cpuset);
    sched_setaffinity(0, sizeof(cpu_set_t), &cpuset);

    // Instruct kernel to match socket assignment to current CPU ID
    setsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu_id, sizeof(cpu_id));

    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_port = htons(port),
        .sin_addr.s_addr = INADDR_ANY
    };

    if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
        close(fd);
        return -1;
    }

    if (listen(fd, 1024) < 0) {
        close(fd);
        return -1;
    }

    return fd;
}

Verifying Kernel Performance with perf

To audit whether socket lock contention is present before and after applying SO_REUSEPORT, monitor kernel symbol overhead under load using Linux perf:

sudo perf top -e cycles:k --dsos=vmlinux

Without SO_REUSEPORT, high connection rates force _raw_spin_lock and inet_csk_accept to top the CPU utilization profile. With per-core SO_REUSEPORT and SO_INCOMING_CPU enabled, CPU cycles shift from kernel spinlocks directly to user space application processing.

Production Caveats

While SO_REUSEPORT improves multi-core scaling, two operational details require consideration:

• Unbalanced queues on process exit: If a worker process terminates abruptly, TCP SYN packets arriving in its socket backlog queue will receive TCP RST responses before the kernel removes the dead socket from the SO_REUSEPORT array. Applications should set SO_ATTACH_REUSEPORT_CBPF or use zero-downtime socket migration via systemd socket activation where possible.
• IRQ steering alignment: SO_INCOMING_CPU yields no latency reduction unless NIC hardware queues are explicitly mapped to matching CPU cores using /proc/irq/IRQ_NUMBER/smp_affinity or irqbalance rules.