Three correct-looking bugs left a closed socket open for days

The sockets sat in FIN_WAIT_2 for days. Not minutes, not an hour. Google’s account of the GKE API timeout incident describes control-plane connections parked in that state long after both endpoints believed the conversation was finished, and roughly 800 developers locked out of production deploys while it lasted. The InfoQ write-up reads at first like a story about one kernel bug. It is a story about three bugs in three layers, each of which behaves reasonably in isolation.

Three bugs, each one defensible

Start at the edge. A change in the Private Service Connect NAT path began silently dropping RFC 5961 challenge ACKs, the segments a TCP stack uses to confirm a peer is genuinely on the path before honoring a reset or a sequence jump. Dropping some of them under load is expected; the RFC rate-limits challenge ACKs. Dropping them consistently is not something the sender ever learns about. It just retries, and the counter nobody exports climbs.

One layer down, the virtual NIC reordered packets, so the client’s FIN arrived out of sequence. Reordering alone is survivable. TCP was built for it, and the stack normally processes the FIN once the gap fills, with no harm done.

Then the kernel. A bug in tcp_ack() on 6.1 discarded valid old ACKs during connection close. Eric Dumazet’s fix landed in 6.9.7 and was backported to the 6.6 and 6.1 LTS branches. Ask yourself which of those your nodes actually boot. In a managed cluster you may not have a clean answer without going and looking.

Alone, each bug is noise. Stacked, they produce a socket neither side will ever leave: the FIN never processed, Envoy having issued SHUT_WR and now waiting, and nothing on the path with a timer that will force the state out. Reuse that port for a new connection and the control plane answers the fresh SYN with stale ACKs from the dead session. Requests hang, clients blame the API server, and the actual peer is a zombie socket from Tuesday.

A half-close is not a cleanup mechanism. It says you have nothing more to send. It never says anyone should reap the socket, so when the last segment is missing, the reaping depends entirely on a timer that Envoy’s own shutdown just disabled.

Retries and health checks don’t reach this layer

Every conventional defense was present, in the conventional sense. Clients retried. Controllers backed off and retried again. Dashboards showed p99 climbing and then recovering, which is precisely what a flapping dependency looks like from above. Liveness probes kept passing, because a probe that opens a fresh connection is the one flow that still worked often enough to look healthy.

Here is the rule I took out of it: if a dependency degrades intermittently and every metric you own lives at the application layer, you are blind. Socket state is one command away, and almost nobody runs it.

ss -tan | awk '$1=="FIN-WAIT-2" {n++} END {print n+0}'

Run it on a proxy host that has been up a while and compare against a baseline from a quiet period. A number near zero is boring. A number that only ever climbs is the fingerprint of this failure class, and it outlives restarts of the process that owns the connections whenever the socket survives it.

Ownership is the other half. The PSC team owned the ACK drops, the proxy team owned the half-close, the kernel team owned tcp_ack(). Nobody owned the path between them. Three teams arrived at three locally correct explanations, and the whole route fit on no one’s screen. Google’s write-up admits that rolling back the PSC change restored service without explaining why the sockets stayed behind. That unexplained gap is where the real cost sat.

What an AI layer would and would not have caught

Agents do one thing well in a case like this. They read a lot of dull evidence at once and never get bored. The useful work is correlation: tcpdump captures from both ends, sequence numbers against the kernel build on each node, the timestamp where two timelines stop agreeing. Hours of tedious comparison, by someone with no other calls on the clock. That is the job worth handing off, and it is work no single engineer had context to do alone.

What an agent would not have done is author the fix. That took reading the ACK path in the kernel, knowing which segment the RFC lets you drop, and then getting the patch into two stable trees. A model summarizing logs gets you to the right question sooner. It will not get you to the answer. The answer was in source code and in a backport schedule.

The honest deployment is narrower than the pitch. Point automation at correlation. Let a human hold the state machine. And do the three cheap things before the next incident: record which kernel your nodes run, alert on FIN_WAIT_2 growth instead of treating it as trivia, and put an application-level idle timeout on every long-lived control-plane connection you own. TCP will not collect a connection both sides agreed to close. One line of config is a small price to stop pretending otherwise.

Press ⌘K to search & switch themes