How Three Linux Networking Bugs Caused Kubernetes API Timeouts in GKE
Three failures in three different layers can look like one ordinary timeout. That’s what made this case so disruptive: pods couldn’t reliably reach the Kubernetes API server, and the eventual fix had to reach all the way into the Linux kernel.
The support ticket arrived in January 2026. Its description was brief: intermittent connections to the Kubernetes API server were timing out. The customer had already investigated its own operators. But as the case unfolded, roughly 800 developers were blocked from production deployments for weeks, and GKE clusters across europe-west were affected. Engineers from Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, the kernel team, and the Linux upstream community traced the failures to three separate bugs.
Here’s the part that’s easy to miss: the first suspect was a workload, but the problem ultimately sat in the network path shared by workloads across the clusters. This is how the investigation moved from a familiar timeout to a Linux kernel patch.
One timeout, a growing list of suspects
The customer operated a large, multi-cluster GKE environment in Google Cloud’s europe-west region. Its clusters used Private Service Connect (PSC) to reach the GKE control plane privately. They also ran a range of system workloads, including Twistlock security scanners, External Secrets Operator, Cilium network policy, and internal Kubernetes operators.
The first errors showed up in a user acceptance testing (UAT) cluster in europe-west3. A few days later, a second cluster in the region began seeing them too. Applications reported timeouts to the cluster’s internal kube-apiserver virtual IP, routed through PSC:
TimeoutError: Connect call failed ('1.2.3.4', 443)
dial tcp 1.2.3.4:443: i/o timeout
# 1.2.3.4 is the cluster's internal VIP for the kube-apiserver,
# routed through Private Service ConnectAt first, the failures came in bursts. The customer suspected a custom operator built with the kopf framework, but the errors had begun eight days before it was deployed. Shutting it down didn’t stop them.
“The issue persists even after completely scaling down the External Secrets Operator. The errors are appearing from customer workloads, and system-level pods we haven't touched.”
The investigation worked through the likely causes one by one. The custom operator was ruled out by the timeline and by the fact that errors continued after shutdown. At the customer’s request, the team removed External Secrets Operator; timeouts persisted in workloads that needed Kubernetes control plane (KCP) connectivity and in cluster system pods. Security agents were uninstalled, with no change.
Conntrack exhaustion looked plausible: too many half-open connections might have filled the control plane nodes’ operating system connection tracking table. It hadn’t. The table was at about 50 percent of its default maximum of 1.3 million entries. The team also increased the KCP VM size. kube-apiserver latency remained healthy, but the timeouts continued.
The reach of the failures changed the shape of the case. They appeared in five clusters, across DaemonSets, Deployments, and system pods. Developer laptops and GitHub Actions runners were affected too. If you’re working through an incident like this, that breadth matters: the evidence pointed away from a single broken workload and toward the route between clients and the API server.
From intermittent errors to packet captures
- Day 1: Intermittent I/O timeouts appeared in one UAT cluster in europe-west3 and were easy to dismiss.
- Day 10: The errors became much more frequent and persistent.
- Day 15: Multiple affected clusters were confirmed. Around 800 developers were blocked, production releases were frozen, and the case was escalated as a P0 bug.
- Day 20: The custom operator, security agents, conntrack exhaustion, and control plane resizing had all been ruled out.
- Day 22: Engineers captured packets at both the control plane node’s network interface and inside the kube-api container.
Once the application layer had run out of explanations, the packets became the most useful evidence.
Following the connection to the control plane
A pod’s request to a GKE API server using PSC takes a longer route than the destination IP suggests:
The address 1.2.3.4:443 is a virtual endpoint, not the control plane itself. Google’s networking fabric intercepts the connection, translates it through a PSC endpoint, sends it across an internal passthrough Network Load Balancer, and delivers it to the KCP VM. There, Envoy acts as a local proxy in front of the kube-apiserver process.
PSC also adds a proxy protocol header to each new connection so the kube-apiserver can identify the client’s real IP. That detail would matter later.
Simultaneous captures from the KCP VM’s PIGv interface—a programmable virtual NIC—and from inside the kube-apiserver container revealed an unexpected TCP exchange. The pods’ SYN packets reached the VM. But instead of getting the SYN-ACK that should follow, some clients received an ACK associated with a different, older connection. It wasn’t an RST, and the server wasn’t silent. It was an acknowledgment that belonged to a connection that had already ended.
Wireshark, the network protocol analyzer, marked the streams “TCP Port numbers reused.” One example showed a completed connection, followed by repeated ACK retransmissions. The client then reused source port 53426 for a new connection and got an ACK carrying sequence information from the previous stream instead of a SYN-ACK:
Time Source Port → Destination Port Info
09:13:41 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Seq=0 Win=42600
09:13:41 10.117.0.2 443 → 10.0.13.77 53426 [SYN, ACK] Seq=0 Ack=1 Win=64768
... (data exchange, connection completes)
09:13:42 10.117.0.2 443 → 10.0.13.77 53426 [TCP Retransmission]
09:13:42 10.117.0.2 443 → 10.0.13.77 53426 [TCP Retransmission]
... (many retransmissions of the same ACK)
# 2 minutes later, same port 53426 is reused for a new connection
09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Seq=1498 ← NEW connection
(expected: SYN-ACK)
(received: ACK Seq=1498 Ack=1025 — belongs to PREVIOUS stream)
09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Retransmit...
09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Retransmit...
→ eventually: i/o timeoutThe KCP VM was answering a new SYN with an ACK from an earlier connection that had used the same source port. The old connection’s state hadn’t been cleared properly.
One packet trace captured the close sequence this way:
“The KCP server sends FIN,ACK; client responds with FIN,ACK and then ACK. But KCP receives ACK before FIN,ACK. KCP server never ACKs the FIN,ACK — continuous retransmission of ACK. RST does not clear state.”
The ACK that never got through
While one team examined packet captures, another checked metrics for the Private Service Connect user space packet switch (USPS), Google’s software switching layer for NAT and proxy protocol header injection. In Andromeda, Google’s networking fabric, a counter named “Proxy protocol drop: invalid ACK packet” had stayed flat for months. In February 2026, it jumped to several drops per second in the affected europe-west regions.
The spike coincided with a USPS version rollout. The release addressed a different bug, but it also changed how PSC handled a particular class of packet. The metrics connected the rollout to the failures; they didn’t, by themselves, explain why an ACK was being dropped.
The missing piece was TCP’s challenge ACK, a security mechanism defined in RFC 5961. After a connection closes, the server’s TCP state machine may briefly retain its sequence numbers. If a new connection arrives with the same source and destination ports—the same five-tuple—the kernel can’t safely assume it’s a fresh connection. The packet could also be stale traffic from the old one, or an attempt to inject data into a connection the kernel still considers open.
RFC 5961’s answer is a challenge ACK. The server acknowledges the last known sequence number from the old connection. A legitimate client recognizes that the response belongs to stale state and sends a reset (RST); the server can then accept a subsequent SYN normally.
# Step 1: New client sends SYN on a reused port
Client → Server: SYN (seq=X, same 5-tuple as an old connection)
# Step 2: Server is in FIN_WAIT_2, sends challenge ACK
Server → Client: ACK (ack=old_seq — "is this really you?")
# Step 3: Client recognizes the stale state, resets it
Client → Server: RST
# Step 4: State is cleared. Server accepts the next SYN normally.
Client → Server: SYN (new connection)
Server → Client: SYN-ACK ← handshake succeedsThat recovery depends on the challenge ACK reaching the client. The new USPS behavior blocked it.
The USPS change was intended to address a real security issue in PSC’s proxy protocol handling. It dropped packets classified as “invalid ACKs” during the handshake, including ACKs arriving before the initial SYN-ACK. In ordinary circumstances, those packets would be anomalous. A challenge ACK, though, arrives before the new handshake is complete because it is responding to the SYN. USPS treated it as invalid and silently discarded it.
PSC’s NAT made the edge case more common. Many pods, spread across nodes and namespaces, could share an endpoint and reuse the same source port after translation. That produced frequent five-tuple collisions. The USPS release hadn’t created challenge ACKs; it had started dropping them, interrupting the TCP stack’s existing way of clearing stale connection state.
Mitigation: rolling back the change
The rollback bought time; the kernel held the answer
Rolling back the February USPS change restored the earlier behavior: challenge ACKs passed through instead of being dropped. The “Proxy protocol drop: invalid ACK packet” rate fell toward baseline almost at once, and by early March 2026 the customer confirmed that every affected cluster had recovered, with no more timeout errors. That was relief, not resolution. The rollback restored behavior the system had accidentally come to depend on. It didn’t explain why KCP VMs were keeping connections inFIN_WAIT_2 for more than two minutes. That lingering state gave Private Service Connect’s NAT layer time to reuse a port, setting up the challenge ACK that USPS then dropped. The investigation’s central lesson is that no single component tells the whole story.
Three failures, one stuck connection
Simultaneous packet captures at the KCP VM’s host network interface and at the PIGv—the virtual NIC visible to the VM—revealed the first piece. Packets arrived in order at the host, then were reordered as they passed through USPS and reached the PIGv. Private Service Connect’s miss-thread processing helps explain how. A packet that opens a flow takes a slower path while the system adds the proxy protocol header and sets up NAT state. Later packets on that flow use a faster path. During connection shutdown, that difference can let a FIN or RST arrive out of order relative to ACKs. What the KCP VM received:# Expected close:
Client → Server: FIN, ACK
Server → Client: ACK
Server → Client: FIN, ACK
Client → Server: ACK
# Connection closes.
# Reordered close:
Client → Server: ACK
Client → Server: FIN, ACK
Server → Client: FIN, ACK
Client → Server: FIN, ACK
# The server should acknowledge this final FIN, but doesn't.
The reordered client FIN arrived after the server had sent its own FIN. The server moved through FIN_WAIT_1 and into FIN_WAIT_2 after receiving the client’s ACK. Then the client’s delayed FIN arrived—and the kernel failed to process it.
The second failure kept the socket there. Linux’s tcp_fin_timeout, set to 60 seconds on the KCP VM, is meant to close abandoned sockets in FIN_WAIT_2. But it applies only to orphaned sockets, meaning no process still holds them. Envoy, the proxy in front of kube-apiserver on the VM, had called shutdown(SHUT_WR): it closed its write side while keeping the socket open to receive data. That’s valid TCP behavior, but it also meant the kernel did not treat the socket as orphaned. The timeout never began.
Socket state captured with netstat:Proto Local Address Foreign Address State
tcp 10.128.0.5:443 10.0.13.77:53426 FIN_WAIT_2
# The socket has been in FIN_WAIT_2 for more than 120 seconds.
# tcp_fin_timeout=60 does not apply while Envoy holds the socket
# open with SHUT_WR.
#
# Two minutes later, Private Service Connect NAT reuses port 53426.
# The new SYN hits the existing 5-tuple, prompting a challenge ACK.
# USPS drops that ACK, and the connection times out.
The third failure was in Linux 6.1, the kernel version underlying the Container-Optimized OS (COS) on the KCP VMs. The relevant reference is Google’s [Container-Optimized OS documentation](https://docs.cloud.google.com/container-optimized-os/docs).
RFC 5961 permits certain older ACKs when they fall within an acceptable sequence-number window, helping TCP handle reordering. In one kernel 6.1 code path, tcp_ack() returned zero for an old but acceptable ACK. The FIN processing code treated that result as a reason to stop, and silently discarded the FIN. A healthy kernel should have processed it and moved the socket from FIN_WAIT_2 to TIME_WAIT.
The upstream fix is commit 795a7dfbc3d9, authored by Eric Dumazet:
net: tcp: accept old ack during closing
tcp_ack() returns 0 when ack is old but acceptable wrt rfc5961.
In FIN_WAIT_2, this caused tcp_fin() to never be called for a
valid received FIN, leaving the connection stuck indefinitely.
Fixed: properly handle old-but-acceptable ACKs during connection close.
Introduced in 6.1, fixed in 6.9.7. LTS backports: 6.6.y and 6.1.y.
The bug was fixed upstream in Linux 6.9.7, then backported to the 6.6.y and 6.1.y long-term support branches. GKE’s KCP VMs were using COS builds based on 6.1 and 6.6, so the patch had to be backported and delivered in a COS image update.
Why the failures compounded
Packet reordering alone would have been disruptive, but a healthy kernel could still process the FIN and eventually clean up. Envoy’s half-close is valid TCP behavior; it became a problem because the FIN never got processed. The kernel bug could leave some connections lingering, but without Private Service Connect’s port reuse, its impact would have been more limited. Together, the sequence was much worse: reordering put the FIN out of order; the kernel bug discarded it; Envoy kept the socket from becoming orphaned, so its timeout didn’t start. The socket remained inFIN_WAIT_2. Private Service Connect NAT reused the port, the kernel sent a challenge ACK, and USPS dropped it. The result was an i/o timeout.
The remedy also had three parts. In February 2026, USPS was rolled back to restore the previous behavior, and the customer’s clusters recovered by early March. A separate correctness change in the Private Service Connect proxy protocol layer was designed to handle the challenge ACK case properly; the team added a PacketDrill test to catch regressions. Finally, the Linux kernel fix for old ACKs during connection close shipped in GKE 1.33.5 and later, resolving the issue at the OS level for GKE versions running the updated COS images.
Why packet captures mattered
The timeout looked plausible at several layers, but each explanation was incomplete. The decisive evidence came from comparing packet captures at the host NIC, the PIGv, and inside the container. Those views exposed the reordering. Connecting the captures to the February 18 USPS metric spike identified the dropped challenge ACKs; reproducing the issue in a lab exposed the kernel failure that made the socket stick. This took two weeks of work from engineers across seven or eight specialties, including GKE networking, Private Service Connect infrastructure, Hostnet, Andromeda, KCP, and the kernel community. The case notes, shared bug tracker, and carefully collected captures let people with different parts of the picture work from the same evidence. No one team had the whole cause. The customer returned to production, and Linux gained a fix for a subtle TCP failure. The rollback restored service; the full diagnosis required following one connection across networking layers, proxy behavior, and the kernel state machine. That’s the part worth carrying into the next incident: a familiar timeout can be the last symptom in a chain of failures that only becomes visible when you compare what each layer actually saw.This post is based on real case data from an anonymized customer support engagement and internal bugs. The Linux kernel fix referenced is commit 795a7dfbc3d9, authored by Eric Dumazet. Thanks to the engineers across GKE, Private Service Connect, Hostnet, Andromeda, COS, and the Linux networking community who helped resolve the issue.