TLDR

August 17. GitHub went sideways for 7 hours 47 minutes.

~20% error rate on web and API traffic
~50% on archive and raw content downloads
Issues, Pull Requests, Actions, Copilot, SAML/OIDC, SCIM, and Team Sync all degraded
Copilot Token Service went from 7,000-9,000 RPS to 70,000-100,000 RPS

One Istio sidecar pod in Central US hit its concurrency ceiling and failed to autoscale. That took down four HAProxy nodes, which took down the gateway auth path. Most of GitHub was back by 16:36 UTC and Actions by around 18:03. Copilot limped until 21:02, because a latent retry bug in VS Code turned a slow endpoint into a load generator running on millions of laptops.

The capacity problem was mitigated in the afternoon. The retry storm it set off ran for another five hours.

The sidecar nobody was scaling on

Istio puts a proxy sidecar next to every service pod, and all traffic flows through it. You get mTLS and routing without touching application code. You also get a second set of concurrency limits per pod, sitting somewhere entirely different from the application’s.

The autoscaling policy watched the host service. When Central US hit a new traffic peak, the sidecar saturated first. The autoscaler read host metrics, saw headroom, and left the deployment alone.

A pod that cannot accept connections and cannot be replaced by more pods sheds load. Whatever sits in front of it eats the difference.

The cascade, in order

Sidecar saturates, autoscaler does nothing.
Refused connections pile onto HAProxy in Central US.
Four HAProxy nodes exhaust their flow limits, one after another.
Those four front the gateway auth path, so auth latency climbs.
Callers waiting on a fast dependency start treating slow responses as failures.
Retry logic engages, and the auth path now gets more traffic than it did before it broke.

Auth is the worst place in a system to lose capacity, because nearly every request touches it. GitHub knows this one well already: a 43-second network partition in 2018 cost them 24 hours of degradation, and the trigger was similarly small next to the blast radius.

Recovery required breaking things harder

Some failing traffic was moved out to Northern Virginia and served successfully there while the Central US network failure was debugged. Regional failover did its job for the bulk of user requests. The retry behavior travelled with the clients, so Northern Virginia inherited the herd along with the load.

The Central US fix is the interesting one. Pausing HAProxy on the saturated nodes, all four simultaneously, produced immediate broad recovery.

Staggering that pause would have pushed load onto whichever node was still accepting traffic and kept the saturation alive. A feedback loop has to be broken everywhere at once, which is an uncomfortable thing to do at 16:00 UTC on a Monday with 20% of your traffic already failing.

Byte-Sized Design

GitHub's 8-Hour Outage Was Mostly Retries

Comments on this post are for subscribers