🔌 Stripe's Proxy Was Opening Hundreds Of Thousands Of Connections

A Go proxy, one connection per host, and 3,000 days of latency saved every day

🤔TL;DR

Stripe ran Envoy for almost ten years, then replaced it with their own Go proxy, mesh-proxy. The payoff:

• ~50% less CPU under high load, ~50% lower latency

• ~3,000 days of inter-service latency saved every day

• One connection per host, down from one per worker

• New Ruby load balancing bought 15-20% more burst headroom

Why rebuild at all??? Stripe targets 99.9995% reliability on infra carrying about 1.6% of global GDP. One API call fans out into many internal requests, so a few hundred microseconds in the data plane compounds fast.

The reason they left was connection count. The math is worth walking through even if you never touch a proxy.

Stripe can throw a team at making internal delivery this reliable. The same problem hits one hop out, when you’re pushing events to customers and their agents. Most teams start with a hand-rolled webhook sender, and it holds until retries, signing, idempotency, and fan-out turn it into a second product nobody wanted to own.

That’s where Svix (today’s sponsor) comes in.

---

Svix: Become event-driven in a day

Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.

---

🔌 The connection math that broke Envoy

Envoy runs one worker per CPU core, and under steady traffic each worker opens its own connection to every upstream it can reach. Connections scale with core count. Host count is irrelevant.

Two 8-core clients talking to two servers: every worker on both clients holds a connection to both servers, so each server carries 16 connections for just those two clients.

Now point that at a fleetwide service, say the pipeline ingesting tracing data from everything you run. It takes traffic from roughly every core in the fleet, so it holds roughly one connection per core company-wide. That’s hundreds of thousands. Discord hit the same class of problem from the other side when they cut WebSocket traffic 40% to keep connection overhead from swamping the gateway.

Two ways that bites:

• You size machines for connection count, not request load. Expensive for a service doing little real work.

• Sockets pile up silently until one day there are too many, the service tips over, and you’re debugging an incident with no obvious cause.

Stripe hit the second one.

⚠️ Every upgrade was a gamble

Thousands of distinct services make Hyrum’s Law literal: with enough consumers, every behavior of the proxy is something a downstream depends on. A minor version bump ships a behavior change to all of them at once.

In 2024, an Envoy upgrade pulled in a new nghttp2 with a defense against the HTTP/2 rapid-reset attack. On default settings, that defense started rejecting legitimate traffic to one of Stripe’s services. A QA load test caught it before prod. Miss that catch and the first signal is real requests getting dropped, and one incident like that can spend the whole year’s reliability budget. Same reason Netflix breaks their own services on purpose: the regression you find in a load test is free.

That’s the standing tax of a shared proxy across a big, unusual fleet. You inherit every upstream change and find out in your own traffic whether it breaks you.

🚫 The feature they couldn’t ship

Stripe’s Ruby services run on big, expensive machines. The team had a routing design to spread traffic more evenly across Ruby backends and shrink those machines. Building it into Envoy meant customizing past the point they’d maintain, so it sat on the shelf.

That’s the third, quietest cost of an adopted proxy: it caps what you can build in your own data plane. Every custom idea has to fit the extension points you’re given, and this one didn’t.

🛠️ What mesh-proxy does differently

Mesh-proxy is Stripe’s Go replacement. It ships its own HTTP/1.1 and HTTP/2, talks straight to the service registry for discovery, and supports:

• Retries and hedging (fire a second request when the first is slow)

• Rate limiting

• Dynamic priority-based routing

The core change is concurrency. Instead of one worker per core, mesh-proxy runs one read and one write goroutine per connection and multiplexes many requests over a single shared connection. Goroutines are cheap enough to spin up per connection without thinking about the cost. That flips the earlier math: one connection per host, where Envoy needed one per worker.

Back to the two 8-core clients and two servers. After migrating, each client holds one connection per server, so each server carries two connections instead of sixteen. The tracing service that was drowning in sockets now fits on a smaller machine.

The tradeoff is real. Multiplexing over a shared connection means coordinating request state across goroutines, more work than letting each worker own its own. Stripe took that on because the connection savings paid for it.

They also dropped xDS (Envoy’s push-based config protocol) for reading the registry directly. Their Envoy setup generated more xDS updates than the topology actually changed, burning CPU and memory on every machine. One flooded service shows a step-drop in both the moment mesh-proxy rolled out.

📈 What it bought them

• Half the CPU under high load

• ~50% lower latency (that’s the 3,000-days-a-day figure)

• The shelved Ruby load balancing shipped: 15-20% more headroom before a burst overwhelms a machine

• A rewritten priority-routing system that unlocked changes across deployment and rate limiting

Owning retries and hedging cuts both ways. Client-side retries are the classic path from a small hiccup to a retry storm that takes the whole dependency down. With the retry budget in code Stripe’s own team reviews, they enforce it consistently. They also own getting it right.

🤔 The part I’d push back on

Building in-house beat running battle-tested open source, and for Stripe that reads as true. It’s also the most expensive kind of true.

Look at the nghttp2 incident again. That upgrade existed because nghttp2’s maintainers built the rapid-reset defense, protecting against a protocol-level DDoS that hit the whole industry in 2023. Stripe got it as a free version bump. It broke them once, in QA, and they changed a config.

mesh-proxy has its own HTTP/2 now. When the next protocol-level HTTP/2 attack lands, and there’s always a next one, Envoy shops get a patch from people who do nothing but maintain that proxy. Stripe writes the patch. Every flow-control edge case and every future HTTP/2 CVE is theirs to find and fix, forever.

Defensible at 1.6% of global GDP and a 99.9995% target. Stripe has run this play before, including building their own database instead of bending an existing one to their uptime number. The precondition is a team whose whole job is this one component, at a scale where 50% latency pays for that team many times over.

For almost everyone else it runs the other way. The connection math travels anywhere. The build decision is specific to Stripe’s scale and headcount: copy it without the scale and you get a worse Envoy only your team can debug. If your proxy is costing you money, measure how connections scale first, with cores or hosts, and whether a fleetwide service is quietly holding a socket for every core you own. That’s usually fixable in the proxy you already run.

✅ What to check in your own mesh

Before anyone pitches a rewrite, get three numbers:

• Connections per upstream, and what they scale with. On a worker-per-core proxy, graph a fleetwide service’s open connections against fleet size rather than request rate. If the lines track, you have Stripe’s original problem right now.

• Config-update volume per machine. Push-based planes like xDS can send far more than the topology changed. Measure how much load is pure config churn you could delete.

• Where retry and hedging budgets live. Scattered across client libraries means nobody owns the total, and it only surfaces mid-outage.

None of these need a new proxy. All three separate a data plane you understand from one that surprises you at 3am.