Every multi-agent architecture diagram published this year looks like a microservices diagram with the boxes renamed. Planner, researcher, coder, reviewer, arrows between them, a gateway on top.

The resemblance is earned, because both take a monolith (one giant prompt, one giant service) and split it into narrow units that talk over defined interfaces.

Half the microservices playbook transfers cleanly:

• Contracts between units

• Failure isolation

• Explicit orchestration

• Tracing

The other half breaks, because the physics of the call changed:

• A service hop: ~1 ms, free, same answer every time

• An agent hop: 2 to 20 seconds, a full inference pass, different answer on retry

Circuit breakers, idempotent retries, and shared-nothing state were all designed around the first set of numbers. Copy them wholesale and you get a system that is expensive when it works and undebuggable when it fails.

No settled best practice exists yet. This is the version I would defend in a design review this week.

One thing both agree on: the units have to talk over a contract, and the moment that contract has to reach a customer’s system, it becomes a webhook. Today’s sponsor has spent years making that hop work!

Svix: Become event-driven in a day

Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.

Start Sending Webhook Today!

🏗️ The parallel is real

Start with the half that transfers.

The monolith problem is identical. A single agent with 40 tools and a 6,000-token system prompt degrades the way a 400,000-line Rails app does. Every capability you add makes every other one slightly worse, and nobody can predict which change broke which behavior. Splitting into narrow agents, each with a handful of tools and a prompt that fits on one screen, is the same relief valve as splitting services.

Interfaces over shared state is the second transfer. In 2024, most agent-to-agent handoffs were free text pasted into the next prompt. That is two services sharing a database table. Works fine until one side changes the shape and the other side silently misreads it.

MCP is filling the OpenAPI and gRPC role. A typed tool contract, a discovery mechanism, and a transport. It stopped being one vendor’s project in December 2025, when Anthropic donated it to the Linux Foundation’s Agentic AI Foundation with AWS, Google, Microsoft, and OpenAI as founding members. Over 10,000 published servers at the time.

Treat an MCP server like a service boundary:

• Version the contract

• Own it as a team

• Put real auth in front of it

One gap remains, because MCP only standardizes agent-to-tool. Agent-to-agent contracts (what a worker returns, in what schema, with what confidence) are still whatever your framework does by default, and the default is prose. Write the schema yourself, because nobody else will. A worker that returns {"findings": [...], "confidence": 0.7, "sources": [...]} can be validated, cached, and replaced. Three paragraphs of prose can only be pasted into the next prompt.

📣 Show the community what you built!

Over 43,000 engineers read this newsletter every week, and a lot of you are building things on the side: CLIs, SaaS tools, open-source libraries, courses, the app you shipped at 2am. I want more of that in front of the people who would actually use it.

• Banner/Top placement: a dedicated block right here, after the TLDR, with a thematic lead-in written by me ($200) . One sponsor per edition

• Small placement: new this month. If you’ve Built a tool, an app, a course, or an open-source project? One line, a link, and a small image for a fraction of the sponsorship price ($40) . Made for indie builders and small teams

Reply to this email or write to bytesizeddesigninfo@gmail.com and I will send over rates and open dates.

💥 Containment when the failing unit can talk

The microservices instinct is right: one flaky agent should degrade one capability, and the task should finish anyway. The mechanics differ, because an agent fails in more ways than a service does. It can:

• Time out

• Return garbage that parses

• Return correct-looking output that ignored the instruction

• Loop on a tool call forever, with every individual call succeeding

Three containment mechanisms that survive the translation, in the order I would add them:

1. Budgets before timeouts. A wall-clock timeout catches a hung inference. It misses the worker happily making its 40th tool call. Give every invocation a hard cap on tool calls and tokens, enforced by the harness rather than the prompt. When the budget trips, the orchestrator gets a structured “exhausted” result it can route around instead of a partial answer it will trust.

2. Fallback routing in the model dimension. Services fall back to a replica. Agents get a second axis:

• Fall back to a cheaper model tier with a stricter prompt

• Fall back to a smaller tool set

• Fall back to a deterministic path (an extractive summary instead of a generated one)

Design the degraded path before you need it, the same way you would design read-only mode for a database outage.

3. Retries with a ceiling, only on classified failures. The retry storm math is worse here because each retry is a full inference pass. Amplification bills you in dollars and seconds instead of connections.

• Retry on transport errors and rate limits

• Do not retry “the output failed my validator” without changing something: the prompt, the model, or the inputs

• Same call, same context, and you are paying to roll the dice again (more on why below)

🧭 Pick an orchestration pattern on purpose

Nobody ships microservices without a load balancer and service discovery. Nobody should ship multiple agents without deciding who is in charge. Three patterns cover most production systems today.

(Router: one classifier fans a request to exactly one specialist. Orchestrator-worker: one capable model decomposes a task, dispatches to cheap parallel workers, and synthesizes. Supervisor: one model loops with a set of agents, re-planning after each result.)

Router. A cheap classifier reads the request and forwards it to exactly one specialist, one hop with no synthesis step.

• Use for: support triage, intent routing in a product assistant, any case where the categories are known and the work per category is self-contained

• Router model: your cheapest one, a fine-tuned classifier, or a regex if the categories are clean enough

• Common mistake: letting the router “help” by adding context. It should classify and get out of the way

Orchestrator-worker. One capable model decomposes the task, dispatches subtasks to workers, and synthesizes the results. Workers are stateless, parallel, and can run on a different model than the orchestrator.

This is where the cost story lives:

• Orchestrator-tier and worker-tier models from the same vendor sit 5x to 15x apart on per-token price

• In a decomposed task, most tokens are worker tokens

• The number circulating in vendor write-ups this year is a 40 to 60 percent cost reduction versus running everything on the frontier model. Those write-ups are selling something, but the mechanism is sound

• Anthropic’s public data point: an Opus lead with Sonnet subagents beat single-agent Opus by 90.2 percent on their research eval. The same post reports multi-agent runs burn ~15x the tokens of a chat interaction, which is the other half of that ledger

Two rules for making it pay:

• Workers get a schema, an example, and a budget. Nothing else

• The orchestrator sees results, never raw worker transcripts, or your context bill grows with every worker you add

Supervisor. A loop rather than a fan-out. One model holds the plan, calls an agent, reads the result, revises the plan, calls the next one.

• Sequential and stateful. The supervisor’s context accumulates every result

• Use for: long-horizon tasks where step N depends on step N-1 (coding agents, migration tooling, multi-stage pipelines)

• Wrong for: anything parallelizable. It turns a 4-second fan-out into a 40-second chain

Most teams reach for supervisor first because it is easiest to reason about, then discover their task was a fan-out with a synthesis step and they have been paying sequential latency for nothing. Draw the dependency graph of your subtasks before you pick:

• Wide and shallow: orchestrator-worker

• A chain: supervisor

• One node: router, or no multi-agent system at all

🔍 Nobody can trace this yet

Distributed tracing is what made microservices operable. Fifteen years of Dapper, Zipkin, Jaeger, and OpenTelemetry mean a request ID follows a call through 30 services and the flame graph tells you which one ate the latency. The observability guide covers the mechanics.

Agents need a stranger kind of trace. Latency per hop is the easy part. The questions that matter during an incident:

• Which agent decided to call this tool?

• Based on what context?

• Why did it pick that argument?

A span that says execute_tool: search_orders, 340ms is useless when the bug is that the orchestrator hallucinated a customer ID two hops earlier.

The tooling is behind:

• As of this summer, every gen_ai.* attribute, span, and metric in the OpenTelemetry registry is still marked “Development”

• The agent conventions (invoke_agent, execute_tool) and the MCP conventions were split into their own repository in June, with no tagged release as of mid-July

• Vendors instrument against them anyway, so the schema you adopt today has a fair chance of changing under you

• Datadog’s own AI incident investigator needed bespoke instrumentation to make its reasoning auditable, and that is a company whose product is observability

What to do in the meantime: log the full prompt and full output of every agent hop, keyed by a trace ID that flows through the whole task, into cheap storage with a short retention window. Crude and large, and the only thing that lets you replay a failed run. Anthropic’s research team landed in the same place: full production tracing was what let them diagnose failures, because runs are non-deterministic between attempts even with identical prompts.

Watch the OTel GenAI repo for a stable tag. Until then, own your schema and expect to migrate it.

🚨 Where the analogy breaks

Forward the next three sections to whoever on your team is drawing the diagram.

💸 The cost inversion

Microservices decomposition is cheap at the seams. A gRPC hop is around a millisecond, so the architecture question is purely about ownership and blast radius.

An agent hop is a full inference pass with the entire subtask context re-encoded as input tokens:

Read more

Stripe ran Envoy for almost ten years, then replaced it with their own Go proxy, mesh-proxy. The payoff:

• ~50% less CPU under high load, ~50% lower latency

• ~3,000 days of inter-service latency saved every day

• One connection per host, down from one per worker

• New Ruby load balancing bought 15-20% more burst headroom

Why rebuild at all??? Stripe targets 99.9995% reliability on infra carrying about 1.6% of global GDP. One API call fans out into many internal requests, so a few hundred microseconds in the data plane compounds fast.

The reason they left was connection count. The math is worth walking through even if you never touch a proxy.

Stripe can throw a team at making internal delivery this reliable. The same problem hits one hop out, when you’re pushing events to customers and their agents. Most teams start with a hand-rolled webhook sender, and it holds until retries, signing, idempotency, and fan-out turn it into a second product nobody wanted to own.

That’s where Svix (today’s sponsor) comes in.

Svix: Become event-driven in a day

Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.

Start sending webhooks today

🔌 The connection math that broke Envoy

Envoy runs one worker per CPU core, and under steady traffic each worker opens its own connection to every upstream it can reach. Connections scale with core count. Host count is irrelevant.

Two 8-core clients talking to two servers: every worker on both clients holds a connection to both servers, so each server carries 16 connections for just those two clients.

Now point that at a fleetwide service, say the pipeline ingesting tracing data from everything you run. It takes traffic from roughly every core in the fleet, so it holds roughly one connection per core company-wide. That’s hundreds of thousands. Discord hit the same class of problem from the other side when they cut WebSocket traffic 40% to keep connection overhead from swamping the gateway.

Two ways that bites:

• You size machines for connection count, not request load. Expensive for a service doing little real work.

• Sockets pile up silently until one day there are too many, the service tips over, and you’re debugging an incident with no obvious cause.

Stripe hit the second one.

⚠️ Every upgrade was a gamble

Thousands of distinct services make Hyrum’s Law literal: with enough consumers, every behavior of the proxy is something a downstream depends on. A minor version bump ships a behavior change to all of them at once.

In 2024, an Envoy upgrade pulled in a new nghttp2 with a defense against the HTTP/2 rapid-reset attack. On default settings, that defense started rejecting legitimate traffic to one of Stripe’s services. A QA load test caught it before prod. Miss that catch and the first signal is real requests getting dropped, and one incident like that can spend the whole year’s reliability budget. Same reason Netflix breaks their own services on purpose: the regression you find in a load test is free.

That’s the standing tax of a shared proxy across a big, unusual fleet. You inherit every upstream change and find out in your own traffic whether it breaks you.

🚫 The feature they couldn’t ship

Stripe’s Ruby services run on big, expensive machines. The team had a routing design to spread traffic more evenly across Ruby backends and shrink those machines. Building it into Envoy meant customizing past the point they’d maintain, so it sat on the shelf.

That’s the third, quietest cost of an adopted proxy: it caps what you can build in your own data plane. Every custom idea has to fit the extension points you’re given, and this one didn’t.

🛠️ What mesh-proxy does differently

Mesh-proxy is Stripe’s Go replacement. It ships its own HTTP/1.1 and HTTP/2, talks straight to the service registry for discovery, and supports:

• Retries and hedging (fire a second request when the first is slow)

• Rate limiting

• Dynamic priority-based routing

The core change is concurrency. Instead of one worker per core, mesh-proxy runs one read and one write goroutine per connection and multiplexes many requests over a single shared connection. Goroutines are cheap enough to spin up per connection without thinking about the cost. That flips the earlier math: one connection per host, where Envoy needed one per worker.

Back to the two 8-core clients and two servers. After migrating, each client holds one connection per server, so each server carries two connections instead of sixteen. The tracing service that was drowning in sockets now fits on a smaller machine.

The tradeoff is real. Multiplexing over a shared connection means coordinating request state across goroutines, more work than letting each worker own its own. Stripe took that on because the connection savings paid for it.

They also dropped xDS (Envoy’s push-based config protocol) for reading the registry directly. Their Envoy setup generated more xDS updates than the topology actually changed, burning CPU and memory on every machine. One flooded service shows a step-drop in both the moment mesh-proxy rolled out.

📈 What it bought them

• Half the CPU under high load

• ~50% lower latency (that’s the 3,000-days-a-day figure)

• The shelved Ruby load balancing shipped: 15-20% more headroom before a burst overwhelms a machine

• A rewritten priority-routing system that unlocked changes across deployment and rate limiting

Owning retries and hedging cuts both ways. Client-side retries are the classic path from a small hiccup to a retry storm that takes the whole dependency down. With the retry budget in code Stripe’s own team reviews, they enforce it consistently. They also own getting it right.

🤔 The part I’d push back on

Building in-house beat running battle-tested open source, and for Stripe that reads as true. It’s also the most expensive kind of true.

Look at the nghttp2 incident again. That upgrade existed because nghttp2’s maintainers built the rapid-reset defense, protecting against a protocol-level DDoS that hit the whole industry in 2023. Stripe got it as a free version bump. It broke them once, in QA, and they changed a config.

mesh-proxy has its own HTTP/2 now. When the next protocol-level HTTP/2 attack lands, and there’s always a next one, Envoy shops get a patch from people who do nothing but maintain that proxy. Stripe writes the patch. Every flow-control edge case and every future HTTP/2 CVE is theirs to find and fix, forever.

Defensible at 1.6% of global GDP and a 99.9995% target. Stripe has run this play before, including building their own database instead of bending an existing one to their uptime number. The precondition is a team whose whole job is this one component, at a scale where 50% latency pays for that team many times over.

For almost everyone else it runs the other way. The connection math travels anywhere. The build decision is specific to Stripe’s scale and headcount: copy it without the scale and you get a worse Envoy only your team can debug. If your proxy is costing you money, measure how connections scale first, with cores or hosts, and whether a fleetwide service is quietly holding a socket for every core you own. That’s usually fixable in the proxy you already run.

✅ What to check in your own mesh

Before anyone pitches a rewrite, get three numbers:

• Connections per upstream, and what they scale with. On a worker-per-core proxy, graph a fleetwide service’s open connections against fleet size rather than request rate. If the lines track, you have Stripe’s original problem right now.

• Config-update volume per machine. Push-based planes like xDS can send far more than the topology changed. Measure how much load is pure config churn you could delete.

• Where retry and hedging budgets live. Scattered across client libraries means nobody owns the total, and it only surfaces mid-outage.

None of these need a new proxy. All three separate a data plane you understand from one that surprises you at 3am.

Read More

A production voice feature in 2026 typically ships with four vendor relationships: a transcription API, a PII redaction step somebody built in a sprint, an LLM provider, and a compliance review s…

Read more

TLDR

August 17. GitHub went sideways for 7 hours 47 minutes.

• ~20% error rate on web and API traffic

• ~50% on archive and raw content downloads

• Issues, Pull Requests, Actions, Copilot, SAML/OIDC, SCIM, and Team Sync all degraded

• Copilot Token Service went from 7,000-9,000 RPS to 70,000-100,000 RPS

One Istio sidecar pod in Central US hit its concurrency ceiling and failed to autoscale. That took down four HAProxy nodes, which took down the gateway auth path. Most of GitHub was back by 16:36 UTC and Actions by around 18:03. Copilot limped until 21:02, because a latent retry bug in VS Code turned a slow endpoint into a load generator running on millions of laptops.

The capacity problem was mitigated in the afternoon. The retry storm it set off ran for another five hours.

The sidecar nobody was scaling on

Istio puts a proxy sidecar next to every service pod, and all traffic flows through it. You get mTLS and routing without touching application code. You also get a second set of concurrency limits per pod, sitting somewhere entirely different from the application’s.

The autoscaling policy watched the host service. When Central US hit a new traffic peak, the sidecar saturated first. The autoscaler read host metrics, saw headroom, and left the deployment alone.

A pod that cannot accept connections and cannot be replaced by more pods sheds load. Whatever sits in front of it eats the difference.

The cascade, in order

1. Sidecar saturates, autoscaler does nothing.

2. Refused connections pile onto HAProxy in Central US.

3. Four HAProxy nodes exhaust their flow limits, one after another.

4. Those four front the gateway auth path, so auth latency climbs.

5. Callers waiting on a fast dependency start treating slow responses as failures.

6. Retry logic engages, and the auth path now gets more traffic than it did before it broke.

Auth is the worst place in a system to lose capacity, because nearly every request touches it. GitHub knows this one well already: a 43-second network partition in 2018 cost them 24 hours of degradation, and the trigger was similarly small next to the blast radius.

Recovery required breaking things harder

Some failing traffic was moved out to Northern Virginia and served successfully there while the Central US network failure was debugged. Regional failover did its job for the bulk of user requests. The retry behavior travelled with the clients, so Northern Virginia inherited the herd along with the load.

The Central US fix is the interesting one. Pausing HAProxy on the saturated nodes, all four simultaneously, produced immediate broad recovery.

Staggering that pause would have pushed load onto whichever node was still accepting traffic and kept the saturation alive. A feedback loop has to be broken everywhere at once, which is an uncomfortable thing to do at 16:00 UTC on a Monday with 20% of your traffic already failing.

Read more

Linear Agent, the assistant that lives inside Linear, does most of its valuable work in ways nobody scripted, and that premise cuts against engineering instinct. The engineering effort went into…

Read more

Read more

There is a shift happening across the industry, and GitHub’s engineering blog put a name on it last week: engineering’s oldest reflex is now misfiring on a specific class of work.

The argument goe…

Read more

Netflix wrote Compute Managed Batch (CMB) in 2018 because nothing in the Kubernetes ecosystem did what they needed. Hierarchical tenants. Fair sharing. Capacity reservations. Priority queues. They built all of it themselves.

Seven years later, Kueue does all of that, plus the features CMB never shipped. So Netflix ripped out CMB’s queuing and scheduling logic and replaced it with Kueue. Millions of batch jobs moved over. The production migration took 4 weeks. Users did nothing and noticed nothing.

The interesting part is not the swap. It is how they decided when to stop maintaining their own thing, and how they migrated without ever exposing a new API to customers.

What CMB Actually Was

CMB is a managed batch layer sitting on top of Titus, Netflix’s container platform. Users submit jobs that run to completion. CMB queues them, orders them by priority, and enforces capacity per tenant.

The tenant model is a tree. Internal tenants are branches, they exist to group things and hold capacity for their subtree. Leaf tenants are where work actually lands, and only leaves have queues. An org might run one tenant for everything, or mirror its entire team structure in the tree.

Capacity came in two flavors:

Reserved. A leaf tenant with a reservation partitions that capacity out of the hierarchy. Nobody else can take it. That guarantees throughput.

Shared. The Compute team runs a global burst pool. Any tenant can pull from it. It is fair-shared across tenants.

Here is the flaw, and it is the flaw that eventually forced the rewrite. CMB fair-shared the pool at admission only. There was no preemption. Once a job was admitted, it ran to completion no matter how the demand picture changed underneath it. A tenant that got in early held its slice until it finished. Fair sharing that only applies at the door is not fair sharing. It is a queue with extra vocabulary.

The other cost was structural. CMB sat far away from the actual Kubernetes clusters. Titus federates jobs across cells, so CMB talked to one endpoint and stayed blissfully unaware of cluster topology. Great for simplicity. Terrible when you want to build preemption, because preemption is a scheduler-level concern and CMB was several abstraction layers above the scheduler. Every new feature meant re-implementing something Kubernetes already had a place for.

Why Kueue and Not the Others

Netflix looked at YuniKorn and Volcano too. Both replace the pod scheduler. That was the disqualifier.

Titus has its own scheduling profiles. Swapping out kube-scheduler means fragmenting job placement, and fragmented placement means worse bin-packing, which means burning money on idle capacity. If you have ever watched a fleet run at 40% utilization because two schedulers disagreed about where things go, you know exactly what this costs. Netflix has written before about how much compute efficiency is worth at this scale.

Kueue does not touch pod scheduling. It sits above it and decides whether a job gets admitted, then hands the pods to the existing scheduler. That single design choice made it the only option that composed with what Netflix already ran.

The rest of the case was straightforward:

• Multi-tenant quota over heterogeneous hardware, which is table stakes when your fleet is a zoo of instance types

• Works on v1.Pod and batch/v1.Job, and also on RayJob / RayCluster for the ML workloads coming down the pipe

• Native preemption, all-or-nothing scheduling, topology-aware scheduling. All the things CMB wanted and never got

• Real adoption momentum, which matters more than feature checklists over a five year horizon

The Migration: Change Nothing the User Can See

They called it Netflix Batch. Three tenets, and the first one is the one worth stealing:

1. Zero lift for CMB users. Completely transparent.

2. No regression in container launch rate or max throughput.

3. Replace CMB queuing and scheduling with Kueue.

Read more

A normal C++ function inside OpenAI’s Rockset service runs, finishes, and returns to garbage. Sometimes the saved return address on the stack is NULL. Sometimes the…

Read more

Last year, 230 of you took a survey about a tool to keep your system design knowledge sharp using real case studies like the ones we cover here, and to actually h…

Read more

Read more

Netflix runs on thousands of microservices. When one breaks at 3am, the on-call engineer needs to know what depends on it, what it depends on, and whether the problem is local or upstream. That a…

Read more

Discord’s Persistence Infrastructure team runs every ScyllaDB cluster behind messages, channels, servers, and most of Discord’s user data. Dozens of clusters. Hundreds of nodes. Seven engineers.

For years, the tooling was a pile of Python and bash scripts that worked but required someone who remembered all the landmines. Standing up a full shadow cluster, a complete replica that mirrors production traffic so you can safety-test a new ScyllaDB release, took a day and a half of careful, sequential, do-not-screw-up-step-nine work.

They rebuilt it as the Scylla Control Plane (SCP): composable tasks, YAML-defined workflows, automatic retries, and resumable jobs. The same operation now takes under two hours, and most of that is the engineer doing something else while nodes bootstrap.

The interesting part isn’t that they automated database ops. It’s the specific shape of the automation, and three boring decisions that mattered more than any of the Rust.

Seven Engineers, Hundreds of Nodes

ScyllaDB is Discord’s largest database by scope. We’ve covered how that data layer evolved before, How Discord Indexes Trillions of Messages walked through the search side of the same infrastructure. The team that operates it is seven people.

That ratio sounds fine until you look at what “operating” actually means: rolling restarts after every config change, expanding clusters as servers fill up, rolling OS upgrades across hundreds of nodes with zero downtime, and validating every new ScyllaDB release on a shadow cluster before it touches production.

None of that is fire-and-forget. Each one demands careful sequencing, validation, and someone paying attention the entire time. We talked about why that kind of pre-production validation matters in The Tech Lead’s Guide to Load Testing, shadow clusters are basically load testing with real production traffic and real production stakes.

For years, Discord automated this the way most teams do: incrementally, under pressure, with no long-term plan. A Python script here, a bash script there. It worked. It also required deep institutional knowledge to run safely, and that’s the kind of debt that’s invisible until the one person who understands it is on vacation.

Three Ways the Old Scripts Failed

Read more

Reactive auto-scaling has a fundamental problem: by the time it kicks in, you’re already overloaded. The scaling operation itself takes several minutes. You’ve spent real money on degraded perfor…

Read more

23:50 UTC, Thursday, May 7 2026. A data center hall in Northern Virginia overheats. Cooling units in availability zone az4 give up. EC2 instances and EBS volumes start losing power inside the hour.

Coinbase goes dark. Seven hours. Trading, Prime, International, derivatives, balance updates, all gone. FanDuel goes down at 9pm ET right as Game 2 of Lakers-Thunder tips off. CME Direct logs go sideways for institutional traders.

Coinbase had multi-AZ on most of their stack. But not the matching engine. That one ran in a single zone, on purpose, for latency. They had a backup. The backup didn’t isolate from the failure the way it was supposed to.

This is a story about a tradeoff that finally got tested in production. And about the difference between high availability and disaster recovery, which is not the same thing, no matter how many architecture diagrams pretend otherwise.

What actually broke

The root cause was not software. It was a building. Multiple chillers in a single data center hall failed. Temperatures climbed. AWS lost power to racks in availability zone use1-az4. EC2 instances and EBS volumes on those racks were physically damaged, not “marked unhealthy,” damaged.

AWS shifted traffic away from the zone, but recovery depended on getting cooling back online before they could safely bring damaged hardware up again. That took more than 20 hours. Cooling was stable at pre-event levels by 13:50 PT Friday afternoon.

This is the part of the story no architecture diagram makes visible. The internet still runs in buildings, and buildings can overheat.

If that framing sounds familiar, the October 20, 2025 AWS outage had the same shape from a different angle, 15 hours, 113 services, traced back to one empty DNS record. Different failure mode, same conclusion: at AWS’s scale, the failures that take down half the internet are rarely the ones you’d architect for.

The Coinbase tradeoff

Here’s the part that matters for engineers.

Coinbase had multi-AZ. They said so plainly: most of their systems are designed to survive a single AZ failing. Most of them did.

The matching engine didn’t.

Rob Witoff, Head of Platform, was honest about why. The exchange runs in a single availability zone by design. Latency. Customer co-location. Real money on the line, measured in microseconds. Spreading it across zones would have meant accepting cross-AZ network hops on every order match. For an exchange that competes on speed, that is not a free trade.

So they made the call. Single AZ for the matching engine. And — this is the part everyone glosses over — they built a backup. A distributed copy of the exchange infrastructure, designed to take over if the primary zone died.

The backup did not work as expected.

Witoff was specific: “backup systems did not work as expected during the incident, extending the outage and forcing engineers to manually execute disaster recovery procedures.” Engineers had to develop, test, deploy, and validate a fix while the production system was on fire.

Kafka made it worse. Coinbase runs partitioned Kafka handling thousands of terabytes a day. That couldn’t fail over automatically either. It needed manual recovery. Balance streams lagged behind until replication caught up.

No data was lost. But seven hours of downtime is seven hours of downtime.

“Distributed backup” is not the same as “distributed”

This is the lesson worth dwelling on.

A backup that depends on the same failure domain as the primary is not a backup. It is a copy.

Coinbase had a distributed copy of the exchange. But “distributed” without specifying what failure mode it survives is marketing. If your backup database lives in the same AZ as your primary, an AZ outage takes both. If your backup region depends on your primary region’s IAM control plane, a region outage takes both.

The principle generalizes brutally: your failover only works against failure modes you actually tested it against. Untested failover is not failover. It is a hypothesis.

GitLab learned this the hard way in 2017. When an engineer accidentally deleted 300GB of production data, they discovered their backup system had been broken for weeks. Five backup mechanisms. None of them worked. Different failure mode, same shape: a safety net that nobody had pulled on recently.

The Coinbase incident is the kinder version of the same story. Their backup existed. Their backup partially worked. But the bits that didn’t work showed up only under the specific conditions of the specific failure that actually happened, which is exactly the moment when you can’t afford to discover them.

If your DR plan has not been exercised in the last 90 days, treat it as untested.

High availability and disaster recovery are different problems

Every time something like this happens, the postmortem comment section fills up with “they should have used multi-AZ.” This is half-right and entirely beside the point.

Multi-AZ is high availability. It protects you from a bad day in one zone. Coinbase already had it for most workloads.

Multi-region is disaster recovery. It protects you from a bad day in one region.

AWS treats availability zones as the failure domain for HA. They treat regions as independent on purpose. A thermal event in US-EAST-1 will not move your data to US-WEST-2 unless you have explicitly built that path, paid for the replication, tested the failover, and decided in advance who gets to push the button.

Most production systems need both. Most production systems have one.

The reason is cost. Cross-region replication is real money. Hot standby in a second region doubles your compute bill. And the day-to-day value of that spending is invisible, until the day the chillers fail, at which point the conversation stops being about budget and becomes about how fast you can recover.

This is the part where engineering leadership earns their salary. Resilience investments look expensive right up until the moment they look obvious.

How Coinbase actually recovered

The recovery sequence is worth studying because it shows what disciplined incident response looks like under load.

When the matching engine came back, they did not just re-enable trading and let it rip. They staged it:

1. Cancel-only mode — let customers withdraw existing orders, but no new trades

2. Auction mode — orders accumulate, no immediate matching, gives time to verify books are consistent

3. Live trading — only after product-by-product health checks

This is not glamorous. It is the engineering equivalent of bringing a power plant back online — slow, sequential, fully verified before each step. The temptation in a multi-hour outage is to fix the thing and flip the switch. Coinbase didn’t. They reconciled state first. No data was lost. That outcome is downstream of this discipline, not luck.

The public communication followed the same pattern. CEO Brian Armstrong posted within hours: “never acceptable.” Witoff followed with a technical thread giving the actual timeline, the actual root cause, the actual decision being revisited. No PR-speak. No “we are looking into it.” Specific.

That is what a public post-mortem looks like when the engineering org is the one writing it, not legal.

The SLA is not your business continuity plan

One detail from the AWS side worth internalizing. The standard EC2 SLA pays out around 10% of monthly compute spend on impacted instances. That’s the entire compensation.

Lost revenue: not covered. Customer trust: not covered. Regulatory exposure: not covered.

The 2024 ITIC survey put hourly downtime cost above $300,000 for 90% of mid-to-large enterprises. 41% lose between $1M and $5M per hour. Trading and finance are higher still. Coinbase was down for seven hours during a quarter where they had also just announced a 14% workforce reduction and a $394M net loss. The 10% AWS service credit was not going to move that needle.

Your cloud provider’s SLA is a small refund. It is not insurance. It is not a business continuity plan. The continuity plan is something you build, fund, and test yourself, or you don’t have one.

Three things to do this week

Read more

Hundreds of inference calls per investigation. Megabytes of agent output. A team of LLMs all working on the same alert, none of them sharing memory by default. That’s what Slack’s security investigation service has to wrangle every time an alert fires.

The agents don’t share a brain. The API is stateless. Stuff every transcript into every prompt and you blow the context window. Strip too much out and the agents drift, contradict each other, and stop sounding like a team.

Slack’s answer: stop pretending one shared context can do the job. Build three of them. Each agent reads the channel that fits its role.

It’s one of the cleanest examples I’ve seen of context engineering as a real discipline, not a vibe.

The setup

Slack runs security investigations with a team of agents. A Director orchestrates. A bunch of Experts go gather evidence using domain-specific tools. A Critic reviews what the Experts produced and flags the bits that don’t hold up.

Investigations move through phases. Each phase has rounds. There’s no fixed budget, the Director decides when to stop.

The dependency runs both ways. Each agent needs enough of the wider investigation to stay coherent with the team. But shove the full transcript into every prompt and two things happen: you hit the context window, and the agent starts drifting toward whatever the loudest voice in the history said. Confirmation bias dressed up as continuity.

Same problem every multi-agent system hits. They built three context channels to solve it.

Channel one: the Director’s Journal

The Director keeps a journal. Not a chat log, but a structured one. Six entry types: decision, observation, finding, question, action, hypothesis. Every entry gets a phase, a round, a timestamp, a priority, and optional citations to evidence.

The journaling tool itself is dumb. It appends. The intelligence is in the prompt that tells the Director to use it constantly, and in the schema that forces every thought into a slot.

The Journal goes into every other agent’s prompt. It’s the spine. The Experts read it to understand what they’re being asked and why. The Critic reads it to understand what the Director was trying to prove.

Read enough Journal entries and you can watch the investigation think. One example from a real run: the Director starts by classifying an alert as a kernel module load, lists 4 expert domains it’ll need, notices the cgroup suggests a personal workstation, then realizes the alert is matching the path of a hook script rather than an actual modprobe call. By round five it’s preliminary-false-positive. By round seven it’s confirmed FALSE POSITIVE with all four Experts agreeing.

That progression is only legible because the Journal exists. Without it, you’d have a pile of disconnected expert outputs and no narrative.

This is the same shape of problem covered in a previous edition on designing AI agents that think in real time, different domain, but the structural answer is identical: give the orchestrator a working memory that everyone else can read.

Channel two: the Critic’s Review

Read more

3:43 PM EST to 4:20 PM EST. 37 minutes. Every HubSpot customer lost the ability to click into contact, company, order, or project workflows in the UI. Deal-based and ticket-based workflows still worked. Every backend automation kept firing on schedule. No data lost. No execution missed.

And the whole thing flew under the radar because the endpoint that broke kept returning HTTP 200.

The incident monitoring didn’t catch it. The automated canary checks didn’t catch it. A 60-minute alert threshold meant the tests that did fail weren’t going to page anyone until well after customers had already flooded support.

This is a textbook case of the thing we keep writing about: your observability is only as good as what you’re actually measuring. If you’re measuring “did the server respond,” you’re going to miss every bug that makes the server respond with the wrong answer. HubSpot’s post-mortem is refreshingly direct about this, and there’s a clean lesson in it for anyone running permission systems, feature flags, or anything else where the shape of a correct response matters more than its existence.

If you’ve been around for the Cloudflare July 2025 outage breakdown or the AWS October 20th dissection, this one will feel familiar. Different blast radius. Same category of failure.

So what actually happened?

HubSpot was rolling out a permissions framework update. The goal was reasonable: replace a broad shared scope with narrower, object-type-specific scopes for contact, company, order, and project workflows. Tighter permissions, better isolation. Standard stuff.

The rollout had two pieces:

1. Create the new permission scopes.

2. Promote the user role assignments that map those scopes to the right users.

Piece one made it to production. Piece two didn’t.

The staging environment had both pieces, so staging worked. Production had scopes without role assignments, so production’s access-control system went looking for user-role mappings that didn’t exist. When it couldn’t find them, it did what permission systems are supposed to do: fail closed. Deny access.

From the access-control system’s perspective, this was correct behavior. Users were asking about permissions the system couldn’t verify, so the system returned a restrictive access level.

From the user’s perspective, their workflows vanished.

The 200 that lied

Here’s the part worth dwelling on. The access endpoint returned HTTP 200 the whole time. The server didn’t crash. It didn’t throw. It didn’t log an error. It just returned a technically-valid response that said “this user can barely do anything.” The frontend, doing its job, saw “barely anything” and hid the UI.

Most monitoring treats HTTP status codes as ground truth. 2xx is fine, 4xx is the client’s problem, 5xx pages the on-call. It’s a useful abstraction, and it’s wrong in exactly this scenario. The server is healthy. The payload is garbage.

We covered something very similar in how Twitch caught their invisible failures—streams that terminated “successfully” from the server’s point of view while users saw nothing. Same failure mode, different domain. When correctness lives in the response body rather than the status line, your dashboards need to look inside the response.

Why the canary didn’t save them

HubSpot’s automated test suite did catch failures during the canary window. Those failures fired into a queue that was configured to wait 60 minutes before paging anyone.

Sixty minutes.

The deployment rolled out fully in 33 minutes. The entire incident lasted 37 minutes from first impact to rollback. The alerts would have arrived after the problem was already resolved.

Alert thresholds are a real tradeoff. Too tight and your on-call drowns in noise from flaky tests. Too loose and you get this. The right answer is rarely a single global threshold, it’s threshold in context. Failures during an active deployment window are categorically different from failures on a quiet Tuesday morning, and HubSpot is correctly calling that out in their remediation plan. Correlate the alerts with the deploys. Shrink the window to minutes during rollout.

This is the kind of instrumentation gap that shows up over and over in post-mortems. For more on how to actually write these documents well instead of just surviving them, our tech lead’s guide to writing post-mortems covers the framing that distinguishes a useful post-mortem from a corporate apology.

The split-brain deployment

Read more

That stat is embarrassing for a company whose product is literally communication. But what’s interesting isn’t the problem, it’s why it was so hard to fix.

The system wasn’t broken. It was incoherent.

Desktop and mobile had entirely separate preference systems that had grown apart over years. “Nothing” on mobile meant something different from “Off” on desktop. Not slightly different. Architecturally different. One disabled push notifications. The other disabled in-app badges too. Users changing settings on one device had no predictable effect on the other.

This is how trust erodes. Not with crashes. With settings that don’t do what you think they do.

The core design flaw was a tight coupling between what notifies you and how you get notified. If you wanted fewer interruptions on mobile, your only lever also killed in-app awareness. There was no way to say “show me everything in the sidebar but only push me for mentions.” You had to pick between overload or ignorance.

Four preference systems became one

The old prefs looked like this:

desktop: everything | mentions | nothing   // Push on desktop
mobile:  everything | mentions | nothing   // Push on mobile

The word “nothing” is doing dishonest work there. Users who chose it thought they’d gone quiet. They hadn’t — they still got in-app badges. They just didn’t know it.

The new model decouples the two concerns cleanly:

desktop: everything | mentions // What activity to show
desktop_push_enabled: true | false // Whether to interrupt you
mobile: everything | mentions | nothing

desktop_push_enabled is new. Because it had no prior value in the database, the team could backfill every existing user based on whether they’d previously set “off”, no disruption, no migration emails, no support tickets. “Off” became “mentions with push disabled” at read time, which is exactly what it meant in practice anyway.

That’s a clean migration. Backwards compatible, rollback-safe, and behaviorally honest.

The real difficulty: millions of users, years of state

Read more

Hours-old data. Petabyte scale. Thousands of engineers making decisions on stale numbers.

Uber’s data lake powers Delivery, Mobility, Finance, Marketing Analytics, and Machine Learning for a company with hundreds of millions of users. For years, the ingestion layer ran on Spark batch jobs. Data arrived in the lake hours late. Sometimes a full day late.

That was fine when the business moved slowly. It stopped being fine when data freshness became a competitive bottleneck—when model iteration speed, real-time experimentation, and operational analytics demanded minutes, not hours.

So they rebuilt ingestion from scratch on Apache Flink. The result: freshness dropped from hours to minutes, compute costs dropped 25%, and the system now handles petabyte-scale streaming across thousands of datasets.

This is IngestionNext. And the problems they had to solve to get there are exactly the kind of problems most data teams quietly ignore until they can’t anymore.

The Dirty Secret About Batch Ingestion

Here’s the thing nobody wants to say out loud: batch jobs are slow by design, and most teams have just accepted that as the cost of doing business.

You run a Spark job every hour. Maybe every 30 minutes if you’re ambitious. The job spins up, reads from Kafka or a transactional database, transforms the data, writes it to the lake. Then it tears down. An hour later, it does it all again.

At small scale this is totally fine. Predictable. Easy to debug. The operational overhead is low.

At Uber’s scale, hundreds of petabytes, thousands of datasets—those batch jobs were burning hundreds of thousands of CPU cores every day. Not because the work required that many cores. Because that’s how batch scheduling works. You provision for the peak, the peak is infrequent, and everything in between is wasted capacity.

And even if you ignore the cost problem, there’s no fixing the freshness problem. Batch is batch. If your job runs every hour, your data is up to an hour old. Period.

For model training, that’s a delay in experiment velocity. For fraud detection, that’s a window where bad actors operate undetected. For marketplace analytics, that’s a lag between what happened and when anyone can respond to it.

Uber looked at this and decided hours-old data was no longer acceptable. They needed minutes. That meant streaming.

If you want to see how Uber’s data lake got to 350PB in the first place, and the replication problems that scale created, read Inside Uber’s 350PB Data Lake: The Distcp Rewrite That 5x’d Performance.

Why Flink, Not Just “More Spark”

The obvious question: why not just run Spark Structured Streaming? It exists. It integrates with Kafka. Half the data ecosystem already knows how to use it.

Because Spark Structured Streaming still thinks in micro-batches. It’s better than full batch scheduling, but it’s not true streaming. You’re still dealing with the same fundamental model: accumulate records, process a chunk, commit.

Flink is a different mental model. It processes records as they arrive. Checkpoints are asynchronous, not tied to batch intervals. The state management is first-class. For continuous ingestion at this scale, Flink’s execution model is a better fit.

Uber already had Flink infrastructure. The ecosystem supported it. That made the decision easier, but the architecture challenges were anything but easy.

Pinterest went through a similar reckoning with Spark at scale, rebuilding their entire Hadoop-based platform into a container-native Spark system. Worth reading alongside this one: How Pinterest Runs Spark at Scale with Moka.

The Architecture

Events arrive in Kafka. Flink jobs consume them continuously and write to the data lake in Hudi format.

Hudi is doing serious work here. It provides transactional commits, rollback support, and time travel queries on top of what would otherwise be raw Parquet files on object storage. When a Flink job fails mid-write, Hudi rolls back the uncommitted data. When someone wants to query data as of a specific timestamp, Hudi handles it.

Above the data plane sits a control plane that manages the job lifecycle across thousands of datasets. Create, deploy, restart, stop, delete—all automated. Configuration changes propagate without manual intervention. Health checks run continuously. This isn’t glamorous infrastructure work, but at Uber’s scale, “we have 4,000 ingestion jobs” means operations without a control plane is a full-time fire drill.

There’s also regional failover. If a region goes dark, ingestion jobs reroute or fall back to batch mode. No data loss. No manual intervention required.

The architecture isn’t surprising. The interesting parts are the problems that showed up once it was running.

Problem 1: Streaming Creates a Small File Nightmare

Read more

GitHub Enterprise Server runs search on Elasticsearch. It also runs High Availability with a primary/replica model. For years, those two things could not coexist cleanly. Elasticsearch would move a primary shard to the read-only replica node. If you then took down that replica for maintenance, the whole thing deadlocked. The replica waited for Elasticsearch to recover before it could start. Elasticsearch couldn’t recover until the replica rejoined.

GitHub engineers knew this was broken. They spent years trying to patch around it. It took until Elasticsearch shipped Cross Cluster Replication to actually fix it.

The fix is live in GHES 3.19.1. The lesson underneath it is older than GitHub.

The Original Sin Was a Reasonable Decision

Let’s be precise about what went wrong here, because it’s easy to read this story as “Elasticsearch bad” when the real issue is more interesting.

Read more