Meta built one AI agent that knew everything about onboarding a data column into its recommendation pipelines: three repos, three languages, up to six pipeline stages, config schemas, codegen rules, population patterns for ten-plus data sources. It worked in demos. In production it started hallucinating, applying one subsystem’s naming conventions to another and using the wrong codegen pattern for the wrong data source.

The fix was to take knowledge away from the central agent. What replaced it:

• A 1,300-line orchestrator that does zero domain work. It classifies intent, gathers fields, runs reliability checks, and routes

• 14 specialist subagents, each loaded with only its own domain and nothing else

• Workflow templates, which are fixed state machines that sequence those 14 agents differently for each of 20+ use cases

Column config prep went from three to four days by hand, to about a day with an earlier config-driven system, to under an hour. Around two-thirds of the system’s changes land without significant human edits.

The rule underneath, which transfers to any agent system: for every component, decide whether it’s making a routing decision or a reasoning decision. Routing is deterministic and gets code. Reasoning gets an LLM, inside the smallest context that can do the job. Meta’s orchestrator follows seven steps per request and exactly one of them calls a model.

🫠 More context, worse agent

The pipeline is config-as-code. Python configs drive C++ code generation, which feeds data processing services, which populate analytics systems. One new column means coordinated edits across configuration registries, backend services, and pipelines, in the right order, kept in sync.

The first version was the obvious one. One agent, one giant prompt, every schema and rule and pattern loaded. It had everything it needed, and past some context size that became the problem. Knowledge from one subsystem leaked into decisions about another. The agent had plenty of information and was mixing it up.

A second attempt, still single-agent, got to 70 to 80 percent success, with a catch: an engineer had to hand-write a custom system prompt for each specific job. Take the handholding away and success fell to 10 to 20 percent. A system that only works when a human pre-chews the prompt per task is a system where the human is doing the orchestration.

Meta cites a Mount Sinai preprint on the same effect in a different domain: orchestrated multi-agent systems held 65.3% accuracy at 80-task batches where single agents fell to 16.6%, with up to 65x less token usage, attributed to task isolation preventing context interference. Different field, same shape of failure.

If you’ve read the agents-are-microservices edition, this is the monolith problem. A single agent with 40 tools and a 6,000-token system prompt degrades the way a 400,000-line app does. Every capability you add makes every other one slightly worse.

🚦 A router that refuses to think

The replacement has two layers, and the split between them is the whole idea.

The orchestrator is deterministic and does no domain work at all. It never reasons about code, never generates a config, never touches the codebase. For every request it runs the same seven steps:

1. Read the relevant subagent’s instruction file

2. Check whether the target already exists, to prevent duplicates

3. Gather required fields from the user

4. Auto-discover what can be inferred (repo type, existing configs)

5. Reliability gate: active warnings, required reviewers

6. Delegate to the subagent with full context

7. Handle results, surface errors, confirm success

Steps 1 through 5 and step 7 are plain code. Step 6 is the only place a model reasons, and it does so inside a subagent that has only its own domain loaded.

The 14 subagents are each a complete domain expert with isolated context. A schema agent knows the configuration registries. A population agent knows how to generate code for each of the ten data sources. A validation agent knows build systems and quality checks. No single agent sees the whole picture, which is what stops the cross-contamination that killed v1. The population agent for data source A has never seen data source B’s patterns, so it can’t apply them by mistake.

Read more

A fine-tune of Qwen 27B on Databricks’ AI Runtime took 7 hours 44 minutes with the GPUs reporting 98% utilization. After two changes it took 1 hour 17 minutes. Same model, same data, same hardw…

Read more

Uber has 800+ MCP servers and 5,000+ tools sitting behind a single gateway, with hundreds of teams building agents on top. Almost none of the work was about models. The questions that actually slowed them down were boring platform questions:

• Who owns this tool, and is it safe to expose?

• How does an agent find the right one among 5,000?

• Who checks authorization when the caller is an agent acting for a human?

• How do you give agents access to thousands of existing services without asking every team to write new code?

Their answer is one microservice that sits between every agent and every backend. A registry catalogs the tools, a proxy runs them, and a crawler drafts every tool definition from the Protobuf and Thrift schemas Uber already has. Nothing is exposed until a human owner turns it on.

If you remember one thing: this is the API gateway problem again, with a twist. The client is a model, and a model has a token budget. Hand it 5,000 tool definitions and it has spent most of its context before it starts working.

🧱 The problem turned out to be plumbing

Hundreds of teams at Uber started building agents around the same time. The early ones worked, because an agent that can hit a live service and do something for a user is a lot more useful than one that can only chat.

Then it spread, and you can guess what happened:

• Every team built its own integrations, duplicating infrastructure and fragmenting the tooling

• Tools were hard to discover and harder to run reliably

• Each integration was welded to one service or one agent framework

• Security and observability depended on whoever wrote the glue

If you were around for the first few years of microservices you’ve seen this list before. It’s roughly 2014, a month before somebody proposes an API gateway. Agent architectures have the same problem one level down, with the boxes renamed.

🗺️ Two planes

The gateway has two halves:

• MCP Registry (control plane). The source of truth for which servers and tools exist, who owns them, and whether they’re enabled

• Proxy Gateway (data plane). Takes an MCP call, translates it to HTTP, gRPC, or TChannel, forwards it, and converts the response back into an MCP result

The whole design rests on one bet: the fastest tool you can give an agent is an API you already run. Uber has thousands of services behind HTTP, gRPC, and TChannel. Asking those teams to rewrite anything for agents would have stalled every roadmap in the company, so the gateway adapts to the services and the services stay as they are.

I think that’s right, and I think most teams get it backwards. When MCP shows up, the instinct is to go write MCP servers. You already have the schemas. Wrap them.

🕷️ Discovery is automatic. Exposure is a decision.

Nobody hand-wrote 5,000 tool definitions. A crawler drafts them.

AutoCrawler runs on Cadence. A cron job kicks it off, it scans the IDL registry and some internal service signals for new services, new APIs and schema changes, and for each service or API group it finds it:

1. Upserts a virtual MCP server for the service

2. Parses the Protobuf or Thrift files for method names, request and response schemas, and doc comments

3. Asks an LLM to write agent-friendly tool descriptions from those schemas and comments

4. Translates the schemas into MCP-compatible JSON-RPC 2.0

5. Registers the tools in the registry, switched off

Hand-built servers take a second path. Anything written on MCPFx (Uber’s framework for native MCP servers) emits a heartbeat metric. The crawler sees a new heartbeat, calls listTools, and registers a virtual proxy server with whatever came back. Also switched off.

The disabled-by-default part is the decision I’d copy first. A tool stays dark until the owning team has read the generated definition, fixed whatever the LLM got wrong, and enabled it. After that, any edit to a tool description shows up as a config diff the owner has to approve, and they can roll back to the previous version.

Read more

17,000 merged changes in August, up from roughly 8,100 a year earlier. That is Spotify’s engineering org after a year of AI coding tools, spread across nearly 3,000 production services handling 11 to 12 million backend requests per second.

The part most people skip past:

• Their incident reviews have found no AI-authored code acting as a material direct contributor to a major incident so far

• The strain showed up after the merge, in review, testing, rollout, observability, and rollback

• Change volume grew faster than the controls that verify change

AI moved the constraint downstream. Writing the change got cheap, and proving the change is safe became the expensive step.

This edition walks through one Spotify incident from June that has nothing to do with AI, because it happens to be a perfect physical model of what a 2x change rate does to a delivery pipeline. Then a five-part scorecard you can take to your next staff meeting, since acceptance rate and “percent of code written by AI” will tell you almost nothing about any of this.

Svix: Become event-driven in a day

Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.

Start sending webhooks today!

🎬 June 24: a queue with four things wrong at once

More than 500,000 songs, videos, podcasts, and audiobooks enter Spotify’s processing systems every day. Video podcasts go through a transcoding fleet with a medium-priority queue for new episodes and a low-priority queue for updates to existing ones. Normal processing time is minutes, and on June 24 it became hours. Here is the timeline, in UTC:

• 13:30 Early alerts fire. Nobody reads them as a fleet-wide capacity problem

• 15:00 A spike in video podcast deliveries pushes transcoding close to its ceiling

• 16:35 A scheduled batch job gets stopped to free capacity

• 17:31 First creator report of a stuck episode

• 17:34 Queue-backlog alerts cross threshold and formal incident response starts

• 20:49 A fix ships for a resource scheduling bug

• 00:14 An additional processing cluster comes online

• 01:02 Queues clear

Roughly four hours between the first alert and a declared incident, and about eleven and a half between the first alert and an empty queue. Many creators found out from their audiences before they heard from Spotify.

Nothing in the incident report involves AI, and the word never appears in it. Four ordinary engineering problems stacked on the same afternoon:

• Thin headroom. The transcoding fleet had too little spare capacity to absorb a large delivery spike

• A competing batch job. A routine re-processing run for existing episodes was eating capacity during peak submissions

• A quality win that cost compute. A recent change delivered better video at lower bitrates. Each episode now took more processing, and the capacity plan still assumed the old cost

• A scheduler bug. After a migration to more powerful hardware, a resource scheduling bug left compute underused and cut throughput by about 10%

Any one of these is survivable. A 10% throughput loss on a fleet with 40% headroom is a line item in a weekly review. Put all four on the same afternoon and arrivals exceed service rate, which is the only condition a queue needs to grow without bound.

The monitoring gap is another story. The 13:30 alerts were symptom alerts, and they fired correctly. What was missing was a signal that said “arrival rate is about to exceed service rate for the whole fleet.” Twitch hit the same class of problem with failures their dashboards technically recorded and nobody could interpret. Alerts that detect pain are cheap. Alerts that explain capacity are the ones worth building, and the observability guide covers how to get there.

My one criticism of the engineering: the transcoding quality change should have gone through capacity review. Any change that raises per-item processing cost is a capacity change, whatever the ticket says. Spotify’s fix list suggests they agree. They added about 67% more transcoding capacity, fixed the scheduler, and committed to planning for steady state, burst, and recovery separately, with creator uploads always ahead of background work and backpressure extended through the pipeline.

🔁 Now swap episodes for pull requests

The mapping below is mine, and it’s an analogy. AI had no role in June 24. The shape of the failure is what transfers.

Your delivery pipeline is also a queue with a fixed-ish service rate. Changes arrive, wait for review, wait for CI, wait for a rollout slot, and bake in production while someone watches a dashboard.

  design -> implement -> review -> test -> deploy -> observe -> operate
               ^            ^________________________________________^
        AI made this             capacity here is mostly unchanged
        stage ~2x cheaper

Line the June 24 factors up against a year of AI-assisted development:

• The delivery spike is merged changes going from ~8,100 to ~17,000

• Higher per-item cost is PR size and code complexity, both of which are creeping upward at Spotify. A bigger diff takes longer to review, the same way a higher-quality transcode takes longer to encode

• The competing batch job is automated fleet-wide change. Spotify’s Fleet Management tooling finished a Java migration across backend services in three days. It also shipped an automated dependency upgrade that passed every check and still failed in production with user impact

• Thin headroom is your reviewers, your CI fleet, and your canary slots, all sized for last year’s arrival rate

• Symptom-level monitoring is the AI usage dashboard. It tells you the tools are busy

That dependency upgrade is the cleanest example in the whole September write-up. Authoring the change cost close to nothing. The checks were the entire safety case, and the checks were wrong. The response was all downstream: stronger safeguards, more rollback capacity, and scheduling automated changes during the owning team’s working hours so a human is awake when it lands.

Read more

Every multi-agent architecture diagram published this year looks like a microservices diagram with the boxes renamed. Planner, researcher, coder, reviewer, arrows between them, a gateway on top.

The resemblance is earned, because both take a monolith (one giant prompt, one giant service) and split it into narrow units that talk over defined interfaces.

Half the microservices playbook transfers cleanly:

• Contracts between units

• Failure isolation

• Explicit orchestration

• Tracing

The other half breaks, because the physics of the call changed:

• A service hop: ~1 ms, free, same answer every time

• An agent hop: 2 to 20 seconds, a full inference pass, different answer on retry

Circuit breakers, idempotent retries, and shared-nothing state were all designed around the first set of numbers. Copy them wholesale and you get a system that is expensive when it works and undebuggable when it fails.

No settled best practice exists yet. This is the version I would defend in a design review this week.

One thing both agree on: the units have to talk over a contract, and the moment that contract has to reach a customer’s system, it becomes a webhook. Today’s sponsor has spent years making that hop work!

Svix: Become event-driven in a day

Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.

Start Sending Webhook Today!

🏗️ The parallel is real

Start with the half that transfers.

The monolith problem is identical. A single agent with 40 tools and a 6,000-token system prompt degrades the way a 400,000-line Rails app does. Every capability you add makes every other one slightly worse, and nobody can predict which change broke which behavior. Splitting into narrow agents, each with a handful of tools and a prompt that fits on one screen, is the same relief valve as splitting services.

Interfaces over shared state is the second transfer. In 2024, most agent-to-agent handoffs were free text pasted into the next prompt. That is two services sharing a database table. Works fine until one side changes the shape and the other side silently misreads it.

MCP is filling the OpenAPI and gRPC role. A typed tool contract, a discovery mechanism, and a transport. It stopped being one vendor’s project in December 2025, when Anthropic donated it to the Linux Foundation’s Agentic AI Foundation with AWS, Google, Microsoft, and OpenAI as founding members. Over 10,000 published servers at the time.

Treat an MCP server like a service boundary:

• Version the contract

• Own it as a team

• Put real auth in front of it

One gap remains, because MCP only standardizes agent-to-tool. Agent-to-agent contracts (what a worker returns, in what schema, with what confidence) are still whatever your framework does by default, and the default is prose. Write the schema yourself, because nobody else will. A worker that returns {"findings": [...], "confidence": 0.7, "sources": [...]} can be validated, cached, and replaced. Three paragraphs of prose can only be pasted into the next prompt.

📣 Show the community what you built!

Over 43,000 engineers read this newsletter every week, and a lot of you are building things on the side: CLIs, SaaS tools, open-source libraries, courses, the app you shipped at 2am. I want more of that in front of the people who would actually use it.

• Banner/Top placement: a dedicated block right here, after the TLDR, with a thematic lead-in written by me ($200) . One sponsor per edition

• Small placement: new this month. If you’ve Built a tool, an app, a course, or an open-source project? One line, a link, and a small image for a fraction of the sponsorship price ($40) . Made for indie builders and small teams

Reply to this email or write to bytesizeddesigninfo@gmail.com and I will send over rates and open dates.

💥 Containment when the failing unit can talk

The microservices instinct is right: one flaky agent should degrade one capability, and the task should finish anyway. The mechanics differ, because an agent fails in more ways than a service does. It can:

• Time out

• Return garbage that parses

• Return correct-looking output that ignored the instruction

• Loop on a tool call forever, with every individual call succeeding

Three containment mechanisms that survive the translation, in the order I would add them:

1. Budgets before timeouts. A wall-clock timeout catches a hung inference. It misses the worker happily making its 40th tool call. Give every invocation a hard cap on tool calls and tokens, enforced by the harness rather than the prompt. When the budget trips, the orchestrator gets a structured “exhausted” result it can route around instead of a partial answer it will trust.

2. Fallback routing in the model dimension. Services fall back to a replica. Agents get a second axis:

• Fall back to a cheaper model tier with a stricter prompt

• Fall back to a smaller tool set

• Fall back to a deterministic path (an extractive summary instead of a generated one)

Design the degraded path before you need it, the same way you would design read-only mode for a database outage.

3. Retries with a ceiling, only on classified failures. The retry storm math is worse here because each retry is a full inference pass. Amplification bills you in dollars and seconds instead of connections.

• Retry on transport errors and rate limits

• Do not retry “the output failed my validator” without changing something: the prompt, the model, or the inputs

• Same call, same context, and you are paying to roll the dice again (more on why below)

🧭 Pick an orchestration pattern on purpose

Nobody ships microservices without a load balancer and service discovery. Nobody should ship multiple agents without deciding who is in charge. Three patterns cover most production systems today.

(Router: one classifier fans a request to exactly one specialist. Orchestrator-worker: one capable model decomposes a task, dispatches to cheap parallel workers, and synthesizes. Supervisor: one model loops with a set of agents, re-planning after each result.)

Router. A cheap classifier reads the request and forwards it to exactly one specialist, one hop with no synthesis step.

• Use for: support triage, intent routing in a product assistant, any case where the categories are known and the work per category is self-contained

• Router model: your cheapest one, a fine-tuned classifier, or a regex if the categories are clean enough

• Common mistake: letting the router “help” by adding context. It should classify and get out of the way

Orchestrator-worker. One capable model decomposes the task, dispatches subtasks to workers, and synthesizes the results. Workers are stateless, parallel, and can run on a different model than the orchestrator.

This is where the cost story lives:

• Orchestrator-tier and worker-tier models from the same vendor sit 5x to 15x apart on per-token price

• In a decomposed task, most tokens are worker tokens

• The number circulating in vendor write-ups this year is a 40 to 60 percent cost reduction versus running everything on the frontier model. Those write-ups are selling something, but the mechanism is sound

• Anthropic’s public data point: an Opus lead with Sonnet subagents beat single-agent Opus by 90.2 percent on their research eval. The same post reports multi-agent runs burn ~15x the tokens of a chat interaction, which is the other half of that ledger

Two rules for making it pay:

• Workers get a schema, an example, and a budget. Nothing else

• The orchestrator sees results, never raw worker transcripts, or your context bill grows with every worker you add

Supervisor. A loop rather than a fan-out. One model holds the plan, calls an agent, reads the result, revises the plan, calls the next one.

• Sequential and stateful. The supervisor’s context accumulates every result

• Use for: long-horizon tasks where step N depends on step N-1 (coding agents, migration tooling, multi-stage pipelines)

• Wrong for: anything parallelizable. It turns a 4-second fan-out into a 40-second chain

Most teams reach for supervisor first because it is easiest to reason about, then discover their task was a fan-out with a synthesis step and they have been paying sequential latency for nothing. Draw the dependency graph of your subtasks before you pick:

• Wide and shallow: orchestrator-worker

• A chain: supervisor

• One node: router, or no multi-agent system at all

🔍 Nobody can trace this yet

Distributed tracing is what made microservices operable. Fifteen years of Dapper, Zipkin, Jaeger, and OpenTelemetry mean a request ID follows a call through 30 services and the flame graph tells you which one ate the latency. The observability guide covers the mechanics.

Agents need a stranger kind of trace. Latency per hop is the easy part. The questions that matter during an incident:

• Which agent decided to call this tool?

• Based on what context?

• Why did it pick that argument?

A span that says execute_tool: search_orders, 340ms is useless when the bug is that the orchestrator hallucinated a customer ID two hops earlier.

The tooling is behind:

• As of this summer, every gen_ai.* attribute, span, and metric in the OpenTelemetry registry is still marked “Development”

• The agent conventions (invoke_agent, execute_tool) and the MCP conventions were split into their own repository in June, with no tagged release as of mid-July

• Vendors instrument against them anyway, so the schema you adopt today has a fair chance of changing under you

• Datadog’s own AI incident investigator needed bespoke instrumentation to make its reasoning auditable, and that is a company whose product is observability

What to do in the meantime: log the full prompt and full output of every agent hop, keyed by a trace ID that flows through the whole task, into cheap storage with a short retention window. Crude and large, and the only thing that lets you replay a failed run. Anthropic’s research team landed in the same place: full production tracing was what let them diagnose failures, because runs are non-deterministic between attempts even with identical prompts.

Watch the OTel GenAI repo for a stable tag. Until then, own your schema and expect to migrate it.

🚨 Where the analogy breaks

Forward the next three sections to whoever on your team is drawing the diagram.

💸 The cost inversion

Microservices decomposition is cheap at the seams. A gRPC hop is around a millisecond, so the architecture question is purely about ownership and blast radius.

An agent hop is a full inference pass with the entire subtask context re-encoded as input tokens:

Read more

Stripe ran Envoy for almost ten years, then replaced it with their own Go proxy, mesh-proxy. The payoff:

• ~50% less CPU under high load, ~50% lower latency

• ~3,000 days of inter-service latency sa…

Read more

A production voice feature in 2026 typically ships with four vendor relationships: a transcription API, a PII redaction step somebody built in a sprint, an LLM provider, and a compliance review s…

Read more

TLDR

August 17. GitHub went sideways for 7 hours 47 minutes.

• ~20% error rate on web and API traffic

• ~50% on archive and raw content downloads

• Issues, Pull Requests, Actions, Copilot, SAML/OIDC, SCIM, and Team Sync all degraded

• Copilot Token Service went from 7,000-9,000 RPS to 70,000-100,000 RPS

One Istio sidecar pod in Central US hit its concurrency ceiling and failed to autoscale. That took down four HAProxy nodes, which took down the gateway auth path. Most of GitHub was back by 16:36 UTC and Actions by around 18:03. Copilot limped until 21:02, because a latent retry bug in VS Code turned a slow endpoint into a load generator running on millions of laptops.

The capacity problem was mitigated in the afternoon. The retry storm it set off ran for another five hours.

The sidecar nobody was scaling on

Istio puts a proxy sidecar next to every service pod, and all traffic flows through it. You get mTLS and routing without touching application code. You also get a second set of concurrency limits per pod, sitting somewhere entirely different from the application’s.

The autoscaling policy watched the host service. When Central US hit a new traffic peak, the sidecar saturated first. The autoscaler read host metrics, saw headroom, and left the deployment alone.

A pod that cannot accept connections and cannot be replaced by more pods sheds load. Whatever sits in front of it eats the difference.

The cascade, in order

1. Sidecar saturates, autoscaler does nothing.

2. Refused connections pile onto HAProxy in Central US.

3. Four HAProxy nodes exhaust their flow limits, one after another.

4. Those four front the gateway auth path, so auth latency climbs.

5. Callers waiting on a fast dependency start treating slow responses as failures.

6. Retry logic engages, and the auth path now gets more traffic than it did before it broke.

Auth is the worst place in a system to lose capacity, because nearly every request touches it. GitHub knows this one well already: a 43-second network partition in 2018 cost them 24 hours of degradation, and the trigger was similarly small next to the blast radius.

Recovery required breaking things harder

Some failing traffic was moved out to Northern Virginia and served successfully there while the Central US network failure was debugged. Regional failover did its job for the bulk of user requests. The retry behavior travelled with the clients, so Northern Virginia inherited the herd along with the load.

The Central US fix is the interesting one. Pausing HAProxy on the saturated nodes, all four simultaneously, produced immediate broad recovery.

Staggering that pause would have pushed load onto whichever node was still accepting traffic and kept the saturation alive. A feedback loop has to be broken everywhere at once, which is an uncomfortable thing to do at 16:00 UTC on a Monday with 20% of your traffic already failing.

Read more

Linear Agent, the assistant that lives inside Linear, does most of its valuable work in ways nobody scripted, and that premise cuts against engineering instinct. The engineering effort went into…

Read more

Read more

There is a shift happening across the industry, and GitHub’s engineering blog put a name on it last week: engineering’s oldest reflex is now misfiring on a specific class of work.

The argument goe…

Read more

Netflix wrote Compute Managed Batch (CMB) in 2018 because nothing in the Kubernetes ecosystem did what they needed. Hierarchical tenants. Fair sharing. Capacity reservations. Priority queues. They built all of it themselves.

Seven years later, Kueue does all of that, plus the features CMB never shipped. So Netflix ripped out CMB’s queuing and scheduling logic and replaced it with Kueue. Millions of batch jobs moved over. The production migration took 4 weeks. Users did nothing and noticed nothing.

The interesting part is not the swap. It is how they decided when to stop maintaining their own thing, and how they migrated without ever exposing a new API to customers.

What CMB Actually Was

CMB is a managed batch layer sitting on top of Titus, Netflix’s container platform. Users submit jobs that run to completion. CMB queues them, orders them by priority, and enforces capacity per tenant.

The tenant model is a tree. Internal tenants are branches, they exist to group things and hold capacity for their subtree. Leaf tenants are where work actually lands, and only leaves have queues. An org might run one tenant for everything, or mirror its entire team structure in the tree.

Capacity came in two flavors:

Reserved. A leaf tenant with a reservation partitions that capacity out of the hierarchy. Nobody else can take it. That guarantees throughput.

Shared. The Compute team runs a global burst pool. Any tenant can pull from it. It is fair-shared across tenants.

Here is the flaw, and it is the flaw that eventually forced the rewrite. CMB fair-shared the pool at admission only. There was no preemption. Once a job was admitted, it ran to completion no matter how the demand picture changed underneath it. A tenant that got in early held its slice until it finished. Fair sharing that only applies at the door is not fair sharing. It is a queue with extra vocabulary.

The other cost was structural. CMB sat far away from the actual Kubernetes clusters. Titus federates jobs across cells, so CMB talked to one endpoint and stayed blissfully unaware of cluster topology. Great for simplicity. Terrible when you want to build preemption, because preemption is a scheduler-level concern and CMB was several abstraction layers above the scheduler. Every new feature meant re-implementing something Kubernetes already had a place for.

Why Kueue and Not the Others

Netflix looked at YuniKorn and Volcano too. Both replace the pod scheduler. That was the disqualifier.

Titus has its own scheduling profiles. Swapping out kube-scheduler means fragmenting job placement, and fragmented placement means worse bin-packing, which means burning money on idle capacity. If you have ever watched a fleet run at 40% utilization because two schedulers disagreed about where things go, you know exactly what this costs. Netflix has written before about how much compute efficiency is worth at this scale.

Kueue does not touch pod scheduling. It sits above it and decides whether a job gets admitted, then hands the pods to the existing scheduler. That single design choice made it the only option that composed with what Netflix already ran.

The rest of the case was straightforward:

• Multi-tenant quota over heterogeneous hardware, which is table stakes when your fleet is a zoo of instance types

• Works on v1.Pod and batch/v1.Job, and also on RayJob / RayCluster for the ML workloads coming down the pipe

• Native preemption, all-or-nothing scheduling, topology-aware scheduling. All the things CMB wanted and never got

• Real adoption momentum, which matters more than feature checklists over a five year horizon

The Migration: Change Nothing the User Can See

They called it Netflix Batch. Three tenets, and the first one is the one worth stealing:

1. Zero lift for CMB users. Completely transparent.

2. No regression in container launch rate or max throughput.

3. Replace CMB queuing and scheduling with Kueue.

Read more

A normal C++ function inside OpenAI’s Rockset service runs, finishes, and returns to garbage. Sometimes the saved return address on the stack is NULL. Sometimes the…

Read more

Last year, 230 of you took a survey about a tool to keep your system design knowledge sharp using real case studies like the ones we cover here, and to actually h…

Read more

Read more

Netflix runs on thousands of microservices. When one breaks at 3am, the on-call engineer needs to know what depends on it, what it depends on, and whether the problem is local or upstream. That a…

Read more

Discord’s Persistence Infrastructure team runs every ScyllaDB cluster behind messages, channels, servers, and most of Discord’s user data. Dozens of clusters. Hundreds of nodes. Seven engineers.

For years, the tooling was a pile of Python and bash scripts that worked but required someone who remembered all the landmines. Standing up a full shadow cluster, a complete replica that mirrors production traffic so you can safety-test a new ScyllaDB release, took a day and a half of careful, sequential, do-not-screw-up-step-nine work.

They rebuilt it as the Scylla Control Plane (SCP): composable tasks, YAML-defined workflows, automatic retries, and resumable jobs. The same operation now takes under two hours, and most of that is the engineer doing something else while nodes bootstrap.

The interesting part isn’t that they automated database ops. It’s the specific shape of the automation, and three boring decisions that mattered more than any of the Rust.

Seven Engineers, Hundreds of Nodes

ScyllaDB is Discord’s largest database by scope. We’ve covered how that data layer evolved before, How Discord Indexes Trillions of Messages walked through the search side of the same infrastructure. The team that operates it is seven people.

That ratio sounds fine until you look at what “operating” actually means: rolling restarts after every config change, expanding clusters as servers fill up, rolling OS upgrades across hundreds of nodes with zero downtime, and validating every new ScyllaDB release on a shadow cluster before it touches production.

None of that is fire-and-forget. Each one demands careful sequencing, validation, and someone paying attention the entire time. We talked about why that kind of pre-production validation matters in The Tech Lead’s Guide to Load Testing, shadow clusters are basically load testing with real production traffic and real production stakes.

For years, Discord automated this the way most teams do: incrementally, under pressure, with no long-term plan. A Python script here, a bash script there. It worked. It also required deep institutional knowledge to run safely, and that’s the kind of debt that’s invisible until the one person who understands it is on vacation.

Three Ways the Old Scripts Failed

Read more

Reactive auto-scaling has a fundamental problem: by the time it kicks in, you’re already overloaded. The scaling operation itself takes several minutes. You’ve spent real money on degraded perfor…

Read more

23:50 UTC, Thursday, May 7 2026. A data center hall in Northern Virginia overheats. Cooling units in availability zone az4 give up. EC2 instances and EBS volumes start losing power inside the hour.

Coinbase goes dark. Seven hours. Trading, Prime, International, derivatives, balance updates, all gone. FanDuel goes down at 9pm ET right as Game 2 of Lakers-Thunder tips off. CME Direct logs go sideways for institutional traders.

Coinbase had multi-AZ on most of their stack. But not the matching engine. That one ran in a single zone, on purpose, for latency. They had a backup. The backup didn’t isolate from the failure the way it was supposed to.

This is a story about a tradeoff that finally got tested in production. And about the difference between high availability and disaster recovery, which is not the same thing, no matter how many architecture diagrams pretend otherwise.

What actually broke

The root cause was not software. It was a building. Multiple chillers in a single data center hall failed. Temperatures climbed. AWS lost power to racks in availability zone use1-az4. EC2 instances and EBS volumes on those racks were physically damaged, not “marked unhealthy,” damaged.

AWS shifted traffic away from the zone, but recovery depended on getting cooling back online before they could safely bring damaged hardware up again. That took more than 20 hours. Cooling was stable at pre-event levels by 13:50 PT Friday afternoon.

This is the part of the story no architecture diagram makes visible. The internet still runs in buildings, and buildings can overheat.

If that framing sounds familiar, the October 20, 2025 AWS outage had the same shape from a different angle, 15 hours, 113 services, traced back to one empty DNS record. Different failure mode, same conclusion: at AWS’s scale, the failures that take down half the internet are rarely the ones you’d architect for.

The Coinbase tradeoff

Here’s the part that matters for engineers.

Coinbase had multi-AZ. They said so plainly: most of their systems are designed to survive a single AZ failing. Most of them did.

The matching engine didn’t.

Rob Witoff, Head of Platform, was honest about why. The exchange runs in a single availability zone by design. Latency. Customer co-location. Real money on the line, measured in microseconds. Spreading it across zones would have meant accepting cross-AZ network hops on every order match. For an exchange that competes on speed, that is not a free trade.

So they made the call. Single AZ for the matching engine. And — this is the part everyone glosses over — they built a backup. A distributed copy of the exchange infrastructure, designed to take over if the primary zone died.

The backup did not work as expected.

Witoff was specific: “backup systems did not work as expected during the incident, extending the outage and forcing engineers to manually execute disaster recovery procedures.” Engineers had to develop, test, deploy, and validate a fix while the production system was on fire.

Kafka made it worse. Coinbase runs partitioned Kafka handling thousands of terabytes a day. That couldn’t fail over automatically either. It needed manual recovery. Balance streams lagged behind until replication caught up.

No data was lost. But seven hours of downtime is seven hours of downtime.

“Distributed backup” is not the same as “distributed”

This is the lesson worth dwelling on.

A backup that depends on the same failure domain as the primary is not a backup. It is a copy.

Coinbase had a distributed copy of the exchange. But “distributed” without specifying what failure mode it survives is marketing. If your backup database lives in the same AZ as your primary, an AZ outage takes both. If your backup region depends on your primary region’s IAM control plane, a region outage takes both.

The principle generalizes brutally: your failover only works against failure modes you actually tested it against. Untested failover is not failover. It is a hypothesis.

GitLab learned this the hard way in 2017. When an engineer accidentally deleted 300GB of production data, they discovered their backup system had been broken for weeks. Five backup mechanisms. None of them worked. Different failure mode, same shape: a safety net that nobody had pulled on recently.

The Coinbase incident is the kinder version of the same story. Their backup existed. Their backup partially worked. But the bits that didn’t work showed up only under the specific conditions of the specific failure that actually happened, which is exactly the moment when you can’t afford to discover them.

If your DR plan has not been exercised in the last 90 days, treat it as untested.

High availability and disaster recovery are different problems

Every time something like this happens, the postmortem comment section fills up with “they should have used multi-AZ.” This is half-right and entirely beside the point.

Multi-AZ is high availability. It protects you from a bad day in one zone. Coinbase already had it for most workloads.

Multi-region is disaster recovery. It protects you from a bad day in one region.

AWS treats availability zones as the failure domain for HA. They treat regions as independent on purpose. A thermal event in US-EAST-1 will not move your data to US-WEST-2 unless you have explicitly built that path, paid for the replication, tested the failover, and decided in advance who gets to push the button.

Most production systems need both. Most production systems have one.

The reason is cost. Cross-region replication is real money. Hot standby in a second region doubles your compute bill. And the day-to-day value of that spending is invisible, until the day the chillers fail, at which point the conversation stops being about budget and becomes about how fast you can recover.

This is the part where engineering leadership earns their salary. Resilience investments look expensive right up until the moment they look obvious.

How Coinbase actually recovered

The recovery sequence is worth studying because it shows what disciplined incident response looks like under load.

When the matching engine came back, they did not just re-enable trading and let it rip. They staged it:

1. Cancel-only mode — let customers withdraw existing orders, but no new trades

2. Auction mode — orders accumulate, no immediate matching, gives time to verify books are consistent

3. Live trading — only after product-by-product health checks

This is not glamorous. It is the engineering equivalent of bringing a power plant back online — slow, sequential, fully verified before each step. The temptation in a multi-hour outage is to fix the thing and flip the switch. Coinbase didn’t. They reconciled state first. No data was lost. That outcome is downstream of this discipline, not luck.

The public communication followed the same pattern. CEO Brian Armstrong posted within hours: “never acceptable.” Witoff followed with a technical thread giving the actual timeline, the actual root cause, the actual decision being revisited. No PR-speak. No “we are looking into it.” Specific.

That is what a public post-mortem looks like when the engineering org is the one writing it, not legal.

The SLA is not your business continuity plan

One detail from the AWS side worth internalizing. The standard EC2 SLA pays out around 10% of monthly compute spend on impacted instances. That’s the entire compensation.

Lost revenue: not covered. Customer trust: not covered. Regulatory exposure: not covered.

The 2024 ITIC survey put hourly downtime cost above $300,000 for 90% of mid-to-large enterprises. 41% lose between $1M and $5M per hour. Trading and finance are higher still. Coinbase was down for seven hours during a quarter where they had also just announced a 14% workforce reduction and a $394M net loss. The 10% AWS service credit was not going to move that needle.

Your cloud provider’s SLA is a small refund. It is not insurance. It is not a business continuity plan. The continuity plan is something you build, fund, and test yourself, or you don’t have one.

Three things to do this week

Read more

Hundreds of inference calls per investigation. Megabytes of agent output. A team of LLMs all working on the same alert, none of them sharing memory by default. That’s what Slack’s security investigation service has to wrangle every time an alert fires.

The agents don’t share a brain. The API is stateless. Stuff every transcript into every prompt and you blow the context window. Strip too much out and the agents drift, contradict each other, and stop sounding like a team.

Slack’s answer: stop pretending one shared context can do the job. Build three of them. Each agent reads the channel that fits its role.

It’s one of the cleanest examples I’ve seen of context engineering as a real discipline, not a vibe.

The setup

Slack runs security investigations with a team of agents. A Director orchestrates. A bunch of Experts go gather evidence using domain-specific tools. A Critic reviews what the Experts produced and flags the bits that don’t hold up.

Investigations move through phases. Each phase has rounds. There’s no fixed budget, the Director decides when to stop.

The dependency runs both ways. Each agent needs enough of the wider investigation to stay coherent with the team. But shove the full transcript into every prompt and two things happen: you hit the context window, and the agent starts drifting toward whatever the loudest voice in the history said. Confirmation bias dressed up as continuity.

Same problem every multi-agent system hits. They built three context channels to solve it.

Channel one: the Director’s Journal

The Director keeps a journal. Not a chat log, but a structured one. Six entry types: decision, observation, finding, question, action, hypothesis. Every entry gets a phase, a round, a timestamp, a priority, and optional citations to evidence.

The journaling tool itself is dumb. It appends. The intelligence is in the prompt that tells the Director to use it constantly, and in the schema that forces every thought into a slot.

The Journal goes into every other agent’s prompt. It’s the spine. The Experts read it to understand what they’re being asked and why. The Critic reads it to understand what the Director was trying to prove.

Read enough Journal entries and you can watch the investigation think. One example from a real run: the Director starts by classifying an alert as a kernel module load, lists 4 expert domains it’ll need, notices the cgroup suggests a personal workstation, then realizes the alert is matching the path of a hook script rather than an actual modprobe call. By round five it’s preliminary-false-positive. By round seven it’s confirmed FALSE POSITIVE with all four Experts agreeing.

That progression is only legible because the Journal exists. Without it, you’d have a pile of disconnected expert outputs and no narrative.

This is the same shape of problem covered in a previous edition on designing AI agents that think in real time, different domain, but the structural answer is identical: give the orchestrator a working memory that everyone else can read.

Channel two: the Critic’s Review

Read more