ā” TLDR
Every multi-agent architecture diagram published this year looks like a microservices diagram with the boxes renamed. Planner, researcher, coder, reviewer, arrows between them, a gateway on top.
The resemblance is earned, because both take a monolith (one giant prompt, one giant service) and split it into narrow units that talk over defined interfaces.
Half the microservices playbook transfers cleanly:
Contracts between units
Failure isolation
Explicit orchestration
Tracing
The other half breaks, because the physics of the call changed:
A service hop: ~1 ms, free, same answer every time
An agent hop: 2 to 20 seconds, a full inference pass, different answer on retry
Circuit breakers, idempotent retries, and shared-nothing state were all designed around the first set of numbers. Copy them wholesale and you get a system that is expensive when it works and undebuggable when it fails.
No settled best practice exists yet. This is the version I would defend in a design review this week.
One thing both agree on: the units have to talk over a contract, and the moment that contract has to reach a customerās system, it becomes a webhook. Todayās sponsor has spent years making that hop work!
Svix: Become event-driven in a day
Your customers want to make their agent workflows event-driven, so donāt lose deals trying to implement webhooks yourself.
šļø The parallel is real
Start with the half that transfers.
The monolith problem is identical. A single agent with 40 tools and a 6,000-token system prompt degrades the way a 400,000-line Rails app does. Every capability you add makes every other one slightly worse, and nobody can predict which change broke which behavior. Splitting into narrow agents, each with a handful of tools and a prompt that fits on one screen, is the same relief valve as splitting services.
Interfaces over shared state is the second transfer. In 2024, most agent-to-agent handoffs were free text pasted into the next prompt. That is two services sharing a database table. Works fine until one side changes the shape and the other side silently misreads it.
MCP is filling the OpenAPI and gRPC role. A typed tool contract, a discovery mechanism, and a transport. It stopped being one vendorās project in December 2025, when Anthropic donated it to the Linux Foundationās Agentic AI Foundation with AWS, Google, Microsoft, and OpenAI as founding members. Over 10,000 published servers at the time.
Treat an MCP server like a service boundary:
Version the contract
Own it as a team
One gap remains, because MCP only standardizes agent-to-tool. Agent-to-agent contracts (what a worker returns, in what schema, with what confidence) are still whatever your framework does by default, and the default is prose. Write the schema yourself, because nobody else will. A worker that returns {"findings": [...], "confidence": 0.7, "sources": [...]} can be validated, cached, and replaced. Three paragraphs of prose can only be pasted into the next prompt.
š£ Show the community what you built!
Over 43,000 engineers read this newsletter every week, and a lot of you are building things on the side: CLIs, SaaS tools, open-source libraries, courses, the app you shipped at 2am. I want more of that in front of the people who would actually use it.
Banner/Top placement: a dedicated block right here, after the TLDR, with a thematic lead-in written by me ($200) . One sponsor per edition
Small placement: new this month. If youāve Built a tool, an app, a course, or an open-source project? One line, a link, and a small image for a fraction of the sponsorship price ($40) . Made for indie builders and small teams
Reply to this email or write to bytesizeddesigninfo@gmail.com and I will send over rates and open dates.
š„ Containment when the failing unit can talk
The microservices instinct is right: one flaky agent should degrade one capability, and the task should finish anyway. The mechanics differ, because an agent fails in more ways than a service does. It can:
Time out
Return garbage that parses
Return correct-looking output that ignored the instruction
Loop on a tool call forever, with every individual call succeeding
Three containment mechanisms that survive the translation, in the order I would add them:
1. Budgets before timeouts. A wall-clock timeout catches a hung inference. It misses the worker happily making its 40th tool call. Give every invocation a hard cap on tool calls and tokens, enforced by the harness rather than the prompt. When the budget trips, the orchestrator gets a structured āexhaustedā result it can route around instead of a partial answer it will trust.
2. Fallback routing in the model dimension. Services fall back to a replica. Agents get a second axis:
Fall back to a cheaper model tier with a stricter prompt
Fall back to a smaller tool set
Fall back to a deterministic path (an extractive summary instead of a generated one)
Design the degraded path before you need it, the same way you would design read-only mode for a database outage.
3. Retries with a ceiling, only on classified failures. The retry storm math is worse here because each retry is a full inference pass. Amplification bills you in dollars and seconds instead of connections.
Retry on transport errors and rate limits
Do not retry āthe output failed my validatorā without changing something: the prompt, the model, or the inputs
Same call, same context, and you are paying to roll the dice again (more on why below)
š§ Pick an orchestration pattern on purpose
Nobody ships microservices without a load balancer and service discovery. Nobody should ship multiple agents without deciding who is in charge. Three patterns cover most production systems today.
(Router: one classifier fans a request to exactly one specialist. Orchestrator-worker: one capable model decomposes a task, dispatches to cheap parallel workers, and synthesizes. Supervisor: one model loops with a set of agents, re-planning after each result.)
Router. A cheap classifier reads the request and forwards it to exactly one specialist, one hop with no synthesis step.
Use for: support triage, intent routing in a product assistant, any case where the categories are known and the work per category is self-contained
Router model: your cheapest one, a fine-tuned classifier, or a regex if the categories are clean enough
Common mistake: letting the router āhelpā by adding context. It should classify and get out of the way
Orchestrator-worker. One capable model decomposes the task, dispatches subtasks to workers, and synthesizes the results. Workers are stateless, parallel, and can run on a different model than the orchestrator.
This is where the cost story lives:
Orchestrator-tier and worker-tier models from the same vendor sit 5x to 15x apart on per-token price
In a decomposed task, most tokens are worker tokens
The number circulating in vendor write-ups this year is a 40 to 60 percent cost reduction versus running everything on the frontier model. Those write-ups are selling something, but the mechanism is sound
Anthropicās public data point: an Opus lead with Sonnet subagents beat single-agent Opus by 90.2 percent on their research eval. The same post reports multi-agent runs burn ~15x the tokens of a chat interaction, which is the other half of that ledger
Two rules for making it pay:
Workers get a schema, an example, and a budget. Nothing else
The orchestrator sees results, never raw worker transcripts, or your context bill grows with every worker you add
Supervisor. A loop rather than a fan-out. One model holds the plan, calls an agent, reads the result, revises the plan, calls the next one.
Sequential and stateful. The supervisorās context accumulates every result
Use for: long-horizon tasks where step N depends on step N-1 (coding agents, migration tooling, multi-stage pipelines)
Wrong for: anything parallelizable. It turns a 4-second fan-out into a 40-second chain
Most teams reach for supervisor first because it is easiest to reason about, then discover their task was a fan-out with a synthesis step and they have been paying sequential latency for nothing. Draw the dependency graph of your subtasks before you pick:
Wide and shallow: orchestrator-worker
A chain: supervisor
One node: router, or no multi-agent system at all
š Nobody can trace this yet
Distributed tracing is what made microservices operable. Fifteen years of Dapper, Zipkin, Jaeger, and OpenTelemetry mean a request ID follows a call through 30 services and the flame graph tells you which one ate the latency. The observability guide covers the mechanics.
Agents need a stranger kind of trace. Latency per hop is the easy part. The questions that matter during an incident:
Which agent decided to call this tool?
Based on what context?
Why did it pick that argument?
A span that says execute_tool: search_orders, 340ms is useless when the bug is that the orchestrator hallucinated a customer ID two hops earlier.
The tooling is behind:
As of this summer, every
gen_ai.*attribute, span, and metric in the OpenTelemetry registry is still marked āDevelopmentāThe agent conventions (
invoke_agent,execute_tool) and the MCP conventions were split into their own repository in June, with no tagged release as of mid-JulyVendors instrument against them anyway, so the schema you adopt today has a fair chance of changing under you
Datadogās own AI incident investigator needed bespoke instrumentation to make its reasoning auditable, and that is a company whose product is observability
What to do in the meantime: log the full prompt and full output of every agent hop, keyed by a trace ID that flows through the whole task, into cheap storage with a short retention window. Crude and large, and the only thing that lets you replay a failed run. Anthropicās research team landed in the same place: full production tracing was what let them diagnose failures, because runs are non-deterministic between attempts even with identical prompts.
Watch the OTel GenAI repo for a stable tag. Until then, own your schema and expect to migrate it.
šØ Where the analogy breaks
Forward the next three sections to whoever on your team is drawing the diagram.
šø The cost inversion
Microservices decomposition is cheap at the seams. A gRPC hop is around a millisecond, so the architecture question is purely about ownership and blast radius.
An agent hop is a full inference pass with the entire subtask context re-encoded as input tokens:
