⚡ TLDR
17,000 merged changes in August, up from roughly 8,100 a year earlier. That is Spotify’s engineering org after a year of AI coding tools, spread across nearly 3,000 production services handling 11 to 12 million backend requests per second.
The part most people skip past:
Their incident reviews have found no AI-authored code acting as a material direct contributor to a major incident so far
The strain showed up after the merge, in review, testing, rollout, observability, and rollback
Change volume grew faster than the controls that verify change
AI moved the constraint downstream. Writing the change got cheap, and proving the change is safe became the expensive step.
This edition walks through one Spotify incident from June that has nothing to do with AI, because it happens to be a perfect physical model of what a 2x change rate does to a delivery pipeline. Then a five-part scorecard you can take to your next staff meeting, since acceptance rate and “percent of code written by AI” will tell you almost nothing about any of this.
Svix: Become event-driven in a day
Your customers want to make their agent workflows event-driven, so don’t lose deals trying to implement webhooks yourself.
🎬 June 24: a queue with four things wrong at once
More than 500,000 songs, videos, podcasts, and audiobooks enter Spotify’s processing systems every day. Video podcasts go through a transcoding fleet with a medium-priority queue for new episodes and a low-priority queue for updates to existing ones. Normal processing time is minutes, and on June 24 it became hours. Here is the timeline, in UTC:
13:30 Early alerts fire. Nobody reads them as a fleet-wide capacity problem
15:00 A spike in video podcast deliveries pushes transcoding close to its ceiling
16:35 A scheduled batch job gets stopped to free capacity
17:31 First creator report of a stuck episode
17:34 Queue-backlog alerts cross threshold and formal incident response starts
20:49 A fix ships for a resource scheduling bug
00:14 An additional processing cluster comes online
01:02 Queues clear
Roughly four hours between the first alert and a declared incident, and about eleven and a half between the first alert and an empty queue. Many creators found out from their audiences before they heard from Spotify.
Nothing in the incident report involves AI, and the word never appears in it. Four ordinary engineering problems stacked on the same afternoon:
Thin headroom. The transcoding fleet had too little spare capacity to absorb a large delivery spike
A competing batch job. A routine re-processing run for existing episodes was eating capacity during peak submissions
A quality win that cost compute. A recent change delivered better video at lower bitrates. Each episode now took more processing, and the capacity plan still assumed the old cost
A scheduler bug. After a migration to more powerful hardware, a resource scheduling bug left compute underused and cut throughput by about 10%
Any one of these is survivable. A 10% throughput loss on a fleet with 40% headroom is a line item in a weekly review. Put all four on the same afternoon and arrivals exceed service rate, which is the only condition a queue needs to grow without bound.
The monitoring gap is another story. The 13:30 alerts were symptom alerts, and they fired correctly. What was missing was a signal that said “arrival rate is about to exceed service rate for the whole fleet.” Twitch hit the same class of problem with failures their dashboards technically recorded and nobody could interpret. Alerts that detect pain are cheap. Alerts that explain capacity are the ones worth building, and the observability guide covers how to get there.
My one criticism of the engineering: the transcoding quality change should have gone through capacity review. Any change that raises per-item processing cost is a capacity change, whatever the ticket says. Spotify’s fix list suggests they agree. They added about 67% more transcoding capacity, fixed the scheduler, and committed to planning for steady state, burst, and recovery separately, with creator uploads always ahead of background work and backpressure extended through the pipeline.
🔁 Now swap episodes for pull requests
The mapping below is mine, and it’s an analogy. AI had no role in June 24. The shape of the failure is what transfers.
Your delivery pipeline is also a queue with a fixed-ish service rate. Changes arrive, wait for review, wait for CI, wait for a rollout slot, and bake in production while someone watches a dashboard.
design -> implement -> review -> test -> deploy -> observe -> operate
^ ^________________________________________^
AI made this capacity here is mostly unchanged
stage ~2x cheaperLine the June 24 factors up against a year of AI-assisted development:
The delivery spike is merged changes going from ~8,100 to ~17,000
Higher per-item cost is PR size and code complexity, both of which are creeping upward at Spotify. A bigger diff takes longer to review, the same way a higher-quality transcode takes longer to encode
The competing batch job is automated fleet-wide change. Spotify’s Fleet Management tooling finished a Java migration across backend services in three days. It also shipped an automated dependency upgrade that passed every check and still failed in production with user impact
Thin headroom is your reviewers, your CI fleet, and your canary slots, all sized for last year’s arrival rate
Symptom-level monitoring is the AI usage dashboard. It tells you the tools are busy
That dependency upgrade is the cleanest example in the whole September write-up. Authoring the change cost close to nothing. The checks were the entire safety case, and the checks were wrong. The response was all downstream: stronger safeguards, more rollback capacity, and scheduling automated changes during the owning team’s working hours so a human is awake when it lands.
