The Disney architecture for India Vs. Pakistan World Cup
In a single database
TLDR;
Before 2021, Disney had a problem streaming their content.
There was too much metadata on users and their current infrastructure had issues.
→ Data was duplicated between Redis and Elasticsearch
→ User viewing data was scaling A LOT.
📺 Letting everyone watch the world cup
Disney+’s existing architecture stores view data.
Examples:
If you saw this show already
Reminding you when the world cup is on
Remembering timestamps on a show you already started
🔌 The technologies behind it
To keep track of viewer data, Disney used Redis and Elasticsearch… except it didn’t scale very well. Especially during world cup season.
⚡ So how’d they scale it?
Simple answer:
First they set the data model with the following map pairing.
userId → contentId[]Then they started migrating to ScyllaDB.
So why do it like this?
Because sometimes you want data on a specific content show, not all the shows. Always getting everything all at once would be super slow and kills the database server.
❓ But why ScyllaDB?
It was cheap
It had fast reads and writes
It could be used as a central data store
Instead of being split like elasticsearch and redis
🗓️ How do you seamlessly migrate to a new database?
The common strategy for this is to do work twice.
Meaning write to both databases and only read on the new one. Fallback to reading on the old one if data is missing.
Keep doing this until the migration is done. Then flip the switch and route all reads and writes to the new database.
We can call it a day and drop the old database and all it’s problems
💰 HELP WANTED
This newsletter has grown to 5000+ AMAZING READERS. It’s grown to a scale that a single person can’t maintain all of it on their own.
If you’re interested in being a byte-sized design writer, apply here!
📝 Official Article
(Links to official article and sources are available to paid subscribers. They help maintain and support this newsletter!)
