In the world of streaming, the “Super Bowl” isn’t just a game. It is a distributed systems stress test that happens in real-time before tens of millions of people.

When I manage infrastructure for major events (whether it is the Olympics, a Premier League match or a season finale) I am dealing with a “thundering herd” problem that few systems ever face. Millions of users log in, browse and hit “play” within the same three-minute window.

But this challenge isn’t unique to media. It is the same nightmare that keeps e-commerce CTOs awake before Black Friday or financial systems architects up during a market crash. The fundamental problem is always the same: How do you survive when demand exceeds capacity by an order of magnitude?

Most engineering teams rely on auto-scaling to save them. But at the “Super Bowl standard” of scale, auto-scaling is a lie. It is too reactive. By the time your cloud provider spins up new instances, your latency has already spiked, your database connection pool is exhausted and your users are staring at a 500 error.

Here are the four architectural patterns we use to survive massive concurrency. These apply whether you are streaming touchdowns or processing checkout queues for a limited-edition sneaker drop.

1. Aggressive load shedding

The biggest mistake engineers make is trying to process every request that hits the load balancer. In a high-concurrency event, this is suicide. If your system capacity is 100,000 requests per second (RPS) and you receive 120,000 RPS, trying to serve everyone usually results in the database locking up and zero people getting served.

We implement load shedding based on business priority. It is better to serve 100,000 users perfectly and tell 20,000 users to “please wait” than to crash the site for all 120,000.

This requires classifying traffic at the gateway layer into distinct tiers:

• Tier 1 (Critical): Login, Video Playback (or for e-commerce: Checkout, Inventory Lock). These requests must succeed.
• Tier 2 (Degradable): Search, Content Discovery, User Profile edits. These can be served from stale caches.
• Tier 3 (Non-Essential): Recommendations, “People also bought,” Social feeds. These can fail silently.

We use adaptive concurrency limits to detect when downstream latency is rising. As soon as the database response time crosses a threshold (e.g. 50ms), the system automatically stops calling the Tier 3 services. The user sees a slightly generic homepage, but the video plays or the purchase completes.

For any high-volume system, you must define your “degraded mode.” If you don’t decide what to turn off during a spike, the system will decide for you, usually by turning off everything.

2. Bulkheads and blast radius isolation

Manoj Yerrasani

On a cruise ship, the hull is divided into watertight compartments called bulkheads. If one section floods, the ship stays afloat. In distributed systems, we often inadvertently build ships with no walls.

I have seen massive outages caused by a minor feature. For example, a third-party API that serves “user avatars” goes down. Because the “Login” service waits for the avatar to load before confirming the session, the entire login flow hangs. A cosmetic feature takes down the core business.

To prevent this, we use the bulkhead pattern. We isolate thread pools and connection pools for different dependencies.

In an e-commerce context, your “Inventory Service” and your “User Reviews Service” should never share the same database connection pool. If the Reviews service gets hammered by bots scraping data, it should not consume the resources needed to look up product availability.

We strictly enforce timeouts and Circuit Breakers. If a non-essential dependency fails more than 50% of the time, we stop calling it immediately and return a default value (e.g. a generic avatar or a cached review score).

Crucially, we prefer semaphore isolation over thread pool isolation for high-throughput services. Thread pools add overhead context switching. Semaphores simply limit the number of concurrent calls allowed to a specific dependency, rejecting excess traffic instantly without queuing. The core transaction must survive even if the peripherals are burning.

3. Taming the thundering herd with request collapsing

Imagine 50,000 users all load the homepage at the exact same second (kick-off time or a product launch). All 50,000 requests hit your backend asking for the same data: “What is the metadata for the Super Bowl stream?”

If you let all 50,000 requests hit your database, you will crush it.

Caching is the obvious answer, but standard caching isn’t enough. You are vulnerable to the “Cache Stampede.” This happens when a popular cache key expires. Suddenly, thousands of requests notice the missing key and all of them rush to the database to regenerate it simultaneously.

To solve this, we use request collapsing (often called “singleflight”).

When a cache miss occurs, the first request goes to the database to fetch the data. The system identifies that 49,999 other people are asking for the same key. Instead of sending them to the database, it holds them in a wait state. Once the first request returns, the system populates the cache and serves all 50,000 users with that single result.

This pattern is critical for “flash sale” scenarios in retail. When a million users refresh the page to see if a product is in stock, you cannot do a million database lookups. You do one lookup and broadcast the result.

We also employ probabilistic early expiration (or the X-Fetch algorithm). Instead of waiting for a cache item to fully expire, we re-fetch it in the background while it is still valid. This ensures the user always hits a warm cache and never triggers a stampede.

4. The ‘game day’ rehearsal

The patterns above are theoretical until tested. In my experience, you do not rise to the occasion during a crisis; you fall to the level of your training.

For the Olympics and Super Bowl, we don’t just hope the architecture works. We break it on purpose. We run game days where we simulate massive traffic spikes and inject failures into the production environment (or a near-production replica).

We simulate specific disaster scenarios:

• What happens if the primary Redis cluster vanishes?
• What happens if the recommendation engine latency spikes to 2 seconds?
• What happens if 5 million users log in within 60 seconds?

During these exercises, we validate that our load shedding actually kicks in. We verify that the bulkheads actually stop the bleeding. Often, we find that a default configuration setting (like a generic timeout in a client library) undoes all our hard work.

For e-commerce leaders, this means running stress tests that exceed your projected Black Friday traffic by at least 50%. You must identify the “breaking point” of your system. If you don’t know exactly how many orders per second breaks your database, you aren’t ready for the event.

Resilience is a mindset, not a tool

You cannot buy “resilience” from AWS or Azure. You cannot solve these problems just by switching to Kubernetes or adding more nodes.

The “Super Bowl Standard” requires a fundamental shift in how you view failures. We assume components will fail. We assume the network will be slow. We assume users will behave like a DDoS attack.

Whether you are building a streaming platform, a banking ledger or a retail storefront, the goal is not to build a system that never breaks. The goal is to build a system that breaks partially and gracefully so that the core business value survives.

If you wait until the traffic hits to test these assumptions, it is already too late.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?