One Slow Service Took Salesforce Down for Ten Hours. The Same Pattern Is in Your Code.

The most expensive outages rarely have exciting causes. They have boring causes that were allowed to spread.

On 16 September 2026, Salesforce went down across the US, Japan, India, the UK, France and Germany for about ten and a half hours. Their public statement is one sentence: requests were getting stuck waiting for a reply from an old internal login service, which used up the available server resources and limited how many new requests the platform could handle.

There is no detailed root cause report yet, so everything below is about the pattern that sentence describes, not about Salesforce's specific design. I want to be clear about that line. But the pattern is worth studying, because it is the most common way a healthy system kills itself.

The mechanism: one slow service, no workers left

Start with the simplest model. A service has a fixed number of workers — threads, connections, whatever your runtime calls them. Say 200. Each request uses one worker until it finishes.

Normally a request takes 20 milliseconds. 200 workers at 20 ms each gives you roughly 10,000 requests per second, and most workers sit idle.

Now one service you depend on — login, say — starts taking 30 seconds instead of 20 milliseconds. It is not failing. It is not returning errors. It is just slow. Slow is the dangerous state, because an error frees the worker immediately and a slow success does not.

Your capacity for requests that touch login drops from 10,000 per second to about 6. Requests keep arriving at the old rate. Workers pile up waiting. Within seconds — not minutes, seconds — all 200 workers are stuck on login.

Here is the part people miss. At that moment your service stops serving everything. Not just login. The health check, the settings endpoint, the read-only query that never touches login — all of it waits behind the same empty pool. One slow service has turned into total downtime for every feature you offer.

One slow service turns into total downtime for every feature you offer.

That is the shape of "used up the available server resources and limited how many new requests the platform could handle."

Your load balancer makes it worse

The next thing that happens is that your infrastructure tries to help and speeds up the collapse.

Health checks start timing out, because the health endpoint is waiting behind the same empty pool. The load balancer marks the server unhealthy and removes it. Its traffic moves to the remaining servers, which are already in the same state, which pushes them over faster. Servers drop out one after another.

Autoscaling joins in. New servers start, immediately take traffic, immediately fill up waiting on the same slow login service, and go unhealthy too. You are now paying for more servers that produce more failures.

Clients retry. A retry on a request that was going to time out anyway just multiplies the load — three retries turn one stuck worker-second into four. Whatever room you had left is gone.

None of these parts are broken. Health checks, traffic redistribution, autoscaling and retries are all correct behaviour for the failures they were designed for. They are simply wrong for this one, and together they make it worse.

Timeouts are necessary but not enough

The first instinct is "add a timeout", and yes, you must. Waiting forever on a network call is a bug either way. But do the arithmetic before assuming it saves you.

Put a 1-second timeout on the login call. Capacity for login requests goes from 6 per second to about 200. Thirty times better. If you were serving 10,000 per second, you are still 50 times short of demand, your pool is still full, and you are still down — just with a cleaner error message in the logs.

Timeouts limit the damage per request. They do not limit total resource use when every request is hitting the slow path. To survive, you need something that stops the slow path from taking the whole pool in the first place.

Bulkheads: the control that fits

The bulkhead pattern is named after the sealed compartments in a ship. If the hull is damaged, one compartment floods and the ship stays up, because the compartments are sealed off from each other.

Applied here: give each service you depend on its own limited number of slots, separate from the general pool. Login gets 40. When all 40 are in use, the 41st login request fails immediately. It does not wait for a general worker.

Run the numbers again. Login gets slow. 40 workers get stuck. The other 160 are fine, because they were never allowed to serve login calls. Everything that does not need login keeps working at full speed. Requests that do need it fail fast with a clear error.

You are degraded. You are not down. That difference is the whole point.

The same worker pool, now split by a bulkhead. On the left, a compartment of 40 slots reserved for login is full and marked in danger colour, with a note that the forty-first request fails immediately instead of waiting. A vertical divider labelled 'bulkhead' separates it from the right-hand compartment, where the remaining 160 workers are drawn in accent colour and marked untouched, still serving health checks, reads and everything not behind login at full speed. The caption reads: you are degraded, you are not down. — Give each dependency its own limited number of slots. A flooded compartment does not sink the ship.

The honest costs, because this is not free:

You have to choose the numbers. 40 is a decision about how much of your service one dependency deserves when things go wrong. Too small and you throttle a healthy service. Too large and the bulkhead does nothing.
You get a new kind of error. "Bulkhead full" is a failure your callers have to handle, and it will happen during normal traffic spikes, not only during incidents.
It is hard to add later. In an existing codebase, adding per-dependency limits means finding every call site, and there are always more than the architecture diagram suggests.

Most teams skip it, then find out during their first cascading failure that they were relying on every dependency staying fast forever.

One more thing: a circuit breaker is not a replacement. A breaker trips when the error rate goes up, and a service that is slow while still returning successful responses may never trip one. I wrote about that separately in when circuit breakers lie. Bulkheads and breakers solve neighbouring problems and you want both.

Load shedding: choosing what to drop

Bulkheads protect you from one dependency. Load shedding protects you from more traffic than you can serve.

The rule is to reject early instead of queueing. A request you will not finish in time should be refused at the edge, before it takes a worker. Every millisecond spent on a doomed request is stolen from one you could have completed.

Two things worth building:

Check how long a request has been waiting. Record the arrival time on each request. Before running it, if it has already waited longer than the client's timeout, drop it without doing the work. During an overload, a large share of your queue is requests whose callers gave up long ago. Serving them is pure waste.

Give traffic priority levels. Not all requests are equal. Health checks and internal traffic should have reserved space that user traffic cannot fill. That alone breaks the load balancer loop described above. A logged-in user's save is probably worth more than an anonymous page view. When you have to drop something, you want to decide which, rather than letting arrival order decide for you.

Why ten and a half hours

The failure takes seconds. Recovery took most of a working day across six countries. That gap is the real lesson, and most incident reviews stop before reaching it.

Recovering from total saturation is much harder than recovering from a crash:

Everything arrives at once. While you were down, clients queued work, retried, and built up changes to sync. The moment you come back, all of it lands together — often several times your normal peak. Bring capacity back gradually and you can absorb it. Turn everything on at once and you saturate again, giving yourself a second outage with the same shape.

Caches are empty. Ten hours without traffic means cached data has expired. The restored system asks far more of its databases per request than it did before the incident, exactly when demand is highest. Your real capacity on restart is a fraction of what the old dashboards suggest.

Login is everyone's dependency. Almost every other service needs it, which makes the restart order hard. Login has to come back first, healthy, under the full retry load of everything waiting for it — and it was the part that was already struggling.

Six countries is not one recovery. It is several, competing for the same shared systems and the same engineers.

On the left, a bar chart of demand: a low, steady level before the outage, then a spike several times higher at the moment of restart as queued retries all arrive together, tailing off afterwards. On the right, the three forces that make recovery hard, listed in danger colour: everything arrives at once, caches are empty so every request costs more, and login must come back first under all of it. Below, two options are contrasted — restoring all capacity at once, which saturates again and causes a second outage, and a gradual restore in accent colour, which absorbs the surge and stays up. — The failure takes seconds. The recovery takes hours, because demand returns all at once against cold caches.

If you take one operational lesson from this: practise your recovery, not just your failover. Most teams have tested that one server can die. Far fewer have tested bringing a whole region back from cold while clients retry, and that is the scenario that consumes your ten hours.

September was a bad month generally

This did not happen alone.

On 1 September, a Google Cloud failure in a single zone affected 15 products for 4 hours and 8 minutes. Google's early analysis pointed at routine network maintenance causing unexpected problems in one cluster. On 17 and 18 September, Cloudflare had three separate incidents in about 24 hours.

Two lessons follow, and they point in a different direction from the usual advice:

A single zone is not a unit of reliability. The Google outage was limited to one zone and lasted over four hours. If your system runs in one zone, that zone's availability is your ceiling, and no amount of clever application code raises it. Running across multiple zones is not luxury. It is the baseline the guarantees are written against.

Routine maintenance is a leading cause of outages. Not exotic hardware failure — a planned network change. The changes you think are safe enough to skip a rollback plan are the ones that will page you, because the risky ones already get careful review.

What to check in your own system

Five questions. If you cannot answer any of them for the service you own, that is the finding.

What is the slowest thing on your main request path, and what is its timeout? If the answer is "no timeout" or "whatever the library default is", you have an unlimited wait waiting to happen.
How much of your worker pool can one dependency take? If the answer is "all of it", you have no bulkhead, and one slow service means total downtime.
Does your health check share a pool with normal requests? If yes, slowness becomes server removal becomes cascading failure. A health check should still work when the service is saturated. That is the entire point of it.
What happens to a queued request whose caller already timed out? If you still run it, a large share of your capacity during an incident goes to work nobody will read.
Have you ever restored this system from cold while clients retry? Not failed over — restored. Almost nobody runs that drill.

None of this is advanced. Bulkheads, timeouts, load shedding and reserved health-check capacity are decades old, well documented, and available as libraries in every major language. They get skipped because they take real effort to add and their benefit is invisible on a normal Tuesday.

The takeaway

A slow dependency is more dangerous than a dead one, because an error returns the worker and a slow success does not. Once every worker is stuck waiting on one call, your service stops serving everything — including the health check, which is what turns a local problem into load balancer flapping, autoscaling thrash and retry storms. Timeouts limit damage per request but not total resource use. Only a per-dependency slot limit keeps a bad service inside its own compartment.

The ten hours are the part to plan for. Recovering from saturation is harder than recovering from a crash: queued retries all arrive together, caches are empty so every request costs more, and login has to come back first under the weight of everything waiting on it. Most teams have tested that a server can die. Almost nobody has tested bringing a region back from cold during a retry storm — and that, not the original failure, is the drill that matches the outage you will actually have.