ElevenLabs Kept Counting Calls That Had Already Hung Up

ElevenLabs' Nov 19 incident was the small one. A cache hit its connection limit, ended calls never left the active-call counts, and about 10% of new conversations were refused. A timeout became a wrong number that stayed wrong.

Share
ElevenLabs Kept Counting Calls That Had Already Hung Up

ElevenLabs released its write-up for INC-465 this week, and most readers will care about the big day. On November 18, between 14:49 and 20:55 UTC, a burst of concurrent requests triggered repeated full cache rebuilds on the voice caching cluster. Backend services had no upper bound on cache connections, so the cluster hit its maximum client limit, failed health checks, and entered a repeating restart cycle with an empty cache. At peak, 12% to 50% of requests failed across Voices, Creative, Studio, Music, Reader, and login/workspace routes. Unbounded connection pools are a known story with known fixes.

Day two is the one to study

The second day is odder and more useful. On November 19, a server hosting one of the caches reached its limit for new network connections, and connections to that cache timed out. The write-up calls it "much smaller in scale." But look at what actually failed. ElevenAgents stored its active-call counts in that cache. In the write-up's words: "Calls that ended during this window were not consistently removed from ElevenAgents’ active-call counts which was stored in the impacted cache, so those counts reached customers’ concurrency limits." About 10% of new conversations overall were refused between 14:10 and 15:20 UTC.

Put plainly, customers were refused because the system was still counting calls that had already hung up.

A timeout turned into a wrong number

When a cache request times out, you normally get a slow request or a failed one. You retry or fall back, and once the connection returns, the system is healthy again. Counters don't behave like that. If the increment for a new call goes through and the decrement for a finished call is lost, the count stays high after the network recovers. The cache is reachable again, the value in it is wrong, and nothing in the request path will fix it.

Most health signals can't see this. The cache responds. Latency looks normal. The only symptom is that a customer who should have room under their concurrency limit gets refused, and from the outside that looks the same as a customer who really has reached their limit.

The published timeline fits this reading. The write-up says ElevenLabs cleared the stale counts at 15:17, and the refusals window closes at 15:20 UTC. A follow-up on the incident page says the "initiation errors stopped at 15:20 UTC" and that "These failures had no impact on other products." Recovery lined up with resetting the data. The team also "raised this connection limit on all servers that host these caches."

Timeline of ElevenLabs INC-465 in UTC: Nov 18 impact 14:49 to 20:55 with peak 15:45 to 17:15; Nov 19 conversation refusals 14:10 to 15:20, stale call counts cleared at 15:17.
INC-465 timeline in UTC, drawn from the times stated in the ElevenLabs incident write-up (status.elevenlabs.io).

Admission counters need their own design

A concurrency limit is admission control. It decides whether a paying customer's call gets connected at all. Keeping the counter behind that decision in a cache puts it in a store that teams usually treat as disposable. When a voice cache empties, things get slow for a while. When an active-call count drifts upward, it refuses new calls until someone corrects it.

The preventative actions in the write-up center on capacity and limits: larger caching clusters, limits inside services, a review of isolation, rate limiting, and health checks, and better provisioning for caching and datastore services. That all makes sense. None of the listed items describes changing how the active-call count itself is kept, and that is the part I would look at first in a system of my own.

If you run something similar, such as per-tenant caps on calls, sessions, jobs, or websocket connections, here is what I would check.

  • Make each slot a lease. Store one entry per active call with a TTL that the call refreshes while it is alive, and count live entries. A lost "end" event then expires on its own instead of sitting in the count forever.
  • Reconcile against a source of truth. Something knows which calls are actually connected. Run a periodic job that rebuilds the count from that and alerts when the cached number and the real number disagree.
  • Treat a failed decrement as a real failure. Queue and retry it, and log it loudly. A silently dropped decrement is a slow outage for one customer.
  • Decide on purpose what happens when the counter store is unreachable. Failing closed protects your capacity. Failing open protects your customers. Either can be right, but it should be a written decision rather than whatever the client library does on a timeout.
  • Watch limit refusals across tenants. A spike in concurrency-limit refusals across many customers at once is rarely real demand.

The question to ask of every cached value

November 18 was a capacity failure. November 19 was much smaller, but it left behind bad state that outlived the connection problem, and the fix was clearing that state. For every value in your caches, ask: if writes to this are lost for ten minutes, will it fix itself? For data that can be rebuilt, usually yes. For a counter that gates admission, often no, and that is the value that will quietly refuse customers who did nothing wrong.

ElevenLabs deserves credit for publishing both days, including the small one. The full write-up and the incident updates are short and worth reading together.

0 subscribers
0 average monthly readers