GitHub's Connection Limiter Was Live. It Was Only Logging.
GitHub's 19 October Actions outage came from a retry storm. The connection limiter that would have capped it was already in production, in log-only shadow mode.
On Monday, 19 October, GitHub Actions ran degraded for 88 minutes. Between 15:24 and 16:52 UTC, 5.13% of jobs on GitHub-hosted runners started late, 0.02% of workflow runs failed, and in the first half hour 8.47% of requests to the Actions jobs REST API returned errors. Pages deployments and webhooks from Actions runs backed up behind it. GitHub's incident report on its status page is short, and one sentence in it is more interesting than the outage:
"We were already in the process of rolling out a limit on how quickly this service can open new connections, but it was still in shadow mode (logging only, not enforcing) at the time of the incident."
The control that would have capped the damage was already deployed. It logged the connections it would have blocked and let every one of them through.
What broke
According to GitHub, an internal Actions service that stores logs and run status lost connections to some of its cache servers in one cloud region, and every server running that service retried at once. The burst of new connections overloaded a network link between GitHub's data center and that region. Other services use the same link, so they degraded with Actions. Live updates on GitHub.com and code search were affected for part of the window.
This is a retry storm, and the fix is old knowledge. Marc Brooker explained why synchronized retries pile up and why backoff needs random jitter on the AWS Architecture Blog more than a decade ago. The Google SRE book's chapter on cascading failures covers the same pattern. GitHub's own follow-up list includes "adding backoff to retries," which suggests the retries here had little or none. The report doesn't say which.
Shadow mode measures quiet days
Running a new limiter in shadow mode is a sensible way to ship it. You put it in front of production traffic, log every request it would have blocked, and turn on enforcement once the logs show it won't throttle legitimate work. A new rate limit that causes an outage is worse than no limit.
The catch is what those logs can tell you. A connection-rate limiter matters for a few minutes a year, when something upstream breaks and every client reconnects at once. The rest of the time it should do nothing. A shadow period made of normal days mostly proves that the limiter stays out of the way. It tells you little about a surge, because no surge has happened yet. The rollout is a bet that enforcement goes live before one does.
So the shadow period is a stretch of known exposure. The team has already decided the system needs the protection, which is why someone built it, and production still runs without it. On 19 October the surge arrived first. By the time GitHub posted its report at 16:56 UTC, it had "fully enforced" the limit and capped the service's concurrent connection attempts.
The timeline shows a second delay. Impact started at 15:24 UTC. The first public status post went up at 15:49, 25 minutes later, and the cause appeared at 16:56, four minutes after recovery. The last item on GitHub's follow-up list is better alerting "so we can find the source of connection surges faster," so tracing the surge to its source apparently took a while too. A limiter in shadow mode logs exactly which service exceeded its intended connection rate and when. The report doesn't say whether those logs were part of the investigation.
What to take from it
If you run a rate limiter, circuit breaker, or load-shedding rule in log-only mode, this report suggests two changes.
Give the shadow period an end date and a real test. Replaying a recorded reconnect storm against the limiter, or forcing one in staging, tells you more about a connection limit than another week of ordinary traffic. Once that test passes, more time in shadow mode adds risk and little information.
Alert on the shadow logs while you wait. A limiter reporting that it would have blocked thousands of connections in one minute is describing an incident as it happens. If nothing pages on that number, you are ignoring the earliest signal the system has.
GitHub's follow-ups go beyond this one limiter. It is extending connection limits to other Actions services, adding backoff to retries, and limiting how much shared network capacity any single service can use. That last change matters most to everyone downstream, since code search and live updates went down because of retries from a service they share a link with.
Source: GitHub status page, "Degraded availability for GitHub Actions, GitHub Pages and Webhooks," resolved update posted 19 October 2026, 16:56 UTC. Feature image generated for Endoblog; the timeline chart is drawn from the timestamps and figures in GitHub's report.
