CircleCI's Three-Day Retention Window Replayed Tuesday's Outage on Friday

On October 12, CircleCI published a post-incident report for the outage that ran from 11:15 to 23:47 UTC on Friday, October 9. For almost two hours, pipelines were created but not processed. The UI and API fell up to about two hours behind, and some queued workflows and jobs waited about two hours before they started. The report covers a problem with the Postgres write-ahead log, a retry storm, and an emergency sharding job. One sentence in the middle of it keeps standing out to me: the October 9 outage was set in motion by cleanup from the October 6 outage three days earlier.

The setup

CircleCI runs several core services on Temporal, including the build service, workflow orchestration, notifications and credit charging. Temporal stores the state of its workflows and activities, which it calls history, in a database. At the time, CircleCI ran one clustered Temporal instance backed by a single Multi-AZ Postgres RDS instance. The write-ahead log was stored on an RDS Dedicated Log Volume. According to the report, that volume is limited to 1TB of storage and 3000 IOPS, and AWS hardcodes those limits so they cannot be increased.

Provided archiving keeps up with the write rate, this architecture is fine. If writes outpace archiving, the volume fills, Postgres can no longer write its WAL, and the database goes read-only.

CircleCI also configured Temporal to purge completed workflows and activities from history 3 days after they finish. That retention value is the key to the whole story.

How Tuesday came back on Friday

The October 6 incident had the same cause. One contributing factor was a high rate of Temporal history deletions running at the same time as incoming work. To resolve it, CircleCI changed cleanup settings on Temporal's largest tables and moved the transaction log back to its dedicated volume. In the report's words, those changes "did not consider the additional log writes from scheduled deletions."

Because of the three-day retention period, history from October 6 was scheduled for deletion on October 9. At 08:00 UTC on October 9, Temporal started processing those deletions. Free space on the log volume dropped from 0.8TB at 08:00 to 0.1TB at 10:10. Monitoring alerted at 10:28 and an incident was declared at 10:31. The volume ran out of space at 11:15, which put Postgres into read-only mode, and all Temporal processing stopped.

The team's description of one fix it is making sums it up nicely: "a busy day returns as database load later in the week."

The second wave

At 11:42 the team restarted Postgres to move the WAL onto the data volume, the same move that had mitigated October 6, which would take roughly 60 minutes. While that ran, CircleCI flagged off credit charging updates at 12:29, which made up 12% of Temporal activities. The database was back at 12:59 and the backlog had cleared by 14:40.

Delays returned at 15:15. The tables Temporal uses to track queued and scheduled work normally hold tens of thousands of deleted rows awaiting cleanup. By late afternoon they held tens of millions, and every Temporal operation had to read past them. The team tried moving notifications (18% of Temporal activities) to a separate Temporal instance, finished by 17:07, and shipped a change at 17:16 that shrank its pipeline state representation by 5%. Manual vacuum jobs competed with queries for CPU and were cancelled at 17:45. None of these fully mitigated the problem.

At 20:48 it became clear that Postgres was thrashing on CPU and was no longer bottlenecked on log writes. Upstream services were submitting new work and also retrying work that had failed earlier. The team paused all incoming and retry work, throttled it back up from 22:07, and by 23:47 all queued work had started.

Deletes are writes, and they run on a schedule

The takeaway is simple to explain and simple to overlook: in Postgres, a delete is a write. It goes through the WAL, it leaves dead rows for vacuum to clean up, and a TTL-based cleanup job books those writes in advance. A retention policy of N days replays the traffic profile of day zero on day N, regardless of whatever else the database is doing that day.

GitHub ran into a version of this on October 27. While an installation records table was being migrated between MySQL clusters, a scheduled cleanup job deleted approximately 59 million records from it. The migration's gh-ost replication fell up to an hour behind. Once reads were cut over, authenticated git operations using GitHub App server-to-server tokens mistakenly returned 404 for 1.24% of requests on average and 2.49% at peak. GitHub says its process changes include disabling overlapping cleanup jobs before these migrations.

In both cases a background delete job ran as scheduled, and the surrounding work did not account for its load.

What I would check in my own systems

If you have a queue, workflow engine or event store with TTL cleanup, a few questions are worth answering before another incident answers them for you.

When do your deletes actually land? A retention window converts any spike, including one from an incident you just fixed, into a deletion spike later. After a busy day, look at the calendar and work out what that day's cleanup will cost.

Does your postmortem cover the cleanup? CircleCI's October 6 changes addressed the immediate log pressure but not the deferred deletions. For any write-volume incident, the remediation should ask what the system will delete because of that day, and when.

Are you watching the rate, or only the level? CircleCI says it is adding alerts on how far log archiving falls behind and how fast the database writes logs, so it can act "hours earlier" than it did on October 9. Monitoring alerted at 10:28 on a volume that had been draining since 08:00.

Can you throttle cleanup? CircleCI says it is changing when and how quickly Temporal's deletions run. Rate limiting or spreading out deletes costs some storage and buys real headroom.

CircleCI's structural fix splits work across multiple Temporal instances, each with its own database and worker cluster. That work was already an October 6 action item, scheduled to start October 12 and take two weeks. On October 9 the team built a simpler version in a few hours, completed at 01:45 UTC on October 10. Sharding removes the single log volume as a shared choke point. It does not remove the three-day echo, which is why the deletion schedule is on the list too.

The full report on CircleCI's Discuss forum is worth reading end to end. It is frank about which mitigations failed and why, a part many postmortems leave out.