Releasing a voice stack without dropping live calls.

Every merge to main releases itself once CI passes. On a busy day that’s five or more releases. On 11 October Sneh sent the message that started this work: “every time we update (5 times a day) we get the nginx screen on the console and the API doesn’t respond. This should be hot swap always. Same velocity, no downtime, no new server.”

This is what a release used to cost us.

We measured it from the gateway that day. The error page he’d seen was already gone, fixed by a change the day before. Until then a voice release threw 19 to 55 failed responses on the public API and left a 23 to 25 second gap on the console. After the fix it was zero, across three releases. The rest was still true:

• Every voice release stopped the three services that admit calls for about 22 seconds, plus 4 more for each round of gateway configuration. Inbound calls went unanswered and campaigns stopped dialling while they were down.
• The whole fleet had to drain before every release. That could take up to 15 minutes, and if calls never drained, the release failed.
• All ten workers on the gateway stopped at the same moment.
• 129 old release directories sat on disk, never cleaned up.

A voice release is harder than a web one.

A web request lasts milliseconds. A phone call holds a websocket to one worker process for as long as it lasts, and a call bridged through the telephony adapter can run for half an hour. Outbound calls add their own delay. We pick a worker when we dial, and the phone can ring for up to 45 seconds before anyone answers, so a worker can receive a call it was promised most of a minute ago. Restart any of these processes and you hang up on whoever is using it.

So we started from two rules. Nothing that admits calls stops for a release. Nothing that carries a call stops until the call ends.

Everything that answers requests now runs in slots.

The console, the public API and the telephony adapter each run as two slots behind nginx, three for the adapter. nginx still listens on the old loopback ports and forwards to whichever slot is listed first, so every client of those ports, including the workers, the smoke test and the health timer, kept working without a change.

A slot roll goes like this:

1. Start the idle slot from an immutable release directory with its own Python environment. Nothing serves from the git checkout any more.
2. Check its health and version on its own port.
3. Rewrite the upstream file with the new slot first, run nginx -t, reload. The reload is graceful, so old nginx workers finish the websockets they hold.
4. Check the version again through the front.
5. Keep the old slot running until the whole release has passed. Until then, rollback is writing the file back.
6. Retire the old slot with a stop that waits for whatever it still carries: up to 10 minutes for browser test calls on the console, 30 minutes for phone calls on the adapter, 10 seconds for API requests.

The adapter has a third slot because a retiring adapter can still be bridging a long call when the next release arrives.

The API took a review to get right. During a flip, two API processes run at once. Its background jobs already took Postgres advisory locks, so the overlap was safe for them. A first draft of the plan said the API ran no migrations at start. The review found that it does: it applies the campaign and billing schemas at boot, under advisory locks, so the new slot migrates while the old one is still serving. That only works with additive migrations. They were already the repository’s rule. Now a release depends on it.

Media workers restart a few at a time.

Media workers aren’t behind nginx. A pool in Redis decides which worker gets a call. Each worker heartbeats into it every 15 seconds, and one that stops heartbeating ages out after 45. A worker roll has three steps:

• The controller tells the worker to drain with a grace period. It stops taking new calls at once, but keeps answering calls already dialled to it.
• The controller removes it from the pools, then again 17 seconds later, to beat a heartbeat that was already in flight.
• The worker keeps answering dialled calls for 60 seconds after that second removal, which covers the 45-second ring with some margin. That makes the grace 77 seconds. Then it exits when idle, and systemd starts it again from the new release directory.

A canary worker rolls first and carries the release’s one smoke test: a browser-audio call with a greeting, a spoken reply and a clean ending. It never rings a phone. The rest follow in waves of three, and the second worker host rolls in parallel. A worker still on a call when the drain timeout passes doesn’t fail the release. It’s reported, and it rolls itself when the call ends, because its service already points at the new release.

The services that admit calls never stop. The workflow worker restarts in place, which is safe because the workflow engine buffers tasks and its activities retry. The SIP edge restarts only when its binary changed, and only after its own calls have ended.

A release restarts only what changed.

The controller compares the live commit with the release. A change only under docs, tests, CI or scripts rolls the console slot, so the version endpoint shows the new commit, and touches nothing else. Anything in the worker’s source rolls everything. A change to the adapter’s own files rolls the adapter.

Every release grades itself in one line.

Every release, healthy or rolled back, ends with one line:

Release health: 0 x 5xx on public hosts, admission units never stopped

The 5xx count comes from nginx’s access log over the release window, including the rotated file if logrotate ran mid-release. Loopback requests are counted apart, because a public request proxied through a front is logged twice. Admission stops come from systemd’s job records, where a restart in place is told apart from a stop by its job ID. The report never fails or rolls back a release, since a 5xx can be the customer’s own.

Releases got slower, and a few things went wrong.

Releases got slower. A voice release used to take between 1 minute 50 and 2 minutes 51, plus the 22-second stop and a drain of no fixed length. A full roll now plans for 4 to 5 minutes: staging about 30 seconds, slot rolls 20, canary 55, smoke test 30, three waves about 160. A docs-only commit takes about a minute. Nothing stops while either runs, and a few extra minutes are cheap next to an unanswered call.

The first move onto slots couldn’t be hot. The old single services had been started without a graceful stop and would have died instantly, so the bootstrap release waited for a moment with no calls on the fleet. Moving each old port to nginx cost loopback clients a refusal window of under a second, once.

Graceful shutdown found a sharp edge. The console’s signal handlers run on its event loop, so a console with a blocked loop never sees SIGTERM at all. The health script that restarts a stuck console now sends two SIGTERMs, the second skipping the graceful wait, and falls back to SIGKILL after 15 seconds. A plain restart would have sat out the slot’s 11-minute stop timeout.

An independent review of the controller found twelve problems before it shipped. Two of them: a service rolled twice in one release has to keep its original slot as the revert target, and the node agent must never restart a worker because a health probe failed. Now it tries three times, asks systemd, and reports an active worker that doesn’t answer without touching it.

Two pieces still restart in place: the workflow worker and the SIP edge. And a release never rolls back the database. Once a release is recorded, going back means a revert PR that ships as an ordinary release.