Our cloud vendor went dark for six hours. Our customers never knew.

Our fleet controller talks to several Indian GPU clouds. It knows what every card on every vendor costs per hour, right now, in rupees at the real exchange rate, and it refuses to start a machine it cannot prove it can pay for. It brings a card from bare metal to a live agent by itself: provision, driver, speech recognition up, model up, voice warm, quality gate, wired to the phone network.

To do any of that it has to ask the cloud what exists. One afternoon, one of them stopped answering.

Six hours of nothing

Not errors — silence, then errors, then silence. The provider’s API and its console both went dark for about six hours. From inside our controller, every question about that vendor’s inventory came back unreadable.

The node we cared about was serving production traffic the whole time. It never stopped. The provider’s data plane was fine; it was the part that tells you about the data plane that was gone.

That gap is the interesting part of this story, because the two states look identical from the outside. “The card is gone” and “I cannot see the card” are the same API response.

The line that saved the day

cloud API unreadable — leaving node state untouched

That is what the log said, over and over, for six hours. It is the whole incident report.

A reconciling controller normally does something obvious and wrong here. It holds a desired state, it reads the actual state, and it closes the gap. Read an empty inventory and the arithmetic is brutal: nothing exists, therefore nothing is running, therefore bring everything up — or worse, the version where a node it cannot see is a node it does not own, and the tidy-up begins.

Ours was built the other way round. An unreadable cloud is not evidence about the nodes. It is evidence about the API. The controller is allowed to act on what it knows; it is not allowed to act on what it merely cannot see. So it stopped reconciling that vendor, kept the running node exactly as it was, said so in the log, and waited.

Customers noticed nothing, because nothing happened to them. Calls landed, agents answered, the ledger debited. The only sign anything was wrong was a log line repeating in a terminal nobody had to be watching.

Why we hold multiple vendors

Any single Indian GPU cloud will have a bad day. That is not a criticism; it is what a young market looks like, and we would rather design for it than complain about it. So the fleet software makes vendors interchangeable — one board, one set of primitives, cost per hour in rupees on every row — and no customer-facing promise depends on any one of them being reachable.

This outage is the reason that abstraction was worth building before it looked necessary. On the day it mattered, the useful property was not that we could move workloads. We had no need to move anything. It was that we already treated a provider as an untrusted source of truth about our own infrastructure.

The design rule underneath

Three sentences, and they cost us nothing on the days nothing goes wrong:

• Absence of information is not information. An empty list from a system that is not answering means nothing at all.
• Destructive actions require positive evidence. We tear down a node when something tells us to tear it down, never when something fails to tell us anything.
• Doing nothing is a valid response, and it has to be written down as one. If “wait” is not an explicit branch in the code, the code will eventually invent a worse one on its own.

The last is the one that took discipline. A controller that pauses is harder to reason about than a controller that always converges, and a paused controller is uncomfortable to watch. Six hours of a log line saying it is deliberately not acting looks, at a glance, like a bug.

What we would say to anyone building the same thing

The outage was not our failure and it was never going to be. The failure available to us that day was our own automation reacting confidently to a question it had not actually received an answer to.

Reliability at this layer is mostly a matter of deciding, in advance and in writing, what your software is not permitted to conclude.