Railway is a deployment platform. Its customers store their secrets there as environment variables: database passwords, API keys, registry credentials, the keys to everything their software talks to. In October somebody got in.

Railway's preliminary report gives a window. The attack ran "from October 20th 13:58 UTC to October 22nd 03:32 UTC". The report's introduction says Railway became aware at October 20th 19:53 UTC.

Work the arithmetic. Five hours and fifty-five minutes before anyone noticed. Then thirty-one hours and thirty-nine minutes of attacker presence after Railway knew. Thirty-seven and a half hours in total.

The second number is the uncomfortable one, and it is not necessarily a failure. Evicting an intruder from a build system while preserving forensics and not destroying customer deployments is slow work, and a company that cuts everything off in ten minutes often loses the evidence it needs. But thirty-one hours is a long time to be sharing your build infrastructure with somebody.

What they have said about the decryption

More than the coverage suggested, and still not enough.

Railway identified "potential privilege escalation in our build infrastructure by the attackers", and says it is "not yet able to share the specific details and scale of the exploits". On the secrets themselves: a database query accessed encrypted variables, and "a subset were decrypted utilizing a combination of the compromised credentials". Separately the attacker reached CI, where they "were able to decrypt artifacts that held additional API access".

So the shape is clear enough. The encryption was not broken. The attacker obtained the things that are allowed to decrypt, which is how this almost always goes, and is why "encrypted at rest" has never been the reassurance it sounds like. Data at rest is encrypted. The production system that needs to read it is, by construction, holding something that turns it back.

The sentence that is the actual story

This one, from the report:

"Because the full keys were envelope encrypted, this query alone cannot identify specifically which keys" were affected.

Envelope encryption is the right design. You encrypt each secret with its own data key, encrypt the data keys with a master key, and keep the master key somewhere harder to reach. It limits what one compromise yields and it is what a careful platform does.

And here it means Railway cannot tell its customers which of their secrets were exposed. The query that shows what the attacker touched shows envelopes, not contents. The property that made the storage safer is the property that makes the blast radius unmappable.

So the instruction is the blunt one: everybody rotates everything. At most 1.85 percent of all workspaces were potentially affected, which Railway puts at about 100,000 workspaces, and all of them get the same advice as if they were certainly affected, because nobody can narrow it.

There is a scale figure hiding in that too, which Railway has not published directly: if 1.85 percent is about 100,000, the platform has on the order of 5.4 million workspaces.

The thing worth taking from it

Nobody designs for this. Key hierarchies are evaluated on how well they contain a compromise, and this one contained it. They are not evaluated on whether, after a compromise, the operator can produce a list of exactly what was taken, and that turns out to be the thing every customer needs on the day.

"We cannot tell you which of your secrets leaked, so assume all of them" is a defensible position and an expensive one. Rotating every environment variable across a production estate is hours of work, breaks things, and most teams will do it incompletely. The ones who do it incompletely will not know which part they skipped mattered.

If you build platforms that hold other people's secrets, the question this raises is whether your design can answer "which ones" after an incident, and whether answering it is worth the thing you would give up.

What is not established

  • How the attacker got into the build infrastructure in the first place. Railway says it cannot yet share the details.
  • Where the master keys were held and what reached them.
  • Whether the 100,000 figure is Railway's or derived, and therefore whether the 5.4 million inference is sound.
  • Whether a final report has been published. Railway promised "a follow up to this document as the investigation concludes" and mentioned a final forensic investigation report; check the blog before relying on this piece being current.
  • Whether any customer has reported a downstream compromise traceable to these secrets.
  • The two timestamps in the report, 13:58 and 19:53 on 20 October, are presented differently in the timeline and the introduction, so the detection gap may need clarifying.