Railway confirms customer secrets were decrypted: what to rotate, in what order
Railway's October 21 incident pulled customer environment variables in encrypted form and decrypted a subset; here is what was taken, which credentials Railway already reset, and the order to rotate the rest.
Decision: Railway disclosed on October 21, 2026 that an unauthorized party had reached a subset of its internal systems, read customer environment variables in their encrypted form, and decrypted some of them. 1
The preliminary report published the next day caps the reach at 1.85% of workspaces, about 100,000 of them. 2
That figure describes how far the attacker got. Your checklist still covers every credential you stored on Railway, because Railway cannot say which variables inside a workspace were decrypted. 2
Railway's own instruction is to rotate everything a customer stored there before October 21. 2
So this week's work is a credential rotation. The order you do it in decides how long the attacker keeps a usable key.
The quick read
| Question | Answer | What to do, and when |
|---|---|---|
| Do I need to act? | Railway asks everyone who signed up before October 21 to rotate the keys they stored there. 2 | Start today, with AI and cloud keys. |
| Was my workspace one of the 1.85%? | Railway knows which workspaces it rates high risk. Which keys were read inside one, it cannot say. 2 | Treat the answer as unknowable and rotate anyway. |
| What did Railway already reset? | Railway sessions and tokens, your GitHub connection, the ghcr.io tokens it held, and every bucket's S3 keys. 3 | Reconnect GitHub, then redeploy what reads those credentials. |
| What is left to me? | Everything else in your variables: AI provider keys, cloud keys, Stripe keys, database passwords, personal access tokens. 2 | Follow the rotation order below. |
| Why is my deploy failing? | Commits pushed during the pause were dropped, and CI tokens were revoked. 4 | Reconnect GitHub, then reissue RAILWAY_TOKEN or RAILWAY_API_TOKEN. |
| Should I move off Railway? | The final forensic report is still to come, and build isolation has moved to single-tenant machines. 2 | Decide when that report lands, against the three tests at the end. |
What Railway says happened
Monitoring paged Railway's on-call engineers at 13:58 UTC on October 20 for elevated latency, and the pages turned out to be an attempted privilege escalation by an outside party. 2
The engineering team stopped build pipelines and began deploying mitigations. By 17:03 UTC Railway had declared an incident on its status page. 25
Two hours later, new evidence showed the same party could escalate further, in parallel, across several parts of the system at once. 2
| Time (UTC) | What Railway recorded |
|---|---|
| Oct 20, 13:58 | Monitoring pages on-call for elevated latency; a targeted campaign is underway. 2 |
| Oct 20, 16:51–18:05 | Privilege escalation found in the build infrastructure; builds halted and mitigations deployed. 2 |
| Oct 20, 19:53 | Evidence that the attacker can escalate at higher scale, in parallel. 2 |
| Oct 21, 04:24 | Forensics show secrets pulled in encrypted form, not yet decrypted by the attacker. 2 |
| Oct 21, 05:23–07:25 | Access to the identified systems revoked; the attacker reaches CI credentials and decrypts artifacts holding more API access. 2 |
| Oct 21, 07:25–09:18 | Database evidence of a query against encrypted environment variables; a subset decrypted. 2 |
| Oct 21, 09:25 | Railway gives vendors the potentially exposed keys and asks for them to be revoked. 2 |
| Oct 21, 20:19 | Sessions and tokens invalidated; the public disclosure goes out. 2 |
| Oct 22, 03:32 | Railway confirms the attacker is out of every internal system. 2 |
| Oct 22, 22:49 | API restoration complete; deployments normal on all five plans. 2 |
Railway describes the motive as credential theft for model distillation: the attacker wanted AI provider API keys, and is believed to have had stripped-down models and substantial compute to work with. 2
Distillation trains a model on another model's outputs to copy its behaviour at a fraction of the cost.
An abliterated model is a release with its refusal behaviour removed, which makes it usable for producing that training data where a normal model would decline.
Railway named GitHub, GitLab, Anthropic, OpenAI, OpenRouter, Stripe and Google among the vendors it contacted. 2
What was taken, and what stayed out of reach
Environment variables reached the attacker encrypted, and Railway's keys are envelope encrypted: the key that encrypts each secret is itself encrypted by another. 2
That arrangement is why the blast radius reads as 1.85% rather than the whole platform. 2
It is also why Railway cannot name the affected variables in your workspace. The query returned encrypted values, and the decryption happened elsewhere in the chain. 2
Three more boundaries, as of the October 22 report:
- Railway user accounts were not logged into, though every Railway access token was revoked anyway. 2
- Whether the attacker reached customer source code is unresolved, and Railway says the final report will confirm it. 2
- Running services stayed online throughout. The outage was builds, deployments, dashboard, API and CLI. 2
Two notification lists, and what each one proves
Railway ran two separate notifications, and they mean different things.
The first went to workspaces whose variables may have been decrypted, starting with the highest-risk cohorts on October 22 and continuing into October 23. 5
The second went to vendors. Railway handed over every key it stored for some providers, whether or not that key was reached, so vendors revoked far more keys than were ever exposed. 2
A revocation email from GitHub, Anthropic, OpenAI or Stripe therefore says your key was on Railway's list. It does not say your key was read. 2
Silence works the same way in reverse: the notification campaign was still running when the report went out, so no email yet is no clearance. 5
Railway named two shapes it rates high risk: workspaces holding a very large number of environment variables, and environments holding many AI vendor keys. 2
What Railway already reset, and what it left to you
Railway reset some credentials for every workspace, so a few services stopped reading theirs before you touched anything. 3
| Credential | Reset by Railway | What you still have to do |
|---|---|---|
| Dashboard sessions, Railway tokens | Revoked across all accounts. 2 | Sign in again; reissue project and account tokens for CI. |
| GitHub connection and stored GitHub tokens | Invalidated for your account and services. 3 | Reconnect GitHub, then revoke the old personal access tokens on GitHub itself. |
ghcr.io registry tokens on services | Removed from Railway; still valid on GitHub. 3 | Delete the token on GitHub, issue one with read:packages, paste it into Registry Credentials, redeploy. |
| Storage bucket S3 keys | Regenerated for every bucket in every environment. 3 | Redeploy each service that reads them; the bucket name, endpoint and objects are unchanged. |
| Everything else in your variables | Untouched. 2 | Rotate at the vendor, update the variable, redeploy. |
Workspace admins can see these resets in the audit log, recorded under the actor type Railway Staff, which separates them from changes your own team made. 3
Export the entries if you need them for your own incident record, because retention depends on your plan. 3
The rotation order
Six steps, ordered by what an attacker can spend or mine fastest. Railway publishes a guide per provider; the order below is a delivery sequence across those guides.
- AI provider keys. They were the target, and a leaked one bills you until someone notices. In OpenAI, delete the key in the project that owns it, then check organization-level and admin keys, because they live on separate pages. 6 Deletion takes effect immediately. The usage dashboard groups spend by API key, so a key driving models you never call stands out. 6 Repeat at Anthropic, OpenRouter and every other AI vendor whose key sat in your variables.
- Cloud credentials. These carry the largest bill an attacker can run up in a day. On AWS, create the replacement key first, then deactivate the old one rather than deleting it, so you can reactivate briefly if a service you forgot breaks. 7 CloudTrail keeps 90 days of management events, and you can search by access key ID, one region at a time. 7 The calls to scan for are
RunInstances,CreateUser,CreateAccessKey,AttachUserPolicy,CreateRoleandPutBucketPolicy. 7
On Google Cloud, create the new service account key, then disable the old one with
gcloud iam service-accounts keys disable. The console cannot disable a key. 8Short-lived tokens already issued from that key stay valid for up to an hour. Disable the service account itself if the cutoff needs to be immediate. 8
- Payment keys. Stripe rotates a key with an expiry you choose, and a confirmed leak wants Now so the old key is deleted at once. 9 Request logs are kept per key, including for rotated ones, so the old key's own history shows what it did. 9 Check payouts and bank accounts before charges, and open Webhooks for endpoints you did not create. 9
- Database passwords. On a Railway PostgreSQL service, Regenerate in the Database view changes the password inside the database and updates
POSTGRES_PASSWORD, whichPGPASSWORD,DATABASE_URLandDATABASE_PUBLIC_URLreference. 10 The database restarts to apply it, which drops every open session, including an attacker's. 10 Then redeploy every dependent service, because containers read variables only at startup. 10 ASELECTagainstpg_stat_activityshows any session that reconnected with the new password and does not belong to you. 10 While you are in Settings, turn Public Access off if nothing outside the project needs it. 10 - GitHub and registry credentials. A personal security log covers 90 days, and an organization audit log covers 180. 11 Look past sign-ins: authorized OAuth apps and GitHub Apps, SSH and GPG keys, personal access tokens, deploy keys, repository secrets and webhooks are the ways access survives a password reset. 11 Export the filtered audit log before you revoke anything, so the evidence outlives the credentials. 11
- Everything else. SMTP credentials, other SaaS keys, webhook signing secrets and anything else that can move data or money. These sit last because they are the slowest to abuse at scale, and they still need the same treatment.
A pasted connection string leaves half your services broken after one rotation. A service that takes
${{ Postgres.DATABASE_URL }} picks up the next rotation without an edit. 10Getting deploys working again
Commits you pushed while Railway was not accepting deployments were never queued. Each service still runs its last successful deployment until you trigger a new one. 4
Build errors such as "unable to pull repository" mean the GitHub connection is still missing. 4
- Confirm your GitHub account is connected under Account Settings, and that the Railway GitHub App still has access to the repository. 4
- In the service, open Settings, disconnect the repository, then connect it again. 4
- Run Deploy Latest Commit from the Command Palette. 4
For CLI deploys, clear the revoked session before logging back in. 4
railway logout
railway login
railway link
railway upIn CI, both
RAILWAY_TOKEN and RAILWAY_API_TOKEN were revoked. Project tokens are created under the project's Settings, then Tokens; account and workspace tokens on the account tokens page. 4Services that read storage buckets need a redeploy to pick up the new credentials. A service still logging
InvalidAccessKeyId is running the old deployment or holds a pasted value. 3What Railway changed, and what is still open
Railway rolled all of its keys, then rolled them again after confirming the intruder was out, and replaced the credentials on every storage bucket it runs. 2
It also moved builder infrastructure from third-generation multi-tenant machines to fourth-generation single-tenant ones, a rollout planned for the following week and pulled forward into this one. 2
The platform rebuilt its tooling from validated source, added a way to mass-revoke Railway application permissions, and engaged external forensics with the authorities. 2
Railway rolled every GitHub App installation it holds, so an account whose GitHub email differs from its Railway email may have to sign in with email instead. 2
Still open: the exploit details and its scale, whether customer source code was reached, and which keys in the 1.85% were decrypted. 2
New signups stayed paused while the team worked through notifications. Some ancillary features were still returning on October 23. 5
Does this change your platform decision
You run production on Railway. Rotate first, and keep that work separate from the migration question. The incident exploited long-lived provider keys sitting in environment variables. Any platform that stores them that way carries the same shape.
You are evaluating Railway. Two facts belong beside each other when you weigh the response. Railway told vendors to revoke keys before it told customers, to avoid handing the attacker a fresh set mid-intrusion. 2
Builds were paused for about two days, with Pro and Enterprise back first and Free and Trial last. Running services stayed up throughout. 5
You run somewhere else. Railway believes the attack is part of a series aimed at platform companies, with the goal of reaching the AI providers their customers call. 1
Two questions to ask your own host: which long-lived third-party keys sit in project variables, and can you replace one without a redeploy?
What to do this week
- List every variable in every environment that is a credential, and mark the ones pasted in rather than referenced.
- Rotate in the order above, across the exposure window of October 20, 13:58 UTC to October 22, 03:32 UTC. 2
- After each rotation, read that vendor's own logs for the window: CloudTrail and Cloud Audit Logs, Stripe request logs, the OpenAI usage view, GitHub's security and audit logs.
- Restore the pipeline: reconnect GitHub, reissue CLI and CI tokens, redeploy bucket and registry consumers.
- Close the public network path to any database that only your services use. 2
Rotate now, judge the platform when the report lands
Rotating is this week's decision, and it holds whether or not your workspace was among the 1.85%. 2
The migration decision can wait for the final report. Three tests settle it.
- Whether the report names which keys were decrypted, and whether source code was reachable. 2
- Whether the path from CI credentials to secrets is closed for good, past the single-tenant rollout Railway pulled forward. 2
- Whether anything changes about long-lived keys sitting in a project's variables, since that arrangement is what made the theft worth attempting. 1
Until then, the useful move costs a day of credential work and keeps your options open.
References
- 1Security Disclosure: Wednesday October 21st
blog.railway.com
- 2Preliminary Report: Oct 21st Security Incident
blog.railway.com
- 3
- 4Security Disclosure Recommendation: Recover Failing Deploys
blog.railway.com
- 5Railway Platform Outage
status.railway.com
- 6
- 7
- 8
- 9
- 10
- 11
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.