# The First Hour After a Hacked Dependency: A Runbook

> Two bad versions of axios were live for about three hours in March 2026, installing remote access malware on macOS, Windows and Linux. Three hours is shorter than most teams take to notice. So the question is never 'did we install it' but 'which machines downloaded it during those hours'. Here is the ordered runbook: find the window, list what downloaded, rotate in the right order, then look for what was left behind.

Author: Kishore K Sharma. Published: 2026-09-20. Canonical URL: https://kishorek.dev/writing/first-hour-dependency-compromise. Tags: incident-response, supply-chain-security, security, devsecops, npm.
License: © 2026 Kishore K Sharma. All rights reserved. Reproduction requires attribution and a link to https://kishorek.dev/writing/first-hour-dependency-compromise.

---
On 31 March 2026, two versions of axios — `1.14.1` and `0.30.4` — were published from a hacked maintainer account. Both pulled in a package that installed remote access malware on macOS, Windows and Linux. They were live for about three hours before being removed.

Three hours. Think about what that means for your team.

It does not mean you probably missed it. It means the opposite. Your risk almost certainly has nothing to do with whether someone chose to upgrade axios that afternoon. It depends on how many automated processes downloaded a dependency during one specific three-hour window — CI jobs, nightly Docker builds, update bots, or a developer running a fresh install after switching branches.

Most teams cannot answer that at all.

![A timeline with the bad version live for about three hours, marked in danger colour between a publish marker and a removal marker, and a much later marker showing when you read the advisory. Below it, four kinds of process are ranked by risk: CI runs marked exposed, Docker layer caches marked exposed later too because a cached install layer is reused by builds that never download again, update bots marked exposed because the pull request check installed the proposed version, and builds after removal marked clean. The caption states the question is never whether you installed it but which machines downloaded during those hours.](/writing/dependency-exposure-window.svg "Find the window from publication to removal, then list every automated process that downloaded inside it.")

This is the runbook for the hour after you find out. The order matters, because the usual mistake is doing the right steps in the wrong sequence.

## Step 0: do not rebuild yet

When you read a security advisory, the reflex is to bump the version and redeploy. Wait fifteen minutes. Two reasons.

**You will destroy your evidence.** The best record of what you actually installed is the state you are in right now: lockfiles, CI logs, Docker layer caches, the `node_modules` folder on disk. Rebuilding overwrites the very thing you need to read.

**You may not be fixing anything.** If a bad install script already ran on a machine, upgrading the package does not undo what it did. It removes the delivery method from a machine that has already been emptied. A clean rebuild that turns the scanner green while stolen credentials still work is worse than doing nothing, because it ends the investigation early.

Take a snapshot first. Copy the lockfile, save the CI logs, note the timestamps. Then start.

## Step 1: find the exact window

You need two timestamps: when the bad version was published, and when it was removed. Both are usually in the advisory or in the registry's own data.

Write them down in UTC. Everything that follows is a comparison against that window.

One detail people get wrong: the window is **publication to removal**, not publication to *when you found out*. If the bad version was removed at 13:20 UTC and you read about it three days later, a build that ran on day two downloaded the clean version. Widening the window out of worry turns a manageable investigation into an endless one, and endless investigations get abandoned halfway.

## Step 2: list everything that downloaded during the window

This step is the hard one, for a simple reason: most teams have no record of what installs their dependencies.

Work through these, roughly in the order people forget them:

**CI runs.** Every pipeline run with a timestamp inside the window that installed packages. Not just your main application — the docs site, the internal tool nobody maintains, the scheduled job that runs at 3 AM. Search your CI provider's history by time range.

**Container builds.** Docker caches each layer. An image built after the window can still contain a layer created *during* it. If the `RUN npm ci` layer was cached, a later build reuses the bad packages without downloading anything. Check layer creation times, not image build times.

**Developer machines.** Anyone who installed during the window. In practice you ask in a channel, with the window written in their local time so they can actually answer. Expect this list to be incomplete. Treat it as a minimum, not a maximum.

**Update bots.** Renovate and Dependabot open pull requests automatically, and the CI job on that pull request installs the version being proposed. A bot that proposed the bad version and had its check run is a real exposure, even though no human ever merged it.

**Anything without a strict lockfile.** A lockfile protects you only where it is used. `npm install` without one, a Dockerfile that runs `npm install` instead of `npm ci`, or someone using your library and resolving your version ranges themselves — each of these downloads fresh.

The output of this step is a list of machines and jobs. If the answer is "we do not know", that is your finding, and the rest of this becomes blanket rotation instead of investigation.

## Step 3: decide whether credentials are gone

One question decides how expensive the next few hours are: **did install scripts run?**

If the malware ran through an install hook and your installs run with scripts enabled — the default — then on every machine from step 2, assume every credential reachable from that process is stolen. Not "might be". Assume it.

That includes things people forget are reachable:

- environment variables in the process, including every secret given to the CI job, whether or not the step used it
- `~/.npmrc`, `~/.aws/credentials`, `~/.docker/config.json`, kubeconfigs, SSH private keys
- cloud instance metadata — from a build machine inside your own cloud account, that is one HTTP request away
- shell history, which contains more secrets than anyone admits
- credentials for AI coding tools, which are now a standard target

If installs ran with `--ignore-scripts`, your position is much better. The bad code sat on disk without running, and you are doing a cleanup rather than handling a breach. This single flag is the biggest difference in outcome, and it was decided months earlier. I went through why in [the ChainDrop writeup](/writing/chaindrop-npm-worm-provenance) — the delivery method in these attacks is almost always an install hook.

## Step 4: rotate in the right order

This is the step most often done wrong, and the reason this is an ordered runbook rather than a checklist.

**Rotate credentials in the order that removes the attacker's ability to steal them again.**

Say you start with the obvious one: a production database password. You rotate it. Meanwhile the attacker still holds the CI token stolen from the same build machine. They use it to read your secret store, or they simply wait for the next pipeline run and take the new password out of memory. You did real work and improved nothing.

![Two rows. The top row shows the correct rotation order as four boxes in sequence: identity and CI credentials such as GitHub tokens, cloud trust settings and npm tokens; then cloud credentials such as IAM keys and Vault tokens; then the contents of the secret store; then long-lived session signing keys. The bottom row, in danger colour, shows what happens if you start at step three instead: rotating the database password feels productive, but the attacker still holds the CI token, reads the secret store with it, and the new password is stolen — net progress zero. A caption reads: revoke, do not merely replace.](/writing/credential-rotation-order.svg "Start with the credentials that can reach other credentials. The order decides whether the work counts.")

The correct order runs from credentials that can reach other credentials, outward:

1. **Identity and CI credentials.** GitHub tokens and SSH keys, npm publish tokens, CI provider tokens, and any trust settings that let a pipeline get cloud access. These reach everything downstream, including the pipeline that hands out your other secrets.
2. **Cloud credentials.** IAM keys, service account keys, Vault tokens. Anything that can read a secret store.
3. **Secret store contents.** Now, and only now, the application secrets themselves — database passwords, third-party API keys, signing keys.
4. **Long-lived session material.** Session signing secrets and refresh token keys — anything where holding the old value still works after the credential is replaced.

Two rules throughout:

**Revoke, do not just replace.** Issuing a new token while the old one still works achieves nothing. The action is revocation. The new credential is a side effect of needing to keep working.

**Prefer removing the trust relationship over rotating a shared secret.** Disabling a trust setting or deleting a deploy key is instant and easy to verify. Rotating a shared secret that eleven services read from four places is a hunt for the eleventh service at 2 AM.

## Step 5: pin, override, then rebuild

Now fix the dependency.

Pin to a known-good version, and use your package manager's override setting rather than trusting every package in the tree to have resolved sensibly:

```json
{
  "overrides": {
    "axios": "1.14.2"
  }
}
```

Then reinstall with scripts off, whatever your normal policy is, because you are reinstalling from a state you do not fully trust:

```bash
rm -rf node_modules
npm ci --ignore-scripts
```

Check what you actually got, not what you asked for. The lockfile is the authority, not `package.json`:

```bash
node -e "
const lock = require('./package-lock.json');
for (const [p, m] of Object.entries(lock.packages || {})) {
  if (p.split('node_modules/').pop() === 'axios') console.log(p, m.version);
}"
```

Several entries at different versions is normal, and it is exactly why you check. An override in the top-level `package.json` quietly does nothing if something deeper in the tree pulls its own copy.

## Step 6: look for what was left behind

Rotation removes the access the attacker has now. It does not remove the access they set up for later, and modern malware sets up for later.

![Two panels. On the left, in accent colour, the credentials rotation removes: a stolen CI token, cloud key, npm token and SSH key, each struck through and marked revoked. On the right, in danger colour, what rotation never touches: an editor task file, an agent settings file, a CI workflow the attacker added, and a deploy key they created — all marked still working. A caption reads: you rotated what they stole, you did not rotate what they made.](/writing/persistence-survives-rotation.svg "A credential the attacker created is not on your list of credentials to rotate.")

What to check on every affected machine and repository:

**Files that run automatically.** Editor task files, AI agent settings folders, and git hooks in `.git/hooks`. Attackers like these because nobody reviews them and they survive a dependency rollback completely.

**New or changed CI workflows.** A workflow file added during the window, especially one that touches secrets or writes build files. Compare your workflow folder against a known-good commit rather than reading it. A malicious workflow is written to look boring.

**Commits you cannot explain.** Check the window for commits nobody remembers making, including ones with believable author names. Attackers choose bot-like identities because a commit from a tooling address gets less attention than one from a colleague.

**New tokens, keys and integrations.** A fresh deploy key, a new app authorisation, an added collaborator, an access token created during the window. This is the access that survives every rotation you just did, because the attacker created it rather than stealing it.

**Outgoing connections.** With the window in hand, look for connections from affected machines to places you cannot explain. Do not expect an obviously bad domain. Recent malware routes its early stages through services you already trust, and the suspicious destination only shows up at the point where data is sent out.

## What makes the hour survivable

Everything above is much easier if some things were already true. In order of value:

**Short-lived credentials.** If your CI gets a fifteen-minute cloud credential instead of a permanent access key, most of step 4 disappears — the stolen credential expired before you finished reading the advisory. This is the highest-value preparation available and also the hardest to add later, which is why it keeps not happening.

**CI logs you can still read.** Step 2 only works if you can search what ran and when. The default retention on most providers is shorter than most teams' detection time. Check yours before you need it.

**An answer to "what can this token reach?"** Most teams cannot say what a given CI token has access to, which forces them to rotate everything or, more often, rotate the obvious things and hope. Keeping permissions narrow pays off mainly on the day you have to work this out under time pressure.

**Lockfile history in git.** Your lockfile is a dated record of exactly what you installed. `git log` on it, filtered to the window, answers a question that is otherwise guesswork.

## The real problem

The bad axios versions were live for three hours. Most teams would not have noticed within three days.

That gap is the actual problem, and no amount of incident response discipline fixes it. A runbook makes a bad hour shorter. It does not prevent the hour.

The control that closes the gap is refusing to install anything published recently: a three-to-seven day waiting period, available as `min-release-age` in npm 11.10+ and equivalent settings in pnpm, Yarn and Bun.

Against an attack measured in hours, a waiting period measured in days is not damage control. It removes the whole class of attack. Together with `--ignore-scripts`, it means this runbook stays a document you wrote once and never had to open.

## The takeaway

When a dependency you use is hacked, do not bump the version and redeploy. That overwrites the lockfiles, CI logs and caches you need to work out the damage, and it does nothing about an install script that already ran. Find the window from publication to removal, list everything that downloaded inside it — including Docker layer caches and bot pull request checks — and let one question decide the cost of everything else: did install scripts run? If they did, every credential reachable from those processes is gone, not just the ones the step used.

Then rotate in order. Identity and CI first, because those are what let an attacker steal everything else again, then cloud, then the secret store, then session keys. Revoke rather than replace. Finish by looking for the access that survives rotation: files that run automatically, new workflows, and above all credentials the attacker **created** rather than stole.

None of this prevents the incident. What does is refusing very new versions. A waiting period measured in days, against an attack measured in hours, is not damage control — it removes the whole class, and it keeps this runbook something you never have to open.
---

Originally published at https://kishorek.dev/writing/first-hour-dependency-compromise.
© 2026 Kishore K Sharma. All rights reserved.