Upgrading Node Across 29 Production Services Without Breaking Things

In October, Node.js shipped a security release. I owned getting it onto the trading services at a Canadian fintech: 29 services, running a mix of Node 18, 20, 22 and 24.

On paper, that's 29 one-line Dockerfile changes. In practice, a good part of the work was in places that had nothing to do with Node.

Decide the target, not just the patch

The quickest option was to patch each service within its current major version. That doesn't work for everything. Some services were on major versions that no longer receive security fixes, so patching within the major wasn't an option.

So instead of 29 patches, we standardized on Node 24 LTS, supported until April 2028, at the exact patched version. One target, the same everywhere, with years of runway.

Batch by risk

Not every service carries the same risk. A background job that reconciles a file once a day is not the same as a public API that the mobile app talks to.

I split the 29 into batches. The public APIs still on old Node versions got a full regression pass, because a jump of several major versions on a service with that much surface deserves one. The rest moved on their own track, so they didn't wait behind the slowest QA cycle.

That had a cost I accepted openly: the services with the biggest version jumps and the most surface area were the slowest to land, by design. A proper regression pass was worth more than finishing the batch a week earlier.

One ticket per service, the same checks every time

Every service got its own ticket and PR, with the same structure:

• Why: what version it's on and why that matters.
• Change: the exact Dockerfile line, plus anything else touched.
• Verified locally: the image builds, node -v inside it prints the target version, and unit tests give the same result on the old and new Node.
• Startup comparison: run the old image and the new image with the real start command, and compare. Same logs, same number of modules loaded.
• Test on dev and QA: one concrete business flow for that service, not just "pod is healthy."
• Acceptance: runs on the new version in production with no error-rate regression.

The startup comparison was the most useful check. Many of these services can't fully boot on a laptop without real config, so "it starts" isn't a fair test. But you can check that the old and new images fail at the same point, with identical logs, having loaded the same modules. On the largest API, 280 modules loaded on both versions with no new failures.

When unit tests had existing failures, I recorded them as existing. "10 fail on both versions" is a very different statement from "10 fail."

Surprise 1: floating tags that can't receive the fix

Several Dockerfiles used floating tags like node:24-alpine3.22 or node:22-alpine3.21. The idea behind a floating tag is that rebuilding picks up the latest patch.

Except no patched build existed for those Alpine versions. Rebuilding would produce exactly the same unpatched Node. The tag looked like it would self-update, and it couldn't.

Others used tags like node:22-alpine, which do move, but that means the Node version in production depends on when the image was last built. Two services on the "same" Dockerfile could be on different patch versions.

The fix was the same for both: pin the exact version.

# before
FROM node:24-alpine3.22

# after
FROM node:24.22.1-alpine3.24

Pinning means you have to bump it deliberately next time. That's the point. The version in production should be something you chose, not something you inherited from a build date.

Surprise 2: a Dockerfile that ignored its lockfile

One service couldn't be built at all, on any Node version. Its Dockerfile copied package.json but not the lockfile, so every build installed the newest version of every dependency. One of those dependencies had since published new TypeScript types, and the build failed on a type error.

Nobody had noticed because nobody had deployed it recently. The next deploy, for any reason, would have failed.

COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

I flagged one caveat in the ticket: the lockfile holds the versions developers last tested, which might be older than what the running production image was built with. So QA needed to cover the main flow, not just a smoke test. Another service had the same lockfile gap and still built, but it wasn't reproducible either, so it got a follow-up.

Surprise 3: a config library that crashed on Node 24

One service crashed on startup with only the Node version changed:

TypeError: Utils.isRegExp is not a function

Its lockfile pinned an older version of the config package that called util.isRegExp(), which Node removed in v23. A newer patch version of config doesn't use it, and the other services were already on that version. The bump was within the existing semver range, so only the lockfile changed.

This is the kind of thing that only shows up when you actually boot the new image. Unit tests that mock config would never see it.

Small cleanups that rode along

Most services started with yarn start. On Node 24, that printed a deprecation warning from yarn itself. Since start just ran node index.js, the container now runs that directly. Two details I checked each time:

• If the start script set an environment variable, like a time zone, it moved into the Dockerfile so it wasn't lost.
• For jobs, the exit code on failure stayed non-zero, so Kubernetes still marks a failed run as failed.

Tracking it separately pays off

Having one ticket and one PR per service paid off in a way I didn't plan for. Whenever something unrelated broke in a service mid-rollout, the first question was "did the upgrade cause this?", and the per-service tracking made it a quick answer instead of an investigation.

Takeaways

• Pick a target version, not just a patch. End-of-life runtimes won't get the next fix either.
• Batch by risk. Public, high-surface services get full regression. Don't let them hold up the low-risk ones.
• Pin exact base image versions. A floating tag can silently stop receiving updates.
• Always copy the lockfile into the build. An unreproducible build is a failed deploy waiting for a reason to happen.
• Compare the old and new images side by side. Same logs, same modules loaded. It catches runtime removals that unit tests never will.
• Record existing failures as existing. It keeps the signal clean for the next person.