The OOM That Wasn't a Memory Problem

One morning, a production service that processes account files stopped processing anything. Both pods showed zero consumers on their RabbitMQ queue. Kubernetes had OOM-killed them, they restarted, picked the messages back up, and got killed again.

The obvious fix was more memory. It was also the wrong one.

What happened

An upstream partner dropped five large CSV reports at the same moment. Each pod consumed messages from the queue, and each message pointed to a file the service downloaded and parsed in memory.

The consumers had no prefetch limit. RabbitMQ will happily push every unacknowledged message to a consumer that hasn't said how many it can handle. So each pod grabbed all five files at once, buffered every CSV in memory, and blew through its 256 MiB limit.

Then the cycle repeated: OOMKill, messages redelivered, the restarted pod grabs all five again, OOMKill.

Attempt 1: more memory

We raised the limit to 512 MiB as a stopgap. The new pods died again three minutes later, this time on the V8 heap limit, not the container limit. In our setup, Node's default heap ceiling sat at roughly half the container's memory, so doubling the container didn't double what the process could actually use.

I opened a PR for 1 GiB, looked at it again, and closed it.

More memory doesn't fix "this process loads an unbounded amount of work at once." It just moves the ceiling, and the next burst of six files finds it.

The fix: one line

channel.prefetch(1);

With a prefetch of 1, each pod takes one message, processes it, acknowledges it, and only then gets the next. Five files across two pods are processed two at a time, and memory stays flat no matter how many files land at once.

It was in production within about two hours of the first alert, and memory went back to the original limit.

The part that stuck with me

This was the second time in about a month that the same service broke only when several files arrived together. The first time, concurrent files closed a shared database client under each other. Neither bug shows up when you test one file at a time, which is exactly how we'd been testing.

So the lessons I took away:

1. Any consumer that buffers a whole message's payload needs a prefetch limit. Unbounded prefetch is a default, not a decision.
2. When a pod OOMs, the first question is "what made it load everything at once?", not "how much memory does it need?"
3. A burst belongs in the test plan. If production can send you five files at once, your tests should too.
4. Watch for poison loops. With redelivery, a crash during processing doesn't lose the message; it brings it back to crash the next pod. RabbitMQ eventually dropped a few files after repeated retries, which we had to re-upload by hand. Dead-letter queues and retry limits are worth setting up before you need them.

The most useful fix is often the smallest one. It just takes resisting the urge to make the number bigger.