Every Retry Loop Needs an Alarm on Its Last Attempt

We had alerting on our instant account opening flow. If the custodian rejected an account, a message went to the team. That part worked.

What we didn't have was any signal when the job simply ran out of retries. For about two weeks, some instant account openings ended that way, and nobody heard about it.

How it was found

After our move to a new cloud, I was running a daily scan of production errors across our services. One Tuesday it showed a handful of failures in the account opening job. On its own it looked like a one-off.

I went back further before closing it. It wasn't a one-off. The same failure had been happening since early in the month, at a steady rate. A few accounts from the previous weeks were still waiting with no account created at the custodian.

No alert had fired, because nothing the alerts were watching had happened.

What the job did

When someone opens an account, a background job waits for their profile to be ready before it submits the account to the custodian. It polls the profile 10 times over about 100 seconds.

Most profiles are ready well within that window. Some aren't, because they're still in compliance review, which can take much longer than 100 seconds.

For those, the job reached its last attempt, threw, and nothing caught the error. It became an unhandled promise rejection. No alert, no status change, nothing a person would see unless they went looking.

So there were two outcomes we'd thought about (success, and the custodian saying no) and a third we hadn't: the job giving up.

The fix

The change was small:

• On the last attempt, post an "Instant Account Opening Skipped" alert to the account activation channel.
• Handle the error instead of letting it escape as an unhandled rejection.

Roughly this shape:

for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  const profile = await getProfile(profileId);
  if (profile.isReady) return submitAccount(profile);

  if (attempt === MAX_ATTEMPTS) {
    await alert('Instant account opening skipped', {
      profileId,
      status: profile.status,
    });
    return; // handled: someone now knows
  }

  await sleep(INTERVAL_MS);
}

I also asked operations to check the review status of the accounts that were already stuck. When they went through them, two had already become active through the normal path, and the rest were either still in compliance review or waiting on the member. So the fallback process existed. It just had no way of knowing which accounts needed it.

Why this keeps happening

Retry loops are written with the happy path in mind. You think about the call that fails once and succeeds on the second try. The last iteration, where you stop trying, gets whatever the language does by default: return undefined, or throw into nothing.

That's the worst kind of failure because it's quiet. Around the same time, I'd written down a pattern from a few other incidents: a scheduled job that stopped running for two days and logged nothing, a consumer failing in a loop whose errors arrived at info level, and a weekly job that exited 0 with a green status while processing none of its files. The expensive failures were the ones that produced no error at all.

The weekly job is worth a sentence on its own. Its counter collapsed four different outcomes into one number, and the files that matched no handler were logged at debug level, which production doesn't emit. Fixing it meant making each outcome a separate count and adding a run summary. Once the job could describe its own work, the real problem was obvious within one run.

A checklist for retry loops

When I write or review a retry loop now, I look for these:

1. What happens on the last attempt? It should be explicit code, not fall-through.
2. Who finds out? An alert to a channel someone owns, with enough context to act on (the ID and the last state seen).
3. Is the error handled? An unhandled rejection is not a notification. Depending on the runtime, it's either noise in a log or a crashed process.
4. Is giving up a distinct outcome? Success, rejected and gave-up should be three different things in logs and metrics, not one "failed" bucket.
5. Is the retry window realistic? Ten attempts over 100 seconds is fine for a flaky network call. It's the wrong tool for waiting on a human review process. That case might need a different design, like reacting to the review-completed event.

Takeaways

• A job that gives up has to say so, the same as a job that fails.
• If you already alert on rejection, check whether you alert on exhaustion. They're different code paths.
• Daily, boring error review finds things alerts can't, because alerts only watch for what you expected. The two-week gap here was found by a person reading a scan, not by a monitor.
• Make jobs report what they did, not just whether they finished. "Exit 0" is not the same as "did the work."