Making Scheduled Jobs Survive DST and a Single Killed Run

Scheduled jobs are easy to write and easy to forget. They're also where some of the quietest failures I've dealt with came from: jobs that ran at the wrong time, and jobs that stopped running entirely without a single error log.

This is about two fixes to Kubernetes CronJobs on an investing platform: making time-sensitive jobs DST-proof, and stopping one killed run from stranding a CronJob permanently.

Part 1: UTC doesn't know about market close

One of our jobs builds a USD trade allocation file right after market close. Its CronJob schedule was in UTC:

schedule: "3 20 * * 1-5"  # 20:03 UTC

During daylight saving time, 20:03 UTC is 4:03 PM in Toronto, a few minutes after the market closes. That's what it was meant to be.

When DST ends in November, Eastern time moves from UTC-4 to UTC-5. The same UTC schedule then fires at 3:03 PM Eastern, an hour before the close. The file would miss the last hour of trades.

Until then, the only fix was for someone to remember to edit the schedule at every DST change. That's a process that works until the one time it doesn't.

The fix: schedule in the market's time zone

Kubernetes CronJobs support spec.timeZone. Production already had CronJobs using it with America/Toronto, so this was following an existing pattern, not introducing a new one.

With a time zone set, the schedule is read as Toronto time, and Kubernetes handles the EST/EDT switch:

# values.yaml
cron:
  schedule: "3 16 * * 1-5"      # 4:03 PM Toronto
  timeZone: "America/Toronto"

# cronjob.yaml
spec:
  schedule: {{ .Values.cron.schedule | quote }}
  timeZone: {{ .Values.cron.timeZone | default "America/Toronto" | quote }}

The new schedule is the current run time converted once, so behaviour on the day of the deploy didn't change.

One detail mattered more than the rest: the template change and the schedule change had to ship in the same deploy. With only the new schedule, the job would read 3 16 as UTC and run hours early. With only the time zone, it would read 3 20 as Toronto time and run hours late. Either half alone is a worse bug than the one you're fixing.

I swept all the trading services for schedules pinned to UTC. Four were time-sensitive enough to matter: USD allocation, expected positions, cash allocation and a price push job. Each now schedules in America/Toronto. QA signed off, and all four went to production before the November change, so the next DST switch needs no edits at all.

The real proof is the first run after the switch: each job firing at the right Eastern time on the Monday after.

A related note: one service already handled time zones in its start script (TZ='America/Toronto' node index.js). When we changed how that container starts, the TZ moved into the Dockerfile as an environment variable so it wasn't silently dropped. Time zone settings hide in more places than the schedule.

Part 2: one killed run, two days of silence

The second problem was worse because nothing logged it.

A price notification CronJob in our QA environment ran zero times for two days. No errors, no alerts. Its last run had been killed partway through: the logs showed it processing a batch, reconnecting to RabbitMQ, and then nothing, ever again. No stack trace, because the pod was killed rather than crashing.

One killed run turned into a two-day outage because three settings compounded:

1. restartPolicy: OnFailure kept the Job active while it retried the failing pod.
2. concurrencyPolicy: Forbid then blocked every later schedule while that Job was active.
3. No startingDeadlineSeconds****. Without it, the CronJob controller counts missed schedules since the last scheduled time. Past 100 missed schedules, it stops scheduling and logs too many missed start times.

The job ran every 5 minutes inside a daily window. The missed runs from the rest of that day plus the next morning crossed 100, and the CronJob stranded itself. Recreating it brought QA back, but that only cleared the symptom.

And the trigger wasn't rare. Counting runs that started but never logged completion over about a week, it was roughly one a day across the non-production environments. QA was just unlucky in when its run died.

The fix: bound both windows

The fix went into the chart defaults so every environment inherited it:

startingDeadlineSeconds: 300   # missed-schedule counting can't reach 100
activeDeadlineSeconds: 240     # a stuck Job can't hold the Forbid lock open
failedJobsHistoryLimit: 1      # was 0: keep the last failure to inspect

The first two are independent. startingDeadlineSeconds bounds the window the controller counts missed schedules over, so it can never hit the limit. activeDeadlineSeconds bounds how long a Job can stay active, so a retrying pod can't block Forbid. Normal runs took 1 to 24 seconds, so 240 is about ten times the headroom.

The third one explains why it took two days to notice. With failedJobsHistoryLimit: 0, a failed Job is deleted immediately, so there was nothing left to look at.

A few days after deploying, a QA run hit DeadlineExceeded after a node-level mount timeout, the same kind of event that caused the original outage. This time the CronJob carried on scheduling.

The same defaults applied everywhere, so the fix went into the shared chart rather than one environment. Any pause long enough to rack up 100 missed schedules, like a cluster upgrade, would otherwise strand a CronJob the same way.

Takeaways

• Schedule business-time jobs in the business's time zone. If a job is tied to market close, payroll or a bank cutoff, UTC will drift from it twice a year.
• Ship coupled config changes together. A schedule and its time zone are one change, not two.
• Set startingDeadlineSeconds and activeDeadlineSeconds on every CronJob. The defaults let one bad run stop all future ones.
• Keep at least one failed Job. You can't debug what's been deleted.
• Watch for jobs that stop arriving, not just jobs that fail. A job that never starts produces no errors. That's the gap I'd close next: an alert when a scheduled job is overdue.