Scheduled jobs on EventBridge
Read, recover, and hand over a job that EventBridge Scheduler runs in its own Lambda, where EasyCron used to call a Flask route for it.
Canonical sources. Which jobs exist, their IAM grants and concurrency: SCHEDULED_JOBS in infra/components/scheduled-jobs.ts. Which environment runs which job: scheduledJobs on each row of infra/app/envs.ts. Each schedule's cadence: its <schedule>_schedule key in app/config/environments.yaml. The handler and the job registry: app/api/scheduled_jobs/. What each job's cadence is coupled to: the EasyCron section of the backend-development skill.
How a job runs
Each job is one function built from the monolith's whole tree (the flask-web.zip that infra/scripts/build-flask-web-zip.ts writes), inside the tier's VPC, under a role carrying only what that job calls. It restores from a SnapStart snapshot only when it runs more than 24 times a day (SNAPSTART_ABOVE_RUNS_PER_DAY); a rarer job cold-starts and no one waits on it. Its schedules invoke the live alias directly with {"job": "<schedule>"}; nothing crosses a public URL. The handler builds the same headless app the other worker Lambdas use and runs the job's own code, which writes the same log lines the Flask route wrote.
A schedule never retries, and a tick that finds every reserved slot busy waits at most 60 seconds before Lambda drops it. Reserved concurrency is the job's concurrentRuns plus one for the deploy's probe, so it is also the bound on the job's database connections.
Reading a job's runs
Lambda logs go to CloudWatch, not OpenSearch. Query the job's log group with Logs Insights; the lines are the ones the Flask route writes, with path=eventbridge-scheduler. During a handover, runs EasyCron still triggers are in OpenSearch as before, under the route's own path.
fields @timestamp, @message
| filter @message like /SES VDM enrichment completed/
| filter @message like /skipped_locked=True/ or @message like /deadline_hit=True/
| sort @timestamp desc
| limit 50Each run also ends in Lambda's own REPORT line, which carries its duration and, on a cold start, its restore time under SnapStart or its init time without it. A job without SnapStart exhausts Lambda's 10-second init limit, which runs the init again inside the invocation, so its cold start reads as a long Init Duration and an INIT_REPORT line with Status: timeout; neither is a fault.
Alarms
Prod and EU page; stage builds none. Each alarm's description names the customer impact, the first thing to look at and the safe recovery.
| Alarm | Means | First look |
|---|---|---|
onramp-<env>-job-<job>-errors | runs raised: in two 15-minute windows running for a job that never goes 30 minutes without a run, in one run for any other | the log group, filtered to ERROR; the job's Sentry issue |
onramp-<env>-job-<job>-throttles | a tick found every reserved slot busy, so it ran late or was dropped | the REPORT durations: a run is outlasting its schedule |
onramp-<env>-<schedule>-delivery | EventBridge cannot invoke the alias at all | whether the schedule still targets live, and whether its role may invoke it |
onramp-<env>-<schedule>-missed-run | no run has returned in two of the schedule's longest gaps plus a run's timeout, at most seven days | whether the schedule exists, is ENABLED and targets live; --report for its last run |
onramp-<env>-<schedule>-failed-items | runs returned but failed on items | the run's scheduled-job-summary line: its failed count and the job's own fields |
The failed-items alarm pages on the first such run when the job declares failedItems: FailedItems.LOST in SCHEDULED_JOBS, because no later run retries the item, or when the job runs less often than every 30 minutes; a RETRIED job that runs more often waits for two 15-minute windows running.
Each run that returns publishes one FailedItems datapoint under OnRamp/ScheduledJobs/<env>, dimensioned by Schedule and valued at its summary's failed count. The missed-run alarm counts them and the failed-items alarm sums them. A missed-run alarm stays in INSUFFICIENT_DATA until its schedule's first run publishes, so a new job or alarm never pages before it has run; a schedule that has never run once is the one stop it cannot see, and the delivery and error alarms cover that.
How stale a job may get is one rule, FRESH_GAPS of its schedule's longest gaps: fresh_gaps in app/api/scheduled_jobs/alarm_contract.json, with longest_gap in app/api/scheduled_jobs/cadence.py. The EasyCron handover and the missed-run alarms both read it, so removing the handover leaves both files in place.
Safe recovery for a delivery or missed-run alarm is re-running the deploy, which rebuilds the schedule, its role and the alias together. Every job survives a doubled tick, so invoking live with a schedule's input to run it sooner is always safe.
Deploys
Each newly published version must answer {"probe": true}, which reaches the database and runs no job, before the alias moves to it. A version that fails fails the apply, and the previous version keeps serving. Job functions ship only on a release; an out-of-band infra dispatch holds them at their deployed code (infra/components/co-release.ts). A release whose infra leg leaves any job function on other code than its own fails that leg and names each function. Re-run the failed leg once the cause is fixed: it ships the release's own job package, so no hotfix is needed for the jobs alone.
Handing a job over from EasyCron
A job's schedules start with the release that lists it on an environment's row, and EventBridge never goes through app.onramp.us or eu.onramp.us, so neither hostname's CloudFront cutover is a step for it. What the cutover needs is only that every long-running job has already moved: its console calls are then instant no-ops, below any edge timeout.
Nobody switches anything. Each EventBridge run that succeeds records a heartbeat (or_scheduled_job_heartbeat), and the job's EasyCron route, behind yields_to_eventbridge (app/api/scheduled_jobs/handover.py), answers 200 at once without running while a run succeeded within two of its schedule's longest gaps. Otherwise, which covers the first ticks after the release that adds the schedule, an environment whose infra apply failed, and a function whose runs keep failing, the route runs the job as it always did. So a console job can be deleted whenever it is convenient, and its calls stay instant, and below any edge timeout, meanwhile. The tick on which both still run is safe because each job's PR shows it survives a doubled tick.
Validating an environment
bun run scheduled-jobs.smoke <env> checks, on your own AWS credentials, that each job's live alias answers its probe, and that each schedule exists, targets that alias with its own input, and fires on the expression and in the state the config map resolves. --run adds one real tick of every schedule; --plan lists the checks and calls nothing. Stage deploys every job it can probe. Where stage must not run one on a schedule, as with SES enrichment, which shares prod's SES insights quota, its schedules are paused with <schedule>_schedule_state: DISABLED, and --run-paused is how it runs there. --run alone skips paused schedules: a tick that succeeds makes the job's EasyCron route stand aside for two gaps, which on an environment whose console still runs the job would leave it unrun until the schedule is enabled.
Did the job do its work?
Every run returns a summary (job_summary, app/api/scheduled_jobs/verification.py): how many items it examined, acted on, skipped and failed on, how many messages it sent or enqueued, then its own figures. Every run logs it as one scheduled-job-summary {...} line in the job's log group. Each job declares a Verification beside its probe (app/api/scheduled_jobs/jobs.py); one that declares none gets the default.
| Tier | A healthy verify tick |
|---|---|
| Every job | returns a summary with all five counts, fails on nothing, advances its heartbeat; a tick straight after it sends nothing; its log group holds both summary lines and no error |
| Critical | also moves each database count it declares by what its summary reports, and in --report fails once EventBridge has stopped running it |
--verify runs those ticks, twice per schedule, on stage only: it refuses an environment whose email_destination_policy or sms_destination_policy is not internal_allowlist. --verify --run-paused includes the paused schedules.
bun run scheduled-jobs.smoke stage --verify --run-paused--report ticks nothing. For prod and gdpr-prod after a release it prints each schedule's heartbeat, its last run's summary and duration, and the errors in its log group over two of its longest gaps.
bun run scheduled-jobs.smoke prod --report