Skip to content

Journey recovery worker ​

How to read the journey recovery worker's records and alarms, and what to do when a journey phase is not advancing.

Lambda · onramp-<env>-journey-outbox-workerSchedule · <env>-journey-outbox-recoveryAlarms route to the tier's pipeline topic

Canonical sources. The handler is app/api/journeys/worker_lambda.py and the pass it runs is app/api/journeys/services/journey_outbox_processor.py. The function, roles, schedule and alarms are infra/components/journey-outbox-worker.ts. Its cadence, deadline, memory and backlog threshold are the journey_outbox_* keys in app/config/environments.yaml. The zip is built by infra/scripts/build-journey-outbox-worker-zip.ts from app/api/journeys/worker_bundle.json.

TL;DR ​

What it does

It drains the journey outbox. A committed change queues a row, and the web app wakes a pass right after the commit. That wake is best effort. This worker runs the same pass on a schedule, so a row whose wake was lost still advances.

It also starts the clock's work. A phase whose time-in-stage finish line has passed is queued by the pass, so it advances with no project change.

What it never does

It never skips the feature check. A row of a vendor with customer_journeys off waits. The worker counts it as paused, and it never counts toward the backlog alarm. That vendor's due phases are left untouched too.

Safe to run twice

Rows and due phases are claimed with SKIP LOCKED. A manual run beside a scheduled one cannot advance a phase twice. The transition's own locks and dedupe still decide.

How it runs ​

EventBridge Scheduler invokes the function every journey_outbox_recovery_schedule (5 minutes by default) with {"job": "drain-journey-outbox"}. Nothing else may invoke it. Neither the scheduler nor Lambda retries a run: the next tick is the retry.

Each invocation builds a headless Flask app, connects with the worker database login, and runs one bounded pass across every vendor. The pass stops taking rows when its budget is spent. The budget is the smaller of journey_outbox_pass_budget_seconds and the time left in the invocation, less one capped statement and a few seconds to report. With the defaults (a 125 second deadline, a 30 second statement cap and 5 seconds to report) a pass gets its full 90 second budget. The deploy refuses a deadline too short for that.

Due phases ​

When a phase is entered, next_due_at on its or_journey_phase_instances row records the next moment time alone could change its outcome: the first time-in-stage finish line it has not passed yet. Each pass, before it takes any outbox row, finds open phases whose next_due_at has passed for vendors with journeys on. For each one it queues an outbox row through one of the phase's own linked projects and moves next_due_at to the phase's next such moment, or clears it when none is left, in one transaction. It takes at most 1,000 phases a pass, 100 at a time, and stops early when the budget runs out.

Those rows are then processed in the same pass, oldest first beside rows from project changes, with the same locks, exit evidence and feature check. So a due phase never jumps ahead of project work, and a row that fails stays pending like any other. A phase with no linked project is not queued; its due moment still moves on, and linking a project evaluates it.

The outgoing lead's handoff confirmation (POST /api/journeys/instances/<id>/handoff-confirmation) also queues a row, in the same transaction that records it. Confirming does not advance the journey on its own: the phase still waits for its other finish lines.

Handoff recaps are not this worker's ​

Every stage transition, automatic or Advance now, also writes one kind = 'handoff_recap' row for vendors with Aero account reviews on. This worker never takes, counts or alarms on those rows: its pass, remaining, paused and oldest_pending_seconds cover kind = 'project_change' rows only. The account CSM worker writes and delivers recaps; see docs/runbooks/account-csm-worker.md, "Handoff recaps".

A project the worker creates can wake the CRM field-sync worker and hand it Salesforce file uploads, so the worker may invoke that one function. It may invoke nothing else.

On a durable tier the login is or_app, read from SSM: host, port and name under database_ssm_prefix, and user and password under app_db_ssm_prefix. This is the same login the CRM field-sync worker uses, so its runbook's login fixes apply here too (docs/runbooks/crm-field-sync-worker.md, "Its database login"). A preview connects to its own Neon branch.

Reading the records ​

Every run writes two records to the function's log group, /onramp/<env>/lambda/onramp-<env>-journey-outbox-worker. The function logs in Lambda's JSON format, so each field is a top-level key.

text
fields @timestamp, event, outcome, advanced, consumed, left_pending, remaining, paused, oldest_pending_seconds, due_queued, due_remaining, oldest_due_seconds, duration_ms, error_category, build_id
| filter record = "journeys.recovery"
| sort @timestamp desc
| limit 50
FieldMeaning
eventrun.started, then run.finished
outcomesucceeded, out_of_time (the budget ran out with rows left), or failed (the run raised)
advancedRows whose transition created the next project
consumedRows handled with no transition (their phase did not meet its exit rule)
left_pendingRows that failed or were skipped this run. They stay pending for the next run
remainingRows still pending for vendors with journeys on, when the run ended
pausedRows pending for vendors with journeys off. They wait on purpose
oldest_pending_secondsAge of the oldest pending row of a vendor with journeys on. Absent when there is none
due_queuedDue phases this run queued a row for
due_remainingPhases of vendors with journeys on that are due but not queued yet, when the run ended
oldest_due_secondsHow long the oldest of those has been due. Absent when there is none. It is not the outbox backlog: a queued phase counts in remaining instead
error_categoryOn a failed run, the exception's type
build_idThe release id the zip carries, journey-outbox-worker-<12 hex>, a digest of the zip's content — the same one its Sentry events report

A row that fails is logged at ERROR with its id (journeys: outbox row <id> left pending) and the traceback. A batch of due phases that cannot be queued is logged at ERROR as journeys: could not queue due journey phases; those phases stay due for the next run. Filter the log group to $.level = "ERROR" to see them.

The alarms ​

Alarms exist only on tiers whose row in infra/app/envs.ts sets pages: true (prod and gdpr-prod today). Every tier still counts runs and backlog as metrics in OnRamp/JourneyRecovery/<env>.

Errors: two failed runs in a row

The function raised or timed out twice running. Read the run.finished record's error_category and the ERROR records. A WorkerDatabaseCredentialsUnavailable names the SSM parameter it could not read. A ModuleNotFoundError is a package the zip leaves out, which app/api/journeys/tests/test_worker_bundle.py should have caught. Fix the cause, then let the next tick run.

Heartbeat: no finished pass in 15 minutes

No run finished in three schedule periods. Either the schedule is not firing, the function cannot start, or every run fails. Check the schedule's TargetErrorCount and the function's Invocations and Errors. Re-running the infra deploy rebuilds the schedule, its role and the function together.

Backlog: oldest row over 30 minutes

A row of a vendor with journeys on has waited longer than journey_outbox_backlog_alarm_seconds, across two runs in a row. Either one row keeps failing, or passes run out_of_time and cannot keep up. If remaining is falling, wait. If one row id keeps appearing in the ERROR records, that row is the problem.

Due: oldest due phase over 30 minutes

A phase of a vendor with journeys on has been due for longer than journey_outbox_backlog_alarm_seconds without being queued, across two runs in a row. Its time-in-stage finish line passed, but it will not advance until a project changes. Either queueing due phases keeps failing (look for could not queue due journey phases in the ERROR records), or passes run out_of_time before reaching them. If due_remaining is falling, wait.

What does not page

Rows and due phases of vendors with customer_journeys off. Rows appear only in paused; due phases appear nowhere. Turning a vendor's flag off is a deliberate pause, not an incident.

Looking at the queue ​

The pending rows, in the order the next pass will try them:

sql
SELECT id, vendor_id, project_id, created_at, last_attempted_at
FROM or_journey_outbox
WHERE processed_at IS NULL AND kind = 'project_change'
ORDER BY COALESCE(last_attempted_at, created_at), id
LIMIT 50;

The phases due and not yet queued, oldest first:

sql
SELECT id, vendor_id, instance_id, entered_at, next_due_at
FROM or_journey_phase_instances
WHERE exited_at IS NULL AND next_due_at <= now()
ORDER BY next_due_at, id
LIMIT 50;

last_attempted_at is when a pass last tried the row. It is never an acknowledgment and never a lease: a row is done only when processed_at is set. A row whose last_attempted_at keeps moving while processed_at stays empty is failing every time; its error is in the worker's ERROR records.

A frozen claim ​

A pass holds a row inside a transaction while it runs the transition. If the process holding it freezes, PostgreSQL ends that transaction after journey_outbox_idle_in_transaction_timeout_ms (45 seconds by default), and a statement that runs longer than journey_outbox_statement_timeout_ms is cancelled. Either way the row's lock is released and the next pass takes it. Other rows keep moving meanwhile, because every pass skips rows that are locked.

To see a holder that has not been released yet:

sql
SELECT pid, usename, application_name, state, now() - xact_start AS open_for, left(query, 120) AS query
FROM pg_stat_activity
WHERE state LIKE 'idle in transaction%'
ORDER BY xact_start;

A holder older than the idle timeout should not exist. If one does, check that the session's role still carries the timeout (it is set per transaction with SET LOCAL). Ending the holder with pg_terminate_backend(<pid>) is safe: the transaction rolls back, the row stays pending, and the next pass retries it.

Running a pass by hand ​

A manual run is safe at any time, including beside a scheduled one:

bash
aws lambda invoke \
  --function-name onramp-<env>-journey-outbox-worker \
  --cli-binary-format raw-in-base64-out \
  --payload '{"job": "drain-journey-outbox"}' \
  /dev/stdout

The response is the same summary the run.finished record carries. Any other payload is refused.

Pausing ​

  • One vendor. Turn off that vendor's customer_journeys flag. Their rows wait and show up in paused, and their due phases are not queued. When the flag is turned back on, the next pass queues those phases and processes the rows, oldest attempt first. If the waiting rows are older than the backlog threshold and the first pass cannot drain them, the backlog alarm can fire until they clear.
  • The whole worker. Disable the schedule (aws scheduler update-schedule with --state DISABLED, or the console). This is an emergency step only: the schedule is owned by Pulumi, so the next infra deploy turns it back on. Rows keep queueing and the in-app wake still runs. When the schedule is back, the worker drains what is left.

Deploy notes ​

  • The worker deploys with the tier's infra, after the release's migrate gate. Its code moves only on a release run, the one that built the app image. On an infra dispatch the function keeps the code it already runs, and the deploy log says so (journey recovery worker: holding worker ...). A dispatch built from a newer commit could otherwise run code the database's schema does not support yet.
  • A worker is only ever created by a release whose migrate gate passed. An infra dispatch, or a hotfix promoted with the migrate gate skipped, on a tier with no worker yet creates none and says so in the log (journey recovery worker: not creating it this apply); the next release whose gate passes creates it. The deploy looks the worker up first and refuses the apply if it can't tell whether one exists.
  • A preview gets its own worker and schedule on its pr-<N>-pipeline stack. Deleting the stack removes both, and the nightly preview sweep removes any previews-pr-<N>-* schedule a stack left behind.

Rolling back ​

  • Roll back the authoring UI and the processor together. Do not put back a processor that does not understand native exit conditions while native rules are active: an old processor can consume a native row as a no-op, and that row will not be tried again.
  • Downgrading the database migration discards recorded exit evidence. It is not a routine rollback.
  • Pausing the worker or a vendor's feature does not lose work. Pending rows stay pending and are processed when the pause ends.

Internal documentation — gated behind Cloudflare Access.