Account CSM worker
How the account CSM rail is wired, what each alarm means, and what to do when accounts are not being reviewed.
Canonical sources. The handlers are app/api/account_csm/fanout_lambda.py and app/api/account_csm/worker_lambda.py; the queue message contract is app/api/account_csm/account_runs.py. The functions, roles, queues, schedule and alarms are infra/components/account-csm-worker.ts. Cadence, deadlines, memory, concurrency and the queue-age threshold are the account_csm_* keys in app/config/environments.yaml. The zip is built by infra/scripts/build-account-csm-worker-zip.ts from app/api/account_csm/worker_bundle.json.
How it runs
EventBridge Scheduler invokes the fan-out every account_csm_schedule (one hour by default) with {"job": "account-csm-fanout", "queue_url": "<the account queue>"}. The fan-out queues one message per account. Messages are grouped by vendor, so one vendor's accounts run one at a time, and deduplicated per account per tick, so a send retried inside the queue's window lands once. The worker takes one message per invocation, up to account_csm_max_concurrency at once.
Neither the scheduler nor Lambda retries a function: the next tick selects the accounts again. A message the worker fails three times is parked on onramp-<env>-account-csm-accounts-dlq.fifo for four days.
The worker connects with the worker database login (host, port, name under the tier's migrate prefix; user, password under its app_db prefix) and, on tiers that compose an account-context database, reads its connection and app_user login leaves under /onramp/<env>/account-context; both functions run on the executor's VPC placement, the same network path the account-context pipeline worker uses to reach that cluster. It reads the agent runtime registry under /onramp/<env>/agent-runtimes and may invoke only the tier's intel AgentCore runtime. The fan-out may send to the account queue and read the same account-context leaves, and nothing else.
Handoff recaps
A journey stage transition writes one kind = 'handoff_recap' row on or_journey_outbox, in the transition's transaction, for vendors with Aero account reviews on. Each tick the fan-out sends one {"job": "handoff-recaps", "vendor_id": N} message for every enabled vendor with a recap to take, ahead of that vendor's account runs. The worker then takes the vendor's pending recaps one at a time (up to five a run, oldest-tried first, skipping a row another run holds), and starts one only while a full Aero read still fits in the invocation. For each it resolves the incoming stage's lead (the role's holder, else the journey's owner), has the intel runtime write the recap from the outgoing stage's projects, tasks and comments that lead may see, and commits the in-app Message Center message with the row's processed_at. A recap is therefore delivered once, however often it is retried.
A recap stays pending, and is tried again 30 minutes after its last try, when:
- its stage's lead can't receive it (departed, inactive or unable to view accounts): the worker logs
is waiting for its stage's lead. Fix the role's holder or the journey's owner, and the next tick delivers it to them. - Aero failed or answered with something invalid: the worker logs
handoff recap <id> failedwith the cause.
SELECT id, vendor_id, created_at, last_attempted_at, payload->>'instance_id' AS instance_id
FROM or_journey_outbox
WHERE kind = 'handoff_recap' AND processed_at IS NULL
ORDER BY COALESCE(last_attempted_at, created_at), id
LIMIT 50;Each recap run logs one recaps.finished record with the vendor and a count per outcome (delivered, no_recipient, recipient_changed, taken, failed). Acknowledging a recap only sets the row's acknowledged_at; it never moves a journey.
Reading the records
Both functions log in Lambda's JSON format, so each field is a top-level key. The fan-out writes account_csm.fanout records and the worker writes account_csm.worker records, each to its own log group (/onramp/<env>/lambda/onramp-<env>-account-csm-{fanout,worker}).
fields @timestamp, record, event, outcome, queued, vendor_id, account_id, build_id
| filter record in ["account_csm.fanout", "account_csm.worker"]
| sort @timestamp desc
| limit 50The alarms
Alarms exist only on tiers whose row in infra/app/envs.ts sets pages: true (prod and gdpr-prod today). Every durable tier still counts finished ticks as RunCompleted in OnRamp/AccountCsm/<env>.
No fan-out tick finished in three schedule periods. Either the schedule is not firing, the function cannot start, or every run fails. Check the schedule's TargetErrorCount and the fan-out's Invocations and Errors. Re-running the infra deploy rebuilds the schedule, its role and the functions together.
The worker or the fan-out failed or was throttled in a period. Filter the function's log group to $.level = "ERROR". Nothing is lost: the next tick selects the accounts again.
An account run failed three deliveries in a row. The parked message's body names the vendor and account. Fix the cause, then purge the dead-letter queue: nothing needs redriving, because the next tick sends the account again.
A run has waited over account_csm_queue_age_alarm_seconds without the worker taking it. Check that the queue's event source mapping is enabled and that the worker is not throttled or failing. Re-enable the mapping if it was switched off.
Scheduler cannot invoke the fan-out. Check that the schedule still targets a function that exists and that its role may invoke it. Re-running the deploy rebuilds both.
Safe recovery
Everything on this rail is re-selected by the next tick, so recovery is: fix the cause, purge the dead-letter queue if it holds messages, and let the next tick run. A manual run is safe at any time:
aws lambda invoke \
--function-name onramp-<env>-account-csm-fanout \
--cli-binary-format raw-in-base64-out \
--payload '{"job": "account-csm-fanout", "queue_url": "<the account queue url>"}' \
/dev/stdoutPausing
- One vendor. Turn off that vendor's
aero-account-csmflag. - The whole rail. Disable the schedule (
aws scheduler update-schedulewith--state DISABLED, or the console). This is an emergency step only: Pulumi owns the schedule, so the next infra deploy turns it back on.
Deploy notes
- The functions' code moves only on a release run, the one that built the app image. On an infra dispatch they keep the code they already run.
- A preview gets its own rail on its
pr-<N>-pipelinestack. Deleting the stack removes it, and the nightly preview sweep removes anypreviews-pr-<N>-*schedule a stack left behind.