CRM field-sync worker
The worker runs the three field-sync rails, rebuilds the HubSpot interest list the integrations acker filters by, and delivers the values a save queued. The outbound rail's and the interest list's schedules invoke it directly. The poll's and the rotation's schedules invoke the fan-out instead, which reads the tenants the rail runs for and sends one message per tenant to the tenant queue; the worker takes those off the queue one tenant per invocation, so a slow tenant spends only its own invocation. The fan-out reads Postgres and sends to that queue, and reaches no CRM.
Both are plain handlers on the Python 3.12 arm64 managed runtime, from the same zip, with no Flask application at all: app/api/crm_field_sync/runtime.py builds their engine from the same config class the web app is built from and binds one session per invocation. Anything on its path that reaches for the web app raises create_app() was called in a process already serving the crm-field-sync worker rather than booting one — that message in its log is a code defect on the path it names, not a configuration fault.
It ships as a zip built by infra/scripts/build-crm-field-sync-worker-zip.ts and loaded from the tier's onramp-<env>-crm-field-sync-code bucket. The zip carries the monolith's locked production dependencies less the packages app/api/crm_field_sync/worker_bundle.json names; a ModuleNotFoundError in the log is the worker reaching one of those, and app/api/crm_field_sync/tests/test_worker_bundle.py is the test that should have caught it.
Outbound — every 5 minutes, plus a wake after each committed save. Sends the current desired value for each CRM destination.
Poll — every 15 minutes, one run per tenant. Asks each mapped CRM object what changed since this tenant's watermark. An object with more changes than one pass reads is read again in the same run, while the run has time.
Rotation — nightly, one run per tenant. Re-reads linked records whether or not they changed, which is the only way a formula or rollup field ever moves.
What a customer sees when it is down
Nothing in the app errors. Values simply stop moving: a field saved in OnRamp stays unsent, and a field changed in the CRM stays unapplied. Nothing is lost — outbound writes stay PENDING in or_crm_field_sync_record_state, and each inbound rail resumes from its own cursor — so recovery is catching up, never reconstructing.
Which rail is broken
SELECT job_name, status, started_at, finished_at, totals, error
FROM or_crm_field_sync_job_run
ORDER BY id DESC
LIMIT 20;Every tick writes a row here — the worker's for outbound, the fan-out's for the poll and the rotation — including one that finds no enabled vendors, so an absent heartbeat means the schedule did not fire, a different fault from a run that failed. A fan-out row's totals say how many tenants it found and how many runs it queued. One tenant's own poll or rotation writes no row: its event=tenant_run.finished record in the worker's log group names the tenant, the rail and the outcome.
The rails are safe to run twice. Outbound claims its rows with FOR UPDATE SKIP LOCKED, so a manual invoke alongside a scheduled tick cannot double-send. The poll and the rotation run once per tenant at a time: a run that finds its tenant's poll or rotation already running reads nothing and reports tenant_busy.
Its database login
On a durable tier the worker logs in as or_app, the web app's own login. It reads host, port and name from the tier's database_ssm_prefix, and user and password from its app_db_ssm_prefix (/onramp/<tier>/app_db/<target>), both in app/config/environments.yaml. Those two values are a copy of the Flask service's own login, written by devtools/seed-app-db-creds.ts. Never change or_app's password: every tenant's Flask service connects with it.
A worker log full of password authentication failed for user "or_app" means the copy is stale, because the Flask service's password changed. Re-run the script for that target:
AWS_PROFILE=admin bun devtools/seed-app-db-creds.ts --apply --interactive <target>WorkerDatabaseCredentialsUnavailable means the worker could not read one of those parameters, or its tier has no app_db_ssm_prefix. The message names the parameter. The worker does not start without them, and the next invocation tries again.
The worker runs across every vendor with the ALL_VENDORS tenant scope, the bypass that or_app's row-level-security policies honour.
Following one run across the Worker and the CRM Connector
A scheduled run's records carry run_id, the id of its row in or_crm_field_sync_job_run. Every CRM Connector call the run makes carries the same id, so the Connector's own event=crm.request caller=crm-field-sync records join the Worker's records on run_id, rail and vendor_id. A Connector record also gives the call's duration_ms, the CRM's own provider_status_code and an error_category. The Event Processor's field-sync step writes event=run.finished rail=event to the pipeline executor's log group. None of these records carries a field value, a payload or a response body.
Dashboard
onramp-<env>-crm-field-sync, from top to bottom:
- Scheduler and Worker. Did each schedule fire, and are invocations failing, throttling or near their limits?
- Rails. Completions for each scheduled rail, the event rail's completions and failures, and the lagging-cursor and stuck-write counts.
- Outbound queue. Rows waiting, waiting after a failed attempt, terminally failed, and past the recovery window, plus the age of the oldest waiting write. The Worker publishes these on each unscoped recovery pass, every 5 minutes.
- CRM Connector, field sync's calls. Latency (p50, p95, max) and counts of requests, failures, rate limiting and upstream 5xx, all read from the Connector's own metrics under
Caller=crm-field-sync.
The cadence each rail runs on is its <job name>_schedule key in app/config/environments.yaml, for example crm_field_sync_poll_changes_schedule.
Alarms
| Alarm | What it means | Safe recovery |
|---|---|---|
…-crm-field-sync-worker-errors | Invocations are failing. | None for a transient failure — the next tick retries. If it persists, turn off the vendors' crm_field_sync flag while it is fixed. |
…-crm-field-sync-worker-throttles | Invocations are being dropped, not delayed. | Reserve concurrency for the function. |
…-crm-field-sync-fanout-errors | Poll or rotation ticks are failing, so some or all tenants are not being queued. The tick's or_crm_field_sync_job_run row says how many runs it found and how many it could not queue. | None for a transient failure — the next tick re-reads every tenant. A refused send usually means the tenant queue or the fan-out's grant on it is gone; re-run the deploy. |
…-crm-field-sync-fanout-throttles | A poll or rotation tick was dropped. | Reserve concurrency for the fan-out; one execution is enough. |
…-crm-field-sync-<rail>-delivery | Scheduler cannot invoke the rail's function — the worker for outbound, the fan-out for the poll and the rotation — so that rail is not running. | Re-run the deploy, which rebuilds the schedule, its role and the grant together. |
…-crm-field-sync-<rail>-heartbeat | The rail's tick has not completed in three of its own intervals. For the poll and the rotation that is the fan-out's tick; a tenant's own run failing shows on the two tenant-run alarms instead. Unlike the others, silence fires this one. | Invoke the function the rail's schedule targets with the schedule's own input: aws scheduler get-schedule --name <env>-<job name> --query Target. |
…-crm-field-sync-tenant-runs-waiting | A tenant's run has waited three of the most frequent fanned-out rail's intervals on the tenant queue without the worker taking it. | Re-enable the queue's event source mapping if it was switched off. If the worker is healthy and has more tenants than it runs at once, raise tenantConcurrency on the tier's row in infra/app/envs.ts. |
…-crm-field-sync-tenant-runs-dead-lettered | A tenant's run failed every delivery and was parked on the tenant queue's dead-letter queue. | Nothing needs redriving — the next tick sends that tenant again. Read the parked bodies and the worker's level=ERROR records, fix the cause, then purge the dead-letter queue. |
…-crm-field-sync-poll-lagging | A tenant's watermark is further behind the CRM than this environment allows. | Run that tenant's poll from the admin panel. If the lag does not fall, the CRM is refusing the change query. |
…-crm-field-sync-outbound-stuck | A saved value has been queued past the recovery window. | Nothing, if attempts is still climbing — it is backing off. If not, read the heartbeat alarm: the rail is not claiming. |
…-crm-field-sync-crm-unavailable | Sustained rate limiting or upstream failure from the CRM itself. | Nothing — the rail backs off on the CRM's own Retry-After. If one tenant is the whole of it, turn their flag off to stop spending their API budget. |
The last three count records the worker emits about itself, so their thresholds are crm_field_sync_cursor_lag_alarm_seconds and crm_field_sync_outbound_stuck_alarm_seconds in app/config/environments.yaml, not settings on the alarm. An environment that sets neither opts out of both.
Before enabling a canary vendor
No tier's field-sync alarms exist until its worker row in infra/app/envs.ts sets pages: true, so a tier's first release pages nobody while its schedules run ahead of migrate. An alarm only counts as coverage once a page from it has been seen arriving. On prod and gdpr-prod, before the first canary vendor there is enabled:
Confirm the tier's worker has completed a run of every rail since its release: each rail's
RunCompletedcount on the field-sync dashboard is above zero, including the nightly rail after 03:00 UTC.Set
pages: trueon that tier's worker row and deploy it. The deploy creates the alarms.Force one alarm into
ALARMwith a reason that names the test:bashaws cloudwatch set-alarm-state --alarm-name onramp-prod-crm-field-sync-worker-errors --state-value ALARM --state-reason "synthetic delivery check"gdpr-prod's alarms are in
eu-west-1, so add--region eu-west-1there.Confirm the notification arrives in the tier's pipeline Slack channel, delivered through SNS and Amazon Q.
Nothing needs resetting. The alarm returns to
OKon its next evaluation, and that transition notifies the channel too.Record the tier, the time and a link to the Slack message in the PR or rollout ticket that enables the canary vendor.
Stopping it
The rollout gate is the crm_field_sync feature flag in the features / vendor_features tables, read by the worker, the event processor and the save path alike. Turning it off for a vendor stops every rail for that vendor without touching infrastructure; turning off the global default stops it for every vendor without an explicit override.
Disabling the flag does not drain what is already queued. Outbound rows stay PENDING and are deferred on each pass until the vendor is enabled again — which is deliberate, so a pause does not discard a customer's edits.
To stop the worker taking tenants' runs off its tenant queue without touching any vendor's flag, switch its event source mapping off. Runs already sent stay on the queue until it is switched back on, or until they age out:
UUID=$(aws lambda list-event-source-mappings --function-name onramp-prod-crm-field-sync-worker --query 'EventSourceMappings[0].UUID' --output text)
aws lambda update-event-source-mapping --uuid "$UUID" --no-enabledWhere it is composed
infra/components/crm-field-sync-worker.ts, as a Tier-1 member on the env's crmFieldSyncWorker row in infra/app/envs.ts. Beside the worker it builds the fan-out function, the tenant queue (onramp-<env>-crm-field-sync-tenants.fifo) the fan-out alone may send to, its dead-letter queue, and the event source mapping that hands the worker one tenant's run per invocation, at most tenantConcurrency at once. An environment composing it also names the function in app/config/environments.yaml, which is what turns a save's post-commit wake on; one that does not sends inline on the request thread instead, and none of its rails gets a schedule at all.