Acker receiver cutover
Move each tier's CRM webhooks from the Python receiver to the Node Web Function one tier at a time, and move a tier back if it misbehaves.
Canonical sources. infra/scripts/pipeline/acker-receiver.ts (run it with --help), .github/workflows/acker-receiver.yml, and the edge switch in infra/components/acker-switch.ts. Pipeline-wide behaviour (shadow mode, the brake, the dashboard) is in Integrations pipeline: shadow mode, comparison, brake.
TL;DR
Both receivers ship and the switch is on Python. Each tier's events hostname keeps answering from the Python Lambda until you flip it.
use node and use python, with no AWS credentials. The command runs a GitHub workflow that holds the tier's AWS access and prints the result. use python is the rollback.
The event filter only counts until a customer's flag is on. Compare its counts with executor results before turning any flag on.
Before you merge
Apply the CI identity stack from this branch, following the ci-identity-apply skill. The release adds a role per tier, onramp-deploy-<tier>-acker-web, that the infra apply and the acker-receiver workflow both assume. Until it is applied the Node receiver cannot be created and the workflow cannot run. Do not merge while Declared deploy-role boundaries are applied is red.
What the release deploys
On every durable tier (stage, prod, gdpr-prod):
- The Node receiver, a Lambda Web Function named
onramp-<tier>-pipeline-acker-web, beside the Python receiveronramp-<tier>-pipeline-acker. Both stay deployed. - A second origin on the tier's acker CloudFront distribution, plus a CloudFront Function and a key value store, both named
onramp-<tier>-pipeline-acker-switch. The function reads the keyreceiveron each request:nodesends the request to the Node receiver; any other value, or any error, leaves it on the Python receiver. The key starts aspythonand no later deploy resets it. - A job in the CRM field-sync worker that rebuilds, every minute, the list of HubSpot object types each portal uses, and writes it to the bucket
onramp-<tier>-pipeline-interest. Only the Node receiver reads it.
Previews have no CloudFront. Their smoke test calls the Node receiver directly.
Check where a tier stands
bun infra/scripts/pipeline/acker-receiver.ts status --env prodThe command dispatches acker-receiver.yml on main, follows the run with gh run watch, and prints the job's report. It needs gh signed in and nothing else. It accepts stage, prod and gdpr-prod.
The report shows:
- switch: the value of the key
receiver. - answering: which receiver answers
GET /healthon the tier's public hostname. Only the Node receiver reports"runtime":"node". The two disagree for a few seconds after a flip and no longer. - a table of the last 15 minutes in 5-minute buckets, newest last and still filling: CloudFront requests and 4xx/5xx rates, Node requests and 5xx, Python invocations and errors, and messages sent to the pipeline queue. A bucket with no data point reads
0for a count and-for a rate.n/ameans the metric could not be read, and a note under the table says why.
If status fails because the key value store or the distribution is missing, the release's infra apply has not landed on that tier. Look at its deploy-infra run.
Flip order
| Order | Tier | When |
|---|---|---|
| 1 | stage | After the merge, once the release's infra apply is green on it. Clean before you promote to production |
| 2 | prod | After the production promotion, once its infra apply is green |
| 3 | gdpr-prod | 30 minutes after prod, if prod is clean |
One tier at a time. Run status first; the switch should read python. Then:
bun infra/scripts/pipeline/acker-receiver.ts use node --env stageThe command prints receiver python -> node, waits up to 90 seconds for /health to answer as node, and prints the same report as status. If /health still answers as python after that, the run fails. The key is already written, so run status once more and roll back if the two still disagree.
What to watch after a flip
Watch for 30 minutes. Run status as often as you like; every run is a fresh read.
| Signal | Where | Healthy |
|---|---|---|
| 5xx on the new receiver | the Node web function 5xx row, and hubspot_enqueue_failed lines in the Node log group | 0 |
| CloudFront errors to HubSpot | the CloudFront 5xx rate and CloudFront 4xx rate rows | At or below the numbers before the flip |
| Request counts match | CloudFront requests against Node web function requests, and Python lambda invocations | The first two track each other and the Python row falls to about 0 |
| Queue sends | the SQS messages sent row. For the same hour on earlier days, the Queue widget on the onramp-<tier>-pipeline-shadow-bake dashboard with the time range widened | Inside the range of earlier days at that hour |
| Executor results | the Executor and DLQ widgets on the same dashboard | Errors flat and the DLQ empty |
| Salesforce acks | salesforce_delivery log lines (query below), and the Outbound Messages monitor in Setup for the orgs wired to the tier | An ack on every delivery and no growing retry queue |
| Cold starts and latency | the first requests after a quiet period in the Node log group, against the Python receiver's log group | No slow tail on those first requests |
Log groups, in the tier's home region (eu-west-1 for gdpr-prod): the Node receiver writes to /aws/lambda/web/onramp-<tier>-pipeline-acker-web and the Python receiver to /onramp/<tier>/lambda/pipeline-acker.
fields @timestamp, org, trigger, notifications, envelopes_enqueued, ack
| filter msg = "salesforce_delivery"
| sort @timestamp desc
| limit 50Rolling back
Roll a tier back when any row above looks wrong and a few minutes of looking has not explained it:
bun infra/scripts/pipeline/acker-receiver.ts use python --env prodThe run waits until /health answers as python. Envelopes the Node receiver already put on the queue stay there and the executor processes them. Senders retry deliveries that were not acknowledged. The Python receiver stays deployed throughout, so a rollback is never a deploy.
If GitHub Actions is unavailable, an AWS admin sets the key receiver to python in the CloudFront console (Functions, KeyValueStores, onramp-<tier>-pipeline-acker-switch).
Turning the event filter's dropping on
HubSpot sends every subscribed event for every portal. The Node receiver compares each event's object type with the portal's entry in the interest list and counts the events no customer behind that portal uses. It drops them only for a portal whose enforce setting is on, and it keeps everything when it has no usable list (missing, unreadable, or older than 15 minutes), when the portal is not on the list, and when an event carries no object type. Salesforce is never filtered.
A dropped event is gone. It is never queued, so turning the flag off later does not replay it. Check the counts first.
The setting is the crm_event_filter_enforce feature flag, per organization, off for everyone until you turn it on. A portal enforces only when every organization connected to it has the flag on. Per customer:
Read what the filter would drop for the customer's portal, over a period that covers its normal activity. Replace
PORTAL_IDwith the HubSpot portal id.fields portal, events_received, events_would_drop, interest | filter msg = "hubspot_delivery" and portal = "PORTAL_ID" | stats sum(events_received) as received, sum(events_would_drop) as would_drop by interestOnly
interest=listedcan produce a would-drop count. The other values areportal_not_listed,no_list,expired_listandunreadable_list, and each keeps every event.Compare with executor results. Every event the filter would drop should be one the executor already finishes without acting on: the
no_candidatesoutcome, on the dashboard's "Dropped before evaluation" widget. If the executor created executions or field-sync writes from an object type the filter would drop, the interest list is missing something for that portal. Leave the flag off and fix the list.In Admin, Feature gates, turn
crm_event_filter_enforceon for the customer's organization. The list is rebuilt every minute, so the change reaches the receiver within a few minutes.Confirm that
enforcedis true andevents_dropped.filteredis above 0 on new deliveries.fields @timestamp, portal, events_received, events_would_drop, events_dropped.filtered, enforced | filter msg = "hubspot_delivery" and portal = "PORTAL_ID" | sort @timestamp desc | limit 20
To stop, turn the flag off for that organization. Dropping ends after the next list rebuild.
Refusing deliveries that skip the edge
Both receivers' origins, the Python Function URL and the Node Web Function endpoint, take unauthenticated calls, so a caller who finds either can go around the events hostname and its WAF. The edge sends both receivers a secret only it and they hold, as the x-onramp-edge-secret origin header; Pulumi generates it and seeds it at /onramp/<tier>/pipeline/edge_secret. A receiver refuses a delivery without it only while /onramp/<tier>/pipeline/edge_secret_enforcement holds enforce. No deploy writes that parameter, and each receiver re-reads it every 30 seconds.
Until then, a receiver lets such a delivery through and logs edge_secret_unverified with the reason (absent, mismatch or unverifiable), the method, the path's first segment and the user agent. The EdgeSecretUnverified metric in OnRamp/IntegrationsPipeline/<tier> counts 1 for each of those lines from either receiver and 0 for every other line they log.
Per tier, stage first:
Confirm that nothing legitimate reaches a receiver without the secret. Over at least a day of normal traffic, the metric's Sum must be 0 while its SampleCount is above 0, which shows traffic flowed. If the Sum is above 0, see who sent those deliveries in
/onramp/<tier>/lambda/pipeline-ackerand/aws/lambda/web/onramp-<tier>-pipeline-acker-web:fields @timestamp, @message | filter @message like /edge_secret_unverified/ | sort @timestamp descA scanner or a stray request can be refused. A CRM sender, a smoke test or another OnRamp service cannot: point it at the events hostname first.
As an AWS admin, write
enforceas a String to/onramp/<tier>/pipeline/edge_secret_enforcementin the tier's region.Within a minute, a POST straight to either origin answers 403
Forbidden, andGET https://<events hostname>/healthstill answers 200. The distribution'sacker-function-urlandacker-web-functionorigins name the two origin hostnames.
To stop enforcing, write observe or delete the parameter. A receiver that has never read the parameters, such as a new one started while SSM is unreachable, does not enforce until a read succeeds.
Follow-ups
- Remove the Python receiver a week after the last tier is on Node: its Lambda, its CloudFront origin and the switch.
use pythonis no longer a rollback after that. - Once every tier enforces the edge secret, make enforcing unconditional and retire the enforcement parameter.