Skip to content

Acker receiver cutover ​

Move each tier's CRM webhooks from the Python receiver to the Node Web Function one tier at a time, and move a tier back if it misbehaves.

Owner: Integrations pipelineWhen to use: the release that adds the Node receiver, and any later flip or rollback

Canonical sources. infra/scripts/pipeline/acker-receiver.ts (run it with --help), .github/workflows/acker-receiver.yml, and the edge switch in infra/components/acker-switch.ts. Pipeline-wide behaviour (shadow mode, the brake, the dashboard) is in Integrations pipeline: shadow mode, comparison, brake.

TL;DR ​

🅰 The release changes nothing for senders

Both receivers ship and the switch is on Python. Each tier's events hostname keeps answering from the Python Lambda until you flip it.

🅱 One command moves a tier

use node and use python, with no AWS credentials. The command runs a GitHub workflow that holds the tier's AWS access and prints the result. use python is the rollback.

🅲 Dropping events is a separate switch

The event filter only counts until a customer's flag is on. Compare its counts with executor results before turning any flag on.

Before you merge ​

Apply the CI identity stack from this branch, following the ci-identity-apply skill. The release adds a role per tier, onramp-deploy-<tier>-acker-web, that the infra apply and the acker-receiver workflow both assume. Until it is applied the Node receiver cannot be created and the workflow cannot run. Do not merge while Declared deploy-role boundaries are applied is red.

What the release deploys ​

On every durable tier (stage, prod, gdpr-prod):

  • The Node receiver, a Lambda Web Function named onramp-<tier>-pipeline-acker-web, beside the Python receiver onramp-<tier>-pipeline-acker. Both stay deployed.
  • A second origin on the tier's acker CloudFront distribution, plus a CloudFront Function and a key value store, both named onramp-<tier>-pipeline-acker-switch. The function reads the key receiver on each request: node sends the request to the Node receiver; any other value, or any error, leaves it on the Python receiver. The key starts as python and no later deploy resets it.
  • A job in the CRM field-sync worker that rebuilds, every minute, the list of HubSpot object types each portal uses, and writes it to the bucket onramp-<tier>-pipeline-interest. Only the Node receiver reads it.

Previews have no CloudFront. Their smoke test calls the Node receiver directly.

Check where a tier stands ​

bash
bun infra/scripts/pipeline/acker-receiver.ts status --env prod

The command dispatches acker-receiver.yml on main, follows the run with gh run watch, and prints the job's report. It needs gh signed in and nothing else. It accepts stage, prod and gdpr-prod.

The report shows:

  • switch: the value of the key receiver.
  • answering: which receiver answers GET /health on the tier's public hostname. Only the Node receiver reports "runtime":"node". The two disagree for a few seconds after a flip and no longer.
  • a table of the last 15 minutes in 5-minute buckets, newest last and still filling: CloudFront requests and 4xx/5xx rates, Node requests and 5xx, Python invocations and errors, and messages sent to the pipeline queue. A bucket with no data point reads 0 for a count and - for a rate. n/a means the metric could not be read, and a note under the table says why.

If status fails because the key value store or the distribution is missing, the release's infra apply has not landed on that tier. Look at its deploy-infra run.

Flip order ​

OrderTierWhen
1stageAfter the merge, once the release's infra apply is green on it. Clean before you promote to production
2prodAfter the production promotion, once its infra apply is green
3gdpr-prod30 minutes after prod, if prod is clean

One tier at a time. Run status first; the switch should read python. Then:

bash
bun infra/scripts/pipeline/acker-receiver.ts use node --env stage

The command prints receiver python -> node, waits up to 90 seconds for /health to answer as node, and prints the same report as status. If /health still answers as python after that, the run fails. The key is already written, so run status once more and roll back if the two still disagree.

What to watch after a flip ​

Watch for 30 minutes. Run status as often as you like; every run is a fresh read.

SignalWhereHealthy
5xx on the new receiverthe Node web function 5xx row, and hubspot_enqueue_failed lines in the Node log group0
CloudFront errors to HubSpotthe CloudFront 5xx rate and CloudFront 4xx rate rowsAt or below the numbers before the flip
Request counts matchCloudFront requests against Node web function requests, and Python lambda invocationsThe first two track each other and the Python row falls to about 0
Queue sendsthe SQS messages sent row. For the same hour on earlier days, the Queue widget on the onramp-<tier>-pipeline-shadow-bake dashboard with the time range widenedInside the range of earlier days at that hour
Executor resultsthe Executor and DLQ widgets on the same dashboardErrors flat and the DLQ empty
Salesforce ackssalesforce_delivery log lines (query below), and the Outbound Messages monitor in Setup for the orgs wired to the tierAn ack on every delivery and no growing retry queue
Cold starts and latencythe first requests after a quiet period in the Node log group, against the Python receiver's log groupNo slow tail on those first requests

Log groups, in the tier's home region (eu-west-1 for gdpr-prod): the Node receiver writes to /aws/lambda/web/onramp-<tier>-pipeline-acker-web and the Python receiver to /onramp/<tier>/lambda/pipeline-acker.

fields @timestamp, org, trigger, notifications, envelopes_enqueued, ack
| filter msg = "salesforce_delivery"
| sort @timestamp desc
| limit 50

Rolling back ​

Roll a tier back when any row above looks wrong and a few minutes of looking has not explained it:

bash
bun infra/scripts/pipeline/acker-receiver.ts use python --env prod

The run waits until /health answers as python. Envelopes the Node receiver already put on the queue stay there and the executor processes them. Senders retry deliveries that were not acknowledged. The Python receiver stays deployed throughout, so a rollback is never a deploy.

If GitHub Actions is unavailable, an AWS admin sets the key receiver to python in the CloudFront console (Functions, KeyValueStores, onramp-<tier>-pipeline-acker-switch).

Turning the event filter's dropping on ​

HubSpot sends every subscribed event for every portal. The Node receiver compares each event's object type with the portal's entry in the interest list and counts the events no customer behind that portal uses. It drops them only for a portal whose enforce setting is on, and it keeps everything when it has no usable list (missing, unreadable, or older than 15 minutes), when the portal is not on the list, and when an event carries no object type. Salesforce is never filtered.

A dropped event is gone. It is never queued, so turning the flag off later does not replay it. Check the counts first.

The setting is the crm_event_filter_enforce feature flag, per organization, off for everyone until you turn it on. A portal enforces only when every organization connected to it has the flag on. Per customer:

  1. Read what the filter would drop for the customer's portal, over a period that covers its normal activity. Replace PORTAL_ID with the HubSpot portal id.

    fields portal, events_received, events_would_drop, interest
    | filter msg = "hubspot_delivery" and portal = "PORTAL_ID"
    | stats sum(events_received) as received, sum(events_would_drop) as would_drop by interest

    Only interest = listed can produce a would-drop count. The other values are portal_not_listed, no_list, expired_list and unreadable_list, and each keeps every event.

  2. Compare with executor results. Every event the filter would drop should be one the executor already finishes without acting on: the no_candidates outcome, on the dashboard's "Dropped before evaluation" widget. If the executor created executions or field-sync writes from an object type the filter would drop, the interest list is missing something for that portal. Leave the flag off and fix the list.

  3. In Admin, Feature gates, turn crm_event_filter_enforce on for the customer's organization. The list is rebuilt every minute, so the change reaches the receiver within a few minutes.

  4. Confirm that enforced is true and events_dropped.filtered is above 0 on new deliveries.

    fields @timestamp, portal, events_received, events_would_drop, events_dropped.filtered, enforced
    | filter msg = "hubspot_delivery" and portal = "PORTAL_ID"
    | sort @timestamp desc
    | limit 20

To stop, turn the flag off for that organization. Dropping ends after the next list rebuild.

Refusing deliveries that skip the edge ​

Both receivers' origins, the Python Function URL and the Node Web Function endpoint, take unauthenticated calls, so a caller who finds either can go around the events hostname and its WAF. The edge sends both receivers a secret only it and they hold, as the x-onramp-edge-secret origin header; Pulumi generates it and seeds it at /onramp/<tier>/pipeline/edge_secret. A receiver refuses a delivery without it only while /onramp/<tier>/pipeline/edge_secret_enforcement holds enforce. No deploy writes that parameter, and each receiver re-reads it every 30 seconds.

Until then, a receiver lets such a delivery through and logs edge_secret_unverified with the reason (absent, mismatch or unverifiable), the method, the path's first segment and the user agent. The EdgeSecretUnverified metric in OnRamp/IntegrationsPipeline/<tier> counts 1 for each of those lines from either receiver and 0 for every other line they log.

Per tier, stage first:

  1. Confirm that nothing legitimate reaches a receiver without the secret. Over at least a day of normal traffic, the metric's Sum must be 0 while its SampleCount is above 0, which shows traffic flowed. If the Sum is above 0, see who sent those deliveries in /onramp/<tier>/lambda/pipeline-acker and /aws/lambda/web/onramp-<tier>-pipeline-acker-web:

    fields @timestamp, @message
    | filter @message like /edge_secret_unverified/
    | sort @timestamp desc

    A scanner or a stray request can be refused. A CRM sender, a smoke test or another OnRamp service cannot: point it at the events hostname first.

  2. As an AWS admin, write enforce as a String to /onramp/<tier>/pipeline/edge_secret_enforcement in the tier's region.

  3. Within a minute, a POST straight to either origin answers 403 Forbidden, and GET https://<events hostname>/health still answers 200. The distribution's acker-function-url and acker-web-function origins name the two origin hostnames.

To stop enforcing, write observe or delete the parameter. A receiver that has never read the parameters, such as a new one started while SSM is unreachable, does not enforce until a read succeeds.

Follow-ups ​

  • Remove the Python receiver a week after the last tier is on Node: its Lambda, its CloudFront origin and the switch. use python is no longer a rollback after that.
  • Once every tier enforces the edge secret, make enforcing unconditional and retire the enforcement parameter.

Internal documentation — gated behind Cloudflare Access.