Integrations pipeline — shadow mode, comparison, brake

The new event pipeline (acker → SQS FIFO → executor) runs beside the legacy Prismatic/JS path with zero customer impact until the cutover: legacy keeps processing every event, and the executor only records what it would have done. This runbook covers reading those observations, running the comparator, and the emergency brake. Design rationale (flag placement, flip-back semantics, graft topology) lives in the "Shadow Mode & Cutover" decision doc.

Known: the preview "Deploy preview (pipeline)" smoke check has been red fleet-wide since #11310 (merged 2026-08-24). Before that PR, a hydration read that came back 404/502 was logged and treated as "fields absent" — the claim proceeded anyway. _require_success now admits only a 200, so the SAME 404/502 raises HydrationUnavailable instead, the executor never claims, and SQS keeps redelivering. The preview fixture vendor behind the pre-claim full hydrate has no live CRM connection (see docs/runbooks/preview-crm-backends.md for the two connections that are wired — this fixture isn't one of them), so on every preview this now hard-fails instead of degrading. The PR's own "Known follow-ups" called this out ("The preview pipeline smoke check is red, and cannot go green from this side") but it wasn't written down anywhere a future PR would see it.

Signature in infra/scripts/smoke/preview-pipeline-e2e.ts output — the same four checks, every time, regardless of what the PR touches:

FAIL  execution reaches a terminal state
FAIL  a non-matching trigger condition creates no execution   (inconclusive)
FAIL  re-delivering the identical event still yields exactly one execution
FAIL  an oversized event is parked in S3 and still creates its project

while the VPC/ENI checks and the direct-address Salesforce Outbound Message checks (a different code path, not gated on this hydrate) keep passing. If your PR's preview shows exactly this shape, it is not your regression — don't chase it. It stops being fleet-wide, and is worth root-causing on its own PR, the moment a run shows a different smoke failure shape or the VPC/ Salesforce checks start failing too. The real fix is wiring the pre-claim hydrate's fixture vendor to a live preview CRM connection (the same shape as preview-crm-backends.md's two pins) rather than reverting _require_success — the strict status handling is correct per #11310's own reasoning; the fixture is what's incomplete.

The lever — bun run pipeline

Move a tier with one command, and it takes no arguments. Everything the rest of this runbook describes as a sequence — the two feature rows, the HubSpot webhook target, every live Salesforce ramp's Outbound Message, the rollout parameter — is performed by the cutover CLI, in the order whose consequences make it an order.

bun run pipeline

It opens on a board of every tier: where each provider sits, which layer that came from, and whether the tier's own acker agrees. From there it offers three things — move a tier, look at one, re-read them all — and each choice states its cost as you make it. There is nothing to memorise and no flag to discover.

To type pipeline from anywhere instead, once:

cd devtools/pipeline-cutover && bun link && bun link @onramp/pipeline-cutover

Friction is proportional, and deliberately not symmetric. A customer-facing tier (demo, prod, gdpr-prod), or any move to live on any tier, asks you to type the tier's name. Everything else is one y. A rollback to legacy is always one y, on every tier — a brake that is harder to pull than the thing it undoes is a worse brake.

There is deliberately no sweep command and no setup step. A condition the toggle can reach is an effect inside it; a condition it must not reach — either ingress origin, the HubSpot app id — is a reviewed, deployed value, so it is a BLOCKER naming the config row rather than something the lever edits.

The sections below are the mechanism: what the switch means, what each side effect does to a provider, and the manual commands the CLI drives. Read them to understand or to diagnose a tier; run the CLI to move one.

The switch

A write requires a live stamp; everything else is observe-only.

Switch	Home	Semantics
Ownership stamp (EventEnvelope.mode)	derived from rollout_state — see below	Stamped per event at ingress, per provider. Decides which pipeline owns the event. Never re-decided at process time. Missing stamp ⇒ shadow.

Current values: every durable environment (dev, stage, demo, prod, gdpr-prod) stamps shadow at ingress — dev and stage run the same legacy pipeline prod does, so they walk the same shadow → flip motion, with stage as the dress rehearsal. The only live surfaces are the two with no legacy to shadow: previews (greenfield; the e2e smoke asserts real writes) and local workstations (the test suite writes). A durable env goes live by moving rollout_state and nothing else — a runtime lever, no accompanying deploy; see the next section.

A second per-environment write ceiling used to sit above the stamp and was removed: it moved by reviewed deploy while rollout_state moves at runtime, so the two drifted into the one state where the forward had stopped and the executor still could not write. It also carried the executor's default_transaction_read_only=on, so shadow containment is now Python-level — the dry-run harness's rollback, egress gate, and leak re-query — rather than Postgres-enforced.

Moving a tier — the rollout_state lever

rollout_state has two homes, and which one wins is the point:

Layer	Where	Role
Shipped value	integrations-acker/environments.yaml	The reviewed floor. Travels in the deploy zip, changes by PR.
Runtime override	SSM /onramp/<env>/pipeline/rollout_state/<provider>	What an operator moves to flip a tier without a deploy. Read with a 30s TTL.

It is per provider — hubspot and salesforce move independently, because their rollback costs are not comparable (one provider-side write vs one CRM write per ramp inside each customer's org). A bare string in the YAML is shorthand for both.

aws ssm put-parameter --name /onramp/<env>/pipeline/rollout_state/<provider> \
  --type String --value shadow --overwrite --region us-west-2

Takes effect within the TTL — no deploy, no container action. The same command with --value legacy is the rollback, and it is equally fast.

Absent is the normal case and means "no override, use the shipped value". Nothing creates these parameters: Pulumi deliberately does not, because creating one requires a value and any value it picked would BE an override shadowing the reviewed file. The first put-parameter creates it.

An unreadable parameter keeps the last value that WAS read. A warm execution environment holds its last successful override and extends the TTL; the YAML is the fallback only for a process that has never read the parameter. After a flip that distinction is the whole game — SSM is then the only record the tier is live, so reverting to the file on a throttle would resume forwarding to a legacy that no longer owns those events and stamp them SHADOW, and the executor would observe and drop them. Do not read a throttled tier as having reverted to its file value.

A transient read failure logs a WARNING once per TTL window; a terminal one (AccessDeniedException — the shape a never-applied ssm:GetParameters grant takes) logs at ERROR, because that means the lever will never work rather than that it blipped. A malformed value (a typo like shadowing) is ignored with an ERROR log rather than adopted or fatal — raising would take the ingress down over a hand-edited parameter. An override in effect logs one INFO per window naming what it shadows.

What the TTL means operationally: it is your worst-case rollback time, not a throughput cost. The cache is per Lambda execution environment, so it is one batched read per window whatever the traffic, and exactly one request per window pays the ~10–30ms.

Mid-flip, containers disagree — and that is safe. Ownership is stamped at ingress and never re-decided, so an execution environment holding a stale value produces a correct, complete outcome; just the previous mode's. Every event is handled exactly once by exactly one pipeline. Expect a few unmatched comparator rows straddling a flip and exclude that window rather than chasing them.

The graft — wiring real traffic in (HubSpot, dev first)

The graft puts this acker in front of the legacy pipeline: swap the env's HubSpot app central webhook target URL (one URL per app, applies to every portal at once) from the Prismatic inbound_trigger flow to the acker's /hubspot. While the environment's rollout_state still gives legacy the event AND it has a legacy_ingress_origin in integrations-acker/environments.yaml, the acker forwards every delivery — raw bytes + the provider's own headers, before its own verification — to {base}{the delivery's own path}, so legacy receives exactly what it received before the swap, one hop later.

Those are two independent facts and the split is load-bearing.legacy_ingress_origin is an ORIGIN and a durable property of the environment: set it once, leave it set, it stays true after cutover. rollout_state is the only toggle, and it has THREE values because the migration has three stages — legacy (provider still delivers to legacy; we should receive nothing), shadow (we receive and forward), live (we receive and own it). The two-valued ownership stamp is derived from it, so nothing can claim live while still forwarding. Cutover is therefore one flip rather than deleting the address of the pipeline you are cutting over from, and rollback needs no config change at all: restore the provider target and the acker simply stops receiving.

Because the base is an origin and the route appends the delivery's own path, one value serves both providers — legacy's /hubspot and /trigger/{flow_token}/{trigger_id} are routes on the same host. It is the same value as that environment's integration_api_endpoint configuration row, duplicated only because the acker has no database to read it from.

Three origins, and only one of them ever moves. The identity above is easy to misread as "cutover flips the integration_api_endpoint row", so state it explicitly:

Origin	Home	Names	Moves at cutover?
legacy_ingress_origin	integrations-acker/environments.yaml	where legacy receives	no — a durable fact
integration_api_endpoint	configuration table	the legacy API, and the Prismatic instance's OnRampApiBaseUrl	no, and this is load-bearing
acker_ingress_origin	app/config/environments.yaml	where THIS acker receives	it is the target, not a thing that moves

The middle row is the one that bites. It is also the base every legacy Prismatic flow calls back into OnRamp with (integrator_marketplace_service writes it into the instance's OnRampApiBaseUrl / OnRamp Connection. endpointUrl config vars). Retargeting it at the acker would aim every one of those callbacks at a service that serves two webhook routes and 404s the rest — so the Salesforce graft mints Outbound Message endpoints from acker_ingress_origin instead, and that row keeps naming legacy forever.

Salesforce does NOT graft this way — it has its own section below, and the difference is not a variation on these steps. A HubSpot app has one central webhook, so both pipelines can see every delivery and one provider-side write moves all of them. An Outbound Message has exactly ONE destination, is minted per trigger by us, and pointing it at the acker takes legacy out of the path entirely — the forward is the only thing putting it back. There is no configuration in which both receive it independently, so a Salesforce forward failure is strictly more consequential than a HubSpot one. It acks <Ack>false</Ack> (Salesforce's equivalent of the HubSpot route's 502) so Salesforce redelivers.

Response semantics while grafted — legacy owns every event, so the only fatal failure is the forward itself:

Outcome	Response to HubSpot	Why
Forward failed	502 — HubSpot redelivers; nothing enqueued	The one way the graft can hurt a customer. The retry is fresh for OUR side (FIFO dedup absorbs overlap); for legacy it is fresh only when the forward truly failed — a forward that timed out AFTER legacy received it makes the retry a duplicate there, which legacy already tolerates from HubSpot's own retries.
Forward ok, OUR verification refused	200, verified: false, ERROR log	A 401 would only re-deliver to a legacy that already has it. Nothing unverified is enqueued. Firing at any rate means the shadow is blind. The ERROR separates the two causes: an unreadable or still-placeholder SSM parameter names itself; anything else is a genuine mismatch, i.e. a wrong secret or a wrong ACKER_PUBLIC_BASE_URL.
Forward ok, enqueue failed	200, enqueued: false, ERROR log	Raising would re-deliver to a legacy that already processed it. The shadow loses the event — visible as ONLY_LEGACY_GENERATED in the comparator.
Forward ok, all good	200, accepted: N	The tap working.

Watch the forward itself: filter @message like /"kind": "legacy_forward"/ on /onramp/<env>/lambda/pipeline-acker — ok=false count must be ~zero.

Where each value the swap needs lives. The HubSpot app id used to be recorded only here; it is a per-tier, non-secret structural value, so it now sits in the config map with the other two:

Value	Home
HubSpot app id	hubspot_app_id, app/config/environments.yaml — filled in per tier as each is grafted; null is what the cutover refuses on
Legacy origin	legacy_ingress_origin, integrations-acker/environments.yaml
Acker ingress (the swap target)	acker_ingress_origin, app/config/environments.yaml, plus /hubspot

The CLI reads all three and derives the swap itself, so there is nothing here to copy into a command. Its opening board names any tier still missing one, grouped by the file it belongs in — no tier has to be chosen first, and no per-tier list is kept here to go stale against the files. It also refuses a live target that is neither this tier's legacy ingress nor its acker — a tighter guard than --expect-current, which can only catch a value that has gone stale relative to what you typed.

The swap target is a hostname we own, never the acker's Function URL. One CloudFront distribution per durable environment fronts the Function URL — without OAC, which is the only reason it can: OAC SigV4-signs the origin request and a CloudFront Function cannot read a body to compute x-amz-content-sha256, so it rejects every webhook POST. With the Function URL left authorizationType NONE there is nothing to sign. The point is that the provider-side target outlives the Lambda: replacing the acker becomes an origin change, not a HubSpot repoint across every portal on the app.

Because CloudFront hands the origin its own host rather than the public one, the acker takes the URL a signature covers from ACKER_PUBLIC_BASE_URL (injected by integrations-pipeline.ts, read by config.get_public_base_url) and falls back to the request's Host only where nothing fronts it — a workstation, or a PR preview whose smoke POSTs the Function URL directly. If that value is ever wrong, every delivery answers verified: false, which looks exactly like a wrong signing secret — check it first. A signing secret that is missing or still unprovisioned is the one cause that does say so: the acker logs the parameter name and the remedy instead of reporting a mismatch, so rule that out from the log before comparing URLs.

Sequence for an environment (dev first; stage repeats it; prod repeats stage):

1. Preflight: confirm the legacy Prismatic inbound_trigger flow does NOT verify the HubSpot signature (inspect the live flow config; v2/v3 cover the request URL, which changes when we forward, so a verifying legacy would reject every forwarded delivery and go dark — discovered at step 5, after the swap, unless checked here). Then: env deployed at a commit carrying the graft; SSM /onramp/<env>/pipeline/hs-signing-secret holds the env's HubSpot app client secret (synced by the deploy — see below, and bun run pipeline reports it per tier); an AWS session that can read SSM — the swap script sources the developer-account API key from /onramp/shared/hubspot/developer-api-key itself ("shared" because HubSpot allows one key per developer account and that account owns every environment's app; per-env scoping lives in --app-id). One-time setup: copy the Active API Key from the HubSpot Developer Portal (Keys → Developer API Key — do NOT regenerate it; the existing integrations tooling uses the same key) and store it: aws ssm put-parameter --name /onramp/shared/hubspot/developer-api-key --type SecureString --value "$(pbpaste)" --region us-west-2 — the region is pinned on both the write and the script's read, so the singular key has one home whatever the operator's ambient profile says.

   The signing secret is synced by the deploy, not by hand. It lives in each GitHub environment as HS_APP_CLIENT_SECRET, and .github/scripts/sync-hs-signing-secret.ts writes it into /onramp/<env>/pipeline/hs-signing-secret on every infra deploy of that tier, after pulumi up. The deploy is the only context that can: the value has to MATCH the HubSpot app's client secret so nothing generates it, and GitHub's secrets API returns names and timestamps only.

   Ordering is handled — the sync runs after the apply, because the parameter's overwrite: true seed lands on its CREATE. A value the acker would refuse fails that step loudly instead of being written, and an absent secret warns without failing the deploy.

   What the parameter held before this existed, and why both cases are refused. A tier created after the ownership split holds the placeholder literal. dev, stage, prod and gdpr-prod predate it, so each was created holding a 48-character alphanumeric random — ignoreChanges means no apply would ever replace that with the placeholder, so the acker refuses on the SHAPE as well as on the literal (unusable_signing_secret_reason), and so does the sync script before writing. bun run pipeline reports which tiers still hold one.

   Until it lands, the acker refuses every delivery — nothing unverified is ever enqueued — and logs an ERROR naming the parameter and this step rather than reporting a signature mismatch. That distinction is the whole reason both shapes are recognized: verified: false on its own reads identically for an unprovisioned parameter, a wrong secret, and a wrong ACKER_PUBLIC_BASE_URL. Check the acker log group (/onramp/<env>/lambda/pipeline-acker) for hubspot delivery refused before working through the other two — the line says which of the two unprovisioned shapes it found, or that the signature genuinely did not match.

   Previews and local workstations are the exception and need no write at all: Pulumi generates their value, the e2e smoke signs with the same parameter the acker reads back, and nothing outside the stack holds a copy. The shape rule is skipped there for exactly that reason.

2. Read the live target: bun infra/scripts/pipeline/hubspot-target-swap.ts --app-id <envAppId> --get

3. PR: set that URL's ORIGIN (scheme + host, no path) as the env's legacy_ingress_origin in integrations-acker/environments.yaml; deploy it. The route appends each delivery's own path, so a value carrying one concatenates into /hubspot/hubspot and every forward 404s — the config loader rejects a path at boot rather than letting that ship. Leave rollout_state: shadow in the SAME PR — that is what actually arms the forward, and it stays armed until cutover. Recording the origin while leaving rollout_state: legacy is the safe intermediate: the value is captured, nothing is armed, and a delivery arriving early is forwarded and logged rather than dropped. 3b. Deploy the edge, and prove it before any provider knows about it. PREREQUISITE, once for the estate and not per environment: the *.onrampapps.com certificate must exist in us-east-1 — see infra/README.md "The acker ingress certificate". The apply READS it (it never mints one), and fails loud at the lookup if the bootstrap has not run. The apply itself creates the distribution and the A/AAAA alias records, then injects ACKER_PUBLIC_BASE_URL. Budget ~15 minutes for the distribution to reach Deployed; the alias answers NXDOMAIN until it does. Nothing is at risk during that window — the provider still points at legacy and the acker receives nothing — which is exactly why the verification belongs HERE rather than after the swap:

   • dig +short <that tier's acker hostname> returns the distribution.

   The hostnames, because two tiers break the pattern.resolveAckerPublicHostname (infra/components/integrations-pipeline.ts) is the source of truth; substituting <env>-events for prod or gdpr-prod yields a name nothing answers, and an Outbound Message minted against it has one destination and no fallback:

   Tier	Acker hostname
   dev / stage / demo	<env>-events.onrampapps.com
   prod	events.onrampapps.com
   gdpr-prod	eu-events.onrampapps.com

   • A signed synthetic POST to https://<env>-events.onrampapps.com/hubspot answers 200 with verified: true. Sign it against the public hostname, not the Function URL — that is the whole property being tested, and an unsigned probe proves only that DNS resolves.
   • A 200 carrying verified: false means ACKER_PUBLIC_BASE_URL disagrees with the hostname just published. Fix that before step 4; after the swap the same fault is indistinguishable from a bad signing secret, on live customer traffic.

4. Swap: ... --set https://<env>-events.onrampapps.com/hubspot --expect-current <read value> (the script refuses if the live target moved; it re-writes the full settings object with only targetUrl changed and prints the exact rollback command).

5. Verify within minutes: legacy_forward lines all ok=true, legacy dev still processing (its executions keep appearing), observation lines flowing on the executor log group.

6. Failback at any point: run the printed rollback — one PUT, all portals. Failback is unchanged by the edge: it happens one hop ABOVE CloudFront, at the provider, so a wedged distribution is never in the rollback path. Rollback is now provider-side only. Restoring the target is the whole procedure — leave legacy_ingress_origin set. It describes where legacy lives, which does not stop being true because we stopped pointing at it, and an environment that still forwards while receiving nothing costs nothing. The config-ordering hazard this section used to carry is gone with the toggle that caused it.

   One one-way rule remains: never retire the hostname, the distribution, or the certificate while the provider still points at it. DNS goes NXDOMAIN, the provider's retries all fail, and events are lost with nothing to forward them to. Provider target first, always.

   Clearing the base to null on an environment whose provider still points here is a misconfiguration rather than an ungraft — both routes log loudly on the first such delivery instead of dropping it silently.

The graft — wiring real traffic in (Salesforce)

Everything above is HubSpot-shaped: one central webhook, one provider-side write, both pipelines seeing every delivery. Salesforce is a different operation, and the sequence is not a variation on the HubSpot one.

An Outbound Message endpoint is minted per trigger, by us, from config, and carries the ramp uuid in its URL. So moving Salesforce traffic is O(ramps), each write lands inside a customer's own Salesforce org, and there is no state in which both pipelines receive a delivery independently — the acker's forward is the only thing keeping legacy whole. That is why a Salesforce forward failure is worse than a HubSpot one, and why it acks <Ack>false</Ack> (see the ack table below).

Two switches, both required. A vendor's new Outbound Messages target the acker only when both are true:

Switch	Home	Why it is separate
crm_lambda_trigger_provisioning	per-vendor feature flag	Decides HOW the artifact is written. The legacy Prismatic create_trigger flow derives the endpoint from its own instance context and discards anything Flask passes, so only the broker path can aim it — a graft is impossible without this.
crm_acker_om_endpoint	per-vendor feature flag	Decides WHERE it points. Per-vendor, not per-env, because one tier can hold several unrelated Salesforce orgs — dev holds three — and a graft is a write into one customer's org at a time.

Plus the environment's acker_ingress_origin (app/config/environments.yaml), which names where that tier's acker receives and is armed by nothing. It is set only where BOTH halves hold — the tier composes pipeline (so a distribution answers the name) and its integrations-acker/environments.yaml block carries legacy_ingress_origin (so a delivery is forwarded rather than observed and dropped). Today that is dev and stage — demo, prod and gdpr-prod all compose a pipeline now, but none of their acker blocks carries a legacy origin yet.

Null is what ARMS the protection, so the ordering is not cosmetic: a grant on a tier with no origin logs a warning and stays on legacy, while a syntactically valid origin is used as-is. Recording a name before both halves hold therefore converts a harmless premature grant into Outbound Messages whose deliveries reach nobody — an OM has one destination and no fallback. A cross-file test (test_an_origin_is_set_only_where_that_tier_can_forward_to_legacy) fails the PR if the two files disagree.

Nothing moves on grant. An existing Outbound Message keeps its endpoint until that ramp is republished or repointed. That is deliberate: it makes each graft an explicit, per-ramp, reversible operation instead of a config edit with CRM side effects, and it means the two grants above can be reviewed and landed well before anything is written to an org.

Sequence for an environment (dev first):

1. Preflight, and unlike HubSpot's this is mostly a Salesforce-side question:
   • Is Flow publishing working in the target org at all? It has broken before on Metadata API permissions. Confirm the org's ONRAMP_% Flows exist and the integration user (or_integration.service_user_name) still holds ModifyMetadata / CustomizeApplication / ApiEnabled. A demoted integration user fails every repoint, and the failure reads as a broker bug.
   • acker_ingress_origin is set for the env and the edge answers: a GET to that tier's acker hostname + /trigger/x/y returns 405 with allow: POST (the route is POST-only; a 404 means it is not mounted). Use the hostname from the table below — not <env>-events — because prod and gdpr-prod do not follow that pattern.
   • The env is deployed at a commit carrying both flags and the origin.
2. Inventory, write-free. GET /api/ramps/ops/salesforce-triggers?vendor_id=<v> lists every ramp that owns a live endpoint, with endpoint_origin (where the next provisioning would point) and endpoint_is_ours_to_set (false for a vendor still on the Prismatic path — the origin shown is then a description of a path not taken). Always pass vendor_id: unfiltered the listing spans the whole environment, which on dev means three different Salesforce orgs.
3. Grant both flags for the one vendor. Confirm by re-reading step 2 — the origin must now be the acker's and endpoint_is_ours_to_set true.
4. Repoint ONE ramp: POST /api/ramps/ops/salesforce-triggers/<ramp_uuid>/repoint with {"vendor_id": <v>}. The broker reconciles rather than replaces — it PATCHes WorkflowOutboundMessage.Metadata whole, keyed on the trigger-derived name — so this patches the endpoint in place and does not cut a new Flow version.
5. Verify before touching a second ramp: fire that ramp's trigger, then check legacy_forward lines are ok=true on the acker log group, legacy still wrote its execution, and an observation line appeared on the executor's.
6. Repeat per ramp for as long as you are still proving the path. Deliberately not a loop over the whole listing: each call writes into a different org, and one org rejecting must not decide anything about the next.
7. Once the shape is proven, stop doing this by hand — bun run pipeline sweeps the tier, one lane per org, and moves up to 25 of an org's endpoints per round trip through the batch route. Read the endpoints back afterwards (next section); the sweep's own report is the code that did the writing reporting on itself.
8. Failback: revoke crm_acker_om_endpoint and repoint the affected ramps, which re-mints legacy endpoints from the unchanged integration_api_endpoint. Still O(ramps) where HubSpot's rollback is one PUT, but no longer O(ramps) in time — the batch route's cost is flat in record count, so a rollback is round trips rather than minutes. Read back afterwards: on the way out the flag must be off BEFORE the repoint, and a repoint that ran too early returns repointed: true having moved nothing.

Reading the endpoints back — the check that used to be missing

This section used to say the check was impossible. It said nothing in this repo could tell you where an Outbound Message points, and that reading one needed an sf session on the org. The first half was true of the code as it stood; the second was never true of Salesforce. The endpoint is Tooling-API only, which is not the same as unreadable — and because the sentence read as "don't bother", nobody looked. A tier then sat with one workflow of twenty-two actually moved while every report said it was grafted.

There is now a read-back, and it is the only check in the tool that consults the customer's org rather than our own record of it:

POST /api/ramps/ops/salesforce-triggers/read-back
{"vendor_id": <v>, "ramp_uuids": ["…"]}

bun run pipeline offers it automatically after any sweep. It compares three values per ramp, and every pair of them can disagree:

Value	Where it comes from	What it means
observed	the org, via Tooling	where the Outbound Message actually points, right now
recorded	provisioned_endpoint on the workflow's trigger	where we wrote down that we pointed it
wanted	endpoint_origin on the listing	where this configuration would point it next

Observed vs wanted says whether the tier is where it should be. Observed vs recorded says whether our record is TRUE, and that is the pair nothing could evaluate before — which is exactly how a false record survived long enough to be believed. A sweep resumes from the record, so a record that disagrees with the org is a ramp the next run skips while it sits at the wrong endpoint. The CLI reports that case as record-wrong and names the customer.

Origins only, never the whole URL: the path of a minted endpoint carries the vendor's flow token, which is the secret authenticating that vendor's Salesforce deliveries.

By hand, for one org. Verified against foo9 — SOQL will return Metadata for a single row:

sf data query --use-tooling-api -q "SELECT Name, Metadata FROM WorkflowOutboundMessage WHERE Name = 'ONRAMP_<uuid-no-dashes>'" -o <org> --json

For many at once, SOQL refuses — "the query qualifications must specify no more than one row for retrieval" is what selecting Metadata or FullName across rows gets you. Batch the ids with normal SOQL, then fetch Metadata by id through Tooling composite (25 subrequests per call, each returning full Metadata including endpointUrl):

POST /services/data/v64.0/tooling/composite
{"allOrNone": false, "compositeRequest": [
  {"method":"GET","url":"/services/data/v64.0/tooling/sobjects/WorkflowOutboundMessage/<id>","referenceId":"a"}
]}

That is the same shape the broker's read-back route uses, and it is what independently confirmed the dev ungraft had landed — two workflows both reading the legacy origin. The negative check from our own side still holds and is cheaper when it applies: if the acker's log group has never recorded a /trigger/… delivery, nothing is grafted yet.

Ramps that would MINT rather than patch

The listing excludes paused, archived and draft ramps because none of those owns an Outbound Message — but a published ramp whose Flow was never created (or was deleted org-side) passes every filter. configure_trigger on one of those does not patch anything: it CREATES a Flow and an Outbound Message inside the customer's org, slowly, for a trigger nobody inspected. It is not hypothetical; it holds for a minority of the listed ramps on dev.

Two things changed, and neither is a fix to the listing:

• The read-back reports them. Such a ramp comes back absent, named, with the warning that provisioning it would mint. That is how you find them.

• The sweep can no longer do it — while the batch route answers. That route moves endpointUrl on an Outbound Message that already exists and does nothing else; it has no Flow payload to create one from, so a ramp with no Outbound Message is reported and skipped.

  The fallback re-arms it. Flask and crm-broker deploy independently of the CLI, and on a tier missing the batch route the sweep writes every ramp through the single-ramp route instead — which mints. The run says so in its own record. So on a tier that fell back, read the endpoints back BEFORE sweeping: the ramps this reports absent are the ones that would get artifacts.

The single-ramp route (step 4 above) still mints, deliberately: a human aiming at one ramp may want exactly that. So the old advice stands wherever you use it — list the org's ONRAMP_% Flows, or read back first, and repoint one-at-a-time only what you can see a Flow for.

Reading observations in CloudWatch

Log group: /onramp/<env>/lambda/pipeline-executor (previews: /onramp/preview-<PR>/lambda/pipeline-executor). The executor emits exactly one JSON line per envelope, at every terminal outcome — no_integration, unroutable, proof_failed, no_candidates, unresolvable, no_matches, matched. An envelope with no observation line is an executor bug, not an uneventful event.

All observations in a window (Logs Insights):

filter @message like /"kind": "pipeline_observation"/
| parse @message '"outcome": "*"' as outcome
| parse @message '"mode": "*"' as mode
| stats count(*) by outcome, mode

Trace one CRM object:

filter @message like /"kind": "pipeline_observation"/ and @message like /"external_id": "12345"/
| fields @timestamp, @message
| sort @timestamp asc

Matched events and their would-claim verdicts:

filter @message like /"kind": "pipeline_observation"/ and @message like /"outcome": "matched"/
| parse @message '"account_ref": "*"' as account_ref
| parse @message '"all_values_hash": "*"' as payload_hash
| fields @timestamp, account_ref, payload_hash
| sort @timestamp desc

The same lines carry "mode": "live" after the flip with the actual outcome per ramp (claimed + execution uuid / skipped / failed), so these queries watch the cutover itself.

Two verdicts per ramp — read verdict, diagnose with verdict_raw

During the bake legacy claims within seconds of the webhook while the shadow reads from behind the FIFO queue, so a naive dedup read finds legacy's own row and answers skip_settled for the very events the bake exists to measure. Each shadow ramp entry therefore carries:

Field	Meaning
verdict	The counterfactual answer — what the claim would do as the event's owner: rows created at or after the envelope's received_at (legacy handling this same delivery) are excluded; older rows are genuine prior claims and still settle it. This is what the comparator judges.
verdict_raw	The unexcluded point-in-time read. verdict=create, verdict_raw=skip_settled is the expected bake shape — legacy got there first.
claim_boundary	The received_at the exclusion used. null (a pre-v6 envelope, including a redrive of one — a redriven v6 envelope keeps its original stamp) means the exclusion never ran and dedup-shaped verdicts prove nothing; a boundary far staler than the queue lag (a long-parked redrive keeps its original stamp) is equally untrustworthy. The comparator counts both shapes as unverifiable and they gate its exit code.

The dry run — the full chain, for real, then discarded

The claim boundary above only answers "would this ramp claim the event". A human deciding whether the flip is safe needs the rest of the chain to agree too — account resolution, the CRM-account link, data-field definitions, the project itself — and RampsProjectCreationService has no dry-run mode of its own; it commits as it goes. So dry_run.py's harness runs the ENTIRE downstream chain for real — claim_execution → RampsExecutionsService. process_execution → RampsProjectCreationService's twelve steps → ProjectCreateService.try_create_project_from_data — inside one transaction that is unconditionally rolled back, for every ramp whose claim preview says it would attempt something (verdict is create or resume; every other verdict is dry_run_status: not_attempted and never reaches the harness at all).

Containment is three independent layers, so a hole in one does not become a durable write: DB.deferred_commits() turns every commit reachable through the DB wrapper — how this whole domain chain commits, including RampsProjectCreationService's own error-handling finally — into a flush; a before_commit sentinel raises on any db.session.commit() that reaches the session some other way; and outside the harness entirely, the connection-level default_transaction_read_only=on (mode.py / runtime.py) means even a bug that defeats both of the above still hits a loud Postgres 25006, never a silent write. The harness is the one place that lifts that default, and only for its own transaction (SET TRANSACTION READ WRITE, verified on dev01 to cover exactly one transaction and nothing after rollback).

Egress is suppressed, not merely unrouted. CreateProjectInputDTO. suppress_external_events=True gates the outbound queue call, project-created events/automations, and the project-created email. Everything else downstream of a side effect — the data-field CRM-sync emission, internal/customer invite emails — passes through egress.egress_checkpoint(kind, detail): a no-op outside a dry run, and inside one, fail-closed — every kind but crm_read raises DryRunEgressViolation before the underlying call fires. CRM reads are the one exception (decision D3): they're idempotent, and counting them (egress.dry_run_egress()'s live tally, surfaced as the observation's flow_invocations) IS the flow-usage telemetry that later justifies retiring a legacy Prismatic flow — a real answer to "does anything still call this", not a guess.

What the fingerprint covers. fingerprint.execution_fingerprint projects the four durable artifacts an execution produced (or would have): the execution row (status_code, all_values_hash_v8, whether it has a customer account, whether it produced a project), the project itself (name, account name, owner, dates, playbook, value, module/task counts), the entity links attached to both, and the project's data-field values. Every value is a sha256 hash over canonical JSON, keyed by field NAME — never a raw customer value (decision D4): a mismatch names which field diverged without a CloudWatch line ever holding what it diverged to or from. The same function runs on both sides of the comparison — inside the harness against the flushed-but- uncommitted dry-run state, and via fingerprint_cli.py against a legacy row legacy actually committed — so there is exactly one implementation of the projection, not two that can drift.

Known noise source: project.start_date/end_date resolve from the ramp's workflow-relative date offsets against "today" at execution time. A dry run and its legacy comparison point straddling midnight can legitimately diverge on exactly those two fields with no bug behind it — expected, and worth excluding from a bake's pass/fail read rather than chasing.

The comparator — "these agree" as a query

bun infra/scripts/pipeline/shadow-compare.ts --env dev --since-minutes 60 --database-url-file ~/.pgpass-dev

Joins observations against the executions legacy actually wrote (or_ramp_executions) on (vendor_id, ramp_id, external_id) and prints one verdict per event.

Verdict	Meaning
BOTH_GENERATED_IDENTICAL	Same ramp, and every non-synthetic all_values field agrees. The only silent verdict
BOTH_GENERATED_NEW_HAS_EXTRA	Same ramp, and the only differences are fields the new pipeline populated while legacy left them empty. Does not fail the run, but is printed with vendor, ramp, external id and field names. all_values is read back BY FIELD NAME downstream — lookup_bound_value for a task-prepopulation binding, _field_value for a start date or an account key, and the condition evaluators where absence-is-null flips a branch — so a field only the new pipeline fills can still change what a customer sees. Chase one by checking BOTH homes a binding can live in: the workflow's own trigger/actions config, and the pre-population binding on a task-step form element in the playbook the project is built from (or_task_steps.step_configuration, prepopulateBinding or binding). Absence from workflow config alone proves nothing -- the task-step binding is playbook content and is not reachable from the workflow row
BOTH_GENERATED_NEW_MISSING_FIELDS	Both acted, but at least one field legacy populated came back empty on the new pipeline's side — a regression
BOTH_GENERATED_CONFLICTING_VALUES	Both acted, but at least one field both sides populated holds two different values (listed per field)
ONLY_ONE_PIPELINE_CREATED_PROJECT	Both sides claimed the same ramp and ran the chain to the end, but only one produced a project (fingerprint:execution.produced_project). Legacy-true is a customer's project silently not existing after the cutover; shadow-true is one appearing that legacy never made. Outranks all four BOTH_GENERATED_* classes — an event that loses a project entirely must never be reported as a value conflict, which is what happened before this class existed. Only reachable under --fingerprints; see The project-loss question needs --fingerprints below
ONLY_LEGACY_GENERATED	Legacy wrote an execution no observed delivery accounts for — nothing received, a delivery declined before this ramp, a delivery that matched other ramps, or one already answering a different row. Which, and on what evidence, is in the detail — see Reading an ONLY_LEGACY_GENERATED detail below
ONLY_NEW_PIPELINE_GENERATED	Shadow would have claimed; legacy wrote nothing for the key at all
NEW_PIPELINE_WOULD_FAIL	The shadow's claim verdict is one that, live, raises or strands the object rather than settling it — never agreement, whatever legacy wrote
NEW_PIPELINE_WOULD_DUPLICATE	A sibling delivery contends for legacy's one row under a mode where the live claim CAN insert alongside it — post-cutover this delivery would have claimed a duplicate
COMPARISON_BLOCKED_BY_HARNESS	Not comparable, and live is not implicated — printed, never counted as a failure
COMPARISON_DECLINED	Not a verdict — the count of shadow lines the comparator refused to judge, printed in the tally so it is not visible only inside a warning. A refusal verifies nothing and gates the exit code: a dedup-shaped verdict with no claim boundary, a boundary-decided verdict implausibly stale for the queue lag, or a late-claimed row another delivery of the same object could equally own
DEFERRED_BINDS	Not a verdict either — how many rows the deferred pass bound past the attribution ceiling, printed only when non-zero. Never gates: it counts comparisons that HAPPENED. Each one is printed in full whatever it concluded, and the gap in its detail is what to read

COMPARISON_BLOCKED_BY_HARNESS has one producer today. Two CRM event batches for the same object, arriving further apart than the executor's queue lag, both hydrate past the trigger's condition and both match the same ramp. The dry run stages each claim inside a transaction it rolls back, so the second delivery reads an empty table and answers create where live — which commits the first claim before processing it — would have found the row. Under ONE_TIME every status that row could hold is one the live claim refuses to insert alongside, so there is no duplicate to report; under the other two modes there can be, and the verdict is NEW_PIPELINE_WOULD_DUPLICATE. Neither is compared: legacy holds one payload for two shadow creates, so a field diff would be against the wrong delivery.

A create whose attribution window held nothing is only ONLY_NEW_PIPELINE_GENERATED when legacy wrote nothing for the key AT ALL. When legacy claimed the same (vendor, ramp, object) later than the ten-minute window allows, legacy wrote — so the line is held back to a deferred pass that runs once every other delivery has bound the rows it owns.

The ceiling is standing in for "no other delivery could own this row", and that is a question the window can be asked directly. Legacy's claim latency is set by a Prismatic queue this platform does not own, so no fixed span answers it for every window; what the ceiling actually protects is unique consumption, and that survives being asked. The deferred pass names every delivery of the same CRM object that could be the one legacy answered, and:

• No other candidate — the row binds, and the line is compared like any other. The verdict carries deferred bind: and its gap, and the run's tally gains DEFERRED_BINDS=N. This is the two slow HubSpot claims a bake sees per half-day, and it no longer reds the run.
• Any other candidate — nothing binds. One COMPARISON_DECLINED line is reported, carrying the same root-cause sentence the sweep would have written, and legacy's row is not also reported as ONLY_LEGACY_GENERATED. One event, one gate.

A delivery is a candidate unless it already bound a row for this ramp, or its envelope demonstrably arrived after the row was written. Arrival comes from the claim boundary where the line has one — a matched line names only the triggers the gauntlet selected, so one that omits this ramp still states its arrival through the ramps it did name. A declined line names no ramp at all, so only its observed_at is available and it is ruled out only when it is further past the row than the --legacy-pad-minutes receipt lag. There is no lower bound in either case: nothing is too early to be what legacy answered.

Two edges refuse a bind however cleanly the candidates count out, both of them the trailing sweep's own: a row newer than the pad may still be answering a delivery whose observation has not reached this window yet, and a row older than the sweep cutoff is one the sweep declines to report at all. More than one late row for the key is legacy having written twice, which nothing here can attribute either — each row stays its own sweep line.

What a bind cannot see. A delivery the shadow never received leaves no observation, so it is invisible to this count, and legacy's row for it is data-identical to legacy simply having been slow. Binding is the optimistic reading of that ambiguity. That is why a deferred bind prints even when it agrees, while every other agreeing verdict stays silent: a wide gap in a green run is the only thing left pointing at the other reading, and it is worth opening legacy's own handler log over. The unverifiable count still gates the exit code exactly as a divergence does, so a contested late claim keeps a bake red — once.

Field values are masked — reading a diff line you can paste

A field diff's values are whatever the CRM held, and a bake result gets pasted into Slack, a ticket, a PR or a screenshot. So the comparator masks them by default and prints a characterisation instead:

  · pain_points: legacy=<text len=56 #6864275c> shadow=<empty> — dropped by the new pipeline
  · hs_lifecyclestage: legacy=<empty> shadow=<token len=8 #4452bad2> — extra on the new side, not a divergence
  · CaseNumber: legacy=<digits len=4 #c1a883f1> shadow=<digits len=8 #5a5fbb55> — conflict — equal but for leading zeros

Everything a bake decision turns on survives the mask:

• The field name, in the clear. It is schema rather than data, and it is what makes a finding actionable.
• The shape, named in words — dropped by the new pipeline gates, extra on the new side does not, conflict gates. <empty> is the one state absent, null and "" collapse into, exactly as the comparison does.
• A characterisation of each value: its class (text, digits, token, email, uuid, datetime, url, number, json), its length in characters, and a short digest.
• A normalisation note on a conflict, where the two masked tokens alone would not be enough to triage — equal but for leading zeros, equal but for case, equal but for whitespace, equal but for punctuation and spacing, one side is a truncation of the other. This is the part that keeps a finding like CaseNumber readable: the note IS the bug.

The digest is salted per run. It answers one question — does this value recur, across the events in this run or the fields of one event, which separates one systemic mapping bug from N unrelated ones. It is not stable across runs and is not meant to be: eight hex characters unsalted is a lookup table away from a phone number or a first name, which is the leak the mask exists to close.

Two kinds of VALUE stay verbatim, because neither can be personal data: a boolean, and a sha256 digest — every customer-derived fingerprint leaf is one, and it is the only thing you can diff against a legacy row by hand. An amount, a status code and a date all look safe and none of them is, so they mask.

Two fingerprint leaves are exempted by name rather than by shape, because they are platform metadata and nothing about 200 says so: fingerprint:schema and fingerprint:execution.status_code. Masked, a legacy 200 against a shadow 500 reads as two opaque digit counts and the finding says nothing.

Identifiers are not masked. A verdict still prints vendor=… ramp=… external_id=… in the clear, and the set-aside block still names vendors — those are what you open the object with. "Masked" is a claim about field VALUES, so a masked result is shareable only to the extent those identifiers are.

--unmasked prints the raw values, and --json masks the same way unless you pass it. Use the flag when you are diagnosing rather than reporting, and treat its output as customer data: redirect it to a file rather than letting it into a terminal you will scroll back through.

--unmasked is refused outright on an EU run. EU personal data may not leave the region, so masked field values are the only form in which an EU bake result can be reported — to Slack, to a ticket, to a PR, or to anyone outside the region.

The refusal reads the run's RESOLVED sources, not the --env slug: the effective region (--region, else the tier's) and the --log-group's tier segment. --env alone would not hold, because both of those override it and the header documents pointing a log group outside its tier as a real way to run this — so --env prod --region eu-west-1 --log-group /onramp/gdpr-prod/… --unmasked would have read EU rows and printed every value. It fires before the credential file is opened or a single row is read.

The region arm is the one that holds unconditionally — any eu-* region, however the operator spelled it, refuses. The log-group arm resolves its tier through the same region table, so it covers a tier once that tier has a row there; a slug with no row falls back to the default region and the effective region is what refuses it.

Reading an ONLY_LEGACY_GENERATED detail

One CRM object takes several deliveries in a window — a create, then whatever property changes follow it — so the shadow's observations for an object are a set, and the detail describes the set rather than one member of it. What separates the members is that emit_observation fills ramps on matched alone: a line naming this row's ramp was evaluated against it, a matched line naming other ramps ran the gauntlet without selecting this one, and any other outcome names no ramp at all and so cannot say which trigger it concerned.

Which sentence you get is the first thing to read, because each starts the investigation somewhere different:

The detail says	What it means	Where to start
emitted no observation for this object in the window	Nothing reached the executor for the object at all	A lost delivery: the ramp's provisioned_endpoint, then the acker
the one unanswered delivery … gave verdict=X, observed Nm before legacy's row	One delivery was evaluated against this ramp and no other row answered it	That verdict, and the stated gap — a wide gap is legacy's claim latency, not a shadow fault
N unanswered deliveries reached ramp … and all gave verdict=X	Several deliveries, one story	The same, times N
N unanswered deliveries … and they DISAGREE	Several deliveries answering differently	The set, not one line — no single verdict is the cause
ramp N was not among the ramps they matched	The delivery arrived and was evaluated; the gauntlet picked other ramps of the object	Trigger-condition evaluation for this ramp. Not a lost delivery
none names a ramp (…)	The object was delivered and declined before any ramp was selected	Either the gauntlet declined THIS ramp on one of those lines, or this row's delivery never arrived — the line cannot tell you which
already answered another row for this key (execution N)	Every delivery for this ramp is accounted for by a different row	Legacy's extra row — nothing observed is behind it
arrived after this row was written, so none can be its cause	Every delivery for this ramp postdates the row	An earlier delivery this window did not cover — widen --since-minutes

The last two are the ones to read carefully. A declined delivery carries no ramp whichever trigger it concerned, so the ABSENCE of a ramp is a property of the emitter and never evidence that the delivery was someone else's — which is why that row's sentence offers both readings instead of asserting a lost event.

Anything else the window held is appended as also in the window: … with counts and outcomes. That is context, not the root cause.

The SIZE of the gap is not a discriminator: legacy's claim latency is set by a queue this platform does not own, so the gap between a delivery and legacy's row is reported and never used to rank which delivery is the row's. Its DIRECTION is, and on the CLAIM BOUNDARY rather than observed_at — the boundary is when the envelope arrived, so a delivery whose boundary postdates the row cannot be what legacy claimed. observed_at says only when the executor got round to the line, and routinely trails legacy's own claim, so it settles nothing.

A delivery is also read per TRIGGER, not per line: one shadow line carries an entry for every ramp the gauntlet matched, so a row bound by one trigger of a delivery leaves that same delivery available as another trigger's row's cause.

Root-causing one starts with legacy's claim latency — its usual driver is legacy's own retry cadence after its chain fails mid-flight — but check for a missing observation too: a delivery legacy handled and the shadow never received produces the same shape, and that is a lost event rather than a slow one.

Exit codes: 0 judged clean, 1 judged and something diverged, 2 no verdict at all — a usage or shell-out failure, or a window that held nothing to compare. An unverifiable line counts toward 1 exactly as a divergence does. That last case used to exit 0: the check summed failure signals, and a window with no events had none. A quiet window is not a passed bake, and the go-live bar below reads this exit code. An event a NOT_CUT_OVER_VENDORS entry set aside reaches neither 0 nor 1 — it is counted nowhere, so a window holding nothing else exits 2 for having judged nothing.

A busy window is read in slices. One Logs Insights query returns at most 10,000 rows and offers no cursor, so a window over that used to be refused outright — which took the comparator out exactly during a redrive, where a 24h US prod window has held 47,000 observation lines. The script now halves an over-cap range and reads each half, de-duplicating on @ptr. The cost is wall clock and a few re-scans of the same bytes, not correctness; a wide prod window takes minutes rather than seconds.

The region comes from --env (resolveTier), so gdpr-prod queries eu-west-1 without the caller setting AWS_REGION. --region overrides it.

Notes: synthetic metadata: true entries (lastReceivedTime, objectLink) are excluded on both sides. The script only ever SELECTs; for prod, hand it a read-only connection and follow the prod-DB access procedure. The known gauntlet hydration short-circuit (met == total skips the CRM fetch for action-referenced fields) will surface here as BOTH_GENERATED_NEW_MISSING_FIELDS field diffs — expected until it is fixed during the bake.

What the two sides are NOT held to

The pipelines reach all_values by different routes, and some differences are transport, not payload. All but one are discounted on BOTH sides — the hash in canonical_all_values_hash (observation.py) and the field diff through comparableEntries + diffValues (shadow-compare.ts):

Discounted	Why
LastViewedDate, LastReferencedDate	Salesforce stamps them on read, so they depend on who read the record and when. The Outbound Message declares them; legacy's Prismatic flow does not store them.
A field with no value	An OM declares a field list and this pipeline keeps an entry per declared field; legacy keeps only the populated ones. One EU delivery: 68 entries against 14, agreeing on all 14. Absent, null and "" are one state.
The encoding of a scalar	Legacy arrives via Prismatic into a JSON column and keeps false and 1000.0; this pipeline parses the delivery and gets "false" and "1000". Both render as JS String() would.
Transport escaping, at any depth	Legacy stores what its chain produced — &, <br>, CRLF, a line break spelled as the text \n / \r\n, and one encode layer more than this pipeline (& against &); this pipeline stores what its parser decoded. Both sides expand XML's five predefined entities plus numeric character references, expand the line-break escapes, and normalize CRLF to LF — repeatedly, until nothing changes, because the two chains differ in depth as well as in which encoder ran. Deliberately not the whole HTML5 entity table: TypeScript has no built-in that would match it.
An email local part — field diff only	install_email_redaction wraps the LogRecord factory process-wide, so every observation line reaches CloudWatch already redacted while the legacy row is read raw. The field diff has no choice but to redact both halves. The HASH does not redact: both of its inputs are raw — the legacy fingerprint reads the row out of Postgres, the shadow one hashes the dry-run row in process — so it compares local parts honestly and is the only thing in the bake that can see two pipelines resolve different addresses. An address reaches a task through lookup_bound_value, so that is a divergence, not noise. The domain is comparable at both depths.
Numeric TEXT vs the number a JSON column coerced it to	Salesforce sends every OM field as text. Legacy's JSON column holds 0 where this pipeline holds "0.0", and 15550100199 where this one holds the E.164 "+15550100199". A leading zero, exponent form, and anything past the double round-trip ceiling are deliberately left uninterpreted, so a zip code is never equated with its integer. Being one-sided, it equates two TEXT readings of one number too ("1.50" and "1.5"); that class had no instances across 890 minutes of US prod.
A compound object value	Salesforce BillingAddress arrives as an object-valued entry, which String() flattened to [object Object] — making every compound field compare EQUAL to every other. Rendered as canonical JSON with sorted keys instead, over leaves that come back through the same rendering, so key order is not a divergence and a nested address or coerced number reduces like any other.

Why "both sides" is load-bearing rather than tidy. A rule the hash lacks makes it differ on every event carrying that shape, and a hash that differs everywhere names nothing and cannot gate. The numeric rule was doing exactly that: written as a pairwise test, which a hash has no pair to run, it was unshareable by construction until it moved into the rendering.

Redaction is the opposite case, and the distinction is the one to hold on to. It is asymmetric because the two depths genuinely receive different inputs — one half of the field diff is redacted before the comparator can read it, and neither half of the hash is — so an unredacted hash differs only when two addresses really differ. Mirroring the redaction into the hash would buy a symmetry it does not need and spend the detector.

So past that one rule the two depths differ only in the key set they cover, never in how a value was read. The hash sees two further things: an email local part, and its INPUT — it reads the row claim_execution persisted, past the claim boundary, where the field diff reads the payload the same observation line reported.

What is not discounted is a difference in the characters once every layer of transport encoding is off both sides. A leading zero is the one that recurs: legacy's JSON column coerced the value to a number, so a zip code reads 1234 where this pipeline holds "01234", and after the cutover lookup_bound_value really does hand downstream a different string. That is a real payload difference and it stays BOTH_GENERATED_CONFLICTING_VALUES — the numeric rule refuses it deliberately, and reporting it is the comparator working.

A line break spelled as its own escape used to sit here too, on the same reasoning. It does not any more: measured over 13 hours of US production, legacy holds no real line break at all on the Salesforce path across 68 affected field instances, including free text of 9,488 characters, while the HubSpot rows beside it hold real ones and no escape text — and the shape that settles it is one CRLF stored as 
 beside the text \n, an XML-escaped carriage return next to a JSON-escaped line feed. One break split across two encoders is the transport, not a customer keystroke, so it belongs in the table above.

Two things pin all of this, and they cover different ground. the two exclusion sets agree compares the field-NAME sets by parsing the Python source.

The rest is a shared fixture — app/api/integrations_pipeline/tests/fixtures/all_values_parity.json — holding value pairs and the equal/not-equal answer both implementations must give. Each suite runs its own half in the job that has that toolchain: this side of the parity fixture holds in test_shadow_mode.py over canonical_all_values_hash, and the case of the same name in shadow-compare.test.ts over comparableEntries/diffValues. That is what catches a rendering divergence, since the two can agree on which names to drop and still disagree on whether 1000.0 equals "1000". Adding a pair means adding it once and both halves pick it up — but a rule only one side implements needs a pair that distinguishes them, or the fixture passes on both. A pair the field diff answers DIFFERENTLY states agree_field_diff alongside agree, so the asymmetry above is pinned from both ends rather than exempting either side.

A field one side holds with a value and the other does not still diverges — unless the Workflow never asked for it. That exception is the whole of the next section.

A narrowed read is judged on what the Workflow declared

On HubSpot the executor no longer asks for every property the object type defines. It asks for the fields the Workflow actually references — its trigger conditions, its own actions, and the CRM fields its reachable task forms bind — which is what lets one call replace the seventeen a 1,622-property object needed, and what stops a record written mid-read from coming back with empty fields.

Field-set parity was never the goal; value correctness was. So the comparator diffs the fields the Workflow declared, and a field legacy populated that this side lacks is a finding only when that list holds it:

Shape	Verdict
Legacy had a value, this side is empty, the Workflow declared the field	Reported and gates, as before
Legacy had a value, this side is empty, the Workflow never referenced it	Counted, never reported — the read did not ask for it
Both sides hold a value and they differ	Reported. A narrowed read cannot explain a conflict: this side clearly fetched the field
Only this side holds a value	Reported as richer, as before

The count rides the summary as its own number, never folded into the missing count:

fields: 0 conflicting, 0 missing on the new side, 2 extra on the new side, 158 narrowed out of the read by a workflow's declared field list

That is what separates "narrowed by 158 fields as designed" from "lost one it needed". Without the split, every comparable event under a narrowed read would report hundreds of dropped fields, and the comparator would be unreadable at precisely the moment the cutover depends on reading it.

A read that was not narrowed is compared field for field exactly as before — Salesforce, which reads whole records by design, and any HubSpot event whose enumeration gave up.

What this gives up, and what covers it instead. An incomplete enumeration — a field the Workflow really does read that the enumerator failed to list — is undeclared by definition, so this suppresses exactly the defect class the projection's own review round twice found (a superseded Playbook id, and tasks under a linked library module). Payload comparison could not have caught it either way once the read is narrowed: the field is genuinely not fetched, so both sides agree it is absent. What does catch it is the fingerprint, which compares the artifacts a customer would see — the Project, its data-field values, the answers prepopulated onto tasks — and is untouched by this gate.

--fingerprints is therefore required on any window that narrows, and the comparator enforces it rather than asking: a run that suppressed a field and never asked for the fingerprint exits 1 and says so, alongside the other classes that mean a comparison silently did not happen. A run that narrows nothing is unaffected.

One thing the fingerprint comparison does NOT hold a narrowed read to is its own all_values hash leaf. That hash covers the whole map each side persisted — legacy's every fetched property against the shadow's declared ones — so it differs on every narrowed event by construction, and letting it gate would move the noise one depth down and render it as a hash mismatch that names nothing. Every other leaf still gates, which is what keeps the end-of-line check a real one. The same exemption, for the same reason, already applied to an event the new pipeline populated MORE of.

the declared-field key agrees in shadow-compare.test.ts parses the Python source and pins the two sides' key name equal, the same way the exclusion sets above are pinned. It is the quietest of those pins: a rename on one side alone is not a type error, it is a gate that stops applying while the tally still prints zeros.

Is the narrowing doing anything at all?

Enumeration fails open — any unresolved reference, any raised exception, or a non-trivial Workflow config that enumerated nothing gives up the whole projection and reads the full record. If most Workflows fail open the projection is a no-op, and the cutover buys none of the behaviour it was built for. The comparator counts it per provider, from the same lines it already reads:

hubspot: 41/47 event(s) read only the fields their workflow declared, 6 read the whole record because field enumeration gave up — a narrowing that mostly gives up is one in name only

It gates nothing — a fallback is correct behaviour, not a divergence. It is a rollout-readiness number, and a high one is a reason to widen the enumeration before soaking rather than to soak harder. A provider that never narrows is absent from the line rather than reading as a total fallback.

Start a US prod window at or after 2026-09-16T03:32:21Z. The release before that one carried the projection but it was inert: roughly three quarters of eligible events hit an unresolvable Playbook reference, discarded the field list by design, and read the whole record. A window opened earlier is not comparing a narrowed read against a full one — it is comparing a full fetch against a full fetch, and it reads as agreement for the wrong reason. v4.51.3 is the release that closed it.

The other half of the question — whether a record's seventeen calls really collapse to one — is not the comparator's to answer; it reads the executor's log group, not the broker's. The broker logs properties_requested per call on /onramp/<env>/lambda/crm, and the read is classified per request, never per line:

filter @message like /crm=hubspot batch_read/
| parse @message /properties_requested=(?<props>\d+)/
| stats count() as calls, sum(props) as properties by @requestId

One call carrying tens of properties is a projected read. More than one call is a full fetch, sliced. Two things make a per-line reading wrong, which is why this groups by request: a sliced fetch's LAST slice carries only the remainder, so it looks narrow on its own; and a single read of one or two properties is the targeted-hydrate path that predates all of this, not the projection.

The tell that the narrowing has stopped is in the same log group and needs no query — a falling back to a full fetch line. There should be none.

The default question — and --include-not-cut-over

By default the comparator does not compare the vendors that are not on the new pipeline. A workflow still pointing at legacy delivers nothing here, so every event of its lands on ONLY_LEGACY_GENERATED with "the shadow emitted no observation for this object in the window" — sitting in the same count as an event that would genuinely lose a customer project, which is the number the cutover decision turns on. Those vendors are listed in NOT_CUT_OVER_VENDORS (infra/scripts/pipeline/shadow-compare.ts), keyed by tier, and their events are set aside instead:

set aside as not cut over: 2 event(s) across vendor 427 (…), vendor 506 (…) — counted nowhere above, and gating nothing; --include-not-cut-over judges them too
  [would have been ONLY_LEGACY_GENERATED] vendor=427 ramp=104 external_id=…
    legacy wrote execution … and the shadow emitted no observation for this object in the window (never delivered to the new pipeline?)

# the whole picture instead — those events counted and gating
bun infra/scripts/pipeline/shadow-compare.ts \
  --env prod --since-minutes 780 --database-url-file ~/.pgpass-prod \
  --include-not-cut-over

Set aside is not dropped. They leave every verdict count and the exit code, and print in their own section with the verdict each would have received and that verdict's own detail. This output is read as evidence for a cutover decision, so "no issues" and "what was set aside" have to appear on the same screen.

An entry only excuses the class it asserts, on the ramps it asserts it for. The classes that never gate anyway are set aside — agreement, richer payloads, a harness-blocked comparison — so a vendor nobody is evaluating cannot keep a window out of the judged-nothing exit. ONLY_LEGACY_GENERATED is set aside too, but only on a ramp no observation in the window named. That class covers two shapes and an entry asserts only one: the detail separates "the shadow emitted no observation for this object" from a delivery the shadow did evaluate whose ramp bound no row — "a trigger-condition divergence, not a missing delivery". An observation naming a ramp is proof that ramp reaches the new pipeline, since the prefilter selects candidates on vendor, pause, archive and integration and never on cutover state.

The key is the ramp, never the envelope: repointing is per workflow, so one object legitimately arrives via ramp A while ramp B still points at legacy, and an envelope-level test would gate the very state an entry exists to sit through.

Everything else keeps gating, and that is the load-bearing half. Every other verdict class requires an observation to exist, which means the new pipeline did act on the event — the entry's own premise being false. A vendor part-way through a cutover, delivering on some ramps while others still point at legacy, is the state an entry is meant to sit through, so a payload divergence on one of its repointed ramps still fails the run. Same rule for a declined line, a raised dry run and an uncompared fingerprint: the new pipeline acted and the comparator could not finish the job, which an entry does not speak to.

Why the list is stated, and what keeps it honest

Cutover state cannot be derived here, and the reason is specific.provisioned_endpoint null means unrecorded, never "points at legacy" — devtools/pipeline-cutover/ops-api.ts says so outright, and read-back.ts carries an unrecorded verdict for a ramp that is at the target with nothing written down. without_provisioning_keys also strips the record on an ordinary save, so a live cut-over ramp can lose it. On US prod, active Salesforce workflows hold only null (284) or the acker origin (248) — there is no "points at legacy" value to key on at all, and the field is only ever written for Salesforce triggers, so it says nothing about HubSpot's 101 ramps.

So no query here can tell a quiet not-cut-over vendor from a cut-over vendor whose deliveries have stopped. A derived set would suppress both, which means a rotated signing secret would read as "not cut over" and the bake would exit clean over a total loss of the new pipeline for that tenant. A stated list suppresses only the vendors a human named, and nothing else, ever — that bound is the whole reason it is a list.

What keeps an entry from rotting is checked on every run: a listed vendor that produced a judgeable event in the window is reported as looking cut over. Delivering observations is not the threshold — a vendor whose every observation stops short of a verdict is exercising no comparison, so counting those would warn on every run and mean nothing.

WARNING: vendor 506 (…) is set aside but produced 3 judgeable event(s) this window — it looks cut over, so take it off NOT_CUT_OVER_VENDORS and let its events gate again

Reported, never auto-removed — a vendor part-way through a cutover legitimately delivers on some ramps while others still point at legacy, so the call is a human's. But a stale entry can no longer sit there silently, which is the one real hazard of a checked-in list.

Adding or removing an entry is a PR against that map. The stronger question — which workflows still point at legacy — is the endpoint read-back, which asks the customer orgs directly rather than trusting anything recorded here.

The end-of-line diff — --fingerprints

bun infra/scripts/pipeline/shadow-compare.ts \
  --env dev --since-minutes 60 --database-url-file ~/.pgpass-dev --fingerprints

Your checkout must match the deployed executor image. This flag is the one place the comparator holds two sides that are the same code at possibly different revisions: the SHADOW fingerprint is read verbatim off a CloudWatch line the DEPLOYED executor wrote, while the LEGACY one is computed here, by your checkout, through fingerprint_cli.py. Run a newer checkout against a tier still on the previous image and the two hash byte-identical payloads differently.

The leaf is versioned (all_values_hash_v8) so that skew announces itself as a leaf one side does not carry, rather than as a payload divergence — but the version only moves when the reduction changes. After merging a change to canonical_all_values_hash, wait for the tier to deploy before trusting a --fingerprints run against it; the claim-time field diff is unaffected and stays usable in the meantime.

Extends the same join past the claim boundary. With the flag set, the script shells out to the fingerprint CLI against the SAME database file (there is no separate flag for it — one legacy database, one file):

python -m app.api.integrations_pipeline.fingerprint_cli \
  --database-url-file ~/.pgpass-dev --since-minutes 60 --json

— the legacy side of execution_fingerprint, run headlessly (no monolith Flask app; a minimal one bound only to the file's URL, forced read-only) and read-only at the Postgres level. It prints execution_fingerprint(row) for every non-debug or_ramp_executions row in the window — the same rows the comparator's own legacy_executions count covers, so the two reconcile; the comparator joins that against each observation ramp entry's own fingerprint on (vendor_id, ramp_id, external_id) and diffs the two structures FIELD BY FIELD. A hash diff can only ever say "these differ", never by how much or why, so the field name is the entire report — fingerprint:project.owner, fingerprint:entity_links, and so on — with the two raw hashes still carried in field_diffs for anyone diffing lines by hand. A fingerprint mismatch can upgrade an otherwise-agreeing verdict to BOTH_GENERATED_CONFLICTING_VALUES: the fingerprint covers strictly more of the chain than all_values does, so it can catch a divergence the claim-time comparison never sees. One leaf does more than upgrade — a fingerprint:execution.produced_project disagreement REPLACES whatever payload verdict the event held with ONLY_ONE_PIPELINE_CREATED_PROJECT.

The project-loss question needs --fingerprints

A bake run without --fingerprints cannot answer "did any customer silently lose a project?" at all. Not "answered it and found nothing" — the question is never asked. produced_project lives only in the fingerprint, so without the flag there is nothing in the run that compares whether the two pipelines ended in a project.

ONLY_LEGACY_GENERATED=0 does not answer it either, and reading it as though it does is what a 2026-09-10 US prod bake did. That count means "a legacy execution row with no shadow observation behind it". An event where the shadow observed, claimed, and ran the whole chain — and produced no project where legacy produced one — has a shadow observation, so it is not in that count. It landed on BOTH_GENERATED_CONFLICTING_VALUES instead, on the strength of an unrelated one-field payload difference, and read as a cosmetic nit.

The tally line says which of the two you are looking at rather than printing a zero that reads as "nothing lost":

ONLY_ONE_PIPELINE_CREATED_PROJECT=0                          # asked, and clean
ONLY_ONE_PIPELINE_CREATED_PROJECT=not-asked(no --fingerprints)  # not asked

It is printed at zero on every run, unlike the other zero-suppressed classes, precisely because an operator reading the gate has to see that this question was put and came back clean.

A =0 covers the events that were compared. COMPARISON_BLOCKED_BY_HARNESS is included: whether the dry run produced a project is fingerprinted the same way a bound delivery's is, so it is compared — but against the object's outcome, not the contended delivery's alone. ONE_TIME's dedup is what produces a harness-blocked delivery, so that delivery's own "no project" is its correct, expected result, not evidence about the object; the comparison instead asks whether ANY shadow delivery sharing the same (vendor, ramp, external_id) key produced a project, and only reclassifies to ONLY_ONE_PIPELINE_CREATED_PROJECT — counted there instead of here — when none of them did while legacy's current row did, AND every delivery for that key reached a completed, fingerprinted dry run. A sibling still on blocked_by_egress, or on a line predating the dry-run schema, leaves that disjunction unable to say "none of them did" — the object's outcome is unknown rather than false, so this never reclassifies to a loss on the strength of it; it counts under fingerprint_unverifiable instead, because the question could not be put, not because it was put and came back clean. Read the zero alongside COMPARISON_BLOCKED_BY_HARNESS in the same tally: a harness-blocked line can still print with its own produced_project disagreement noted, as long as a sibling delivery covered the object's project.

When it does fire, read fingerprint:execution.has_customer_account on the same line. Diverging the same way, it is named in the detail as the likely upstream cause — an execution that resolved no customer account has nothing to hang a project on — rather than being a second class competing for the event's one verdict slot.

A ramp whose dry run never reached completed cannot be fingerprint-compared at all, and is reported as its own status rather than silently absent from the join: dry_run_raised (the downstream chain raised — never a green bake, and this alone forces exit code 1 regardless of the verdict counts), dry_run_skipped_by_claim (the claim verdict looked attemptable but resolved to nothing once the transaction ran), dry_run_not_attempted (the claim declined before the chain ran — the expected shape for most skip verdicts), or no_legacy_row (fingerprinted fine, but no legacy execution exists to compare against — expect this on every ONLY_NEW_PIPELINE_GENERATED entry, by construction).

The brake — stop the drain without losing it

The ingress stamp fixes future events. If the executor is writing bad data now (post-flip), freeze the drain — nothing is acked, nothing is lost, the queue holds messages durably (4-day retention bound):

aws lambda list-event-source-mappings --function-name onramp-prod-pipeline-executor \
  --query 'EventSourceMappings[0].UUID' --output text

aws lambda update-event-source-mapping --uuid <UUID> --no-enabled

Then decide calmly: fix-forward (patch, --enabled, the drain resumes in FIFO order) or abandon (replay the owned messages into legacy by hand — order breaks, events survive). Re-enable is the same command with --enabled.

Sequence to full live (prod)

1. Shadow deployed (this change) — prod stamps SHADOW at ingress and the executor's writes are contained by the dry-run rollback, observations flowing once the graft lands.
2. Graft + flip lever (separate change, gated on the decision doc's D2/D3).
3. Bake, and the flip does not happen until every one of these is true — run --fingerprints daily, not just the claim-level comparator:

   • Output-fingerprint MATCH rate over a stated window and volume (record both — "100% of 4 events" and "99.98% of 40,000" are not the same claim). project.start_date/end_date mismatches straddling midnight are the one class allowed to be waived rather than root-caused; every other field name that shows up in a mismatch gets burned to a root cause before it counts toward the rate.

   • Zero unexplained ONLY_NEW_PIPELINE_GENERATED / ONLY_LEGACY_GENERATED. Every occurrence has a written root cause, not a shrug — these are the two classes that mean one pipeline acted (or would act) and the other didn't, which is exactly the failure mode shadow mode exists to catch before it is a customer's missing project. A COMPARISON_DECLINED line owes a root cause on the same terms, and so does a DEFERRED_BINDS line with a wide gap — the comparator bound that row because nothing else could have owned it, which is the optimistic reading of a slow legacy, and the pessimistic one is a delivery the shadow never received. Neither is decidable from the comparator's own inputs: legacy's rows carry no receipt time. What settles it is onramp-prismatic-api-handler's log group, which records whether legacy received the same delivery the shadow did. Reading that log group is a manual step today; teaching the comparator to join it is scoped separately, since it is a new read with its own failure modes.

     Two root causes are already written. The first is a legacy fault rather than a shadow one. onramp-prismatic-api-handler runs at 128 MB and its postInbound callback occasionally exhausts it — Status: error, Error Type: Runtime.OutOfMemory, Max Memory Used equal to the size. Legacy drops that event and writes no execution row; the provider was acked when the trigger returned, so nothing retries. Ten occurrences in ~54,000 EU invocations over 30 days, the first shadow window included. Confirm one by request id in that lambda's log group before counting it, and check that the OOM predates the graft rather than assuming it: this is an ONLY_NEW_PIPELINE_GENERATED that argues for cutting over, since live takes the handler out of the path entirely.

     The second covers a late legacy claim, and it is not a fault at all — but which of its two readings you are looking at is not decidable from the comparator's own inputs, so both end here.

     Measured, prod over a 13-hour window on 2026-09-09: 112 matched create observations, legacy claim latency p50 12.6s and p95 32.9s, then exactly two outliers at 661s and 771s. hubspot-processor received both webhooks within milliseconds of the shadow's own claim boundary and handed them to a Prismatic instance flow whose queue took 11-13 minutes to call back. The ceiling is not systematically wrong; the queue setting the latency is third-party and unbounded, which is why the deferred pass counts candidate deliveries rather than widening the span.

     Resolve either reading in the legacy handler's log, which the comparator does not read. Legacy's rows carry no receipt time — lastReceivedTime is synthesized by this pipeline only — so find the object's deliveries in that window and read whether legacy acted on the same one the shadow did. Worked example, prod 2026-09-09, vendor 492 ramp 784 ticket 48439625849 (execution 24607): two deliveries seconds apart, the shadow declining one and matching the other. The handler logged did not meet any rule conditions for the delivery the shadow declined and One or more workflow trigger conditions met in full for the one it matched, 11 minutes before the row appeared — so the two pipelines AGREE and the line is explained, not a lost event.

     Which line you are chasing says how urgent it is. A DEFERRED_BINDS line with a wide gap already bound and compared, so the run is green and the handler log only confirms it; a gap the log does NOT account for is the other reading, and is a lost delivery to chase. A COMPARISON_DECLINED line is the contested case — another delivery of the object could equally own that row, so nothing bound and the run is not green. Neither is a divergence; both are a comparator blind spot with a procedure.

   • ONLY_ONE_PIPELINE_CREATED_PROJECT=0, printed as a zero and not as not-asked. This is the only line in the run that answers "did any customer silently lose a project?", and it exists only under --fingerprints — which is why the daily run needs the flag rather than the claim-level comparator alone. ONLY_LEGACY_GENERATED=0 does not cover it: an event where both sides claimed and ran the chain has a shadow observation, so it is not in that count.

   • Zero dry runs that raised. shadow-compare.ts --fingerprints folds this into its own exit code already — a bake is not green while dry_run_raised is nonzero, full stop, independent of the MATCH rate.

   • The recovery kit exists and has been drilled in stage: a sweeper for executions stuck or failed post-flip, a way to replay an observation line as a live claim (for a message the brake caught mid-drain), and a CRM backfill path for whatever the queue's retention window can't cover on its own. "Exists" means exercised against a real stage failure injected on purpose, not merely designed on paper.

   • Per-flow invocation counts justify each Prismatic flow you intend to DELETE. flow_invocations (the dry run's crm_read tally) is the usage evidence — a flow with nonzero invocations across the bake window is still doing something and is not a deletion candidate yet. This is what gates the Prismatic teardown specifically, and that teardown is a separate decision on a separate clock — it does not gate the flip itself, and the flip does not imply it.

   • The DLQ-depth alarm reaches a human. The alarm itself is codified (dlqAlarm on the tier's pipeline member), but the topic it notifies only reaches Slack once it is added to the Amazon Q channel configuration that feeds #eng-alerts — see "Binding a tier's alert topic to Slack" below. An unbound topic raises an alarm nobody receives. Executor-error and divergence-rate alarms land here too.

Binding a tier's alert topic to Slack

Pulumi creates one SNS topic per alerting tier, onramp-<env>-pipeline-alerts, in that tier's own region, and that tier's depth alarm notifies it. Nothing reaches Slack until the topic is added to the account's Amazon Q channel configuration, which authorizes a Slack workspace and is deliberately not Pulumi-managed: no deploy role holds chatbot:*. Until then the alarm transitions normally and nobody is told.

The procedure is in infra/README.md under "Model-EOL alerter (prod alarms member)", with the three traps that make it worth reading rather than improvising: the update replaces the WHOLE configuration rather than just its topic set, so anything not passed back reverts; UserAuthorizationRequired is a flag pair whose wrong half quietly changes who may run commands from the channel; and a payload that is neither a supported AWS service event nor the custom-notification schema is dropped as unsupported while the publish still reports success.

One configuration carries topics from more than one region, so gdpr-prod needs no second binding. A CloudWatch alarm is a supported service event, so a depth alarm needs only the binding — the payload trap applies to anything hand-published to these topics, not to the alarms themselves. 4. Flip the ingress state in a low-traffic window. There is no accompanying PR: the stamp is the only gate, so this one operation stops the forward and starts the writes together. Flag-back returns new events to legacy; the queue drains what it owns; the brake covers haywire.

What the bake does NOT certify

The dry run proves the chain up to its egress checkpoints — nothing past them. suppress_external_events and egress_checkpoint mean the project- created email, project-created automations, the outbound queue publish, and every internal/customer invite email are never actually sent during a dry run; a fingerprint MATCH says the chain would have reached the point of sending them with the right inputs, not that sending itself works. Anything whose correctness lives downstream of a suppressed egress call — a broken email template, a misconfigured automation, a queue consumer that mishandles the payload — is invisible to this bake by construction and needs its own verification before the flip, not an inference from a green comparator.