Reviving the Preview CRM Backends
Every PR preview reviews CRM behaviour against two shared Prismatic instances. They are pinned by instance id in four places. When one breaks, the fix is to revive that instance — not to connect a replacement.
The two backends
| CRM | Backend | CRM org id | Prismatic instance id |
|---|---|---|---|
| Salesforce | foo9 dev org (foo9-dev-ed.develop.my.salesforce.com) | 00DDo000001B9fNMAS | SW5zdGFuY2U6NzMxZTBlODAtNTA1NC00MTYwLWIyMzQtNGU5NGZkOTUzYzc1 |
| HubSpot | "OnRamp CRM - Nonprod" portal | 51740383 | SW5zdGFuY2U6ZDFhMmE0YjctYTUxZC00YWMzLWE1MWItODhlMmE3MDZiNjVh |
They belong to the dev test vendor and are shared by the entire preview fleet — every preview reads the same two connections. That is deliberate and is why a single revive brings every preview back at once.
Both are load-bearing. HUBSPOT is a first-class workflow trigger step type alongside SALESFORCE, so HubSpot is half the surface previews exist to review, and playwright/workflows/tests/hubspot.spec.ts drives its cached schema — it is the CRM the E2E suite actually exercises. What differs is the provider side: Salesforce publish provisions a Flow in foo9, HubSpot publish provisions nothing.
What previews cover, and what they don't
Previews exist to review workflow creation against a live CRM: the trigger step's object / field / option pickers, saving, and publishing (which provisions a real Salesforce Flow in foo9 and removes it on unpublish).
Which code provisions that Flow depends on a flag. Previews run crm_lambda_trigger_provisioning on, so publishing goes through the CRM broker's own port of the logic. The seeded Prismatic create_trigger / delete_trigger flows are the legacy path the flag falls back to — they exist so turning it off on a preview, which is how that rollback gets exercised, still works. Either way the artifact in foo9 is the same ONRAMP_<uuid-no-dashes> Flow, so a revive is verified the same way.
Deliberately out of scope, and not worth "fixing" when it appears broken:
- Trigger events. Inbound delivery is not wired —
inbound_trigger,inbound_laneand the write-back flows are omitted from the seed on purpose, so those paths fail fast rather than mutating CRM objects from an ephemeral environment. - Integration configuration. Connecting or reconnecting a CRM from inside a preview is not supported. Settings → Integrations shows disconnected, matching dev. Previews also reject every POST at the CloudFront → Function URL boundary, so an OAuth callback could not complete there anyway.
Where the instance ids are pinned
Four artifacts, one identity. A Prismatic instance id is minted fresh on every connect, and a get_access_token flow-config id is scoped to its instance. Repointing one artifact at a replacement instance while another still names the old one leaves previews half-wired — fixtures reading instance B while the broker mints against instance A — which is harder to diagnose than a clean failure.
| Artifact | What it pins | Consumed by |
|---|---|---|
devtools/db/build_seed.sh (SEED_SF_INSTANCE, SEED_HS_INSTANCE) | which source rows the golden-seed fixtures are harvested from | seed regeneration |
devtools/db/seed_data.sql (or_integration 38 / 39 + flow rows) | the fixtures every preview clone inherits | the app on a preview |
devtools/db/crm-fixtures.ts (CRM_FIXTURES[].instanceId) | the expected pin, plus the org id and flow count | reseed-neon.ts at reseed time, where it cannot read the generator |
infra/app/envs.ts (ONRAMP_MINT_WEBHOOKS) | each instance's get_access_token flow | the per-PR CRM broker, at Pulumi deploy time |
build_seed.sh is the single home for the ids, the org ids and the flow sets: the pytest parses them out of it rather than restating them, and crm-artifact-pins.test.ts asserts the other three agree with it.
Diagnosing
Symptoms, in the order they usually surface:
- The Workflows builder's CRM trigger step offers no objects, or field pickers come back empty on one CRM only.
- Publishing a Salesforce-triggered Workflow 400s.
- A seed regeneration or the "Reseed Preview DB" workflow fails with
the pinned Salesforce Prismatic instance is gone from the source DB.
Confirm against the source DB (~/.pgpass-dev) — an empty result for either instance means it no longer exists:
select i.id, s.type_code, i.integrator_id, i.service_org_id,
i.additional_org_config->'crm_broker_mint_coords'->>'webhook_url' as mint_url
from or_integration i
join or_integration_service_type s on s.id = i.service_type_id
where i.integrator_id in (
'SW5zdGFuY2U6NzMxZTBlODAtNTA1NC00MTYwLWIyMzQtNGU5NGZkOTUzYzc1',
'SW5zdGFuY2U6ZDFhMmE0YjctYTUxZC00YWMzLWE1MWItODhlMmE3MDZiNjVh');To check what a preview actually holds, query its own Neon branch — never shared dev, which returns plausible rows from an unrelated database. GET /integration on the preview is the stronger check: it must return both service types with the org ids above. connected_integrations in /api/app/config is not that check — it serializes vendor_platform_integrations (direct-OAuth and e-sign) and is legitimately empty on previews and on dev with both CRMs bound.
Access you need
Required. Reviving is a Prismatic operation; everything else only confirms it.
| Need | Why | Set up |
|---|---|---|
prism CLI, logged into US prod | There is no Prismatic MCP. The CLI is the only scriptable path; the Prismatic web UI is the manual equivalent. | bun add -g @prismatic-io/prism, then prism login. Verify with prism me — Endpoint URL must be https://app.prismatic.io. An EU-sandbox login cannot see these instances. |
| Dev Postgres | Confirms the revived instance's rows are what the seed harvests. | ~/.pgpass-dev (see the onramp-db skill). Read-only queries; never print the connection string. |
gh CLI | Dispatches "Reseed Preview DB" so the fleet picks up the rebuilt seed. | gh auth status |
Optional: mcp__salesforce-sobject-all__*, already scoped to foo9. It confirms the ONRAMP_<uuid-no-dashes> Flow a preview publish provisions actually landed — the one artifact a revive creates that the app cannot show you. An absent MCP is not an outage; note it once and carry on.
Not needed: the HubSpot MCP. A HubSpot publish provisions nothing on the provider side, so there is no artifact to go looking for, and the HubSpot read path is already proven by the app serving objects and fields through the broker. The one thing the MCP would be reached for is invalid here: the HubSpot fixture takes its identity from portal 51740383 but its cached schema from a richer dev portal on purpose, so diffing live portal properties against the cache compares two different portals and reports a confident false failure. HubSpot assertions on a preview have to stay cache-internal.
Is a pin actually dead?
One command, no prism login, no database — it POSTs every flow config the artifacts name and reads the status back:
bun devtools/db/check-crm-pins-live.ts401 is a healthy flow demanding the org API key it is supposed to demand. 404 is a flow config that no longer resolves: the instance is gone or was replaced. Anything else is reported as no answer and exits non-zero — a rate limit or a change to Prismatic's auth layer would otherwise be indistinguishable from health.
It probes only the read-only flows. The trigger flows are excluded on the same grounds as the prism guardrail below: they mutate the shared foo9 org, and "the request carries no payload" is not the same as "the request is refused".
The same script takes explicit ids, which is how you confirm a candidate instance's flow set before repinning anything at it. It runs nightly as "CRM Pin Liveness" and pages Slack on the crm-pin-dead keyword; crm-artifact-pins.test.ts is the offline half and cannot see any of this.
Reviving
Prismatic instances live under customer "OnRamp Dev", named Salesforce and Hubspot. Classify first:
prism instances:list -c "$(prism customers:list --columns id,name --output json | python3 -c "import json,sys;print(next(c['id'] for c in json.load(sys.stdin) if c['name']=='OnRamp Dev'))")" -x --output json| What you see | Meaning | Do |
|---|---|---|
enabled: false | disabled, id intact | revive below |
| enabled, but flows missing or stale | needs a redeploy on a current integration version | revive below |
| absent from the list | deleted — the id cannot come back | last resort below |
present, wrong service_org_id in the DB | repointed at a different CRM org | reconnect to the org in the table above, then revive |
Then revive under the same id — instances:deploy preserves it, which is the whole point:
prism instances:enable SW5zdGFuY2U6NzMxZTBlODAtNTA1NC00MTYwLWIyMzQtNGU5NGZkOTUzYzc1prism instances:deploy SW5zdGFuY2U6NzMxZTBlODAtNTA1NC00MTYwLWIyMzQtNGU5NGZkOTUzYzc1Re-run the diagnosis query. The instance must show its expected service_org_id and a full flow set: Salesforce needs list_objects, list_object_fields, list_object_field_options, search_objects_by_single_field, fetch_objects_by_ids, create_trigger and delete_trigger; HubSpot needs the reads plus list_related_objects and no trigger flows.
With the instance id unchanged the committed seed is already correct — no code change, the fleet just needs the reseed:
gh workflow run reseed-preview-db.ymlreseed-neon.ts verifies the fixtures before resetting any branch, so a still-broken backend fails the run instead of propagating.
prism guardrails. Never pass -t — it streams and never self-exits, and macOS has no timeout binary to cap it. Never prism instances:flow-configs:test a flow you have not identified: create_trigger, delete_trigger and update_object_by_id mutate the real CRM (get_access_token is read-only and safe). Never instances:delete either of these — that is the failure this procedure exists to avoid. foo9 is shared with every open preview.
If the instance genuinely cannot be revived
Only then does a new instance id enter the repo, and all four artifacts move together in one PR:
- Reconnect the backend to the same CRM org — the org ids above do not change.
SEED_SF_INSTANCE/SEED_HS_INSTANCEindevtools/db/build_seed.sh.- The matching
instanceIdindevtools/db/crm-fixtures.ts. - Regenerate the seed —
bash devtools/db/build_seed.sh "$(cat ~/.pgpass-dev)"— which harvests the new instance'sget_access_tokenURL into the fixture. - That same URL into
ONRAMP_MINT_WEBHOOKSininfra/app/envs.ts.
A full regeneration also picks up every unrelated drift the source DB has accumulated since the last one, which lands in the same PR. When that diff is large enough to bury the repin, swapping the instance id and the flow webhook URLs in place is equivalent for everything load-bearing — take the replacement URLs from the source DB's own or_integration_workflow rows so they are the same values a regeneration would harvest, and say in the PR that you did. Diff a scratch regeneration against the result to prove nothing else moved.
bun test devtools/db/ must pass before the PR goes up; crm-artifact-pins.test.ts fails while any artifact still names the old instance, which is the point. Then merge (the reseed fires on the seed change) and redeploy any preview needing the new broker config.
Looks like this, isn't this
A Salesforce publish can 400 with the instance perfectly healthy: provisioning a Flow needs Metadata API permissions on the foo9 integration user, and that user has been demoted before. The nested Salesforce error never reaches the toast — read the Flask log. That is a foo9 permissions fix, not a revive.
The "Deploy preview (pipeline)" job failing on execution reaches a terminal state / oversized event is parked in S3 is not this either — that fixture vendor is a separate, third connection behind the integrations-pipeline executor's pre-claim hydrate, not one of the two pinned here. See docs/runbooks/integrations-pipeline-shadow.md's "Known: the preview...smoke check has been red fleet-wide" callout.
The gates that catch a drift
| Gate | Runs | Catches |
|---|---|---|
build_seed.sh source checks | seed regeneration | pinned instance gone, repointed at a different CRM org, or missing a required flow |
build_seed.sh artifact checks | seed regeneration | fixtures or flow rows that never reached the generated file |
app/api/integration/tests/test_seed_crm_fixtures.py | every PR | a committed seed that lost a fixture, a flow, or grew a mutating one |
devtools/db/crm-artifact-pins.test.ts | every PR | the four artifacts naming different instances, org ids or flow sets |
devtools/db/reseed-neon.ts verification | the reseed | a rebuilt golden parent whose CRM surface is inert, before the fleet resets onto it |
Each one prints a paste-into-Claude prompt on failure — with the offending instance id and a pointer back here — so a red CI log is enough to start from.