Prismatic Decoupling — First Step (v1)

The scoped first slice toward the Integrations North Star: own trigger evaluation (and execution) inside OnRamp, behind a provider-agnostic seam, and tune up the workflow-config sidebar. This reduces Prismatic to an auth broker only — OAuth consent, refresh-token storage, and a mint-access-token endpoint our backend calls; it no longer executes inbound flows. Draining Prismatic's tokens and removing it entirely is a later phase, not this one. Higher-reach increments (custom-field polling, a Lambda-Web-Adapter pipeline sidecar) are scoped as pull-in stretches, so a strong core can land first and the rest gets pulled in as time allows.

Owner: Ross RasmussenCreated: 2026-06-22

💬 Feedback wanted — draft. A GitHub Discussion (Ideas) will be linked here for comments before we commit. Open questions are flagged in Risks & open questions.

Scope: ~1 engineer, ~1 month for the core. A deliberately narrow committed core, with a stretch menu to pull from if the core lands early. Phased rollout, idempotency focus, and canary + cutover discipline throughout.

What the core delivers

🅰 Own the foundation

Broker tokens + secure ingress

Consume Prismatic's mint-access-token endpoint (cache + single-flight the short-lived tokens), and verify HubSpot webhook signatures. The base everything else stands on.

🅱 Own evaluation, behind a seam

Envelope + one adapter

Relocate trigger evaluation into OnRamp behind a provider-agnostic envelope + adapter interface (HubSpot). Makes polling and Salesforce additive later, not rewrites.

🅲 Workflow sidebar UX

Tune-up, not rehaul

Same look and feel, better form inputs; collapse the two near-duplicate CRM trigger components into one schema-driven control. (Goal 3.)

Goal mapping (be honest about what's core vs pull-in): the sidebar UX (goal 3) lands in the core. The marquee goals — evaluate-in-OnRamp end-to-end and custom-field-change triggers (goals 1 & 2, retiring the onramp_condition_* hack) — are closed by the custom-field polling stretch; the core builds the evaluation + seam (and wires OnRamp to the Prismatic broker for access tokens) that the stretch needs. Caching (goal 4) is a stretch.

What we reuse vs. build

The foundation is mostly in place — the core consolidates and secures it. Net-new is deliberately narrow.

Capability	In place today	Core work
CRM access tokens	Prismatic stores the refresh token + mints access; OnRamp also self-refreshes for some Flask calls (dataloaders.py:59-90)	Consume Prismatic's mint-access endpoint; cache + single-flight access tokens (encrypt the cache at rest)
Direct CRM REST clients	HubspotDataLoader / SalesforceDataloader call the CRM APIs directly, no Prismatic	Reuse (polling stretch leans on these)
Predicate evaluator	CRM-agnostic condition logic in workflow-trigger-helper.js	Relocate behind the seam; add a CRM-agnostic discovery path (it hardcodes HUBSPOT)
Schema cache	Objects / fields / related cached in or_integration.schema_definition	Reuse (caching stretch extends to picklists + owners)
Inbound auth	No HubSpot webhook signature verification	Add it (Pillar 0)
Async execution	async-automation-handler creates execution records + invokes Flask	Relocate its logic; keep the boundary
Lambda infra	DuploCloud + Serverless Framework; SQS FIFO	Reuse as-is

Core net-new is small: the envelope/adapter seam, a CRM-agnostic routing + discovery path, the unified trigger component, and the security/correctness fixes. The poller, watermark state, and transition-identity dedup come with the polling stretch.

Core pillars

Sequenced so each ships independently — chip away, and a slip in one doesn't sink the rest.

Pillar 0 — Foundation (broker tokens + secure ingress)

The base layer. Small, high-value, mergeable on its own in week 1.

• Consume the Prismatic broker — call its mint-access-token endpoint for short-lived CRM access tokens (build the Prismatic flow for it if absent). Prismatic keeps the OAuth consent flow + refresh-token custody this phase; OnRamp does not take refresh tokens yet (that's Phase 5, the sunset).
• Cache + single-flight the access tokens — short-TTL cache with a pg advisory lock or Redis SETNX so concurrent callers resolve to one mint call, not a stampede on Prismatic. Encrypt the cached tokens at rest (codified config — no new env vars).
• HubSpot webhook signature verification — verify X-HubSpot-Signature-v3 (HMAC over raw body + timestamp + URL, keyed by app client secret) and reject stale timestamps in the acker. Documented open security hole (hubspot-acker/handler.js:68-73).
• Fix the MANY_TIMES dead-case typo (workflow-trigger-helper.js:399, request_process_workflow.js:502).

Pillar 1 — Own evaluation behind the seam

Relocate trigger evaluation into OnRamp behind a provider-agnostic contract, and build the routing seam so the polling stretch drops in additively.

flowchart TB
    T[Trigger config] --> R{Routing seam}
    R -->|webhook-able| W[Webhook-owned path]
    R -.->|stretch: custom-field signal| P[Poll-owned path]
    W --> E[Shared CRM-agnostic evaluator]
    P -.-> E
    E --> X[Execute, dedup by transition identity]

• Formalize the runtime envelope (CloudEvents context attributes + canonical type vocab) over today's de-facto {source, objectType, objectId, objectDetails} shape. It is less normalized than it looks — dispatcher-stage objectDetails items carry {name, value} only; type appears after hydration.
• Define the adapter interface with HubSpot as the single implementation. No generic multi-source framework — one adapter on the interface.
• Relocate the evaluator behind the seam and add a CRM-agnostic ramp-discovery path (today getActiveRamps hardcodes trigger.code:'HUBSPOT').
• Build the routing seam (webhook-owned vs poll-owned) but wire only the webhook side in the core; the poll side is the stretch.
• Owning evaluation removes the Prismatic round-trip in the trigger path (today: processor → Prismatic inbound_trigger flow → callback → async-automations.fifo → async-automation-handler → Flask). That round-trip is what forces the redundant queue + lambda; collapsing them is stretch B.
• Reserve envelope room for later phases (origin for write-back echo suppression) without implementing them.

Pillar 2 — Workflow sidebar UX + one trigger component — Goal 3

A light tune-up, explicitly not a workflow rehaul.

• Collapse HubspotTriggerStep.vue and SalesforceTriggerStep.vue (two ~450-line near-duplicates with divergent operator vocab) into one schema-driven trigger-config component fed by a normalized field-schema descriptor + canonical operator vocab — the design-time face of the envelope. v1 implements the descriptor for HubSpot; SF rendering through the same control follows.
• Improve the form inputs and interaction in the sidebar — same general look and feel, better UX. Build on the org PrimeVue design system (auto-loaded UI rules apply).

Stretch menu — pull in if the core lands early

Each is self-contained and independently valuable. Pull in any combination as time allows; the core's seam makes them additive, not rewrites.

⭐ A — Custom-field polling (the marquee)

Closes goals 1 & 2 and retires onramp_condition_*. Wires the poll-owned side of the routing seam: a scheduled poller (per vendor × object-type × watched-custom-field-set) pulls records modified since a persisted watermark, runs the shared evaluator, and dispatches matches. Adds the MANY_TIMES transition-identity dedup across push and poll. Depends on: Pillar 0 + Pillar 1. Scope discipline: HubSpot only, in-use fields only, cadence tuned to rate limits.

⭐ B — Simplify the pipeline (Lambda Web Adapter)

Collapse the redundant inbound plumbing. Run the webhook-processing slice as the Python Flask app on Lambda via Lambda Web Adapter — reusing the same SQLAlchemy ORM, services, and evaluator, with no TS re-implementation and no HTTP hop to a separate Flask server. Net: drop the async-automations.fifo queue + a lambda; relocate async-automation-handler's logic (create execution records + invoke the executor) into the Lambda. Self-contained in serverless-land end to end. Long-term seed: this is the integration domain's first rung on the Domain Lambdaliths ladder (docs/proposals/domain-lambdaliths.md: container LWA → per-domain FastAPI → managed runtime + SnapStart) — so build it SnapStart-ready (lazy DB connect, no native deps at import, published alias). This iteration is just the processing sidecar, not a fleet-wide migration. Pairs with A — the poller lives in the same Python-on-Lambda runtime.

C — Caching audit (goal 4)

Workflows-page performance: picklist values and owner/entity lookups are uncached and hit Prismatic on every render (integration_service.py:403), while object/field/related reads are already cache-backed. Add a TTL'd cache for the uncached reads — the Postgres cache pattern already exists; extend it.

D — Seed the monorepo

Retire the crm-integrations meta-repo (a bun workspace stitching ~9 sibling JS/TS repos) by folding the integration fabric into main-web-application — see the North Star's Repo topology section. The core already authors new code in main-web-application; this pulls in the existing pieces it touches. Wholesale elimination (all repos + their Duplo/Serverless wiring) is a follow-on.

Sequencing

flowchart LR
    P0[Pillar 0<br/>Foundation] --> P1[Pillar 1<br/>Own eval + seam]
    P0 --> P2[Pillar 2<br/>Sidebar UX]
    P1 --> CORE{{Core done}}
    P2 --> CORE
    CORE -.pull in.-> SA[Stretch A<br/>Polling]
    CORE -.pull in.-> SB[Stretch B<br/>LWA pipeline]
    CORE -.pull in.-> SC[Stretch C<br/>Caching]
    CORE -.pull in.-> SD[Stretch D<br/>Monorepo]

Pillar 0 unblocks everything and ships first. Pillar 2 (frontend) runs in parallel with Pillar 1 (backend). Once the core is done, pull stretches in any order — each stands alone.

Non-goals (this initial effort)

• Sunsetting Prismatic. This phase reduces Prismatic to the auth broker (consent + refresh storage + mint-access endpoint) and moves inbound execution into OnRamp — but it does not drain Prismatic's refresh tokens or remove it. Outbound metadata reads also still flow through Prismatic. Draining tokens → own auth server → full sunset is Phase 5.
• HS / SF trigger parity. Salesforce keeps its in-CRM Flow triggers for now; relocating SF evaluation is a later phase.
• Standing up the domain-lambdalith fleet. The long-term direction is a few domain-scoped LWA lambdaliths (the Domain Lambdaliths RFC; North Star Runtime direction), but v1 only stands up the integration domain's processing sidecar (stretch B) — not the fleet, and not a big-bang migration.
• Write-back, agent/MCP surface, generic connector framework — later phases.

Definition of Done

Core:

• [ ] CRM tokens encrypted at rest; refresh is single-flight; custody path per (vendor, CRM) is unambiguous.
• [ ] HubSpot inbound webhooks signature-verified; stale timestamps rejected; MANY_TIMES typo fixed.
• [ ] Trigger evaluation runs in OnRamp behind the envelope + adapter interface (HubSpot), with a CRM-agnostic discovery path and the webhook-side routing wired.
• [ ] One schema-driven trigger-config component renders HubSpot triggers; sidebar form UX improved.
• [ ] The seam demonstrably would accept the poll side, Salesforce, and write-back.
• [ ] No regressions: existing webhook path and execution lifecycle intact.

Per stretch (only the bar for what's pulled in):

• [ ] A — Polling: HubSpot custom-field-change triggers fire via the poller; onramp_condition_* retired; no double-fire with the webhook path; MANY_TIMES deduped on transition identity.
• [ ] B — Pipeline (LWA): webhook processing runs as Flask-on-Lambda via Lambda Web Adapter, reusing the ORM/services; the async-automations queue + async-automation-handler are removed (their logic relocated); no execution-lifecycle regression.
• [ ] C — Caching: picklists + owners TTL-cached; workflows-page render no longer round-trips Prismatic for them.
• [ ] D — Monorepo: the integration pieces v1 touched live in main-web-application and build/deploy from there.

Risks & open questions

HighPoll volume exhausts HubSpot rate limits

(If stretch A is pulled in.) Polling for custom-field changes adds API load (HS Pro ≈ 100 req / 10s, 250k/day).

Mitigation: poll only non-subscribed-custom-field triggers (the routing seam keeps the webhook majority off the poller); watermark-scoped queries; caching is load-bearing; tune cadence before canary expansion.

HighPartition double-fires (webhook + poll)

(If stretch A is pulled in.) A trigger handled by both paths fires twice.

Mitigation: strict ownership routing per trigger; MANY_TIMES transition-identity dedup as backstop; canary on one vendor first.

MedPrismatic broker on the token hot path

The mint-access-token endpoint now sits in the path of every CRM call — if Prismatic is slow, down, or rate-limits, CRM access degrades.

Mitigation: short-TTL access-token cache + single-flight (Pillar 0) so steady-state traffic rarely hits Prismatic; alert on mint-endpoint errors. Refresh-token-store divergence is a Phase-5 (sunset) concern, not v1.

MedScope creep from the sidebar tune-up

"Better UX" silently becomes a workflow rehaul.

Mitigation: same look and feel is a hard constraint; descriptor-driven component + form inputs only.

Open questions

Poller language, placement & schedule? (Stretch A.) Python (Flask scheduled endpoint or Flask-on-Lambda via LWA, reusing dataloaders.py + the evaluator) vs TS (scheduled Lambda reusing the JS evaluator + onramp-database-js). If stretch B (LWA) is in play, Python-on-Lambda is the natural pick — one runtime end to end. Also picks the first monorepo brick. Decide at the spike.

Cutover granularity? Per-trigger-type parallel-run + shadow-compare vs. per-vendor canary. Lean parallel-run for confidence.

Sunset-phase auth server — custom vs. Nango? Deferred to Phase 5, not v1. The drain (a Prismatic flow exporting refresh tokens) means no re-auth migration tax, so both are viable; OnRamp's existing self-refresh (dataloaders.py) is evidence the custom path is cheap. v1 keeps Prismatic as the broker and doesn't touch this.

Watermark store? (Stretch A.) A small Postgres table keyed on (vendor, object-type, trigger) vs. reuse of an existing state surface.

Cross-cutting

• Canary — one vendor, vendor-scoped flag; same canary set across pillars (different populations break attribution).
• Observability — poll lag, dedup hit/miss, signature-verify failures, cache hit ratio, DLQ depth; surface in existing dashboards.
• Security — tokens encrypted at rest; signature verification on ingress; OAuth scope mismatch → graceful "reconnect" UX, not a 500.

References

• North Star: Integrations North Star
• Goals sourced from PO scoping discussion, 2026-06-22.
• Grounded in the current CRM integration code (crm-integrations/ + app/api/integrations/).