RFC: Serverless PR Previews

Stand up a full, throwaway preview of the OnRamp web app (Flask API + Vue SPA) per pull request on AWS serverless — one container-image Lambda plus a Neon database branch, fronted by a single CloudFront distribution with a per-PR subdomain.

Owner: Ross RasmussenCreated: 2026-06-05

✅ Phase 1 (straw man) and Phase 2 (lifecycle automation) are both implemented. The preview fleet is live at preview.onrampapps.com. As-built findings and status notes are woven throughout; an As-built addendum at the end captures what the implementation revealed.

TL;DR

🅰 Compute

One Lambda per PR

The existing app as a single container-image Lambda (a "lambdalith") behind Lambda Web Adapter. Scale-to-zero, VPC-less.

🅱 Data

One Neon branch per PR

A golden-seed Neon branch over its pooled endpoint. The Lambda needs nothing but the connection string.

🅲 Routing

One distro, subdomain per PR

A single CloudFront distribution on a wildcard cert. A CloudFront Function picks the PR's origin per request — adding a PR is one KeyValueStore row.

Why, and what this is not

PR review today has no running full-stack environment — reviewers read diffs, or spin up a heavyweight env. This gives every PR a real, isolated, throwaway URL for a few cents. The preview stack is self-contained AWS serverless, provisioned by its own IaC and independent of the existing dev/stage/prod infrastructure — it sits alongside what's there and adds nothing to maintain on it. This RFC is about previews only.

✅ Goals

• Per-PR full-stack preview (API + SPA + DB) at a stable URL.
• Auto-create on PR open, auto-destroy on close.
• Cheap (scale-to-zero) and low-maintenance.
• Self-contained AWS stack, provisioned by its own IaC (SST/Pulumi).

🚫 Non-goals

• Not the prod/stage architecture (could be reused later, not proven here).
• No changes to existing dev/stage/prod infrastructure — previews sit alongside it.
• Not behaviorally 100% faithful (see Known limitations).
• Not a production security tier — synthetic data only.

Build vs buy

This isn't build-vs-buy in the abstract — it's build (this RFC) vs. buy a managed preview product (Render / Shipyard-class). Build wins here, for situation-specific reasons (not as a universal rule).

Why not buy — and the Kubernetes overhead this avoids. The in-house, non-serverless way to do previews is per-PR namespaces on managed Kubernetes, which carries standing operational overhead that scale-to-zero serverless sidesteps entirely:

• control-plane / cluster version upgrades on a cadence
• node patching and OS/AMI maintenance
• capacity + autoscaling tuning, paying for idle headroom
• CNI / ingress / service-mesh / RBAC wiring
• always-on cost — you pay for idle nodes, but preview traffic is bursty

A managed preview SaaS hides that overhead but re-vendors it — swaps the work for a third-party platform dependency and per-seat cost. For a stack we're deliberately building on first-party AWS primitives, either buy path is a poor fit.

Why build wins here:

• Expertise — deep in-house AWS / CloudFront / IaC, so the team can specify and review what gets built.
• AI collapses the build cost — an agent writes the lifecycle glue and orphan sweeper; that leverage is strongest exactly where there's expertise to review the output.
• Dual-purpose — the IaC modules can be reused beyond previews (see open questions), so the build isn't single-use.
• Cost — scale-to-zero beats always-on k8s nodes or per-seat SaaS for bursty preview traffic.
• No re-vendoring — stays on first-party AWS primitives.

Trade-offs we accept by building (price these consciously — they aren't free):

• We own preview ops forever — half-failed provisions, concurrent-push races, stale KeyValueStore rows, cert renewal, the orphan sweeper. A vendor amortizes that across thousands of customers; here it's our pager.
• Non-engineer access — PMs / PO / UX want a dashboard (status, logs, one click); the floor here is a PR-comment URL. An "anyone can see why pr-123 is broken" layer is extra to build.
• Latency as an adoption tax — slow image builds + multi-second cold starts can make previews feel sluggish enough that people quietly stop using them. Watch adoption, not just cost.

Guardrail — this verdict is situation-specific. Cheaper AI-assisted build is not a reason to build in domains we lack the expertise to evaluate (auth providers, payment rails — buy those). Build where we can review what we built; previews qualify.

Architecture

Request flow — one distribution serves both the SPA and the API on the PR's subdomain, so the browser sees a single same-origin host (no CORS):

flowchart LR
    R["Reviewer browser<br/>pr-123.preview.onramp.us"] --> CF["CloudFront<br/>*.preview.onramp.us<br/>wildcard cert"]
    CF -->|"default"| FD["CF Function<br/>map Host to S3 prefix"]
    FD --> S3["S3 bucket<br/>pr-123/ (Vue SPA)"]
    CF -->|"/api/*"| FA["CF Function (runtime 2.0)<br/>updateRequestOrigin"]
    FA -.->|"Host lookup"| KVS["CloudFront<br/>KeyValueStore"]
    FA --> L["Lambda Function URL<br/>OAC-signed, container, VPC-less"]
    L --> NEON["Neon branch pr-123<br/>pooled endpoint"]

PR lifecycle — open provisions, close tears down:

flowchart TD
    OPEN["PR opened"] --> GHA["GitHub App / Action"]
    GHA --> B1["Neon: branch from golden seed"]
    GHA --> B2["Build + push image to ECR"]
    GHA --> B3["Create Lambda + Function URL (OAC)"]
    GHA --> B4["Build Vue, upload to S3 pr-123/"]
    GHA --> B5["Add KeyValueStore row for pr-123"]
    B5 --> LIVE["Live at pr-123.preview.onramp.us"]
    CLOSE["PR closed / merged"] --> TEAR["Teardown: Lambda, image,<br/>S3 prefix, Neon branch, KVS row"]

Hard requirements

These are the non-negotiable constraints. Everything else is the implementer's call (see Recommended approach).

1. Subdomain routing + wildcard cert. Not path-based. PRs are addressed as pr-<n>.preview.onrampapps.com, served by one CloudFront distribution with an ACM wildcard cert *.preview.onrampapps.com (ACM certs for CloudFront must live in us-east-1). The preview.onrampapps.com child zone is delegated from onrampapps.com. Path-based routing (/pr-123/) is rejected — it forces a non-root Vue Router / Vite base and URL-rewrite gymnastics.
2. Container-image Lambda, not zip. The app's dependency closure is well over the 250 MB unzipped zip-Lambda cap, and includes native libraries (e.g. xmlsec, pymupdf, lxml) — so a container image is mandatory. Run the whole app as one function (lambdalith) behind Lambda Web Adapter.
3. VPC-less Lambda. Neon is reached over its public TLS endpoint; nothing the Lambda touches is private. Keep the function out of a VPC — this avoids a NAT Gateway (an always-on cost) and the VPC cold-start penalty. Do not add a VPC for previews.
4. Neon pooled endpoint + transaction-mode-safe SQLAlchemy. Use Neon's -pooler (PgBouncer) host. Because that's transaction-mode pooling, the app must use NullPool and disable prepared-statement caching, or connections will break.
5. Golden-seed / synthetic data only — never production data. The relaxed network posture (public DB endpoint, lower security tier) is safe only on synthetic data. A preview branch must never be hydrated from prod.
6. Function URLs locked to CloudFront via OAC. Per-PR Lambda Function URLs are not world-open; restrict them with Origin Access Control (CloudFront signs requests).
7. Teardown on PR close/merge. Provisioning is reversible: deleting the Lambda, image, S3 prefix, Neon branch, and KeyValueStore row must be automatic. Orphans cost money and leak surface.

Recommended approach (adapt freely)

These are suggestions, not requirements — pick what fits.

• Dynamic origin dispatch: a CloudFront Function (JavaScript runtime 2.0, import cf from 'cloudfront') at viewer-request, calling updateRequestOrigin() keyed off the Host subdomain, with the pr-N → Function URL map in a CloudFront KeyValueStore. This is the chosen design because adding/removing a PR is a single KVS row — no distribution redeploy. (Alternatives — a per-PR distribution, or Lambda@Edge — work but are heavier; only reach for them if KVS dispatch doesn't pan out.) Implemented: infra/pr-previews/cf-fn-api-dispatch.js.
• App packaging: run the existing Flask app behind Lambda Web Adapter as-is. The planned a2wsgi Flask-under-FastAPI mount is the forward direction but is not required for previews to work. Implemented as-is.
• IaC: Pulumi TypeScript (infra/pr-previews/). Three stack shapes: hello (Gate 1 smoke test), shared (long-lived infra), pr-<N> (ephemeral per-PR). Modules are structured for reuse.
• Lifecycle trigger: GitHub Actions, every PR. .github/workflows/preview-deploy.yml handles open/synchronize/closed; a nightly sweeper (.github/workflows/preview-sweeper.yml) catches orphans. Implemented — see Phase 2 addendum below.
• Image build: GitHub Actions on native amd64 runners — QEMU emulation is gone. A prebuilt base image (docker/flask/flask-lambda-base.dockerfile, content-hash tagged) absorbs apt + uv sync, so PR builds are a FROM base COPY . . layer. Implemented.
• SPA hosting: one shared S3 bucket, per-PR prefix (pr-123/); the default CloudFront behavior maps Host → prefix via originPath.
• Lambda sizing + cold starts: start around 1–2 GB memory (the lambdalith's import weight), then tune. Note SnapStart is not available — it doesn't support container images (requirement 2) or provisioned concurrency. Container cold-start levers are image-size trimming, memory tuning, and provisioned concurrency (always-on, so it fights scale-to-zero — use sparingly). For throwaway previews, accepting the cold start is usually fine.
• Not all app endpoints live under /api/*. Some Flask routes (OAuth callbacks, portal paths, etc.) are outside the /api/* path pattern. The default CloudFront behavior must therefore also proxy to Flask, not only serve S3. Implemented.

Authentication

OnRamp's primary login is cookie-based session auth, not OAuth — so dynamic subdomains don't break it. Login is a username/password form that sets a Flask session cookie plus a double-submit CSRF-TOKEN cookie (and a JWT identity), all scoped to the request host. Because requirement 1 serves the SPA and API on one same-origin host per PR, each preview's cookies are host-scoped and self-contained — no cross-subdomain cookie sharing, no per-PR redirect-URI registration. The redirect-URI problem only exists for the optional OAuth SSO path, addressed second below.

Two tiers, build the second only when a PR needs it:

• Default — seeded users + the normal login form. The golden seed ships dummy users with known passwords (admin@example.com, customer@example.com), so a reviewer just logs in through the standard form; the app sets its session + CSRF cookies on pr-<n>.preview.onrampapps.com and that's it. Cookies default to SameSite=None, which requires Secure — previews are HTTPS under the wildcard cert (requirement 1), so set USE_SECURE_COOKIES=true in the preview env and it holds (same-origin would also work at SameSite=Lax). This covers the large majority of PRs with zero OAuth machinery.
• When a PR exercises the real @onramp.us Google SSO path — broker it through AWS Cognito. OnRamp's existing SSO (per-org Google / SAML in app/api/integrations/sso/) needs pre-registered redirect URIs that dynamic subdomains can't satisfy. Rather than register a callback per PR, a shared Cognito user pool as a broker fronts Google SSO for the onramp.us domain: one fixed callback on the Cognito hosted-UI domain covers the whole fleet. See PR-preview Google SSO for the as-built details.

Known limitations (accepted)

Previews are roughly 95% behaviorally faithful. The gaps below are accepted — don't treat them as bugs to fix, and don't let them block the build. They cluster in async side-effects, file storage, and integrations.

• In-process background work doesn't complete. Lambda freezes the execution environment after the response, so Flask-Executor (submit_stored), app/api/utils/background_jobs.py, and fire-and-forget Thread() calls (CRM/user sync, thumbnails, analytics) silently stall. Moving that work to a queue + worker is a prod concern, out of scope here.
• Local-filesystem files don't persist. File download serves from file.stored_path on local disk; on Lambda that path won't survive across invocations. File up/download is degraded unless backed by EFS or S3 — out of scope for previews.
• One SSE endpoint degrades (playbook edit-lock). Low blast radius.
• ~6 MB Function URL response cap — large exports (e.g. project ZIP) may fail.
• EasyCron-triggered endpoints aren't wired — scheduled jobs don't run in previews.

Cost

• Compute is near-free — scale-to-zero; preview traffic likely sits inside the Lambda free tier. Function URLs add no charge (no API Gateway).
• The thing that would blow the budget is a NAT Gateway (~$32/mo per AZ, always-on). Staying VPC-less (requirement 3) avoids it — this is the single biggest cost lever.
• Small standing costs: ECR image storage (expire old images via lifecycle policy) and Neon branch storage (autosuspends; destroyed on PR close).
• Ballpark: under ~$10–20/mo of AWS for a modest fleet, VPC-less and cleaned up. Break either of those and it stops being cheap.

Implementation phases

Build a straw man first, automate second. Prove the whole request path works for a single hand-built PR before writing any automation — the automation is wasted effort if the underlying stack doesn't serve traffic correctly.

1. ✅ Straw man — one PR, by hand. Manually stand up the full path for a single PR: Neon branch, build + push image, Lambda + Function URL + OAC, S3 upload, CloudFront distro + wildcard cert + CF Function + one KVS row, and a seeded-user login (USE_SECURE_COOKIES=true). Goal: a working pr-<n>.preview.onrampapps.com end to end. No automation yet. Done — PR #9178.
2. ✅ Automate the lifecycle. GitHub Actions on every PR, open/synchronize/closed → provision/teardown; KVS row add/remove; PR comment with preview URL + status; nightly orphan sweeper. GitHub OIDC identity (infra/ci-identity/, role onramp-deploy-previews) — no stored AWS secrets. The legacy infra/pr-previews/ci-identity/ directory and pr-previews-deployer role are retired by the shared-apply cutover. Neon branch management via devtools/db/preview-branch.ts. Done — .github/workflows/preview-deploy.yml, .github/workflows/preview-sweeper.yml.
3. Harden + polish. Orphan sweeper is running; cost guardrails in place. Cognito SSO broker (only if a PR needs real SSO) is deferred — seeded-user login has covered all PRs to date. Post-merge preview revival is not yet implemented.

Risks & open questions

HighProd data in a preview

The relaxed posture (public DB endpoint, lower tier) is only safe on synthetic data.

Mitigation: golden-seed-only invariant (requirement 5); provisioning refuses any non-seed source; never clone prod into a preview branch.

MedPgBouncer transaction-mode breakage

Neon's pooled endpoint breaks prepared statements / session state if the ORM isn't configured for it.

Mitigation: NullPool + disable prepared-statement caching; verify against the pooled endpoint early (requirement 4).

MedCold-start latency

The lambdalith's heavy imports make cold starts multi-second, and sporadic preview traffic rarely stays warm.

Mitigation: trim image size + tune memory. SnapStart can't help (no container-image support); provisioned concurrency works but is always-on, fighting scale-to-zero. Cold starts are accepted for throwaway previews.

LowOrphaned resources → cost leak

A missed teardown leaves a Lambda, image, branch, or KVS row billing.

Mitigation: teardown on close + a scheduled orphan sweeper + ECR lifecycle policy.

Open questions (resolved)

IaC tool — SST or Pulumi? ✅ Resolved: Pulumi TypeScript (infra/pr-previews/), structured for reuse.

Image build location + speed. ✅ Resolved: GitHub Actions on native amd64 runners. Prebuilt base image (flask-lambda-base) absorbs the slow apt/uv sync layer; PR builds are a COPY-layer build. Straw-man was ~15 min (QEMU under colima); Phase 2 is ~1–2 min for Python-unchanged PRs.

Integration stubbing. Partially resolved: CRM, outbound email, and EasyCron are disabled via env flags in the preview environment. Full enumeration lives in the preview stack config.

Neon branch seeding. ✅ Resolved: branch from a golden-seed parent; Neon's reset_to_parent preserves endpoints (no Lambda config churn on reseed). Connection string injected as a Lambda env var by devtools/db/preview-branch.ts.

Is seeded-user login enough for all previews? ✅ Resolved in practice: all PRs to date have used seeded-user login. Cognito SSO broker deferred indefinitely.

Author the IaC modules for reuse, or preview-only? ✅ Resolved: modules structured for reuse (components/ in infra/) but extracted on second use, not speculatively.

As-built addendum

These are empirical findings from Phase 1 (straw man) that shaped the Phase 2 implementation. Each one broke the straw-man preview and was fixed before the lifecycle automation was built.

OAC for dynamically selected Lambda Function URL origins

CloudFront's updateRequestOrigin() (CF Function runtime 2.0) accepts an inline originAccessControlConfig that controls SigV4 signing toward the dynamically selected Lambda Function URL. This inline config is the only legal OAC mechanism for this pattern. Attaching a lambda-type OAC to the static placeholder origin (the distribution's configured origin before the CF Function rewrites it) is rejected at distribution create with IllegalOriginAccessConfiguration: origin type and OAC origin type differ — the placeholder is a custom origin, not a lambda origin, so the type check fails. The CF Function approach bypasses this by configuring signing dynamically at request time. Source and implementation: infra/pr-previews/cf-fn-api-dispatch.js header comment.

CloudFront OAC SigV4 does not hash request bodies

CloudFront's OAC signing for Lambda Function URLs omits the request body from the SigV4 hash (it always signs with UNSIGNED-PAYLOAD). Lambda Function URLs validate the x-amz-content-sha256 header on every non-GET/HEAD request — including bodyless POSTs — and reject requests where it is absent or inconsistent. The viewer must supply a correct x-amz-content-sha256 header before CloudFront signs the request. For bodyless requests (empty-string body), the correct value is the SHA-256 of an empty string. Implemented in app/onramp/services/previewBodyHash.js.

Lambda Function URLs reject raw RFC3986-reserved characters in query strings

Lambda Function URLs return an empty 400 response (before the function code runs) for any query string containing unencoded RFC3986-reserved characters: { } [ ] :. Browsers and axios leave these unencoded in query strings (they are technically legal in query components but Lambda's URL routing rejects them). The app/api/views/ filter serialization uses JSON-in-querystring for complex filter objects, which regularly produces these characters. Fixed by percent-normalizing every query component (decode → re-encode) in the viewer-request CF Function before the origin rewrite. Implementation: infra/pr-previews/cf-fn-api-dispatch.js.

Default CloudFront behavior must also reach Flask

Not all app endpoints live under /api/*. OAuth callbacks, portal paths, and other Flask routes sit outside that path pattern. The default CloudFront behavior (which the RFC's original diagram showed routing to S3 only) must also proxy to the Lambda when a KVS row is present for the PR. Otherwise those endpoints return a 403 or 404 from S3.

Lambda rejects OCI image-index manifests — use --provenance=false

docker buildx by default pushes a multi-platform OCI image index (a manifest list) even for single-arch builds when provenance attestations are enabled. Lambda's CreateFunction / UpdateFunctionCode reject OCI image-index manifests; they require a single docker-media-type manifest. The fix is --provenance=false on every docker buildx build --push in CI. This is enforced in .github/workflows/preview-deploy.yml.

SPA redeploy ordering: image first, then S3

Flask (running inside the Lambda image) renders the SPA shell, so the shell's <script> chunk references are baked at image-build time. Running s3 sync --delete before the new Lambda image is live deletes the chunks the currently-running shell still references — the preview hard-breaks until update-function-code lands. The correct order is: build the frontend, bake the image (which captures the built dist/), push the image, call update-function-code, and only then run s3 sync. This is enforced in the lifecycle workflow; the manual runbook in infra/README.md documents the same ordering for hand-built stacks.

GitHub OIDC identity and fork PRs

Phase 2 CI uses GitHub OIDC (no stored AWS secrets) via the onramp-deploy-previews IAM role (infra/ci-identity/). The legacy pr-previews-deployer role and its source directory (infra/pr-previews/ci-identity/) are retired by the shared-apply cutover. Fork PRs cannot mint OIDC tokens — GitHub restricts id-token: write to non-fork workflows for security. Preview deploys therefore only work for PRs from branches within the OnRampTech org. This is an accepted limitation; external contributors don't need preview access to the OnRamp AWS account.

Neon: reset_to_parent preserves endpoints

Reseeding a preview Neon branch via reset_to_parent preserves the branch's connection endpoint (the hostname stays the same). This means the Lambda's DATABASE_URL env var doesn't change on reseed — no update-function-code churn just to reflect a fresh seed. Per-PR branches are created and deleted by devtools/db/preview-branch.ts.

PR-preview Google SSO

✅ Implemented — the Cognito user pool and per-PR client are provisioned by infra/pr-previews/shared.ts and infra/pr-previews/preview.ts. The CognitoSSO strategy (app/api/integrations/sso/strategy/sso_strategies.py) is active only in preview environments where COGNITO_DOMAIN is configured.

Two login identities, one role each

Identity	Password	Where it works	Tenant	Role
admin@example.com	OnRampLocal!23	everywhere (local, CI, previews)	vendor 0	OnRamp admin
customer@example.com	OnRampLocal!23	everywhere (local, CI, previews)	vendor 1	customer-tenant owner
any @onramp.us address	real Google account	preview only (COGNITO_DOMAIN configured)	matches seeded user row	varies

@example.com is RFC-2606-reserved and never routes real mail — safe to commit as a public dev credential. @onramp.us is the Google-SSO trigger domain.

The SSO routing is config-gated on COGNITO_DOMAIN. In local dev, CI, and prod — where COGNITO_DOMAIN is absent — entering an @onramp.us address falls back to password login (the normal form), never produces a broken redirect. Only preview Lambdas (where COGNITO_DOMAIN is injected via the Lambda env block) route @onramp.us through Cognito → Google.

GCP setup (one-time, shared across all PRs)

1. In Google Cloud Console, create an OAuth 2.0 Web client under the same GCP project already used for OnRamp Google login.
2. Add exactly one Authorized redirect URI:

   https://onramp-pr-previews.auth.us-west-2.amazoncognito.com/oauth2/idpresponse

   This is the Cognito hosted-UI domain. The single registration covers the entire preview fleet — no per-PR entry.
3. Copy the Client ID and Client Secret.

Setting the credentials

# Non-secret: commit-safe client ID on the shared stack
cd infra/pr-previews
pulumi config set googleClientId <client_id>

# Secret: store client secret in SSM (NOT committed, NOT an env var).
# Encrypt with the onramp-pulumi-secrets CMK so it matches the deploy role's
# scoped kms:Decrypt grant (GoogleSsoSecretDecrypt). Omitting --key-id uses the
# default aws/ssm key, which the grant does NOT cover -> deploy-time decrypt fails.
# --overwrite so re-running updates the existing parameter.
aws ssm put-parameter \
  --name /onramp/pr-previews/google_sso_client_secret \
  --type SecureString \
  --key-id alias/onramp-pulumi-secrets \
  --overwrite \
  --value <client_secret>

Pulumi reads googleClientId from the committed stack config (infra/pr-previews/Pulumi.shared.yaml). The client secret is read at deploy time from SSM with withDecryption: true and never touches the repo or a Lambda env var.

How a tester signs in via Google SSO

1. Navigate to the PR preview URL (https://pr-<N>.preview.onrampapps.com/dashboard/login).
2. Enter an @onramp.us email address in the email field and click Next.
3. The login page detects the @onramp.us domain, sees COGNITO_DOMAIN is configured, and redirects to the Cognito hosted-UI, which forwards to Google.
4. Authenticate with the matching real @onramp.us Google account.
5. Google redirects back to Cognito → Cognito redirects to https://pr-<N>.preview.onrampapps.com/sso/redirect/cognito → the CognitoSSO strategy exchanges the code, looks up the user by email in the DB, and establishes the session.

The tester's @onramp.us email must match a seeded user row in the preview Neon branch (provisioned from the golden-seed parent). If the email does not exist as an or_users record the login returns a 401; add the user via the golden seed or the golden-seed-update skill.