Release & Deploy Architecture

How code gets from a merged PR to a running environment under OnRamp's trunk-based release process. This is the system reference; the day-to-day "how do I ship / hotfix" lives in Releasing & Hotfixing.

Status	Live — trunk model is the active release process
Audience	Engineers shipping, releasing, hotfixing, or debugging a deploy
Lives in	.github/workflows/, .github/actions/, .github/scripts/determine-environment.js

Mental model

main is the only long-lived branch. Merging ships to dev only. Staging and prod are reached by running a workflow, not by merging into an environment branch — promotion ≠ merge. This removes branch drift (dev/staging/prod diverging), rebuild-per-env, and the hotfix back-merge ritual.

Three facts hold it together:

1. The commit is the identity. The same SHA flows dev → staging → prod unchanged — no per-environment rebuilds.
2. The git tag is the release value. Clean vM.N.0/vM.N.P tags are minted only by the promote workflows. dev/staging carry a computed vM.N.x-<count> snapshot (with a throwaway snapshot/… tag for visibility), never a release tag.
3. GitHub Deployments are the state — the authoritative record of "what's deployed where," written after every deploy.

Branch & promotion flow

flowchart LR
    PR["Merged PR"] --> MAIN["main (trunk)"]
    MAIN -->|"push to main (auto)"| DEV["dev01"]
    DEV -.->|"Promote to Staging (blank = latest main)"| STAGE["stage"]
    STAGE -.->|"Promote to Production (approved)"| PROD["prod / demo / gdpr-prod"]

    classDef auto fill:#d1fae5,stroke:#059669,color:#065f46
    classDef manual fill:#fee2e2,stroke:#dc2626,color:#991b1b
    class DEV auto
    class STAGE,PROD manual

• Merge to main → dev. Automatic, the only automatic deploy. Each dev build computes its version and creates a snapshot/<version> snapshot tag (e.g. snapshot/v4.2.x-13) used as the deployment ref — so the version is visible in the Deployments UI and selectable later.
• Promote to Staging (🧪 Promote to Staging, promote-stage.yml, workflow_dispatch). The staging twin of promote-prod, on the shared promote model: the workflow always runs from main ("Use workflow from" is guarded), a blank version input promotes the latest main, and a snapshot/v4.2.x-13 tag promotes that exact snapshot (invalid/pruned tags fail fast and list the current ones). It also runs the promotion gates (below) and refuses to promote a main that is red on E2E or the Python unit suite. It's a control plane: after validating it dispatches the deploy worker (⚙️ Deploy to Staging (internal), promote-staging.yml) at the chosen ref — so the build pipeline that runs belongs to the commit being deployed, while the promote logic is always current. No release tag minted — the stage deployment is recorded against the same snapshot/<version> tag (created on the spot if the commit never went through the dev flow), so the Deployments UI shows the snapshot you can later promote to prod, not a bare sha.
• Promote to Production (🚀 Promote to Production, promote-prod.yml; the human gate is the deliberate dispatch itself — see Environments below, required reviewers are not available on the org's plan). Two jobs: resolve decides what ships and runs the promotion gates (below); publish mints and ships it. It promotes the staging-validated commit — resolved by .github/scripts/resolve-stage-deployment.ts (not main-tip, since staging lags main), mints the next clean vM.N.0 tag on that commit, publishes a Release (the audit artifact), and explicitly dispatchesdeploy-release.yml at the tag ref → prod + demo + gdpr-prod. The dispatch is required: releases created with the repo GITHUB_TOKEN never fire release:-triggered workflows (GitHub's recursion guard). A human-published Release still deploys via the release: published event.

Both promotes are driven by agent skills as the primary entry point — /promote-stage and /promote-prod (deliberately separate, so stage and prod can't be fat-fingered into each other). The skills gather context (what's on each environment, the commit delta), ask default-vs-explicit-target, validate, dispatch, and watch the runs. The workflows stay directly runnable from the Actions UI as the no-agent fallback; the runbook documents both paths in that order.

The promotion gates

Two post-merge suites gate promotion, and both are read the same way.

Playwright E2E runs post-merge only — on every push to main (and hotfix/**), alongside the dev deploy. It is not a PR check: the suites weren't catching enough defects to justify running per-PR, let alone per-commit. What they gate instead is promotion.

The Python unit suite (python-test.yml) is different: it is a required PR check. It is gated anyway, because a green PR check does not imply a green main. Two PRs that each forked before the other can each be green, merge cleanly because they touch different files, and still leave the trunk red — a caller updated on one side, a call site added on the other. Only the push-to-main run sees the combination, so that run is the only evidence the trunk is actually green. On a push its path filter is forced open, so a success there is always a real suite run.

Both promote workflows call .github/actions/main-suite-gate once per suite before doing anything irreversible, each with its own skip_* override so waiving one does not blind the other. The gate reads a stored verdict — it never triggers a suite and never waits for one, so promotion wall clock is unchanged:

1. List the commits reachable from the promotion target (bounded at 200).
2. Take the newest conclusive push-to-main run of that suite among them. success, failure, and timed_out are conclusive; cancelled and startup_failure are not — a merge train supersedes in-flight runs under the workflow's cancel-in-progress concurrency, so reading those as failures would block on noise.
3. failure/timed_out → fail the promotion with a link to the run. success → proceed. No conclusive run in range → warn and proceed; the gate only pushes back on a known failure.

A failure announces itself rather than waiting to be discovered. The E2E workflow's notify-failure job posts to #eng-alerts on any red push run, leading the message with the keyword e2e-failure — add that to Slack > Preferences > Notifications > My keywords to be paged personally. The alert names the specific red suites, links the commit and the run, and states whether the block is on promotion (main) or on the prod path (hotfix/**).

Its firing condition mirrors the required check (e2e suites) rather than the gate's conclusive/inconclusive split, and has to keep mirroring it: anything that reddens that check also blocks promotion, so a case it catches and the alert doesn't is a blocked trunk nobody hears about. Two consequences follow. A non-success preflight alerts even when it never reached the suites. And a suite rollup of cancelled alerts too — with fail-fast: false a single leg can lose its runner while its siblings finish, reddening the check without the run being cancelled. What stays quiet is a run-level cancel, which is the merge train: a superseded run is cancelled whole, the job never starts, and the noise the gate ignores at promotion time is ignored here as well. Dispatches on feature branches stay quiet too — the job is push-only.

Consequences worth internalising:

• A run still in flight never blocks. Promoting a commit merged two minutes ago falls back to the last conclusive ancestor. The result of the in-flight run is read by the next promotion — and prod re-runs the gate on the staging-validated commit, by which point its run has concluded.
• Fixing forward clears the gate immediately — a green run on a newer commit outranks an older red one. Re-running just the failed suite to green works too.
• Rollback is never gated. Redeploying an already-released tag is the recovery path; main's current state says nothing about it.
• skip_e2e_gate on either promote workflow ships anyway, and records the override on the run. It is for an outage in the E2E harness itself, not for a real suite failure — that failure is the build you are about to promote.
• Hotfixes bypass this gate entirely (they never touch main). Their coverage is the hotfix/** push run, read by promote-hotfix's "CI green on the tip" check.

Version scheme

Standard semver, patch slot reserved for hotfixes. Clean release tags mint only on prod promotion; dev/staging carry a computed snapshot.

Stream	Identifier	Created
Dev + staging	v4.2.x-13 (release line + commits-since)	Computed at build; a snapshot/v4.2.x-13 tag is created
Trunk release	v4.2.0, v4.3.0, … (minor bump, patch 0)	Prod promotion (mint-version)
Hotfix	v4.2.1, v4.2.2, … (patch increments)	Promote-hotfix, derived from the branch name
Prod re-deploy / rollback	same release tag	No new tag

Reading v4.2.x-13 → 13 commits past the v4.2.0 release. That count is the release-size signal. The x patch slot is deliberate: a snapshot is an unreleased build of the 4.2 line, not the v4.2.0 release — writing v4.2.0-13 made it read like a build of v4.2.0. There's no trailing -<sha> — on the squash-merge trunk the count is unique to one commit, and the snapshot/v4.2.x-13 tag links straight to it.

Snapshot tags are namespaced + ephemeral. dev + stage deploys create snapshot/<version> (under snapshot/, not the v* release namespace), so the release tag list stays clean and the Release Tags ruleset doesn't gate them. cleanup-snapshot-tags.yml prunes snapshot/* tags older than 30 days.

How the version is computed

Two composite actions own this (both require fetch-depth: 0; they force-fetch tags internally):

• compute-version — git describe --tags --long --match 'v[0-9]*.[0-9]*.0' (only clean release tags, so snapshot/* and legacy -prod tags don't pollute the count), then drops the -g<sha> segment and rewrites the patch digit to x → v4.2.x-13. On a release commit it emits the bare tag (v4.2.0). Outputs version, sha, is_release.
• mint-version — reads VERSION_MAJOR (tracked file at repo root, currently 4, CODEOWNERS-gated), finds the highest v4.*.0 tag, bumps the minor, pushes the new tag. No fallback — errors loudly if no clean vM.N.0 tag exists.

Component	Owner	When
Major	VERSION_MAJOR file, bumped by reviewed PR	Breaking-change milestone
Minor	mint-version, auto-increment	Prod promotion only
Patch	0 for trunk; hotfix path owns > 0; x on snapshots	—
Snapshot -<count>	compute-version	Build time (dev/staging)

The version is frozen at deploy time on the GitHub Deployment; the snapshot/<version> tag pins the exact commit. Minting v4.3.0 later doesn't rewrite a prior dev build's v4.2.x-13 — the commit is the join key.

Deploy fan-out

build-and-deploy.yml (reusable) is parameterized by target_environment. It builds the Flask + webserver images once, then fans out:

flowchart TD
    BD["build-and-deploy.yml<br/>(target_environment)"] --> IMG["build flask, flask-lambda + webserver images"]
    IMG --> INFRA["deploy-infra (Pulumi, per env stack)"]
    INFRA --> MIG["migrate gate (in-VPC, the release's flask image)"]
    MIG --> DUPLO["duplo-deploy (flask, then webserver)"]
    MIG --> AGENTS["deploy-agents (onramp-agents runtimes)"]
    DUPLO --> REC["record-deployment step, per leg"]
    REC --> GHD["GitHub Deployment + commit status + run summary"]

• target_environment: prod → web app to [prod-usw2, prod-gdpr-ir, demo-usw2] and agents to [prod, demo, gdpr-prod] (parallel, fail-fast: false).
• stage → [stage]; dev → [dev].
• The order is a contract application code depends on. Infra applies before the migrate gate, and both precede the web app and the agent runtimes, which roll in parallel with no order between them. Which of those edges gate and which only sequence — and what each unit may therefore assume about the others mid-release — is .claude/rules/deploy-boundaries.md.
• record-deployment writes a GitHub Deployment (ref = the version tag, so the chip shows snapshot/v4.2.x-13 / v4.3.0 rather than a bare sha), a deploy/<env> commit status carrying the version (visible on the commit + PRs), and a run-summary line. Each rollout leg records its own environment, so the record describes that tenant's rollout and nothing else. Hotfix tooling resolves the live prod SHA from those records via .github/scripts/resolve-prod-deployment.ts — the single rule, which refuses rather than guessing when prod cannot be established.

Tag → environment routing

determine-environment.js (inside deploy-release.yml → environment-setup.yml) is additive: legacy -(prod|stage|dev) suffix branches are preserved, and a non-prerelease clean-semver tag routes to prod.

Tag	Prerelease	Routes to
v4.2.0 (trunk release) / v4.2.1 (hotfix)	false	prod
v4.1.0-prod (legacy)	false	prod
any	true	dev

Environments

ONE Environment exists per tier, the bare <env>, and every deploy job binds it — deploy-infra-reusable.yml, migrate-gate.yml, deploy-agents-reusable.yml, and the app rollout alike.

They can share it because release reporting no longer depends on telling their Deployment records apart. GitHub auto-creates a Deployment record for ANY job that references an Environment, so the infra and migrate legs leave versionless records behind on a cancelled or partially-failed run. Jira's GitHub integration reads deployment history as a release timeline and computes an issue window as "commits since the last record on this environment", which let one of those phantoms become a real release's starting boundary and silently drop every issue merged before it. report-jira-deployment.ts now states the version and its issue keys to Jira EXPLICITLY, so no window is inferred and a versionless record cannot move anything. The environment's Jira type is declared in jira-environment.ts rather than guessed from its name.

Provisioning the Jira credential that reporting needs, and rotating it, is docs/runbooks/jira-release-reporting.md. Reporting is best-effort: with no credential configured the step warns and exits 0, so releases are unaffected and only the Jira release panel goes unpopulated.

The deployment branch/tag allow-list is declared ONCE, in ALLOWED_REFS_BY_ENV (.github/scripts/env-ref-policy.ts) — not read back from the Environment's live GitHub state — and checked in TWO places that are NOT equivalent controls:

• The <env> GitHub Environment's own deployment branch/tag policy — applied from the same ALLOWED_REFS_BY_ENV by onramp-agents/scripts/bootstrap-agent-environments.ts, an out-of-band admin run with nothing in CI sequencing or verifying it. This is the real control: it's enforced by GitHub and cannot be edited from the ref being judged. An Environment GitHub auto-creates on first reference starts with NO allow-list, so it only protects once that script has been run — and re-run after every ALLOWED_REFS_BY_ENV change.
• In the job itself: check-deploy-ref.ts, the first real step of deploy-infra-reusable.yml's deploy job, migrate-gate.yml's migrate job, and deploy-agents-reusable.yml's deploy/deploy-dashboard jobs — the same check, from the same declared list, run a second time inside the job. This runs FROM the checkout of the ref it's judging, so a branch that also edits env-ref-policy.ts (or the guard itself) passes its own check. It's a backstop against an accidental bad ref (wrong branch dispatched, a matrix/choice list that drifted out of sync with ALLOWED_REFS_BY_ENV), not a substitute for the Environment policy against a deliberate one — provisioning that stays a real security prereq, not just an availability one.

stage allows main (branch) plus snapshot/* (tag); prod/demo/ gdpr-prod allow main (branch) plus v* and the legacy v*-prod (tag), so a published release can deploy; dev is unrestricted. Change the allow-list in env-ref-policy.ts (both enforcement points read it), then re-run the bootstrap script for every affected env — never edit either Environment's policy directly in the GitHub UI, which the next bootstrap run reconciles away.

• Required reviewers are NOT configured: that protection rule is Enterprise-only on private repos and the org is on the Team plan (the API attempt 422s — see preview-shared-apply.yml). The human control on promote-prod / deploy-release is the deliberate dispatch itself (actor + ref + timestamp recorded on the run). If the org ever moves to a plan that supports them, reviewers on prod (and the other production-tier envs) would additionally provide a pause-for-approval gate.
• Each env injects a DUPLO_TENANT_NAME variable, applied to <env> by the same bootstrap script. DUPLO_TOKEN is a repository-level secret, not environment-scoped — it resolves the same regardless of which Environment a job binds.

Hotfix flow

Hotfixes never touch main directly — a short-lived branch is cut from the deployed prod SHA, fixed, promoted straight to prod, then forward-ported.

sequenceDiagram
    actor Eng
    participant CH as Create Hotfix
    participant HB as hotfix/v4.N.P
    participant PH as Promote Hotfix
    participant Prod
    participant FP as Forward-port
    participant Main as main

    Eng->>CH: dispatch (ticket)
    CH->>Prod: read live prod SHA (GitHub Deployment)
    CH->>HB: branch off prod SHA, next patch
    Eng->>HB: commit fix
    Eng->>PH: dispatch (hotfix branch)
    PH->>PH: validate name, CI green, smoke
    PH->>Prod: tag v4.N.P, publish Release, deploy
    PH->>FP: trigger forward-port
    FP->>Main: cherry-pick onto forward-port branch, open PR (automation App)
    FP->>Main: approve + auto-merge on green
    PH->>HB: delete branch

• 🔥 Create Hotfix Branch — resolves the live prod SHA via .github/scripts/resolve-prod-deployment.ts, computes the next patch, branches hotfix/v4.N.P.

• 🚑 Promote Hotfix to Production — validates the branch, checks CI green (python-test / frontend-ci / e2e run on hotfix/** pushes so the gate has real checks to read), runs a smoke suite (placeholder until an ephemeral-env harness exists), tags + publishes the Release and dispatches deploy-release.yml at the tag (→ prod, skips staging by design), triggers forward-port, deletes the branch. The skip_forward_port input skips the forward-port PR when the hotfix commits were cherry-picked from an existing PR — merged (the auto cherry-pick back would be an empty no-op) or open (that PR is the path to main and inherits the forward-port SLA; no branch rename — the drift-sentinel keys on the tag, not branch names, so just label the PR forward-port). It requiresforward_port_source_pr, the number of that PR: promote verifies it against the branch and records it as the tag's forward-port-source/* marker, which is the only thing that lets the drift-sentinel clear a skipped forward-port — and only once that PR is merged into main. Atypical; never set it on a fresh fix.

• forward-port-hotfix (workflow_call) — cherry-picks onto forward-port/<ticket> off main; clean → opens a labelled PR, approves it, and enables auto-merge (squash) so it lands as soon as CI is green; conflict → draft PR with markers assigned to the author (never auto-merged).

  Which token opens the PR is load-bearing. GitHub's recursion guard is specific to GITHUB_TOKEN: a PR opened with it never fires pull_request workflows, so its required checks sit at "Expected" forever and only a ruleset bypass could merge it. An App installation token is not GITHUB_TOKEN and its events do start workflow runs — the same property dependabot-lockfile-sync.yml relies on. So the onramp-github-automation App opens the PR, real CI runs, and auto-merge lands it through the front door with no bypass grant. That forces the approval onto the other identity: GitHub rejects a self-approval, so github-actions[bot] approves what the App authored (org setting "Allow GitHub Actions to create and approve pull requests", already on).

  Credentials: AUTOMATION_BOT_APP_ID (App ID or Client ID) + AUTOMATION_BOT_APP_PRIVATE_KEY org secrets; the installation needs Contents + Pull requests R/W, and Workflows R/W for hotfixes that touch .github/workflows/**. Absent, the PR still opens (as github-actions[bot], unchecked) and the auto-merge job fails loudly rather than the fix going missing.

  Enabling auto-merge is not the same as merging: a forward-port touching a CODEOWNERS path needs a code-owner approval the bot's cannot satisfy, and auto-merge would sit armed but never fire. The job checks reviewDecision after arming and comments on the PR when it is still REVIEW_REQUIRED, so that stall surfaces immediately rather than at the drift-sentinel's 24h mark.

Deferred (tracked follow-up): the "destructive migration pending in trunk not in prod" gate on create-hotfix. Interim mitigation: expand-and-contract discipline + manual vigilance — see the migration runbook.

Guardrails

Guardrail	Mechanism	Effect
Hotfix can't land on trunk via merge	forbid-hotfix-merge.yml (PR check, required on main)	Fails any PR whose head is hotfix/*
A red main can't be promoted	.github/actions/main-suite-gate, called once per gated suite (E2E, Python unit) by both promote workflows	Fails the promotion when the newest conclusive push-to-main run of that suite in the target's history failed; reads a stored verdict, never waits
A red main doesn't go unnoticed — E2E only	notify-failure job in e2e.yml → .github/actions/slack-alert	Posts the failed suites to #eng-alerts under the e2e-failure keyword, so an unpromotable trunk surfaces at merge time instead of at the next promotion attempt. python-test.yml has no equivalent, so a unit-red main blocks promotion silently until someone dispatches a promote
Forward-port lands without human toil	forward-port-hotfix.yml auto-merge job (App-authored PR, github-actions[bot] approves, auto-merge squashes on green)	Reconciliation completes without a human; a red check holds the merge, and a setup failure is loud (red run + PR comment pinging the author), never silent
Forward-port can't be silently skipped	drift-sentinel.yml (daily cron)	Backstop for the above: pages if a prod commit has been off main past a 24h SLA; the 7-day scan is backstop breadth. A deliberately skipped forward-port clears only on its marker's source PR being merged into main; a breach that is real but benign is recorded with a bounded drift-ack/* tag, which stops the failure without hiding the breach
Release tags stay clean	Release Tags ruleset (^v\d+\.\d+\.(0|[1-9]\d*)$)	Only well-formed vM.N.0/vM.N.P in the v* namespace; snapshot/* is unaffected
main protected	main ruleset (cloned from the old dev ruleset's checks + forbid-hotfix-merge, squash-only, code-owner review)	Trunk integrity
Hotfix branches created only by automation	branch ruleset on hotfix/*	Creation restricted to the create-hotfix actor
State is authoritative	record-deployment → GitHub Deployments	"What's in prod?" is a query, not a guess

Workflow inventory

Workflow	Trigger	Role
🛠️ Deploy Dev	push main	Build + deploy to dev; tag snapshot/<version>; record
🧪 Promote to Staging	dispatch (blank version = latest main, or a snapshot/* tag)	Validate the selection + promotion gates → dispatch the staging deploy worker at that ref
⚙️ Deploy to Staging (internal)	dispatch (by Promote to Staging, at the target ref)	Build + deploy the dispatched ref to staging
🚀 Promote to Production	dispatch (prod approval; blank ref = staging-validated)	Promotion gates → mint tag on the staging-validated commit → Release → prod fan-out
E2E (Playwright) 🎭	push main + hotfix/**, dispatch	Full Playwright matrix; its stored verdict is one of the two the promote gates read
🔥 Create Hotfix Branch	dispatch	Branch off live prod SHA
🚑 Promote Hotfix to Production	dispatch	Tag + deploy hotfix to prod, forward-port
forward-port-hotfix	workflow_call	Cherry-pick hotfix → main PR, then approve + merge it
forbid-hotfix-merge	pull_request → main	Block direct hotfix merges
drift-sentinel	cron + dispatch	Detect un-forward-ported prod commits
cleanup-snapshot-tags	cron + dispatch	Prune snapshot/* tags older than 30d
build-and-deploy	workflow_call	Build once, fan out, record deployment
deploy-release	release: published	Fires on a minted Release → environment-setup → build-and-deploy

Related

• Releasing & Hotfixing — the day-to-day HOWTO.
• Migrations: expand & contract — the discipline hotfixes rely on.