Release & Deploy Architecture
How code gets from a merged PR to a running environment under OnRamp's trunk-based release process. This is the system reference; the day-to-day "how do I ship / hotfix" lives in Releasing & Hotfixing.
| Status | Live — trunk model is the active release process |
| Audience | Engineers shipping, releasing, hotfixing, or debugging a deploy |
| Lives in | .github/workflows/, .github/actions/, .github/scripts/determine-environment.js |
Mental model
main is the only long-lived branch. Merging ships to stage. Prod is reached by running a workflow, not by merging into an environment branch — promotion ≠ merge. This removes branch drift (dev/staging/prod diverging), rebuild-per-env, and the hotfix back-merge ritual.
Three facts hold it together:
- The commit is the identity. The same SHA flows
main→ staging → prod unchanged — no per-environment rebuilds. - The git tag is the release value. Clean
vM.N.0/vM.N.Ptags are minted only by the promote workflows. Staging carries a computedvM.N.x-<count>snapshot (with a throwawaysnapshot/…tag for visibility), never a release tag. - GitHub Deployments are the state — the authoritative record of "what's deployed where," written after every deploy.
Branch & promotion flow
flowchart LR
PR["Merged PR"] --> MAIN["main (trunk)"]
MAIN -->|"push to main (auto)"| STAGE["stage"]
MAIN -.->|"Promote to Staging (redeploy / rollback hatch)"| STAGE
STAGE -.->|"Promote to Production (last finished stage deploy)"| PROD["prod / gdpr-prod"]
classDef auto fill:#d1fae5,stroke:#059669,color:#065f46
classDef manual fill:#fee2e2,stroke:#dc2626,color:#991b1b
class STAGE auto
class PROD manual- Merge to
main→ stage. Automatic, the only automatic deploy of the app (the infra-onlytoolstier also applies on merge; see Environments).⚙️ Deploy to Staging (internal)(promote-staging.yml) runs on every push tomainand callsbuild-and-deploy.ymlwithtarget_environment: stage— the same reusable a prod release uses. One run builds the images, runs the migrate gate against the stage DB, rolls stage's Duplo tenant, deploys all of stage's AgentCore runtimes (gated on the migrate gate), applies stage's Pulumi stack, mints thesnapshot/<version>tag (e.g.snapshot/v4.2.x-13) used as the deployment ref — so the version is visible in the Deployments UI and selectable later — and records thestageGitHub Deployment. The Pulumi apply is a best-effort rider: a failed apply does not fail the run, it fires a Slack alert. So stage is the first database to run every migration and the first stack to apply every infra change. All refs share one concurrency group withcancel-in-progress: false, because cancelling mid-migration leaves partial autocommit DDL behind; GitHub keeps one pending run per group, so a burst of merges deploys the newest and skips the ones in between (those commits get no snapshot tag). - Promote to Staging (
🧪 Promote to Staging,promote-stage.yml,workflow_dispatch) is the redeploy / rollback hatch, not the normal path — merges already deploy stage. On the shared promote model, the workflow always runs frommain("Use workflow from" is guarded). A blankversioninput redeploys the latest main (e.g. re-runs a merge deploy that failed or was skipped); asnapshot/v4.2.x-13tag rolls stage back to that build until the next merge deploysmainover it (invalid/pruned tags fail fast and list the current ones). It runs the promotion gates (below) and refuses amainthat is red on E2E or the Python unit suite. It's a control plane: after validating it dispatches the deploy worker (promote-staging.yml) at the chosen ref — so the build pipeline that runs belongs to the commit being deployed, while the promote logic is always current. No release tag minted — the stage deployment is recorded against the samesnapshot/<version>tag, which is created if it does not exist yet. - Promote to Production (
🚀 Promote to Production,promote-prod.yml; the human gate is the deliberate dispatch itself — see Environments below, required reviewers are not available on the org's plan). Two jobs:resolvedecides what ships and runs the promotion gates (below);publishmints and ships it. By default it promotes the staging-validated commit — the newest successful stage rollout, resolved by.github/scripts/resolve-stage-deployment.ts, which is the last merge whose stage deploy finished. That is continuous delivery: no minimum soak time, and nobody is assumed to have checked the commit on stage — "validated" means its deploy, migrations included, finished there. It is not themaintip: a newer merge whose stage deploy is still running or failed is not what ships. It mints the next cleanvM.N.0tag on that commit, publishes a Release (the audit artifact), and explicitly dispatchesdeploy-release.ymlat the tag ref → prod + gdpr-prod. The dispatch is required: releases created with the repoGITHUB_TOKENnever firerelease:-triggered workflows (GitHub's recursion guard). A human-published Release still deploys via therelease: publishedevent.
Prod promotion is driven by the /promote-prod agent skill as the primary entry point: it gathers context (what's on each environment, the commit delta), asks default-vs-explicit-target, validates, dispatches, and watches the runs. Stage has no promotion skill, because merges deploy it; a stage rollback goes through /rollback, and a redeploy of the latest main is a blank dispatch of the hatch. The workflows stay directly runnable from the Actions UI as the no-agent fallback; the runbook documents both paths.
The promotion gates
Two post-merge suites gate promotion, and both are read the same way.
Playwright E2E runs post-merge only — on every push to main (and hotfix/**), alongside the stage deploy, on its own seeded stack on localhost (it does not depend on stage). It is not a PR check: the suites weren't catching enough defects to justify running per-PR, let alone per-commit. What they gate instead is promotion.
The Python unit suite (python-test.yml) is different: it is a required PR check. It is gated anyway, because a green PR check does not imply a green main. Two PRs that each forked before the other can each be green, merge cleanly because they touch different files, and still leave the trunk red — a caller updated on one side, a call site added on the other. Only the push-to-main run sees the combination, so that run is the only evidence the trunk is actually green. On a push its path filter is forced open, so a success there is always a real suite run. It runs without coverage, like a PR's whole-suite run. The per-test record that PR test selection reads comes from a separate workflow, pytest-record.yml, that never gates and runs one build at a time for the newest push, so a slow record build cannot cost main its verdict.
Both promote workflows call .github/actions/main-suite-gate once per suite before doing anything irreversible, each with its own skip_* override so waiving one does not blind the other. The merge-triggered stage deploy is not gated: every merge lands on stage, and the gates sit at the two promotions — the stage hatch and prod. The gate reads a stored verdict — it never triggers a suite and never waits for one, so promotion wall clock is unchanged:
- List the commits reachable from the promotion target (bounded at 200).
- Take the newest conclusive push-to-
mainrun of that suite among them.success,failure, andtimed_outare conclusive;cancelledandstartup_failureare not — a merge train supersedes in-flight runs under the workflow'scancel-in-progressconcurrency, so reading those as failures would block on noise. failure/timed_out→ fail the promotion with a link to the run.success→ proceed. No conclusive run in range → warn and proceed; the gate only pushes back on a known failure.- Runs are listed unfiltered, because the listing's branch and event filters are served from a search index that can return an incomplete page. If the page budget runs out before the listing reaches the range, the gate fails the promotion: not seeing
mainis not the same asmainhaving no verdict.
A failure announces itself rather than waiting to be discovered. The E2E workflow's notify-failure job posts to #eng-alerts on any red push run, leading the message with the keyword e2e-failure — add that to Slack > Preferences > Notifications > My keywords to be paged personally. The alert names the specific red suites, links the commit and the run, and states whether the block is on promotion (main) or on the prod path (hotfix/**).
Its firing condition mirrors the run's verdict check (e2e suites) rather than the gate's conclusive/inconclusive split, and has to keep mirroring it: anything that reddens that check also blocks promotion, so a case it catches and the alert doesn't is a blocked trunk nobody hears about. Two consequences follow. A non-success preflight alerts even when it never reached the suites. And a suite rollup of cancelled alerts too — with fail-fast: false a single leg can lose its runner while its siblings finish, reddening the check without the run being cancelled. What stays quiet is a run-level cancel, which is the merge train: a superseded run is cancelled whole, the job never starts, and the noise the gate ignores at promotion time is ignored here as well. Dispatches on feature branches stay quiet too — the job is push-only.
The Python unit suite announces itself the same way, from python-test.yml's notify-failure job, under the keyword pytest-failure. It fires on a push to main whose pytest job concluded failure, which is exactly what the gate reads as red. A job that hits its timeout reports cancelled, not timed_out, and the gate passes over it like any other cancel. It still alerts: notify-timeout posts under the same keyword, titled as a timeout rather than a failure, and names the push left unverified. A manually cancelled run stays quiet, because cancelling the run cancels that job before it starts. The failure alert names:
- the first red push since the newest green run, its PR and the PR's author;
- the failing test ids, read from pytest's short summary and kept on each red run as a
pytest-failuresartifact; - one tag: known flaky when a failing test also failed on a
mainrun pushed before this red streak, in the last 30 days. Otherwise, what the PR's ownpytestcheck did: passed on its own base, or suite skipped on its PR when its path filter skipped the suite.
It is one alert per breakage. A later red run that fails no test the run before it did stays quiet, and notify-recovery posts pytest-recovered on the first green run after a red one. That job is continue-on-error: the gate reads the run's conclusion, so a failed Slack delivery must never redden a green run.
Pushes to main overlap, so the job waits up to 15 minutes for an earlier run still in flight before calling itself the first red, and says in the alert that the first red is not proven when it still cannot tell.
Consequences worth internalising:
- A run still in flight never blocks. Promoting a commit merged two minutes ago falls back to the last conclusive ancestor. The result of the in-flight run is read by the next promotion, including prod's, which re-runs the gate on the commit it resolves from stage.
- Fixing forward clears the gate immediately — a green run on a newer commit outranks an older red one. Re-running just the failed suite to green works too.
- Rollback is never gated. Redeploying an already-released tag is the recovery path; main's current state says nothing about it.
skip_e2e_gateon either promote workflow ships anyway, and records the override on the run. It is for an outage in the E2E harness itself, not for a real suite failure — that failure is the build you are about to promote.- Hotfixes bypass this gate entirely (they never touch
main). Their coverage is thehotfix/**push run, read by promote-hotfix's "CI green on the tip" check. Itsbreak_glass_skip_ciinput waives that check, with the approving engineer and reason required and recorded in the run summary and the GitHub Release; the tip's release artifacts are still built and required.
Version scheme
Standard semver, patch slot reserved for hotfixes. Clean release tags mint only on prod promotion; staging carries a computed snapshot.
| Stream | Identifier | Created |
|---|---|---|
| Staging | v4.2.x-13 (release line + commits-since) | Computed at build; the merge deploy creates a snapshot/v4.2.x-13 tag |
| Trunk release | v4.2.0, v4.3.0, … (minor bump, patch 0) | Prod promotion (mint-version) |
| Hotfix | v4.2.1, v4.2.2, … (patch increments) | Promote-hotfix, derived from the branch name |
| Prod re-deploy / rollback | same release tag | No new tag |
Reading v4.2.x-13 → 13 commits past the v4.2.0 release. That count is the release-size signal. The x patch slot is deliberate: a snapshot is an unreleased build of the 4.2 line, not the v4.2.0 release — writing v4.2.0-13 made it read like a build of v4.2.0. There's no trailing -<sha> — on the squash-merge trunk the count is unique to one commit, and the snapshot/v4.2.x-13 tag links straight to it.
Snapshot tags are namespaced + ephemeral. Stage deploys create snapshot/<version> (under snapshot/, not the v* release namespace), so the release tag list stays clean. cleanup-snapshot-tags.yml prunes snapshot/* tags older than 30 days. A merge deploy mints its tag before it rolls out, so a tag alone does not prove the build reached stage; stage's Deployments record the ones that did.
How the version is computed
Two composite actions own this (both require fetch-depth: 0; they force-fetch tags internally):
compute-version—git describe --tags --long --match 'v[0-9]*.[0-9]*.0'(only clean release tags, sosnapshot/*and legacy-prodtags don't pollute the count), then drops the-g<sha>segment and rewrites the patch digit tox→v4.2.x-13. On a release commit it emits the bare tag (v4.2.0). Outputsversion,sha,is_release.mint-version— readsVERSION_MAJOR(tracked file at repo root, currently4, CODEOWNERS-gated), finds the highestv4.*.0tag, bumps the minor, pushes the new tag. No fallback — errors loudly if no cleanvM.N.0tag exists.
| Component | Owner | When |
|---|---|---|
| Major | VERSION_MAJOR file, bumped by reviewed PR | Breaking-change milestone |
| Minor | mint-version, auto-increment | Prod promotion only |
| Patch | 0 for trunk; hotfix path owns > 0; x on snapshots | — |
Snapshot -<count> | compute-version | Build time (staging) |
The version is frozen at deploy time on the GitHub Deployment; the snapshot/<version> tag pins the exact commit. Minting v4.3.0 later doesn't rewrite a prior stage build's v4.2.x-13 — the commit is the join key.
Deploy fan-out
build-and-deploy.yml (reusable) is parameterized by target_environment. It builds the Flask + webserver images once, then fans out:
flowchart TD
BD["build-and-deploy.yml<br/>(target_environment)"] --> IMG["build flask, flask-lambda + webserver images"]
IMG --> MIG["migrate gate (in-VPC, the release's flask image)<br/>runs only once every rollout prerequisite is green"]
MIG --> DUPLO["duplo-deploy (flask, then webserver)"]
MIG --> AGENTS["deploy-agents (onramp-agents runtimes)"]
MIG --> INFRA["deploy-infra (Pulumi, per env stack)"]
DUPLO --> REC["record-deployment step, per leg"]
REC --> GHD["GitHub Deployment + commit status + run summary"]target_environment: prod→ web app to[prod-usw2, prod-gdpr-ir]and agents to[prod, gdpr-prod](parallel,fail-fast: false).stage→[stage], called bypromote-staging.ymlon every push tomain.- The order is a contract application code depends on. The migrate gate is the point of no return, so it waits for every prerequisite of the rollout — all three images, the SSM keypair, the version and the tenant list — and precedes everything that puts this release's code in front of a database: the web app, the agent runtimes, and the Pulumi apply, which rolls DB-reading runtimes of its own — and those three then roll in parallel with no order between them. Which of those edges gate and which only sequence, and what each unit may therefore assume about the others mid-release, is
.claude/rules/deploy-boundaries.md. record-deploymentwrites a GitHub Deployment (ref = the version tag, so the chip showssnapshot/v4.2.x-13/v4.3.0rather than a bare sha), adeploy/<env>commit status carrying the version (visible on the commit + PRs), and a run-summary line. Each rollout leg records its own environment, so the record describes that tenant's rollout and nothing else. Hotfix tooling resolves the live prod SHA from those records via.github/scripts/resolve-prod-deployment.ts— the single rule, which refuses rather than guessing when prod cannot be established.
Tag → environment routing
determine-environment.js (inside deploy-release.yml → environment-setup.yml) is additive: legacy -(prod|stage|dev) suffix branches are preserved, and a non-prerelease clean-semver tag routes to prod.
| Tag | Prerelease | Routes to |
|---|---|---|
v4.2.0 (trunk release) / v4.2.1 (hotfix) | false | prod |
v4.1.0-prod (legacy) | false | prod |
| any | true | dev |
Environments
ONE Environment exists per tier, the bare <env>, and every deploy job binds it — deploy-infra-reusable.yml, migrate-gate.yml, deploy-agents-reusable.yml, and the app rollout alike.
They can share it because release reporting no longer depends on telling their Deployment records apart. GitHub auto-creates a Deployment record for ANY job that references an Environment, so the infra and migrate legs leave versionless records behind on a cancelled or partially-failed run. Jira's GitHub integration reads deployment history as a release timeline and computes an issue window as "commits since the last record on this environment", which let one of those phantoms become a real release's starting boundary and silently drop every issue merged before it. report-jira-deployment.ts now states the version and its issue keys to Jira EXPLICITLY, so no window is inferred and a versionless record cannot move anything. The environment's Jira type is declared in jira-environment.ts rather than guessed from its name.
Provisioning the Jira credential that reporting needs, and rotating it, is docs/runbooks/jira-release-reporting.md. Reporting is best-effort: with no credential configured the step warns and exits 0, so releases are unaffected and only the Jira release panel goes unpopulated.
The deployment branch/tag allow-list is declared ONCE, in ALLOWED_REFS_BY_ENV (.github/scripts/env-ref-policy.ts) — not read back from the Environment's live GitHub state — and checked in TWO places that are NOT equivalent controls:
- The
<env>GitHub Environment's own deployment branch/tag policy — applied from the sameALLOWED_REFS_BY_ENVbyonramp-agents/scripts/bootstrap-agent-environments.ts, an out-of-band admin run with nothing in CI sequencing or verifying it. This is the real control: it's enforced by GitHub and cannot be edited from the ref being judged. An Environment GitHub auto-creates on first reference starts with NO allow-list, so it only protects once that script has been run — and re-run after everyALLOWED_REFS_BY_ENVchange. - In the job itself:
check-deploy-ref.ts, the first real step of deploy-infra-reusable.yml'sdeployjob, migrate-gate.yml'smigratejob, and deploy-agents-reusable.yml'sdeploy/deploy-dashboardjobs — the same check, from the same declared list, run a second time inside the job. This runs FROM the checkout of the ref it's judging, so a branch that also editsenv-ref-policy.ts(or the guard itself) passes its own check. It's a backstop against an accidental bad ref (wrong branch dispatched, a matrix/choice list that drifted out of sync withALLOWED_REFS_BY_ENV), not a substitute for the Environment policy against a deliberate one — provisioning that stays a real security prereq, not just an availability one.
stage allows main (branch, for merge deploys) plus snapshot/* (tag, for hatch rollbacks); prod/gdpr-prod allow main (branch) plus v* and the legacy v*-prod (tag), so a published release can deploy; docs-prod, the environment the developer docs publish job binds, allows the same refs as prod; tools and release-tags allow main only. Change the allow-list in env-ref-policy.ts (both enforcement points read it), then re-run the bootstrap script for every affected env — never edit either Environment's policy directly in the GitHub UI, which the next bootstrap run reconciles away.
Release tag control
Only the release GitHub App can create, move or delete a v* or snapshot/* tag. The Release Tag Control ruleset enforces it with the App as its one bypass actor, and the App's key is held by the release-tags Environment, which admits main only. Four jobs mint a token from it: the stage deploy's snapshot-tag, the publish jobs of promote-prod and promote-hotfix, and cleanup-snapshot-tags. Releases themselves are still created with GITHUB_TOKEN, since an App-created Release would fire deploy-release.yml a second time.
No workflow deletes a v* tag, on purpose. Deleting one (a hotfix tag that landed on the wrong commit, say) is a repo admin lifting the ruleset for the length of the deletion, then re-enabling it.
tools is the one tier off the release train: an infra-only tier for internal tooling, applied by deploy-infra-tools.yml on every merge to main that touches infra/**. Only that infra job binds it: no app, migrate or agent leg.
- Required reviewers are NOT configured: that protection rule is Enterprise-only on private repos and the org is on the Team plan (the API attempt 422s — see
preview-shared-apply.yml). The human control onpromote-prod/deploy-releaseis the deliberate dispatch itself (actor + ref + timestamp recorded on the run). If the org ever moves to a plan that supports them, reviewers onprod(and the other production-tier envs) would additionally provide a pause-for-approval gate.
Hotfix flow
Hotfixes never touch main directly — a short-lived branch is cut from the deployed prod SHA, fixed, promoted straight to prod, then forward-ported.
sequenceDiagram
actor Eng
participant CH as Create Hotfix
participant HB as hotfix/v4.N.P
participant PH as Promote Hotfix
participant Prod
participant FP as Forward-port
participant Main as main
Eng->>CH: dispatch (ticket)
CH->>Prod: read live prod SHA (GitHub Deployment)
CH->>HB: branch off prod SHA, next patch
Eng->>HB: commit fix
Eng->>PH: dispatch (hotfix branch)
PH->>PH: validate name, CI green, smoke
PH->>Prod: tag v4.N.P, publish Release, deploy
PH->>FP: trigger forward-port
FP->>Main: cherry-pick onto forward-port branch, open PR (automation App)
FP->>Main: approve + auto-merge on green
PH->>HB: delete branch🔥 Create Hotfix Branch— resolves the live prod SHA via.github/scripts/resolve-prod-deployment.ts, computes the next patch, brancheshotfix/v4.N.P.🚑 Promote Hotfix to Production— validates the branch, checks CI green (python-test/frontend-ci/e2erun onhotfix/**pushes so the gate has real checks to read), runs a smoke suite (placeholder until an ephemeral-env harness exists), tags + publishes the Release and dispatchesdeploy-release.ymlat the tag (→ prod, skips staging by design), triggers forward-port, deletes the branch. Theskip_forward_portinput skips the forward-port PR when the hotfix commits were cherry-picked from an existing PR — merged (the auto cherry-pick back would be an empty no-op) or open (that PR is the path tomainand inherits the forward-port SLA; no branch rename — the drift-sentinel keys on the tag, not branch names, so just label the PRforward-port). It requiresforward_port_source_pr, the number of that PR: promote verifies it against the branch and records it as the tag'sforward-port-source/*marker, which is the only thing that lets the drift-sentinel clear a skipped forward-port — and only once that PR is merged intomain. Atypical; never set it on a fresh fix.forward-port-hotfix(workflow_call) — cherry-picks ontoforward-port/<ticket>offmain; clean → opens a labelled PR, approves it, and enables auto-merge (squash) so it lands as soon as CI is green; conflict → draft PR with markers assigned to the author (never auto-merged).Which token opens the PR is load-bearing. GitHub's recursion guard is specific to
GITHUB_TOKEN: a PR opened with it never firespull_requestworkflows, so its required checks sit at "Expected" forever and only a ruleset bypass could merge it. An App installation token is notGITHUB_TOKENand its events do start workflow runs — the same propertydependabot-lockfile-sync.ymlrelies on. So theonramp-github-automationApp opens the PR, real CI runs, and auto-merge lands it through the front door with no bypass grant. That forces the approval onto the other identity: GitHub rejects a self-approval, sogithub-actions[bot]approves what the App authored (org setting "Allow GitHub Actions to create and approve pull requests", already on).Credentials:
AUTOMATION_BOT_APP_ID(App ID or Client ID) +AUTOMATION_BOT_APP_PRIVATE_KEYorg secrets; the installation needs Contents + Pull requests R/W, and Workflows R/W for hotfixes that touch.github/workflows/**. Absent, the PR still opens (asgithub-actions[bot], unchecked) and the auto-merge job fails loudly rather than the fix going missing.Enabling auto-merge is not the same as merging: a forward-port touching a
CODEOWNERSpath needs a code-owner approval the bot's cannot satisfy, and auto-merge would sit armed but never fire. The job checksreviewDecisionafter arming and comments on the PR when it is stillREVIEW_REQUIRED, so that stall surfaces immediately rather than at the drift-sentinel's 24h mark.
Deferred (tracked follow-up): the "destructive migration pending in trunk not in prod" gate on create-hotfix. Interim mitigation: expand-and-contract discipline + manual vigilance — see the migration runbook.
Guardrails
| Guardrail | Mechanism | Effect |
|---|---|---|
| Hotfix can't land on trunk via merge | forbid-hotfix-merge.yml (PR check, required on main) | Fails any PR whose head is hotfix/* |
A red main can't be promoted | .github/actions/main-suite-gate, called once per gated suite (E2E, Python unit) by both promote workflows (not by the merge-triggered stage deploy) | Fails the promotion when the newest conclusive push-to-main run of that suite in the target's history failed, or when the run listing cannot show that history; reads a stored verdict, never waits |
| A release can't overtake an unported hotfix migration | .github/scripts/hotfix-migration-guard.ts, run by resolve in 🚀 Promote to Production (forward promotions only) | Refuses the promotion while the newest release tag carries a migration file the candidate lacks. No override: the fix is merging the hotfix's forward-port |
| A contract migration can't ship with the code that stopped reading what it drops | requires_deployed in the migration, required by make lint.migrations; .github/scripts/contract-migration-gate.ts, run by resolve in 🚀 Promote to Production (forward promotions) and by the contract-order job ahead of the migrate gate in every stage and prod deploy | Refuses while any target environment's running release (its newest successful rollout record) lacks the declared commit. Clear it by deploying that commit first. skip_migrate_gate skips it along with the migrations |
A red main doesn't go unnoticed | notify-failure jobs in e2e.yml and python-test.yml → .github/actions/slack-alert | Posts to #eng-alerts under e2e-failure (the red suites) or pytest-failure (the first red push, its PR author, the failing tests and a flaky-or-own-base tag), so an unpromotable trunk surfaces at merge time instead of at the next promotion attempt. A pytest run that times out posts under pytest-failure too, naming the push left unverified |
| Forward-port lands without human toil | forward-port-hotfix.yml auto-merge job (App-authored PR, github-actions[bot] approves, auto-merge squashes on green) | Reconciliation completes without a human; a red check holds the merge, and a setup failure is loud (red run + PR comment pinging the author), never silent |
| Forward-port can't be silently skipped | drift-sentinel.yml (daily cron) | Backstop for the above: pages if a prod commit has been off main past a 24h SLA; the 7-day scan is backstop breadth. A deliberately skipped forward-port clears only on its marker's source PR being merged into main; a breach that is real but benign is recorded with a bounded drift-ack/* tag, which stops the failure without hiding the breach |
| A stage deploy can't be cancelled mid-migration | promote-staging.yml concurrency: one group for every ref, cancel-in-progress: false | A merge deploy and a hatch redeploy queue instead of racing on the stage DB; a burst of merges deploys the newest and skips the ones between (no snapshot tag for those) |
| Release tags stay clean | Release Tags ruleset (^v\d+\.\d+\.(0|[1-9]\d*)$) | Only well-formed vM.N.0/vM.N.P in the v* namespace |
Only the release workflows create, move or delete v* and snapshot/* tags | A tag ruleset on both namespaces whose one bypass actor is the release GitHub App; the App's key lives in the release-tags Environment, which admits main only | A tag decides which role a job may assume (prod and gdpr-prod admit v*, stage admits snapshot/*), so a writer pushing a tag from an unreviewed commit would otherwise run with a deploy role. The GITHUB_TOKEN of a branch workflow cannot bypass the ruleset, and a branch workflow cannot read the key |
main protected | main ruleset (cloned from the old dev ruleset's checks + forbid-hotfix-merge, squash-only, code-owner review) | Trunk integrity |
| Hotfix branches created only by automation | branch ruleset on hotfix/* | Creation restricted to the create-hotfix actor |
| State is authoritative | record-deployment → GitHub Deployments | "What's in prod?" is a query, not a guess |
Workflow inventory
| Workflow | Trigger | Role |
|---|---|---|
⚙️ Deploy to Staging (internal) | push main; dispatch (by Promote to Staging, at the target ref) | Build + deploy to stage via build-and-deploy (migrate gate, Duplo, agents, best-effort Pulumi); tag snapshot/<version>; record |
🧪 Promote to Staging | dispatch (blank version = redeploy latest main, or a snapshot/* tag = roll back until the next merge) | Redeploy / rollback hatch: validate the selection + promotion gates → dispatch the staging deploy worker at that ref |
🚀 Promote to Production | dispatch (blank ref = staging-validated, the last finished stage deploy) | Promotion gates → mint tag on the commit stage last deployed → Release → prod fan-out |
🤖 Deploy Agents (Stage) | dispatch | Out-of-band agent redeploy; stage's agents otherwise deploy inside the merge deploy |
E2E (Playwright) 🎭 | push main + hotfix/**, dispatch | Full Playwright matrix; its stored verdict is one of the two the promote gates read |
🔥 Create Hotfix Branch | dispatch | Branch off live prod SHA |
🚑 Promote Hotfix to Production | dispatch | Tag + deploy hotfix to prod, forward-port |
forward-port-hotfix | workflow_call | Cherry-pick hotfix → main PR, then approve + merge it |
forbid-hotfix-merge | pull_request → main | Block direct hotfix merges |
drift-sentinel | cron + dispatch | Detect un-forward-ported prod commits |
cleanup-snapshot-tags | cron + dispatch | Prune snapshot/* tags older than 30d |
build-and-deploy | workflow_call | Build once, fan out, record deployment |
deploy-release | release: published | Fires on a minted Release → environment-setup → build-and-deploy |
Related
- Releasing & Hotfixing — the day-to-day HOWTO.
- Migrations: expand & contract — the discipline hotfixes rely on.