Releasing & Hotfixing

The day-to-day HOWTO for shipping code under the trunk-based release process — how to release, promote, and hotfix, plus the answers you want mid-incident. For how the system works see Release & Deploy Architecture.

Audience	Any engineer shipping, releasing, hotfixing, or debugging a deploy
The one rule	main is the only branch. Merging ships to dev. Stage + prod are a /promote-* skill (or a button), not a merge.

Shipping a normal change

Open a PR → review → merge to main. On merge it deploys to dev automatically and tags the build snapshot/<version> (e.g. snapshot/v4.2.x-13). That's the whole loop — no environment-branch merges.

Your change is not in prod until someone promotes it. Merging ≠ releasing.

Releasing to staging

Primary path: the /promote-stage agent skill. It shows what's on stage vs the latest main, asks whether to promote latest main (the common case) or a specific snapshot — listing the current snapshot tags so you never type one from memory — then dispatches and watches the deploy through to the recorded stage deployment.

Fallback (Actions UI): run 🧪 Promote to Staging → Run workflow. Same model as every promote workflow — never touch the "Use workflow from" dropdown (it stays on main; the workflow refuses anything else):

• Leave version blank (default) → promote the latest main. Just hit Run.
• Type a snapshot tag (e.g. snapshot/v4.2.x-13) → promote that exact snapshot. A typo'd or pruned tag fails fast and lists the current snapshots in the run summary.

Either way the workflow validates your selection, checks that main isn't red on either gated suite (see below), then dispatches the actual build + deploy (⚙️ Deploy to Staging (internal)) at the chosen commit. No release tag is minted — the stage deployment is recorded against the snapshot/<version> tag (created on the spot if it doesn't exist yet), so the Deployments UI shows the exact snapshot on staging — the one Promote to Production will ship.

Releasing to production

Primary path: the /promote-prod agent skill. It shows what's on prod, what staging validated, and the commit delta between them, requires an explicit "shipping to PRODUCTION" confirmation, then dispatches and watches the promote + deploy runs.

Fallback (Actions UI): run 🚀 Promote to Production (requires reviewer approval via the prod Environment). Same model — never touch the "Use workflow from" dropdown, and leave the ref input blank — prod ships whatever is currently on staging (resolved by .github/scripts/resolve-stage-deployment.ts — the commit staging has validated, not one it is still deploying), not the tip of main. Staging usually lags main, so this releases exactly what you validated. Promotion then:

1. Checks main isn't red on either gated suite (see below) — skipped on rollback.
2. Mints the next clean release tag (v4.N.0) on the staging-validated commit.
3. Publishes a GitHub Release on that tag.
4. Fans the build out to prod + demo + gdpr-prod.

So the ladder is: merge → dev (auto) → /promote-stage → /promote-prod. Promote to staging first, or prod has nothing to ship.

The promotion gates — "main is in a … failure state"

Two post-merge suites gate promotion, each as its own step with its own override:

Suite	Runs on	Override
E2E (e2e.yml)	push to main + hotfix/** — not on PRs	skip_e2e_gate
Python unit (python-test.yml)	push to main + hotfix/**, and PRs	skip_unit_gate

Both promote workflows refuse to promote while either result is red. The gates read the stored result — they never wait for a suite — so they cost seconds, not minutes.

The unit suite is gated even though it is a required PR check, because a green PR check does not mean main is green. Two PRs that each forked before the other can both be green, merge cleanly because they touch different files, and still leave the trunk red — one adds a call site, the other changes the signature it calls. Only the push-to-main run sees the two together.

If a promotion fails on a gate, the run summary names the failing run and links it. Three ways out, in order of preference:

1. Fix forward. Land the fix on main; a green run on a newer commit outranks the older red one, so the next promote just works.
2. Re-run the failed job. If the failure was environmental, "Re-run failed jobs" on that run flips it green — one cheap job, not the whole matrix. Infrastructure-shaped failures (a psycopg2 connection-limit error from the xdist worker pool, a runner OOM) are usually environmental — re-run once, and treat a repeat as a real regression rather than re-running again.
3. Override. Re-dispatch the promote workflow with that suite's skip_* input checked. This is for an outage in the harness itself (seeded stack, runners) — not for a real failure, which is telling you the build is broken. Each override waives only its own suite, and is recorded on the run.

Things that are deliberately not gated here: rollback (redeploying a released tag is the recovery path), and hotfixes (they never touch main, so main's state says nothing about them — 🚑 Promote Hotfix to Production instead requires every check green on the branch tip, which covers both suites because each also runs on the hotfix/** push).

Promoting a commit that merged moments ago is fine: a run still in flight is never waited on, and the gates fall back to the last conclusive ancestor. A run cancelled because a newer push superseded it carries no verdict and is passed over, not read as a failure.

To get E2E on a branch before merging — worth it for a risky change — dispatch E2E (Playwright) 🎭 on it (gh workflow run e2e.yml --ref <branch>).

Rollback: re-run 🚀 Promote to Production with ref = a previous release tag — v4.N.0 or a hotfix v4.N.P → redeploys it, no new tag, no revert PR. (ref also accepts a specific SHA/branch to force-promote, bypassing staging — advanced.) Before rolling back, check whether the bad release shipped migrations — code rolls back, the database doesn't. The /rollback agent skill walks the whole decision + execution.

Hotfixing production

Hotfixes never go through main first — they're cut from the live prod commit, shipped, then forward-ported back.

1. 🔥 Create Hotfix Branch — pass the ticket. Branches hotfix/v4.N.P off the currently-deployed prod SHA (not main).
2. Push your fix to that branch.
3. 🚑 Promote Hotfix to Production — validates the branch, checks CI, tags v4.N.P, deploys to prod (skips staging by design), and opens and merges the forward-port PR to main automatically.
4. Nothing — reconciliation is automatic. The forward-port PR is opened by the onramp-github-automation App, approved, and set to auto-merge, so it squash-merges as soon as its checks pass. A red promote run means only one thing: auto-merge couldn't be set up, so merge the PR by hand. Three other cases need you while the run stays green — a cherry-pick conflict (a draft PR is opened and assigned to you), a failing check (the cherry-pick hit real drift on main; fix it forward), and a forward-port touching a CODEOWNERS path (the bot approval can't satisfy the code-owner gate, so a code owner must approve). Judge by the PR, not the run colour; the drift-sentinel backstops all three at 24h.

You cannot merge a hotfix/* branch into main directly — forbid-hotfix-merge blocks it. Hotfixes reach trunk only via the forward-port PR, and a daily drift-sentinel pages if a prod commit stays off main past 24h.

Shipping an existing PR (merged or not) as a hotfix

Same flow, two deltas: instead of writing a fix on the hotfix branch, cherry-pick the PR's commits onto it (git cherry-pick -x <sha> — the squash commit for a merged PR, the branch commits for an open one), and promote with skip_forward_port: true and forward_port_source_pr: <number> — the source PR is the path to main, so the generated forward-port PR would be a duplicate (or, for an already-merged PR, an empty no-op).

Three rules keep this honest:

• The source PR number is required with the toggle, and promote refuses before anything ships if it doesn't hold up — the criteria are in that input's own description in promote-hotfix.yml, the step that enforces them. It is recorded as the tag's forward-port marker, and drift-sentinel clears the tag only once that PR is merged into main, so an abandoned one still pages. Pick with git cherry-pick -x so each commit's provenance is recorded; the check reads those trailers.
• An open source PR inherits the forward-port duty: merge it within the same ≤24h SLA, and merge it as shipped — commits added after the promote make main diverge from what prod is running.
• The toggle is only for this case. A fresh fix promoted with it set has no path back to main and is lost on the next release.

No branch rename needed for an open source PR. Nothing keys on the forward-port/* branch name — the drift-sentinel SLA check watches the hotfix tag, and for a skipped forward-port it resolves the tag's marker to the source PR and reads that PR's merge state, so any branch name satisfies it on merge. Do mark the PR so people scanning open forward-ports see it:

gh pr edit <number> --add-label forward-port
gh pr comment <number> --body "Shipped to prod as v4.N.P via hotfix (forward-port skipped). Merge within 24h, as-shipped."

The /create-hotfix and /promote-hotfix agent skills drive the flow end-to-end: /create-hotfix picks the mode, cuts the branch, lands the fix (fresh or cherry-pick), and waits for CI; /promote-hotfix runs the approval-gated promote, the deploy, and the forward-port reconciliation.

Workflow cheat-sheet

Prefer the agent skills — /promote-stage, /promote-prod, /create-hotfix, /promote-hotfix, /rollback — they gather context, validate the target, and watch the runs. The workflows below are what the skills drive; run them directly only when working without an agent. Emoji-prefixed so you can spot (and pin) them in the Actions sidebar.

Workflow	When you run it	Target
🛠️ Deploy Dev	(automatic on merge to main)	dev
🧪 Promote to Staging	Validate a build on stage (blank = latest main, or a snapshot/* tag)	stage
⚙️ Deploy to Staging (internal)	(dispatched by Promote to Staging — don't run directly)	stage
🚀 Promote to Production	Ship a release (approved; blank = what's on staging)	prod + demo + gdpr-prod
🔥 Create Hotfix Branch	Start an urgent prod fix	(branch only)
🚑 Promote Hotfix to Production	Ship + forward-port the fix	prod
🤖 Deploy {Copilot,Intel,Portal} Runtime	Break-glass single-agent deploy	chosen env

FAQ

How do I know what's in prod?

GH_TOKEN=$(gh auth token) GITHUB_REPOSITORY=OnRampTech/main-web-application \
  bun .github/scripts/resolve-prod-deployment.ts

That is the single rule the hotfix workflows use. It reads only records an app rollout actually wrote, and refuses rather than guessing when prod cannot be established. Each commit also gets a deploy/<env> status carrying its version. No git log guessing, and no reading the newest Deployment by hand: most of them are versionless records GitHub creates for any job bound to the environment.

My change merged to main — is it in prod? No. Merging deploys to dev only. It reaches prod when someone runs Promote to Production. The prod resolver tells you what's live.

Did a specific commit/PR ship to prod yet? Find the earliest vM.N.0 release tag that contains the commit, then check whether prod's currently-resolved version is ≥ that tag. The commit shipped when that earliest release reached prod — not when the latest prod release deployed. The release-status agent skill automates this end to end (PR → sha → ship date) and owns the exact commands.

What's this v4.2.x-13 version on dev?

It means 13 commits past the v4.2.0 release. The x patch slot is the tell that it's a snapshot of the 4.2 line, not the v4.2.0 release itself. It's computed at build time and materialized as a snapshot/v4.2.x-13 git tag (click it → jumps to the commit). You never type or compute it. It is not a release — only promote-prod mints clean v4.N.0 release tags; snapshot/* snapshot tags are auto-pruned after 30 days.

How do I roll back prod? Re-run 🚀 Promote to Production with ref set to the previous release tag (v4.N.0 or hotfix v4.N.P). Or let the /rollback agent skill drive it — including the migration-safety check.

Do I need to bump the version? No. Minor auto-increments on prod promotion. Only a breaking-change milestone bumps VERSION_MAJOR, via reviewed PR.

Where are the dev / staging branches? Gone — archived read-only at cutover. You branch from and PR into main.

Related

• Release & Deploy Architecture — how the system works.
• Migrations: expand & contract — the discipline hotfixes rely on.