Releasing & Hotfixing

The day-to-day HOWTO for shipping code under the trunk-based release process — how to release, promote, and hotfix, plus the answers you want mid-incident. For how the system works see Release & Deploy Architecture.

Audience	Any engineer shipping, releasing, hotfixing, or debugging a deploy
The one rule	main is the only branch. Merging ships to dev. Stage + prod are a /promote-* skill (or a button), not a merge.

Shipping a normal change

Open a PR → review → merge to main. On merge it deploys to dev automatically and tags the build snapshot/<version> (e.g. snapshot/v4.2.x-13). That's the whole loop — no environment-branch merges.

Your change is not in prod until someone promotes it. Merging ≠ releasing.

Releasing to staging

Primary path: the /promote-stage agent skill. It shows what's on stage vs the latest main, asks whether to promote latest main (the common case) or a specific snapshot — listing the current snapshot tags so you never type one from memory — then dispatches and watches the deploy through to the recorded stage deployment.

Fallback (Actions UI): run 🧪 Promote to Staging → Run workflow. Same model as every promote workflow — never touch the "Use workflow from" dropdown (it stays on main; the workflow refuses anything else):

• Leave version blank (default) → promote the latest main. Just hit Run.
• Type a snapshot tag (e.g. snapshot/v4.2.x-13) → promote that exact snapshot. A typo'd or pruned tag fails fast and lists the current snapshots in the run summary.

Either way the workflow validates your selection, then dispatches the actual build + deploy (⚙️ Deploy to Staging (internal)) at the chosen commit. No release tag is minted — the stage deployment is recorded against the snapshot/<version> tag (created on the spot if it doesn't exist yet), so the Deployments UI shows the exact snapshot on staging — the one Promote to Production will ship.

Releasing to production

Primary path: the /promote-prod agent skill. It shows what's on prod, what staging validated, and the commit delta between them, requires an explicit "shipping to PRODUCTION" confirmation, then dispatches and watches the promote + deploy runs.

Fallback (Actions UI): run 🚀 Promote to Production (requires reviewer approval via the prod Environment). Same model — never touch the "Use workflow from" dropdown, and leave the ref input blank — prod ships whatever is currently on staging (the last successful staging deployment), not the tip of main. Staging usually lags main, so this releases exactly what you validated. Promotion then:

1. Mints the next clean release tag (v4.N.0) on the staging-validated commit.
2. Publishes a GitHub Release on that tag.
3. Fans the build out to prod + demo + gdpr-prod.

So the ladder is: merge → dev (auto) → /promote-stage → /promote-prod. Promote to staging first, or prod has nothing to ship.

Rollback: re-run 🚀 Promote to Production with ref = a previous release tag — v4.N.0 or a hotfix v4.N.P → redeploys it, no new tag, no revert PR. (ref also accepts a specific SHA/branch to force-promote, bypassing staging — advanced.) Before rolling back, check whether the bad release shipped migrations — code rolls back, the database doesn't. The /rollback agent skill walks the whole decision + execution.

Hotfixing production

Hotfixes never go through main first — they're cut from the live prod commit, shipped, then forward-ported back.

1. 🔥 Create Hotfix Branch — pass the ticket. Branches hotfix/v4.N.P off the currently-deployed prod SHA (not main).
2. Push your fix to that branch.
3. 🚑 Promote Hotfix to Production — validates the branch, checks CI, tags v4.N.P, deploys to prod (skips staging by design), and opens and merges the forward-port PR to main automatically.
4. Nothing — reconciliation is automatic. The forward-port PR is opened by the onramp-github-automation App, approved, and set to auto-merge, so it squash-merges as soon as its checks pass. A red promote run means only one thing: auto-merge couldn't be set up, so merge the PR by hand. Three other cases need you while the run stays green — a cherry-pick conflict (a draft PR is opened and assigned to you), a failing check (the cherry-pick hit real drift on main; fix it forward), and a forward-port touching a CODEOWNERS path (the bot approval can't satisfy the code-owner gate, so a code owner must approve). Judge by the PR, not the run colour; the drift-sentinel backstops all three at 24h.

You cannot merge a hotfix/* branch into main directly — forbid-hotfix-merge blocks it. Hotfixes reach trunk only via the forward-port PR, and a daily drift-sentinel pages if a prod commit stays off main past 24h.

Shipping an existing PR (merged or not) as a hotfix

Same flow, two deltas: instead of writing a fix on the hotfix branch, cherry-pick the PR's commits onto it (git cherry-pick -x <sha> — the squash commit for a merged PR, the branch commits for an open one), and promote with skip_forward_port: true — the source PR is the path to main, so the generated forward-port PR would be a duplicate (or, for an already-merged PR, an empty no-op).

Two rules keep this honest:

• An open source PR inherits the forward-port duty: merge it within the same ≤24h SLA, and merge it as shipped — commits added after the promote make main diverge from what prod is running.
• The toggle is only for this case. A fresh fix promoted with it set has no path back to main and is lost on the next release.

No branch rename needed for an open source PR. Nothing keys on the forward-port/* branch name — the drift-sentinel SLA check watches the hotfix tag (are its commits on main?), not branches, so any branch name satisfies it on merge. Do mark the PR so people scanning open forward-ports see it:

gh pr edit <number> --add-label forward-port
gh pr comment <number> --body "Shipped to prod as v4.N.P via hotfix (forward-port skipped). Merge within 24h, as-shipped."

The /create-hotfix and /promote-hotfix agent skills drive the flow end-to-end: /create-hotfix picks the mode, cuts the branch, lands the fix (fresh or cherry-pick), and waits for CI; /promote-hotfix runs the approval-gated promote, the deploy, and the forward-port reconciliation.

Workflow cheat-sheet

Prefer the agent skills — /promote-stage, /promote-prod, /create-hotfix, /promote-hotfix, /rollback — they gather context, validate the target, and watch the runs. The workflows below are what the skills drive; run them directly only when working without an agent. Emoji-prefixed so you can spot (and pin) them in the Actions sidebar.

Workflow	When you run it	Target
🛠️ Deploy Dev	(automatic on merge to main)	dev
🧪 Promote to Staging	Validate a build on stage (blank = latest main, or a snapshot/* tag)	stage
⚙️ Deploy to Staging (internal)	(dispatched by Promote to Staging — don't run directly)	stage
🚀 Promote to Production	Ship a release (approved; blank = what's on staging)	prod + demo + gdpr-prod
🔥 Create Hotfix Branch	Start an urgent prod fix	(branch only)
🚑 Promote Hotfix to Production	Ship + forward-port the fix	prod
🤖 Deploy {Copilot,Intel,Portal} Runtime	Break-glass single-agent deploy	chosen env

FAQ

How do I know what's in prod? The prod Environment's latest successful Deployment shows the version (v4.N.0) and commit. Each commit also gets a deploy/<env> status carrying its version. No git log guessing.

My change merged to main — is it in prod? No. Merging deploys to dev only. It reaches prod when someone runs Promote to Production. The version on the prod Deployment tells you what's live.

Did a specific commit/PR ship to prod yet? Find the earliest vM.N.0 release tag that contains the commit, then check whether prod's current (successfully-deployed) version is ≥ that tag. The commit shipped when that earliest release reached prod — not when the latest prod release deployed. The release-status agent skill automates this end to end (PR → sha → ship date) and owns the exact commands.

What's this v4.2.x-13 version on dev?

It means 13 commits past the v4.2.0 release. The x patch slot is the tell that it's a snapshot of the 4.2 line, not the v4.2.0 release itself. It's computed at build time and materialized as a snapshot/v4.2.x-13 git tag (click it → jumps to the commit). You never type or compute it. It is not a release — only promote-prod mints clean v4.N.0 release tags; snapshot/* snapshot tags are auto-pruned after 30 days.

How do I roll back prod? Re-run 🚀 Promote to Production with ref set to the previous release tag (v4.N.0 or hotfix v4.N.P). Or let the /rollback agent skill drive it — including the migration-safety check.

Do I need to bump the version? No. Minor auto-increments on prod promotion. Only a breaking-change milestone bumps VERSION_MAJOR, via reviewed PR.

Where are the dev / staging branches? Gone — archived read-only at cutover. You branch from and PR into main.

Related

• Release & Deploy Architecture — how the system works.
• Migrations: expand & contract — the discipline hotfixes rely on.