Skip to content

Releasing & Hotfixing ​

The day-to-day HOWTO for shipping code under the trunk-based release process — how to release, promote, and hotfix, plus the answers you want mid-incident. For how the system works see Release & Deploy Architecture.

AudienceAny engineer shipping, releasing, hotfixing, or debugging a deploy
The one rulemain is the only branch. Merging ships to stage. Prod is the /promote-prod skill (or a button), not a merge.

Shipping a normal change ​

Open a PR → review → merge to main. On merge it deploys to stage automatically (⚙️ Deploy to Staging (internal)) and tags the build snapshot/<version> (e.g. snapshot/v4.2.x-13). Stage's database runs your migration first and its stack applies your infra change first. That's the whole loop — no environment-branch merges.

Deploys queue rather than cancel, and GitHub keeps one pending run, so a burst of merges deploys the newest and skips the ones in between — those commits get no snapshot tag.

Your change is not in prod until someone promotes it. Merging ≠ releasing.

Redeploying or rolling back staging ​

Merges already deploy stage, so you do not promote to it in the normal flow. The staging workflow is the redeploy / rollback hatch, for two cases: re-running a merge deploy that failed or was skipped, and rolling stage back to an earlier build.

Rolling back: the /rollback agent skill lists the snapshot tags from the stage Deployments so you never type one from memory, then dispatches and watches the deploy through to the recorded stage deployment.

Redeploying, or rolling back without an agent: run 🧪 Promote to Staging → Run workflow. Check first that a newer merge deploy is not already running — it would put the latest main on stage anyway. Don't re-run an older failed merge deploy instead: once a newer merge has deployed, that re-run would roll stage back to the older commit. Same model as every promote workflow — never touch the "Use workflow from" dropdown (it stays on main; the workflow refuses anything else):

  • Leave version blank (default) → redeploy the latest main. Just hit Run.
  • Type a snapshot tag (e.g. snapshot/v4.2.x-13) → roll stage back to that exact snapshot. A typo'd or pruned tag fails fast and lists the current snapshots in the run summary. The rollback holds only until the next merge deploys main over it.

Either way the workflow validates your selection, checks that main isn't red on either gated suite (see below), then dispatches the actual build + deploy (⚙️ Deploy to Staging (internal)) at the chosen commit. The run is titled 🧪 Redeploy <version|latest main> → staging. No release tag is minted — the stage deployment is recorded against the snapshot/<version> tag (created if it doesn't exist yet), so the Deployments UI shows the exact snapshot on staging.

Releasing to production ​

Primary path: the /promote-prod agent skill. It shows what's on prod, what stage last deployed, and the commit delta between them, requires an explicit "shipping to PRODUCTION" confirmation, then dispatches and watches the promote + deploy runs.

Fallback (Actions UI): run 🚀 Promote to Production (there is no reviewer gate — dispatching it is the approval). Same model — never touch the "Use workflow from" dropdown, and leave the ref input blank — prod ships whatever stage last deployed successfully (resolved by .github/scripts/resolve-stage-deployment.ts — the newest successful stage rollout, not one still deploying), not the tip of main: a newer merge may still be deploying or have failed. That is the last merge whose stage deploy finished — continuous delivery, with no minimum soak time, and nobody is assumed to have checked it on stage. If a change needs a human look first, look at stage before you promote. Promotion then:

  1. Checks main isn't red on either gated suite (see below), and that the commit carries every migration the newest release does — both skipped on rollback. The second one fails while a hotfix that shipped a migration still has its forward-port open: merge that PR, let its stage deploy finish, then promote again.
  2. Mints the next clean release tag (v4.N.0) on that stage-deployed commit.
  3. Publishes a GitHub Release on that tag.
  4. Fans the build out to prod + gdpr-prod.

So the ladder is: merge → stage (auto) → /promote-prod. Stage deploys every merge, so prod always has a commit to ship; Promote to Staging is only the hatch.

The promotion gates — "main is in a … failure state"

Two post-merge suites gate promotion, each as its own step with its own override:

SuiteRuns onOverride
E2E (e2e.yml)push to main + hotfix/** — not on PRsskip_e2e_gate
Python unit (python-test.yml)push to main + hotfix/**, and PRsskip_unit_gate

Both promote workflows refuse to promote while either result is red. The gates read the stored result — they never wait for a suite — so they cost seconds, not minutes.

The unit suite is gated even though it is a required PR check, because a green PR check does not mean main is green. Two PRs that each forked before the other can both be green, merge cleanly because they touch different files, and still leave the trunk red — one adds a call site, the other changes the signature it calls. Only the push-to-main run sees the two together.

If a promotion fails on a gate, the run summary names the failing run and links it. Three ways out, in order of preference:

  1. Fix forward. Land the fix on main; a green run on a newer commit outranks the older red one, so the next promote just works.
  2. Re-run the failed job. If the failure was environmental, "Re-run failed jobs" on that run flips it green — one cheap job, not the whole matrix. Infrastructure-shaped failures (a psycopg2 connection-limit error from the xdist worker pool, a runner OOM) are usually environmental — re-run once, and treat a repeat as a real regression rather than re-running again.
  3. Override. Re-dispatch the promote workflow with that suite's skip_* input checked. This is for an outage in the harness itself (seeded stack, runners) — not for a real failure, which is telling you the build is broken. Each override waives only its own suite, and is recorded on the run.

Things that are deliberately not gated here: rollback (redeploying a released tag is the recovery path), and hotfixes (they never touch main, so main's state says nothing about them — 🚑 Promote Hotfix to Production instead requires every check green on the branch tip, which covers both suites because each also runs on the hotfix/** push).

Promoting a commit that merged moments ago is fine: a run still in flight is never waited on, and the gates fall back to the last conclusive ancestor. A run cancelled because a newer push superseded it carries no verdict and is passed over, not read as a failure.

To get E2E on a branch before merging — worth it for a risky change — dispatch E2E (Playwright) 🎭 on it (gh workflow run e2e.yml --ref <branch>).

Rollback: re-run 🚀 Promote to Production with ref = a previous release tag — v4.N.0 or a hotfix v4.N.P → redeploys it, no new tag, no revert PR. (ref also accepts a specific SHA/branch to force-promote, bypassing staging — advanced.) Before rolling back, check whether the bad release shipped migrations — code rolls back, the database doesn't. The /rollback agent skill walks the whole decision + execution.

Release notes ​

Each release gets one short line per change, grouped as New, Improved and Fixed. The line is drafted from the bold sentence and paragraph at the top of the PR body, so that lead is what readers of #product-releases and the Notion release page see. Label a PR flag:off when it ships behind a feature flag that is off, or flag:limited when the flag is on for only some organizations; the notes say so only when one of those labels is present. The /pr skill asks about the flag when the diff adds one and applies the label.

Hotfixing production ​

Hotfixes never go through main first — they're cut from the live prod commit, shipped, then forward-ported back.

  1. 🔥 Create Hotfix Branch — pass the ticket. Branches hotfix/v4.N.P off the currently-deployed prod SHA (not main).
  2. Push your fix to that branch.
  3. 🚑 Promote Hotfix to Production — validates the branch, checks CI, tags v4.N.P, deploys to prod (skips staging by design), and opens and merges the forward-port PR to main automatically.
  4. Nothing — reconciliation is automatic. The forward-port PR is opened by the onramp-github-automation App, approved, and set to auto-merge, so it squash-merges as soon as its checks pass. A red promote run means only one thing: auto-merge couldn't be set up, so merge the PR by hand. Three other cases need you while the run stays green — a cherry-pick conflict (a draft PR is opened and assigned to you), a failing check (the cherry-pick hit real drift on main; fix it forward), and a forward-port touching a CODEOWNERS path (the bot approval can't satisfy the code-owner gate, so a code owner must approve). Judge by the PR, not the run colour; the drift-sentinel backstops all three at 24h.

You cannot merge a hotfix/* branch into main directly — forbid-hotfix-merge blocks it. Hotfixes reach trunk only via the forward-port PR, and a daily drift-sentinel pages if a prod commit stays off main past 24h.

Shipping an existing PR (merged or not) as a hotfix ​

Same flow, two deltas: instead of writing a fix on the hotfix branch, cherry-pick the PR's commits onto it (git cherry-pick -x <sha> — the squash commit for a merged PR, the branch commits for an open one), and promote with skip_forward_port: true and forward_port_source_pr: <number> — the source PR is the path to main, so the generated forward-port PR would be a duplicate (or, for an already-merged PR, an empty no-op).

Three rules keep this honest:

  • The source PR number is required with the toggle, and promote refuses before anything ships if it doesn't hold up — the criteria are in that input's own description in promote-hotfix.yml, the step that enforces them. It is recorded as the tag's forward-port marker, and drift-sentinel clears the tag only once that PR is merged into main, so an abandoned one still pages. Pick with git cherry-pick -x so each commit's provenance is recorded; the check reads those trailers.
  • An open source PR inherits the forward-port duty: merge it within the same ≤24h SLA, and merge it as shipped — commits added after the promote make main diverge from what prod is running.
  • The toggle is only for this case. A fresh fix promoted with it set has no path back to main and is lost on the next release.

No branch rename needed for an open source PR. Nothing keys on the forward-port/* branch name — the drift-sentinel SLA check watches the hotfix tag, and for a skipped forward-port it resolves the tag's marker to the source PR and reads that PR's merge state, so any branch name satisfies it on merge. Do mark the PR so people scanning open forward-ports see it:

bash
gh pr edit <number> --add-label forward-port
gh pr comment <number> --body "Shipped to prod as v4.N.P via hotfix (forward-port skipped). Merge within 24h, as-shipped."

The /create-hotfix and /promote-hotfix agent skills drive the flow end-to-end: /create-hotfix picks the mode, cuts the branch, lands the fix (fresh or cherry-pick), and waits for CI; /promote-hotfix runs the approval-gated promote, the deploy, and the forward-port reconciliation.

Workflow cheat-sheet ​

Prefer the agent skills — /promote-prod, /create-hotfix, /promote-hotfix, /rollback — they gather context, validate the target, and watch the runs. The workflows below are what the skills drive; run them directly only when working without an agent. Emoji-prefixed so you can spot (and pin) them in the Actions sidebar.

WorkflowWhen you run itTarget
⚙️ Deploy to Staging (internal)(automatic on merge to main; also dispatched by Promote to Staging — don't run directly)stage
🧪 Promote to StagingRedeploy or roll back stage (blank = redeploy latest main, or a snapshot/* tag)stage
🚀 Promote to ProductionShip a release (blank = what stage last deployed)prod + gdpr-prod
🔥 Create Hotfix BranchStart an urgent prod fix(branch only)
🚑 Promote Hotfix to ProductionShip + forward-port the fixprod
🤖 Deploy {Copilot,Intel,Portal} RuntimeBreak-glass single-agent deploychosen env

FAQ ​

How do I know what's in prod?

bash
GH_TOKEN=$(gh auth token) GITHUB_REPOSITORY=OnRampTech/main-web-application \
  bun .github/scripts/resolve-prod-deployment.ts

That is the single rule the hotfix workflows use. It reads only records an app rollout actually wrote, and refuses rather than guessing when prod cannot be established. Each commit also gets a deploy/<env> status carrying its version. No git log guessing, and no reading the newest Deployment by hand: most of them are versionless records GitHub creates for any job bound to the environment.

My change merged to main — is it in prod? No. Merging deploys to stage only. It reaches prod when someone runs Promote to Production. The prod resolver tells you what's live.

Did a specific commit/PR ship to prod yet? Find the earliest vM.N.0 release tag that contains the commit, then check whether prod's currently-resolved version is ≥ that tag. The commit shipped when that earliest release reached prod — not when the latest prod release deployed. The release-status agent skill automates this end to end (PR → sha → ship date) and owns the exact commands.

What's this v4.2.x-13 version on stage?

It means 13 commits past the v4.2.0 release. The x patch slot is the tell that it's a snapshot of the 4.2 line, not the v4.2.0 release itself. It's computed at build time and materialized as a snapshot/v4.2.x-13 git tag (click it → jumps to the commit). You never type or compute it. It is not a release — only promote-prod mints clean v4.N.0 release tags; snapshot/* snapshot tags are auto-pruned after 30 days.

How do I roll back prod? Re-run 🚀 Promote to Production with ref set to the previous release tag (v4.N.0 or hotfix v4.N.P). Or let the /rollback agent skill drive it — including the migration-safety check.

Do I need to bump the version? No. Minor auto-increments on prod promotion. Only a breaking-change milestone bumps VERSION_MAJOR, via reviewed PR.

Where are the dev / staging branches? Gone — archived read-only at cutover. You branch from and PR into main.

Internal documentation — gated behind Cloudflare Access.