Blacksmith CI Runner Migration (Eval)
TL;DR — Run CI across two free runner buckets instead of one. GitHub Team already bundles 3,000 free Actions minutes/month (standard 2-core — we pay for it whether we use it or not); Blacksmith adds another 3,000 free minutes/month on ~2× faster hardware. So we route by weight: heavy/parallel jobs → Blacksmith (fast, and its own free bucket), light, high-frequency jobs → GitHub's free 2-core bucket. ~6,000 combined free minutes/month; only the overflow bills, cheapest on Blacksmith (~$0.004/min). The whole evaluation costs $0. The only call that needs a budget owner is what happens after we outgrow both free tiers.
Why
Two things line up in 2026:
- GitHub raised Actions prices. GitHub-hosted runner rates changed on 2026-01-01, and self-hosted runners pick up a new $0.002/min control-plane charge from 2026-03-01. CI minutes are no longer close to free.
- Blacksmith is faster and cheaper. Runners are bare-metal (high single-core clocks) — roughly 2× the throughput of GitHub's shared runners — at roughly half the per-minute price. Because jobs finish faster you also burn fewer minutes, so the two effects compound.
The migration itself is a drop-in: Blacksmith speaks the GitHub Actions runner protocol, so the only change per job is the runs-on label. No step, action, or cache config changes.
The cost model — two free buckets
The affordable play is not "everything on Blacksmith." It is using both free allotments, because we already own one of them:
| Bucket | Free / month | Covers | Nature |
|---|---|---|---|
| GitHub Team (this org, private repos) | 3,000 min | standard 2-core only — larger runners bill from minute 1 | already bundled in the per-seat plan — a sunk cost we forfeit if unused (org-shared, resets monthly) |
| Blacksmith | 3,000 min | any size incl. 4 vCPU, no credit card | a second, independent free tier |
That's ~6,000 free CI minutes/month combined. Moving everything to Blacksmith would waste the GitHub 3,000 we pay for regardless — so the cost-optimal design is a hybrid, and GitHub's own rule decides the split for us: its free minutes only cover 2-core standard runners, so heavy/parallel work can't be both free and fast there (2-core is slow and burns 2× the minutes; 4-core isn't free at all).
Routing rule:
- Light, high-frequency 2-core jobs → GitHub
ubuntu-latest— soak up the sunk free bucket; they're 2-core-native, so no speed penalty. - Heavy / parallel jobs → Blacksmith — 2× faster where it matters, drawing Blacksmith's own free bucket.
- Overflow past both free tiers → Blacksmith paid, never GitHub paid.
Marginal cost of one more CI minute, cheapest → most expensive:
GitHub free (2-core)
$0→ Blacksmith free$0→ Blacksmith paid$0.004→ GitHub 2-core$0.006→ GitHub 4-core$0.012(never route here)
The eval stays $0 until real usage crosses both free tiers; then it is pay-as-you-go on Blacksmith (Ubuntu x64 $0.004/min, ARM $0.0025/min, no seats).
One number to confirm: the GitHub 3,000 is org-shared across all private repos and resets monthly, so some is already spent elsewhere. An org admin can read the real remaining balance in Settings → Billing → Actions; that sets exactly where the light/heavy split line should fall.
What changed (two PRs)
This landed in two steps:
- PR #9936 (merged) — moved the whole CI test/build/lint tier onto Blacksmith and proved it green, with the head-to-head benchmark + measured numbers below.
- Follow-up PR (this one) — applies the two-bucket routing: the light, high-frequency jobs move back to GitHub
ubuntu-latestto consume the sunk free bucket, and the visual-regression renderer (the one heavy CI job missed the first time) moves onto Blacksmith.
Deploy, promote, release, preview, and prod-touching workflows (Duplo / AWS-OIDC) stay on ubuntu-latest throughout — not minute-burners, and we don't change the runner under a prod-facing job during an eval.
ubuntu-latest resolves to Ubuntu 24.04 / 2 vCPU, so Blacksmith mappings target ubuntu-2404 for parity. End-state routing:
Heavy / parallel → Blacksmith (fast + its own free bucket):
| Workflow · job | Tier |
|---|---|
e2e.yml · e2e matrix | blacksmith-4vcpu-ubuntu-2404 |
frontend-ci.yml · build | blacksmith-4vcpu-ubuntu-2404 |
frontend-ci.yml · test (vitest) | blacksmith-4vcpu-ubuntu-2404-arm |
python-test.yml · pytest | blacksmith-4vcpu-ubuntu-2404 |
docker-build-check.yml · build-flask, build-webserver | blacksmith-4vcpu-ubuntu-2404 |
visual-pr.yml · regen (new here — x64 to preserve pixel goldens) | blacksmith-4vcpu-ubuntu-2404 |
Light / high-frequency → GitHub ubuntu-latest (sunk free 2-core bucket): the detect/gate jobs across e2e, frontend-ci, onramp-agents-ci, crm-broker-ci; python-lint; migration-lint; infra-ci; pr-validation; uv-lock-check; and the visual decide / base-freshness / approval status gates.
Supporting files:
.github/actionlint.yaml— registers the Blacksmith labels soactionlintstops flagging them as unknown; doubles as the in-repo list of allowed tiers..github/workflows/blacksmith-benchmark.yml— the head-to-head showcase (below); still exercises the 2 vCPU tier so the "faster silicon" delta stays measurable.
How to see the performance benefit
Three independent signals, in order of effort:
This PR's own checks.
frontend-ciande2erun onpull_request, so the moment this branch pushes, those checks execute on Blacksmith. Compare their durations against the same checks on any recentubuntu-latestPR.The benchmark workflow (
⏱️ Blacksmith Benchmark, manualworkflow_dispatch). It runs the same CPU-bound job — the frontend production build — on three tiers side by side and prints a table to the run summary:- GitHub
ubuntu-latest(2 vCPU) — today's baseline - Blacksmith 2 vCPU — isolates the "faster silicon" win
- Blacksmith 4 vCPU — the tier we're migrating heavy jobs to
Only the build step is timed; dependency install happens in an untimed setup step, so cache state does not skew the numbers. It's on
mainnow, so it's dispatchable from the Actions tab (Run workflow); signal (1) is the always-on sample on every PR.- GitHub
Blacksmith's dashboard at app.blacksmith.sh — per-workflow time and cost savings vs GitHub-hosted, once runs accumulate.
First measured run (this PR)
Numbers from this PR's own checks on Blacksmith vs the same workflows' recent ubuntu-latest runs on main. Directional, not a controlled benchmark (different HEAD and cache state) — that's what the benchmark workflow is for — but the shape is already clear on both time and cost:
| Workload | Wall time | Billed job-min | Cost / run¹ | Savings |
|---|---|---|---|---|
pytest — full API suite (1 job, 4 vCPU) | 12m00s → 5m51s | 12.0 → 5.9 | $0.072 → $0.023 | ~2.1× faster · ~68% cheaper |
e2e — all Playwright suites (matrix: wall = slowest suite, cost = Σ jobs) | 5m27s → 4m47s | 27.1 → 20.8 | $0.163 → $0.083 | ~10% faster wall · ~49% cheaper |
¹ Rates (published 2026 pricing): GitHub Linux 2-core ubuntu-latest = $0.006/min; Blacksmith Ubuntu x64 = $0.004/min (flat per-arch — 4 vCPU is billed at the same per-minute as 2 vCPU). Cost = summed job-minutes × rate. GitHub also rounds each job up to the whole minute (not applied above), so its real cost — and the savings — is if anything higher, especially across the many short e2e matrix jobs. And the eval itself is $0: the first 3,000 min/month are free.
Two things the table shows. Cost tracks CPU-boundedness. pytest is compute-bound, so faster cores nearly halve both time and minutes; e2e is dominated by DB seed + app boot + Docker pulls (IO-bound), so wall-clock barely moves — but summed billed minutes still drop ~23% because each suite finishes faster. That split is exactly why heavy jobs got 4 vCPU and gate/lint jobs stayed at 2. Cheaper on both fronts: fewer minutes and a lower per-minute rate compound — the ~49% e2e saving is mostly the rate, the ~68% pytest saving is rate × speed.
Extrapolating the spend
The measured per-run savings scale linearly with volume. A PR that touches both backend and frontend triggers pytest and e2e, saving ~$0.05 + ~$0.08 ≈ $0.13/PR on those two alone (before the smaller lint/gate jobs and before GitHub's per-job round-up, which widens it). Across a month of PRs plus every push to main and each hotfix, the migrated CI tier lands at roughly half its current runner spend — the compute-bound jobs closer to a third — while every job also returns faster. Blacksmith's dashboard reports the actual figure once runs accumulate; the benchmark workflow isolates the hardware delta on demand.
The first 3,000 minutes each month are free, so none of this bills until real usage crosses that line — which is the whole point of running the eval before committing spend.
Risks & rollback
- Reversible in one revert. Every change is a
runs-onlabel; reverting the PR (or unsetting labels) puts CI back on GitHub-hosted runners with no other cleanup. - Blast radius is contained to the CI tier — no deploy/promote/prod job changes runner.
- Dependency on the Blacksmith GitHub App. Jobs pinned to a
blacksmith-*label queue until the App is installed and provisioning the repo. The App is installed, so the migrated jobs run immediately. - ARM parity. The one ARM job (
frontendvitest — jsdom, no pixels) stays ARM; no emulation is introduced. - Visual goldens: one-time regen. The visual renderer (
visual-prregen) moving GitHub→Blacksmith is kept on x64 to minimise pixel drift, but a different renderer can still shift sub-pixel antialiasing. The suite is built for this — the first render-affecting PR after merge regenerates goldens on Blacksmith, commits them as a native image diff, and a human ticks the existingvisual/approvedgate once. No baseline reset needed; the base-freshness and approval gates don't render, so they stay on GitHub untouched.
The decision for the VP
The engineering side is done and free to validate. The only call that needs a budget owner:
Once real CI usage crosses the 3,000 free minutes/month, do we stay on Blacksmith pay-as-you-go (≈$0.004/min Ubuntu x64, no seats) or fall back to GitHub-hosted?
Merge this PR to start the free evaluation; review the benchmark + dashboard numbers after a week of real CI; decide on paid usage from measured savings.