Blacksmith CI Runner Migration
TL;DR — CI runs on Blacksmith wherever a job either scales with cores or runs long enough for the cheaper per-minute rate to matter, and on GitHub-hosted ubuntu-latest for the entry/gate jobs where a 4s start-up beats a 14s one. Both vendors' free allotments are exhausted, so every minute discussed here is a paid minute and the routing rule below is a pure cost decision. Blacksmith prices linearly per vCPU — the single fact the original version of this document got wrong, and the one that decides every sizing call.
Rates
Read from the GitHub billing ledger (GET /organizations/{org}/settings/billing/usage, which returns per-day, per-SKU line items with real pricePerUnit values) and from Blacksmith's published pricing reconciled against the invoice:
| Runner | Rate / min |
|---|---|
GitHub Linux 2-core (ubuntu-latest) | $0.006 |
| GitHub ARM 2-core | $0.005 |
GitHub ARM 4-core (ubuntu-24.04-arm) | $0.008 |
| GitHub Linux 4-core | $0.012 |
| Blacksmith x64, per 2 vCPU | $0.004 |
| Blacksmith ARM, per 2 vCPU | $0.0025 |
Blacksmith is linear in vCPU count, not flat per architecture. So blacksmith-2vcpu-ubuntu-2404 is $0.004/min and blacksmith-4vcpu-ubuntu-2404 is $0.008/min — dearer per minute than GitHub's 2-core runner, not cheaper.
Two consequences that drive everything else:
- A 4 vCPU Blacksmith job must run at least 1.33× faster than the same job on GitHub's 2-core runner just to break even on cost. Below that it is a latency purchase, not a saving.
- Because the price is linear, a job that genuinely halves on twice the cores costs exactly the same and finishes in half the time. The full API suite is $0.040/run on either 2 or 4 vCPU. That makes 4 vCPU free for anything that parallelises — and a pure waste for anything that doesn't.
Both vendors round each job up to a whole minute. A ten-second job and a fifty-second job cost the same, which is what makes short gate jobs expensive out of proportion to their work.
The sizing rule
| Job shape | Tier |
|---|---|
| Scales with cores — test suites, frontend/docs builds, docker builds | blacksmith-4vcpu-ubuntu-2404 |
| Single-threaded, or IO/API-bound — lint, format checks, type checks, control-plane deploys | blacksmith-2vcpu-ubuntu-2404 |
| Finishes in seconds and only gates other jobs | delete it — fold the gate into the job it guards, or use a workflow-level paths: filter |
| A workflow's entry job, where one is unavoidable | GitHub ubuntu-latest |
| ARM | GitHub ubuntu-24.04-arm — never a Blacksmith ARM label |
The live inventory is the workflows themselves; .github/actionlint.yaml is the registry of labels a workflow is allowed to use. Add a row there before using a new size.
Notes on the non-obvious rows:
- Never route ARM to Blacksmith. Moving the vitest suite to
blacksmith-4vcpu-ubuntu-2404-armmade it 40% slower for a month until it was put back on x64. Blacksmith ARM is not a drop-in for GitHub ARM. - Entry jobs stay on GitHub. Blacksmith's start-up penalty is a stable +10s (14s vs 4s median, measured over five weeks and 420 paired job records). It is paid once per dependency level, not once per job — siblings provision in parallel — so a level of four peers moving together costs 10s between them, not 40s. But a workflow's first job has no siblings to share it with, and everything downstream waits on it.
- Putting a non-scaling job on 4 vCPU doubles its cost for nothing. Moving every job to 4 vCPU would raise the bill ~7%.
Gate jobs
A gate job that resolves a paths: filter and hands a boolean to the jobs it guards costs a full billed minute for ~10s of work. Where that minute buys the skipping of several expensive jobs, it pays for itself. Where it guards a single job, it does not — the filter belongs inside that job, gating its steps.
The constraint that decides which shape is legal is the branch ruleset's required status checks, which match on the bare context string repo-wide:
- A required context must report on every PR. A workflow-level
paths:filter means the workflow does not run at all on an untouched PR, so the context never reports and the PR sits stuck "Expected" — unmergeable, forever. - Therefore: a workflow containing a required context must always trigger, and its cost floor is one billed minute per PR. Folding the path filter into the required job hits that floor. A separate gate job doubles it.
- That floor counts MINUTES, not money, so check the tier before assuming a fold is free. Folding into a job on a dearer tier moves the no-op path UP: the full API suite sits on 4 vCPU, so a PR touching no backend path now spends
$0.008where a separateubuntu-latestgate spent$0.006. It still wins, because the fold saves a whole minute whenever the suite does run and the suite ran on 22 of 31 consecutive PR runs against a 36% break-even — but that is arithmetic to do per workflow, not a rule to assume. A fold onto the 2 vCPU tier needs no sums; it is cheaper on both paths. - A workflow containing no required context can use a workflow-level
paths:filter and spend nothing at all on an untouched PR. Anything in such a workflow that later becomes required has to move back behind a job-level gate first.
.github/scripts/lint-workflow-names.ts enforces the repo-wide uniqueness that makes those context strings addressable in the first place.
Free allotments
GitHub Team bundles 3,000 Actions minutes/month on standard 2-core runners. At $0.006/min that allotment is worth $18/month, and it only applies to ubuntu-latest. It is a rounding error against the paid volume, so it is never the reason to leave a high-volume job on GitHub-hosted runners.
It does mean a low-volume, path-gated workflow is not worth retiering: infra-ci.yml fires only on infra/**, so moving it would save cents, and the allotment covers it. That is the whole argument — it is a cost floor, not a strategy.
Nothing needs to be held back to keep the GitHub-hosted path exercised, either: frontend-ci's detect-changes and Required checks both run on every PR on ubuntu-latest because they are required contexts with no sibling to share a start-up with, and ai-review runs there because the swarm needs GitHub's gh build. Those are far better coverage than anything gated to one directory.
Blacksmith's own free tier is exhausted; all Blacksmith minutes bill.
Migration mechanics
Blacksmith speaks the GitHub Actions runner protocol, so moving a job is a one-line runs-on change with no step, action, or cache edits. actions/cache is transparently redirected to a co-located store; the first cutover showed no cold-cache penalty on day one.
Deploy, promote, release, and prod-touching workflows (deploy-release.yml, promote-*.yml, deploy-infra-*.yml, build-and-deploy.yml) stay on ubuntu-latest. They are not minute-burners and their runner is not worth changing under a prod-facing job.
OIDC is not a constraint anywhere: the token is minted by GitHub's token service, not by the runner, so id-token: write works unchanged on Blacksmith. Nothing on the AWS side (role trust policies, the inline identity policy, the permissions boundary, S3 and ECR resource policies) or the Neon side (allowed_ips is empty) allowlists runner egress IPs, so no pinned static IPs are needed — which matters, because they cost $100/IP/month and would swamp the saving they enabled.
Measured
The first cutover's own numbers, recomputed at the correct 4 vCPU rate of $0.008/min:
| Workload | Wall time | Billed job-min | Cost / run | Outcome |
|---|---|---|---|---|
| Full API suite (1 job, 4 vCPU) | 12m00s → 5m51s | 12.0 → 5.9 | $0.072 → $0.047 | 2.1× faster, ~35% cheaper |
| Playwright matrix (wall = slowest suite, cost = Σ jobs) | 5m27s → 4m47s | 27.1 → 20.8 | $0.163 → $0.166 | ~10% faster wall, cost-neutral |
The API suite is compute-bound, so faster cores nearly halve both wall time and minutes and it clears the 1.33× break-even comfortably. The Playwright matrix is dominated by DB seed, app boot and Docker pulls, so it does not — that move bought wall-clock at par, which is a defensible trade but not a saving. Reading those two rows at a flat $0.004/min, as the original version of this document did, overstated the first cutover's saving by roughly 2×.
Blacksmith 2 vCPU is 1.07× GitHub's 2-core in execution time (15 paired jobs, n=3 each) — i.e. no real speed gain. The 2 vCPU tier is a rate play only, and with the +10s start-up it is a latency loss on any job short enough for 10s to matter.
Verification
.github/workflows/blacksmith-benchmark.yml runs the same CPU-bound job on GitHub 2-core, Blacksmith 2 vCPU and Blacksmith 4 vCPU side by side and prints a comparison table to the run summary. Only the timed step is measured; dependency install happens in an untimed setup step so cache state does not skew it.
To confirm where a job actually landed:
gh api /repos/OnRampTech/main-web-application/actions/runs/<id>/jobs \
-q '.jobs[]|[.name,(.labels|join("+"))]|@tsv'The real proof of any routing change is the billing ledger: read the per-day Actions Linux minutes 3–5 days after merge and confirm they moved in proportion.
Rollback
Every routing decision is a runs-on label, so a revert restores GitHub-hosted execution with no other cleanup. The gate-job removals are structural rather than label changes and revert as ordinary code.
Blast radius is the CI tier plus PR previews: preview-deploy.yml's build, deploy and teardown jobs all run on Blacksmith. No promote, release or prod-touching workflow changes runner. Previews keep a GitHub-hosted cleanup backstop either way — preview-sweeper.yml stays on ubuntu-latest and reconciles Pulumi stacks, Neon branches, S3 prefixes and AgentCore runtimes — so a Blacksmith outage delays a teardown rather than orphaning one.
The one hard dependency is the Blacksmith GitHub App: a job pinned to a blacksmith-* label queues indefinitely until the App is installed and provisioning the repo.