Deploy Roles & Environments

How OnRamp's CI deploy roles work, and exactly what to do to create a new environment, stand up its state backend, or change what a deploy role can do — without ever handing a contributor admin or running an infra apply locally.

Owner: Platform / InfraScope: CI deploy roles (ABAC)Account: single Duplo account, tenants-by-tag

Canonical sources. This runbook is the operator HOWTO. The executable truth lives in code — when a detail here and the code disagree, the code wins:

• infra/ci-identity/boundary-policy.json — the shared permissions boundary.
• infra/ci-identity/identity-policy.ts — buildIdentityPolicy (the per-env grant).
• infra/ci-identity/envs.ts — the tier registry.
• infra/ci-identity/index.ts — how each tier is instantiated.
• infra/scripts/bootstrap-deploy-roles.ts — the one admin step.
• infra/ci-identity/deploy-role-policy.test.ts — the ABAC contract tests (the spec).
• infra/ci-identity/DESIGN-deploy-role.md — design rationale + the §6 per-service ABAC-vs-ARN/region/Deny table.

TL;DR — the mental model

🅰 Effective = identity ∩ boundary

Two layers, one ceiling.

A role can do something only if both its per-env identity policy and the one shared boundary allow it, and neither denies it. The boundary is the cap; the identity is the grant.

🅱 The tier tag is the scope

environment=<tier> PrincipalTag.

Every role is tagged with a constant tier (previews, prod, gdpr-prod, stage, demo). ABAC self-scopes mutations to same-tier resources. This is not the app's per-PR ENVIRONMENT_NAME (preview-9412).

🅲 State is isolated per tier

Own bucket + KMS key.

Each tier deploys with its own Pulumi state bucket + secrets key. A role gets a plain resource-scoped grant on its own — real blast-radius isolation, no cross-env state reads.

No admin for contributors, ever. Admin is used exactly once per environment, by an operator, to run one bootstrap script. After that, CI assumes the least-privilege role via GitHub OIDC — no stored AWS keys, no local infra applies.

How the pieces fit

🧱 Shared, one per account

The boundary (onramp-deploy-boundary) is a single customer-managed policy, env- and region-agnostic. It self-scopes via aws:ResourceTag/environment == ${aws:PrincipalTag/environment}, enforces tag-on-create, and carries the anti-escalation rules (see Security invariants). You almost never edit it.

🏭 Per-env, from a factory

createDeployRole({ env, region, subjects, … }) mints one role onramp-deploy-<tier> — bounded, environment=<tier> tagged — and attaches the identity policy buildIdentityPolicy(...) returns. "Another environment" = one more registry entry + one factory call.

🪣 Per-env state plane

onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>. Created by the bootstrap script in the tier's own region. The role gets a plain ARN grant on its own bucket/key only.

🔑 One admin bootstrap

bun run bootstrap.roles ensures the seed bucket, optionally provisions the per-env state plane, imports the GitHub OIDC provider, and runs pulumi up on the identity-tier stack. The only place admin is needed.

The chicken-and-egg (and how it's solved)

Pulumi needs a state backend before it can create anything — including the buckets and roles that are the deploy-role system. The resolution: one bucket is created imperatively; everything else is Pulumi-managed from it.

flowchart TD
  OP["Operator (admin) · bun run bootstrap.roles"] --> SEED["1. Seed state bucket + KMS<br/>onramp-pulumi-state-ACCT (us-west-2)"]
  OP --> PENV["2. Per-env state plane<br/>onramp-pulumi-state-TIER-ACCT + KMS"]
  OP --> OIDC["3. GitHub OIDC provider (imported, never recreated)"]
  SEED --> STACK["4. pulumi up · identity-tier stack<br/>(its OWN state lives in the seed bucket)"]
  OP --> STACK
  STACK --> BND["Boundary policy<br/>onramp-deploy-boundary"]
  STACK --> ROLE["Deploy role<br/>onramp-deploy-TIER (bounded, environment=TIER)"]
  OIDC --> ROLE
  ROLE -->|CI assumes via OIDC| DEPLOY["Env deploys<br/>Lambda · AgentCore · Cognito · KVS"]
  PENV -->|state backend for| DEPLOY
  style STACK fill:#e6f3ff,color:#000

The key move: the identity-tier stack stores its own state in the seed bucket, not in a per-env bucket. So the seed bucket is the single imperative prerequisite; the boundary, the roles, and the per-env state buckets are all Pulumi resources created by that stack. Per-env buckets hold the tier's deploy state (Lambda, AgentCore, …), never the identity-tier stack's own state.

What stops source from outrunning the apply

Because this stack converges only by a manual pulumi up, merging a trust-policy change does not apply one. A PR whose source is internally consistent — workflow binding and declared subject changed together — still breaks every deploy through that role until an operator runs the apply, and nothing about the merge says so.

Every change under infra/ci-identity/** requires a manual apply after the code is final and before merge. Run it from the exact worktree and branch whose declarations should be live. The stack covers every tier in the shared AWS account, and Pulumi deletes managed resources that are absent from the branch being applied.

Four checks make the divergence unmergeable instead. Each is a link between two things that must agree; none of them can be satisfied by editing only one side.

#	Link	Where it runs	Needs
1	a job's environment: binding == its row in .github/scripts/oidc-env-bindings.ts (verbatim), and every binding has a row	bun test .github/scripts/ — Test CI Scripts, every PR	—
2	that registry's required subjects ⊆ what envs.ts / migrate-starter-role.ts declare	bun test in infra/ — Infra CI	—
3	every declared OIDC subject is live, each admitting statement pins aud + the expected provider, and no statement admits callers it cannot scope	Infra CI → Declared OIDC trust is applied	onramp-ci-oidc-verify
4	vars.DUPLO_TENANT_NAME is non-empty at the point of use	.github/actions/duplo-aws-jit, in-job	—

Link 3 is the one that catches an unapplied stack, and it is red until you apply: a PR that adds a subject stays failing until an operator runs the apply below, then goes green on a re-run of the job with no new commit. That inversion is the mechanism — it turns the apply into a merge prerequisite rather than a step in this runbook that someone has to remember. It fails on divergence in either direction, so removing a subject from source also requires the apply before merge: otherwise the repo claims a trust relationship is gone while AWS still honours it.

It reports three distinguishable causes, and the fix differs:

• declared but not applied (or the role is absent) → run the apply.
• applied but no longer declared → run the apply; until you do, a trust relationship the repo says is gone is live.
• admits callers the check cannot scope to a subject → do not just apply. An Allow sts:AssumeRoleWithWebIdentity statement with no sub condition admits every workflow in every repository, and one federated to another IdP or with a different aud is outside what these roles are supposed to trust. Any of those means a live over-broad trust policy, which is a person's problem before it is Pulumi's. Note what this rules out: a subject-list comparison alone would score such a policy as agreement, because an unconditioned statement contributes no subject to compare.

What link 3 does not compare: the rest of the trust document — other condition keys, tags, session policies. A change confined to those and merged without an apply is not caught.

Link 4 covers what an IAM comparison structurally cannot. The agent-deploy jobs assume no AWS role — they take Duplo JIT credentials keyed on a GitHub Environment variable — so a binding to an Environment nobody has provisioned reads as an empty tenant, not an AccessDenied. It is also the closest in-job signal that an Environment was auto-created blank rather than provisioned, which matters because a blank one carries no deployment branch/tag allow-list either — the exact shape a brand-new tier is in before step 1 of "Add a new environment" below has run.

Known gap, deliberate. Nothing here verifies the GitHub side directly — that a tier's Environment exists with the allow-list ALLOWED_REFS_BY_ENV declares, or that its variables are set. Reading either back needs a credential GITHUB_TOKEN does not carry (administration is not one of the permissions: keys, and variables need a fine-grained token), so closing it means adding a GitHub App or PAT to store and rotate. What stands in for it: link 1 refuses a binding to an env with no ALLOWED_REFS_BY_ENV row (so nothing can bind an env that nothing provisions), and link 4 trips on the blank-Environment signature. Neither proves the allow-list is present on a prod-tier Environment — running bootstrap-agent-environments.ts before a new tier's binding ships is still a real security prereq, not just an availability one.

Add a new environment

Adding an environment is a one-line registry edit plus an operator standing up its state plane. Convention derives everything else from the tier name — the state bucket, the KMS key, the role ARN, the CI backend URL.

TL;DR. A new env X is the one line { env: "X" } in envs.ts — presence in the registry means live (there is no enabled flag; delete the line to retire a tier). Order matters: an operator bootstraps X's state plane first, then the one-line entry lands, then pulumi up creates the role. (gdpr-prod adds region: "eu-west-1" + --region eu-west-1 — the one tier that deviates.)

1

Bootstrap the prereq

admin · state plane

2

Add the tier

envs.ts · one line

3

Apply + verify

pulumi up

4

Wire CI

workflow OIDC

1 · Bootstrap the prereq (admin)

A tier's KMS alias must exist before any pulumi up resolves it, so the state plane is the prerequisite — provision it first, with no role yet:

AWS_PROFILE=onramp-admin bun run bootstrap.roles --env-tag demo --no-pulumi-up

The bucket name + KMS alias are derived from --env-tag (the same convention index.ts uses) — nothing to paste. This provisions the per-env bucket + KMS key (tagged environment=demo), ensures the seed bucket + the GitHub OIDC provider, and stops before pulumi up (the role is created in step 3, once the entry exists).

gdpr-prod adds --region eu-west-1 — the one tier that deviates: … --env-tag gdpr-prod --region eu-west-1 --no-pulumi-up. KMS keys are regional and EU data residency requires the EU tier's state + secrets in-region; the script fails closed if you omit it. The seed bucket + identity-tier stack stay in us-west-2 regardless.

2 · Add the tier to envs.ts (one line)

Now that the state plane exists, add the entry — presence == live (no enabled flag):

{ env: "demo" },

resolveTier + index.ts derive the rest by convention: region us-west-2, main-branch-only OIDC, onramp_demo_ prefix, bucket onramp-pulumi-state-demo-111776553558, key alias/onramp-pulumi-secrets-demo, role onramp-deploy-demo. Override a field only when the tier deviates — still one line; gdpr-prod is the heaviest:

{ env: "gdpr-prod", region: "eu-west-1", agentRuntimePrefix: "onramp_GDPR_PROD_" },

Deviations that exist today: region (EU), subjects (previews also accepts PR-triggered OIDC), agentRuntimePrefix (where the real runtime name isn't onramp_<env>_ — owned by the agent deploy layer, normalizing under the AgentCore→Pulumi migration), and the seed state-plane overrides (stateBucketName / stateKmsAlias / legacyStateBucketName, previews only). Don't land the entry before step 1 — every entry is resolved on every pulumi up, and the KMS-alias lookup hard-fails if the alias doesn't exist yet.

3 · Apply, then verify the role is bounded

pulumi up on the identity-tier stack creates onramp-deploy-demo (re-running bun run bootstrap.roles --env-tag demo does the same — it ups after provisioning):

cd infra/ci-identity
export PULUMI_BACKEND_URL=s3://onramp-pulumi-state-111776553558
pulumi up
pulumi stack output deployRoleArns          # map of live tiers → role ARN
# { "demo": "arn:aws:iam::111776553558:role/onramp-deploy-demo" }

Confirm (console or aws iam get-role) that the role carries the environment=demo tag and PermissionsBoundary = onramp-deploy-boundary. If the boundary is missing, stop and fix before wiring CI — the role could mint unbounded roles.

If the tier sets composesPipeline, this apply is a prerequisite for its first app-stack deploy, not a parallel task. The same pulumi up converges lambdaImagePullPolicy — the flask repository policy that authorizes the durable executor's container-image pull. Nothing else can: a grant on the deploy role or on the function's execution role does not satisfy Lambda's check. Apply this stack before the tier's first deploy-infra, or that apply fails at CreateFunction with AccessDeniedException: Lambda does not have permission to access the ECR image — and because the integrations pipeline is a Tier-1 member with no warn-and-skip on a failed create, it takes the whole env stack apply red with it.

A tier homed outside us-west-2 needs its own in-region statement on that region's flask repository; resolveImagePullRegion fails the apply rather than let such a tier look covered.

4 · Point the environment's CI workflow at the role

permissions:
  id-token: write          # required for OIDC
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111776553558:role/onramp-deploy-demo
      aws-region: us-west-2          # the tier's region
  # Pulumi state for this tier lives in its own bucket:
  # PULUMI_BACKEND_URL=s3://onramp-pulumi-state-demo-111776553558

Every value here is conventional — the role ARN (onramp-deploy-<tier>), the region, and the backend URL (onramp-pulumi-state-<tier>-<account>) all follow from the tier name, so there's nothing to look up.

The role only trusts the OIDC sub claims listed in the tier's subjects (envs.ts) — fork PRs can't mint id-token tokens, so they can never assume it. For stacks that mint Lambda exec roles, set the deployRoleBoundaryArn Pulumi config so the minted role inherits the boundary (the boundary denies CreateRole otherwise).

The infra/migrate/agent jobs and the app rollout all bind the SAME <tier> Environment (see the "Environments" section of docs/architecture/release-and-deploy.md for why sharing it is safe, and docs/runbooks/jira-release-reporting.md for the credential that makes it safe plus the one-time cutover onto it). Standing up a NEW tier needs these applied in order, or the workflow's OIDC token gets AccessDenied on step 3's role:

1. bun run onramp-agents/scripts/bootstrap-agent-environments.ts --envs=<tier> — creates <tier> with the tier's deployment branch/tag allow-list and DUPLO_TENANT_NAME. GitHub auto-creates it blank on first use otherwise, with no allow-list (any ref could deploy).
2. bun run bootstrap.roles --env-tag <tier> (step 1 above) — applies the identity-tier stack, which is what actually adds environment:<tier> to subjects in envs.ts / migrateStarterSubjects in migrate-starter-role.ts. ci-identity does NOT converge from any workflow, so this is a manual pulumi up under admin credentials every time either function's subjects change — including this one.
3. Only then does the workflow's environment: <tier> binding (step 4 below) resolve to a trust policy that actually admits it.

You do not have to trust yourself to remember step 2: Infra CI's Declared OIDC trust is applied job compares every declared subject against the live trust policy and fails the PR while step 2 is outstanding (see "What stops source from outrunning the apply"). Step 1 is the one with no automated check — see the known gap in that section.

Skipping step 1 is still a real security hole, not just an availability one: check-deploy-ref.ts (.github/scripts/env-ref-policy.ts) IS the first step of every job that binds <tier>, and it does refuse a ref outside the declared allow-list independent of whether step 1 has run — but it runs FROM the checkout of the ref it's judging, so it only catches an accidental bad ref (wrong branch dispatched), not a deliberate one on a branch that also edits env-ref-policy.ts or the guard itself. Step 1's Environment-level allow-list is enforced by GitHub and can't be edited from the ref being judged — it's the control that actually holds against a deliberate attempt, so don't treat the in-job guard as covering for skipping it.

Change what a deploy role can do

The first question is always which layer — one tier, or all of them?

Grant or tighten a single tier's permissions → edit buildIdentityPolicy in infra/ci-identity/identity-policy.ts. This is the common case (e.g. previews needs a new S3 prefix, AgentCore action, or SSM path).

1. Add/adjust the statement. Keep it within the boundary ceiling — a grant the boundary doesn't also allow is dead (intersection = nothing).
2. Add a contract to deploy-role-policy.test.ts asserting the new allow and the matching cross-env deny.
3. cd infra && bun test → green, then pulumi up the identity-tier stack.

No boundary edit, no new ticket churn — this is the path you want 90% of the time.

Raise the ceiling for every tier → edit infra/ci-identity/boundary-policy.json. Rare and higher-stakes: it's the shared cap, so a mistake widens all roles.

1. Make the change broad enough that adding normal infra never needs another boundary edit (the whole point — no per-action whack-a-mole), but never weaken a Security invariant below.
2. pulumi up creates a new managed-policy version automatically — roles pick it up live; no re-attach.
3. Tests are the spec: update/extend the contracts in deploy-role-policy.test.ts.

Adding a new AWS service the roles deploy? Prefer ABAC (tag-scoped) where the service honors aws:ResourceTag/aws:RequestTag; fall back to ARN + region + explicit Deny where it doesn't (the bedrock-agentcore pattern). A tag condition on a key the service ignores is a silent hole — check the per-service table in DESIGN-deploy-role.md §6 before adding one.

State buckets & KMS

Naming

• Seed (shared, us-west-2): onramp-pulumi-state-<account> + alias/onramp-pulumi-secrets
• Per-tier: onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>

Region

Per-tier plane is created in the tier's region (--region). gdpr-prod ⇒ eu-west-1 (guard-enforced). Seed + identity-tier stack stay us-west-2.

Isolation

Dedicated bucket ⇒ the name is the isolation (role gets stacks/*). Shared seed bucket ⇒ the grant narrows to the tier's stacks/<prefix>/* so a role can't read co-located stacks (incl. the identity-tier stack that defines the roles).

Secrets key

The per-tier KMS key is the Pulumi secrets provider. It must carry environment=<tier> so the boundary's ABAC KMS allow admits it. The bootstrap script tags it for you.

Verify locally

The ABAC evaluator test is a deps-free IAM simulator over the realbuildIdentityPolicy ∩ the boundary — it's the executable spec for every claim in this runbook.

cd infra
bun install                      # standalone package
bunx tsc --noEmit -p tsconfig.json
bun test                         # cross-env, escalation, tag-integrity, region, cutover contracts

You can't pulumi up without admin + the seed backend, so local verification stops at tsc + tests — which is exactly why the contract suite is exhaustive.

Security invariants — never weaken these

These are the load-bearing properties. Any boundary/identity change must keep all of them; the contract tests enforce them and will fail CI if you regress one.

• CreateRole must carry the boundary — minted roles can't escape the cap.
• No iam:PutRolePolicy — no inline-policy escalation.
• AttachRolePolicy is allowlisted — only the named managed policies.
• PassRole is path- + service-scoped — no passing a privileged role.
• Tag-on-create enforced — a resource can't be created outside its tier tag, and the tag can't be stripped or re-tagged to another tier.
• No no-op conditions — only use a tag condition on a service that honors it.
• Per-env state isolation — a role reaches only its own state bucket/key.
• Boundary + OIDC provider are tamper-protected — roles can't edit the policy that bounds them, or the provider that authenticates them.

Troubleshooting

🚫 AccessDenied on a deploy

Decide the layer: if the boundary denies it, every tier is blocked (raise the ceiling — boundary). If only this tier, it's the identity grant. Reproduce it as a contract test first; fix the policy the test points at.

🏷 Create denied unexpectedly

Almost always tag-on-create: the resource is being made without environment=<tier> (or with the wrong tier). Ensure the deploy stamps the tag, or that deployRoleBoundaryArn is set so minted roles inherit the boundary.

🌍 gdpr-prod region errors

The bootstrap guard fires if --env-tag gdpr-prod isn't paired with --region eu-west-1. The boundary already spans both regions; per-tier region confinement is the identity's region arg, not the boundary.

🧪 Tests red after a policy edit

The contract suite is the spec, not an afterthought. A red test means the shipped policy changed behavior — fix the policy, not the assertion, unless the behavior change is intended (then update the contract in the same change).

Out of scope

Retiring deploy.py and migrating prod AgentCore (Knowledge Bases, OpenSearch Serverless, runtime roles) onto Pulumi is a separate effort. This deploy-role system is the safe-deploy enabler for it: once a prod role is stamped here, that migration deploys through it instead of with admin.