Deploy Roles & Environments

How OnRamp's CI deploy roles work, and exactly what to do to create a new environment, stand up its state backend, or change what a deploy role can do — without ever handing a contributor admin or running an infra apply locally.

Owner: Platform / InfraScope: CI deploy roles (ABAC)Account: single Duplo account, tenants-by-tag

Canonical sources. This runbook is the operator HOWTO. The executable truth lives in code — when a detail here and the code disagree, the code wins:

• infra/ci-identity/boundary-policy.json — the shared permissions boundary.
• infra/ci-identity/identity-policy.ts — buildIdentityPolicy (the per-env grant).
• infra/ci-identity/envs.ts — the tier registry.
• infra/ci-identity/index.ts — how each tier is instantiated.
• infra/scripts/bootstrap-deploy-roles.ts — the one admin step.
• infra/ci-identity/deploy-role-policy.test.ts — the ABAC contract tests (the spec).
• infra/ci-identity/DESIGN-deploy-role.md — design rationale + the §6 per-service ABAC-vs-ARN/region/Deny table.

TL;DR — the mental model

🅰 Effective = identity ∩ boundary

Two layers, one ceiling.

A role can do something only if both its per-env identity policy and the one shared boundary allow it, and neither denies it. The boundary is the cap; the identity is the grant.

🅱 The tier tag is the scope

environment=<tier> PrincipalTag.

Every role is tagged with a constant tier (previews, prod, gdpr-prod, stage, demo). ABAC self-scopes mutations to same-tier resources. This is not the app's per-PR ENVIRONMENT_NAME (preview-9412).

🅲 State is isolated per tier

Own bucket + KMS key.

Each tier deploys with its own Pulumi state bucket + secrets key. A role gets a plain resource-scoped grant on its own — real blast-radius isolation, no cross-env state reads.

No admin for contributors, ever. Admin is used exactly once per environment, by an operator, to run one bootstrap script. After that, CI assumes the least-privilege role via GitHub OIDC — no stored AWS keys, no local infra applies.

How the pieces fit

🧱 Shared, one per account

The boundary (onramp-deploy-boundary) is a single customer-managed policy, env- and region-agnostic. It self-scopes via aws:ResourceTag/environment == ${aws:PrincipalTag/environment}, enforces tag-on-create, and carries the anti-escalation rules (see Security invariants). You almost never edit it.

🏭 Per-env, from a factory

createDeployRole({ env, region, subjects, … }) mints one role onramp-deploy-<tier> — bounded, environment=<tier> tagged — and attaches the identity policy buildIdentityPolicy(...) returns. "Another environment" = one more registry entry + one factory call.

🪣 Per-env state plane

onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>. Created by the bootstrap script in the tier's own region. The role gets a plain ARN grant on its own bucket/key only.

🔑 One admin bootstrap

bun run bootstrap.roles ensures the seed bucket, optionally provisions the per-env state plane, imports the GitHub OIDC provider, and runs pulumi up on the identity-tier stack. The only place admin is needed.

The chicken-and-egg (and how it's solved)

Pulumi needs a state backend before it can create anything — including the buckets and roles that are the deploy-role system. The resolution: one bucket is created imperatively; everything else is Pulumi-managed from it.

flowchart TD
  OP["Operator (admin) · bun run bootstrap.roles"] --> SEED["1. Seed state bucket + KMS<br/>onramp-pulumi-state-ACCT (us-west-2)"]
  OP --> PENV["2. Per-env state plane<br/>onramp-pulumi-state-TIER-ACCT + KMS"]
  OP --> OIDC["3. GitHub OIDC provider (imported, never recreated)"]
  SEED --> STACK["4. pulumi up · identity-tier stack<br/>(its OWN state lives in the seed bucket)"]
  OP --> STACK
  STACK --> BND["Boundary policy<br/>onramp-deploy-boundary"]
  STACK --> ROLE["Deploy role<br/>onramp-deploy-TIER (bounded, environment=TIER)"]
  OIDC --> ROLE
  ROLE -->|CI assumes via OIDC| DEPLOY["Env deploys<br/>Lambda · AgentCore · Cognito · KVS"]
  PENV -->|state backend for| DEPLOY
  style STACK fill:#e6f3ff,color:#000

The key move: the identity-tier stack stores its own state in the seed bucket, not in a per-env bucket. So the seed bucket is the single imperative prerequisite; the boundary, the roles, and the per-env state buckets are all Pulumi resources created by that stack. Per-env buckets hold the tier's deploy state (Lambda, AgentCore, …), never the identity-tier stack's own state.

Add a new environment

Adding an environment is a one-line registry edit plus an operator standing up its state plane. Convention derives everything else from the tier name — the state bucket, the KMS key, the role ARN, the CI backend URL.

TL;DR. A new env X is the one line { env: "X" } in envs.ts — presence in the registry means live (there is no enabled flag; delete the line to retire a tier). Order matters: an operator bootstraps X's state plane first, then the one-line entry lands, then pulumi up creates the role. (gdpr-prod adds region: "eu-west-1" + --region eu-west-1 — the one tier that deviates.)

1

Bootstrap the prereq

admin · state plane

2

Add the tier

envs.ts · one line

3

Apply + verify

pulumi up

4

Wire CI

workflow OIDC

1 · Bootstrap the prereq (admin)

A tier's KMS alias must exist before any pulumi up resolves it, so the state plane is the prerequisite — provision it first, with no role yet:

AWS_PROFILE=onramp-admin bun run bootstrap.roles --env-tag demo --no-pulumi-up

The bucket name + KMS alias are derived from --env-tag (the same convention index.ts uses) — nothing to paste. This provisions the per-env bucket + KMS key (tagged environment=demo), ensures the seed bucket + the GitHub OIDC provider, and stops before pulumi up (the role is created in step 3, once the entry exists).

gdpr-prod adds --region eu-west-1 — the one tier that deviates: … --env-tag gdpr-prod --region eu-west-1 --no-pulumi-up. KMS keys are regional and EU data residency requires the EU tier's state + secrets in-region; the script fails closed if you omit it. The seed bucket + identity-tier stack stay in us-west-2 regardless.

2 · Add the tier to envs.ts (one line)

Now that the state plane exists, add the entry — presence == live (no enabled flag):

{ env: "demo" },

resolveTier + index.ts derive the rest by convention: region us-west-2, main-branch-only OIDC, onramp_demo_ prefix, bucket onramp-pulumi-state-demo-111776553558, key alias/onramp-pulumi-secrets-demo, role onramp-deploy-demo. Override a field only when the tier deviates — still one line; gdpr-prod is the heaviest:

{ env: "gdpr-prod", region: "eu-west-1", agentRuntimePrefix: "onramp_GDPR_PROD_" },

Deviations that exist today: region (EU), subjects (previews also accepts PR-triggered OIDC), agentRuntimePrefix (where the real runtime name isn't onramp_<env>_ — owned by the agent deploy layer, normalizing under the AgentCore→Pulumi migration), and the seed state-plane overrides (stateBucketName / stateKmsAlias / legacyStateBucketName, previews only). Don't land the entry before step 1 — every entry is resolved on every pulumi up, and the KMS-alias lookup hard-fails if the alias doesn't exist yet.

3 · Apply, then verify the role is bounded

pulumi up on the identity-tier stack creates onramp-deploy-demo (re-running bun run bootstrap.roles --env-tag demo does the same — it ups after provisioning):

cd infra/ci-identity
export PULUMI_BACKEND_URL=s3://onramp-pulumi-state-111776553558
pulumi up
pulumi stack output deployRoleArns          # map of live tiers → role ARN
# { "demo": "arn:aws:iam::111776553558:role/onramp-deploy-demo" }

Confirm (console or aws iam get-role) that the role carries the environment=demo tag and PermissionsBoundary = onramp-deploy-boundary. If the boundary is missing, stop and fix before wiring CI — the role could mint unbounded roles.

4 · Point the environment's CI workflow at the role

permissions:
  id-token: write          # required for OIDC
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111776553558:role/onramp-deploy-demo
      aws-region: us-west-2          # the tier's region
  # Pulumi state for this tier lives in its own bucket:
  # PULUMI_BACKEND_URL=s3://onramp-pulumi-state-demo-111776553558

Every value here is conventional — the role ARN (onramp-deploy-<tier>), the region, and the backend URL (onramp-pulumi-state-<tier>-<account>) all follow from the tier name, so there's nothing to look up.

The role only trusts the OIDC sub claims listed in the tier's subjects (envs.ts) — fork PRs can't mint id-token tokens, so they can never assume it. For stacks that mint Lambda exec roles, set the deployRoleBoundaryArn Pulumi config so the minted role inherits the boundary (the boundary denies CreateRole otherwise).

Change what a deploy role can do

The first question is always which layer — one tier, or all of them?

Grant or tighten a single tier's permissions → edit buildIdentityPolicy in infra/ci-identity/identity-policy.ts. This is the common case (e.g. previews needs a new S3 prefix, AgentCore action, or SSM path).

1. Add/adjust the statement. Keep it within the boundary ceiling — a grant the boundary doesn't also allow is dead (intersection = nothing).
2. Add a contract to deploy-role-policy.test.ts asserting the new allow and the matching cross-env deny.
3. cd infra && bun test → green, then pulumi up the identity-tier stack.

No boundary edit, no new ticket churn — this is the path you want 90% of the time.

Raise the ceiling for every tier → edit infra/ci-identity/boundary-policy.json. Rare and higher-stakes: it's the shared cap, so a mistake widens all roles.

1. Make the change broad enough that adding normal infra never needs another boundary edit (the whole point — no per-action whack-a-mole), but never weaken a Security invariant below.
2. pulumi up creates a new managed-policy version automatically — roles pick it up live; no re-attach.
3. Tests are the spec: update/extend the contracts in deploy-role-policy.test.ts.

Adding a new AWS service the roles deploy? Prefer ABAC (tag-scoped) where the service honors aws:ResourceTag/aws:RequestTag; fall back to ARN + region + explicit Deny where it doesn't (the bedrock-agentcore pattern). A tag condition on a key the service ignores is a silent hole — check the per-service table in DESIGN-deploy-role.md §6 before adding one.

State buckets & KMS

Naming

• Seed (shared, us-west-2): onramp-pulumi-state-<account> + alias/onramp-pulumi-secrets
• Per-tier: onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>

Region

Per-tier plane is created in the tier's region (--region). gdpr-prod ⇒ eu-west-1 (guard-enforced). Seed + identity-tier stack stay us-west-2.

Isolation

Dedicated bucket ⇒ the name is the isolation (role gets stacks/*). Shared seed bucket ⇒ the grant narrows to the tier's stacks/<prefix>/* so a role can't read co-located stacks (incl. the identity-tier stack that defines the roles).

Secrets key

The per-tier KMS key is the Pulumi secrets provider. It must carry environment=<tier> so the boundary's ABAC KMS allow admits it. The bootstrap script tags it for you.

Verify locally

The ABAC evaluator test is a deps-free IAM simulator over the realbuildIdentityPolicy ∩ the boundary — it's the executable spec for every claim in this runbook.

cd infra
bun install                      # standalone package
bunx tsc --noEmit -p tsconfig.json
bun test                         # cross-env, escalation, tag-integrity, region, cutover contracts

You can't pulumi up without admin + the seed backend, so local verification stops at tsc + tests — which is exactly why the contract suite is exhaustive.

Security invariants — never weaken these

These are the load-bearing properties. Any boundary/identity change must keep all of them; the contract tests enforce them and will fail CI if you regress one.

• CreateRole must carry the boundary — minted roles can't escape the cap.
• No iam:PutRolePolicy — no inline-policy escalation.
• AttachRolePolicy is allowlisted — only the named managed policies.
• PassRole is path- + service-scoped — no passing a privileged role.
• Tag-on-create enforced — a resource can't be created outside its tier tag, and the tag can't be stripped or re-tagged to another tier.
• No no-op conditions — only use a tag condition on a service that honors it.
• Per-env state isolation — a role reaches only its own state bucket/key.
• Boundary + OIDC provider are tamper-protected — roles can't edit the policy that bounds them, or the provider that authenticates them.

Troubleshooting

🚫 AccessDenied on a deploy

Decide the layer: if the boundary denies it, every tier is blocked (raise the ceiling — boundary). If only this tier, it's the identity grant. Reproduce it as a contract test first; fix the policy the test points at.

🏷 Create denied unexpectedly

Almost always tag-on-create: the resource is being made without environment=<tier> (or with the wrong tier). Ensure the deploy stamps the tag, or that deployRoleBoundaryArn is set so minted roles inherit the boundary.

🌍 gdpr-prod region errors

The bootstrap guard fires if --env-tag gdpr-prod isn't paired with --region eu-west-1. The boundary already spans both regions; per-tier region confinement is the identity's region arg, not the boundary.

🧪 Tests red after a policy edit

The contract suite is the spec, not an afterthought. A red test means the shipped policy changed behavior — fix the policy, not the assertion, unless the behavior change is intended (then update the contract in the same change).

Out of scope

Retiring deploy.py and migrating prod AgentCore (Knowledge Bases, OpenSearch Serverless, runtime roles) onto Pulumi is a separate effort. This deploy-role system is the safe-deploy enabler for it: once a prod role is stamped here, that migration deploys through it instead of with admin.