Skip to content

Deploy Roles & Environments ​

How OnRamp's CI deploy roles work, and exactly what to do to create a new environment, stand up its state backend, or change a boundary guard — without ever handing a contributor admin or running an infra apply locally.

Owner: Platform / InfraScope: CI deploy roles and permissions boundariesAccounts: production + non-prod (previews)

Canonical sources. This runbook is the operator HOWTO. The executable truth lives in code — when a detail here and the code disagree, the code wins:

  • infra/ci-identity/boundary-policy.ts — buildBoundary, the production boundary template and its guards.
  • infra/ci-identity/nonprod-boundary-policy.ts — the non-prod account's boundary.
  • infra/ci-identity/identity-policy.ts — the one broad identity statement.
  • infra/ci-identity/envs.ts — the tier registry.
  • infra/ci-identity/deploy-environments.ts — what each tier's boundaries and roles are.
  • infra/scripts/bootstrap-deploy-roles.ts — the one admin step.
  • The boundary guard tests in infra/ci-identity — the executable spec of each guard.
  • infra/ci-identity/DESIGN-deploy-role.md — design rationale, the guard table, and what the guards do not cover.

TL;DR — the mental model ​

🅰 Effective = identity ∩ boundary

Broad identity, one gate.

Every deploy role carries the same Allow *:* identity. Its permissions boundary is the only gate: in production, allow everything and then deny four guards; in the non-prod account, allow everything and let the account be the fence.

🅱 The guards, not a list

No escalation, no customer data, no other environment, no account controls.

The boundary names no resource a tier deploys, so a new resource needs no boundary change and no pre-merge apply. The boundary changes only when a guard does.

🅲 State per tier

Own bucket + KMS key.

Each tier deploys with its own Pulumi state bucket and secrets key, both tagged environment=<tier>, so no other tier's deploy role can touch them. The identity stack's own state sits in the seed bucket, and no deploy role can touch it.

No admin for contributors, ever. Admin is used exactly once per environment, by an operator, to run one bootstrap script. After that, CI assumes the bounded deploy role via GitHub OIDC — no stored AWS keys, no local infra applies.

How the pieces fit ​

🧱 One template per environment

The boundary (onramp-deploy-boundary-<env>) is one customer-managed policy per environment, rendered from a single template keyed on env and migrateTargets. The seven plane boundaries carry the same document. You change it only when a guard changes (see Change a guard).

🏭 Per-env, from the registry

Each ENVS entry in envs.ts mints one role onramp-deploy-<tier> — bounded, environment=<tier> tagged — with the broad identity. "Another environment" is one more registry entry.

🪣 Per-env state plane

onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>. Created by the bootstrap script in the tier's own region.

🔑 One admin bootstrap

bun run bootstrap.roles requires the seed bucket, optionally provisions the per-env state plane, ensures the GitHub OIDC provider, and runs pulumi up on the account's identity stack. The only place admin is needed.

The chicken-and-egg (and how it's solved) ​

Pulumi needs a state backend before it can create anything — including the buckets and roles that are the deploy-role system. The resolution: one bucket is created imperatively; everything else is Pulumi-managed from it.

mermaid
flowchart TD
  OP["Operator (admin) · bun run bootstrap + bootstrap.roles"] --> SEED["1. Seed state bucket + KMS<br/>onramp-pulumi-state-ACCT (us-west-2)"]
  OP --> PENV["2. Per-env state plane<br/>onramp-pulumi-state-TIER-ACCT + KMS"]
  OP --> OIDC["3. GitHub OIDC provider (looked up, never recreated)"]
  SEED --> STACK["4. pulumi up · identity stack<br/>(its OWN state lives in the seed bucket)"]
  OP --> STACK
  STACK --> BND["Boundary policy<br/>onramp-deploy-boundary-TIER"]
  STACK --> ROLE["Deploy role<br/>onramp-deploy-TIER (bounded, environment=TIER)"]
  OIDC --> ROLE
  ROLE -->|CI assumes via OIDC| DEPLOY["Env deploys<br/>Lambda · AgentCore · Cognito · KVS"]
  PENV -->|state backend for| DEPLOY
  style STACK fill:#e6f3ff,color:#000

The key move: the identity stack stores its own state in the seed bucket, not in a per-env bucket. So the seed bucket is the single imperative prerequisite; the boundaries and the roles are Pulumi resources created by that stack, and the per-env state planes are created by the bootstrap script. Per-env buckets hold the tier's deploy state (Lambda, AgentCore, …), never the identity stack's own state.

What stops source from outrunning the apply ​

Because this stack converges only by a manual pulumi up, merging a trust-policy change does not apply one. A PR whose source is internally consistent — workflow binding and declared subject changed together — still breaks every deploy through that role until an operator runs the apply, and nothing about the merge says so.

Every change under infra/ci-identity/** requires a manual apply after the code is final and before merge. Run it from the exact worktree and branch whose declarations should be live. The stack covers every tier in the shared AWS account, and Pulumi deletes managed resources that are absent from the branch being applied.

Four checks make the divergence unmergeable instead. Each is a link between two things that must agree; none of them can be satisfied by editing only one side.

#LinkWhere it runsNeeds
1a job's environment: binding == its row in .github/scripts/oidc-env-bindings.ts (verbatim), and every binding has a rowbun test .github/scripts/ — Frontend CI → TypeScript unit tests (bun), every PR—
2that registry's required subjects ⊆ what envs.ts declares, and the infra/app roles' pinned subjects equal what the registry requiresbun test in infra/ — Infra CI—
3every declared OIDC subject is live, each admitting statement pins aud + the expected provider, and no statement admits callers it cannot scopeCI Identity Applied → Declared deploy-role boundaries are applied (required; reads AWS when the PR changes declared trust)onramp-ci-oidc-verify
4every rendered boundary (the main boundary, the plane boundaries and the runtime-role boundaries) says exactly what its live default version saysCI Identity Applied → Declared deploy-role boundaries are appliedonramp-ci-oidc-verify

Link 3 is the one that catches an unapplied stack, and it is red until you apply: a PR that adds a subject stays failing until an operator runs the apply below, then goes green on a re-run of the job with no new commit. That inversion is the mechanism — it turns the apply into a merge prerequisite rather than a step in this runbook that someone has to remember. It fails on divergence in either direction, so removing a subject from source also requires the apply before merge: otherwise the repo claims a trust relationship is gone while AWS still honours it.

It reports three distinguishable causes, and the fix differs:

  • declared but not applied (or the role is absent) → run the apply.
  • applied but no longer declared → run the apply; until you do, a trust relationship the repo says is gone is live.
  • admits callers the check cannot scope to a subject → do not just apply. An Allow sts:AssumeRoleWithWebIdentity statement with no sub condition admits every workflow in every repository, and one federated to another IdP or with a different aud is outside what these roles are supposed to trust. Any of those means a live over-broad trust policy, which is a person's problem before it is Pulumi's. Note what this rules out: a subject-list comparison alone would score such a policy as agreement, because an unconditioned statement contributes no subject to compare.

What link 3 does not compare: the rest of the trust document — other condition keys, tags, session policies. A change confined to those and merged without an apply is not caught.

Link 4 is link 3 for the permission boundaries, and is red until you apply in the same way. It compares statements, and fails in both directions:

  • declared, not live → a guard source renders is not enforced yet (the live boundary is broader than source says), or the policy does not exist. Run the apply.
  • live, not declared → the live boundary holds a statement source does not render. Either a guard source removed is still enforced, so a deploy source allows is refused, or an allowance source removed is still live. The usual cause is another open PR that applied from its branch. An apply writes whole documents, so applying from here removes that PR's statements: let it merge, rebase onto main, then apply.

It reports on every PR, so it can be a required check, and it only reads AWS for a PR that renders some boundary differently from its base; every other PR passes without a credential. A green run is a snapshot: another branch's later apply does not re-run it. What keeps that apply from happening is the guard below.

bun run bootstrap.roles runs ci-identity/guard-apply.ts before pulumi up and refuses when a live boundary holds statements that neither origin/main nor the checkout being applied declares — the state another branch's pre-merge apply leaves behind, which this apply would silently remove. It names those statements, the live version's creation time, and the open infra/ PRs whose diffs most likely added them, with authors. The owner merges first and this branch then rebases onto main and applies, or this branch rebases onto theirs when the two changes must ship together. --skip-apply-guard overrides the guard once that owner agrees the statements can go.

Known gap, deliberate. Nothing here verifies the GitHub side directly — that a tier's Environment exists with the allow-list ALLOWED_REFS_BY_ENV declares, or that its variables are set. Reading either back needs a credential GITHUB_TOKEN does not carry (administration is not one of the permissions: keys, and variables need a fine-grained token), so closing it means adding a GitHub App or PAT to store and rotate. What stands in for it: link 1 refuses a binding to an env with no ALLOWED_REFS_BY_ENV row (so nothing can bind an env that nothing provisions). It does not prove the allow-list is present on a prod-tier Environment — running bootstrap-agent-environments.ts before a new tier's binding ships is still a real security prereq, not just an availability one.

The non-prod account ​

PR previews live in their own account, onramp-nonprod (375255020707); every other environment lives in production (111776553558). infra/components/aws-accounts.ts says which environment lives where, and everything account-bound derives from it.

Each account has its own ci-identity stack, applied with that account's admin credentials: identity-tier for production, identity-nonprod for non-prod. In non-prod every boundary allows everything (nonprod-boundary-policy.ts); the organization's guard policy on the NonProd unit, which no deploy role can edit, is the fence. The non-prod stack also owns the account-level resources previews pin by id or name: the agents' execution role, the preview.onrampapps.com zone and its certificate, the agents' guardrail and their knowledge base.

Apply it (the bootstrap is idempotent; the apply guard reads production's boundaries, so it is skipped here):

bash
AWS_PROFILE=nonprod-admin bun run --cwd infra bootstrap
bash
AWS_PROFILE=nonprod-admin bun run --cwd infra bootstrap.roles --stack identity-nonprod --skip-apply-guard

After the first apply, and again whenever the guardrail or knowledge base is replaced:

  • Copy guardrailId, guardrailVersion and knowledgeBaseId from pulumi stack output into the preview block of onramp-agents/environments.yaml.
  • Load the knowledge base: dispatch Refresh preview knowledge base on main. It mirrors the pages the public sites' sitemaps list into S3 and re-indexes them; run it again whenever the public content has changed enough to matter.
  • The certificate validates through the new zone once production's onrampapps.com zone delegates preview.onrampapps.com to previewZoneNameServers; to issue it before that, add the same validation record to the zone the subdomain currently resolves through.

The previews stack's config, infra/app/Pulumi.previews.yaml, is keyed to the non-prod account's alias/onramp-pulumi-secrets and carries the Google client secret as committed secure: config.

Previews send SMS from production's toll-free number through a resource share the production stack declares (infra/ci-identity/preview-sms-share.ts). It needs resource sharing with AWS Organizations enabled in the management account.

Known gap, deliberate. The CI checks that compare live trust and boundaries with the source read the production account only. In non-prod they would guard nothing: the previews deploy role allows everything there, so the same role any pull request assumes can rewrite its own trust and boundary, or create a role that outlives the job. What bounds it is the account itself and the guard policy on the NonProd unit, which no deploy role can change. Inside the account a pull request job reaches preview data and Bedrock spend; the only production resources it reaches are the shared SMS number and its opt-out list, which it can use to send and can edit. Sends still honour the number's own opt-out list, which is not shared.

Add a new environment ​

Adding an environment is a one-line registry edit plus an operator standing up its state plane. Convention derives everything else from the tier name — the state bucket, the KMS key, the role ARN, the CI backend URL.

TL;DR. A new env X is the one line { env: "X" } in envs.ts — presence in the registry means live (there is no enabled flag; delete the line to retire a tier). Order matters: an operator bootstraps X's state plane first, then the one-line entry lands, then pulumi up creates the role. (gdpr-prod adds region: "eu-west-1" + --region eu-west-1 — the one tier that deviates.)

1
Bootstrap the prereq
admin · state plane
2
Add the tier
envs.ts · one line
3
Apply + verify
pulumi up
4
Wire CI
workflow OIDC

1 · Bootstrap the prereq (admin) ​

A tier's KMS alias must exist before any pulumi up resolves it, so the state plane is the prerequisite — provision it first, with no role yet:

bash
AWS_PROFILE=admin bun run bootstrap.roles --env-tag <env> --no-pulumi-up

The bucket name + KMS alias are derived from --env-tag (the same convention index.ts uses) — nothing to paste. This provisions the per-env bucket + KMS key (tagged environment=<env>, the bucket with attribute-based access control on so the cross-environment guard fences it), requires the seed bucket to exist, ensures the GitHub OIDC provider, and stops before pulumi up (the role is created in step 3, once the entry exists).

gdpr-prod adds --region eu-west-1 — the one tier that deviates: … --env-tag gdpr-prod --region eu-west-1 --no-pulumi-up. KMS keys are regional and EU data residency requires the EU tier's state + secrets in-region; the script fails closed if you omit it. The seed bucket and the identity stacks stay in us-west-2 regardless.

A new resource needs nothing ​

Adding a service, bucket, queue, function, alarm or schedule to infra/app or infra/components needs no boundary change and no ci-identity apply. The deploy role is allowed everything except four guards, and the boundary names no resource a tier deploys. Do not add an allowance to boundary-policy.ts or to a plane boundary for a new resource; if a deploy is refused, a guard denied it (see Troubleshooting), and the fix is not an allow.

Change a guard ​

A boundary changes only when a guard does: a deploy role must be barred from something new, or must be allowed something a guard denies. All production-account environments render the same template, so the change reaches every environment's main and plane boundaries at once.

  1. Edit buildBoundary in infra/ci-identity/boundary-policy.ts, and add or update the guard's test among the boundary guard tests in infra/ci-identity, in the same change.

  2. A deny's condition must name a key the service actually sets. A tag condition on a service that ignores it matches nothing, so the guard is a silent hole; check the service's authorization reference before relying on a tag. Keep the guard independent of what a tier composes, or the boundary grows with every deploy.

  3. cd infra && bun test and bunx tsc --noEmit -p tsconfig.json pass.

  4. Finish the whole infra/ci-identity/** diff, then have an admin apply the stack from this branch right before merge, following the ci-identity-apply skill:

    bash
    AWS_PROFILE=admin bun run bootstrap.roles

    The apply writes a new default version of each changed boundary; roles pick it up live, with no re-attach.

  5. Re-run Declared deploy-role boundaries are applied; it goes green once the live policies match what the branch renders.

The guards are listed under Security invariants, and DESIGN-deploy-role.md has the full table, including what they do not cover: S3 buckets without attribute-based access control, untagged resources, actions with no resource tag in the request, and creates.

State buckets & KMS ​

Naming
  • Seed (shared, us-west-2): onramp-pulumi-state-<account> + alias/onramp-pulumi-secrets
  • Per-tier: onramp-pulumi-state-<tier>-<account> + alias/onramp-pulumi-secrets-<tier>
Region

Per-tier plane is created in the tier's region (--region). gdpr-prod ⇒ eu-west-1 (guard-enforced). The seed bucket and the identity stacks stay us-west-2.

Isolation

Each tier deploys against its own bucket. The bucket is tagged environment=<tier> with attribute-based access control on, so S3 shows IAM the tag and the cross-environment guard (G3) keeps every other tier's deploy role out of it. A bucket created before bootstrap.roles enabled that is open to other tiers until bootstrap.roles --env-tag <tier> --no-pulumi-up is re-run. The identity stack's own state, in the seed bucket, is fenced by name (G1).

Secrets key

The per-tier KMS key is the Pulumi secrets provider. It must carry environment=<tier>: that tag is what lets G3 keep other environments' deploy roles off it. The bootstrap script tags it for you.

Verify locally ​

The boundary guard tests in infra/ci-identity evaluate the real identity ∩ boundary locally with no AWS credentials. They are the executable spec for the guards in this runbook.

bash
cd infra
bun install                      # standalone package
bunx tsc --noEmit -p tsconfig.json
bun test                         # guard tests, trust and registry tests

You can't pulumi up without admin + the seed backend, so local verification stops at tsc + tests.

Security invariants — never weaken these ​

These are the four guards (G1 escalation, G2 customer data, G3 other environments, G4 account controls), as the properties a boundary change must keep. The guard tests enforce them and fail CI if you regress one.

  • Roles carry this environment's boundary (G1) — a role a deploy mints, or re-boundaries, can only carry one of this environment's boundaries.
  • Only this environment's roles (G1) — a deploy role creates, changes, passes and assumes only roles named onramp-deploy-<env>-*; every other role, Duplo's and hand-made admin roles included, is out of reach.
  • The identity stack is read-only to every deploy role (G1) — no IAM write or PassRole on anything tagged onramp:project=ci-identity, no change to the boundary policies, the GitHub OIDC provider or the identity stack's state, and no way to remove a boundary.
  • No IAM users, access keys, login profiles or groups (G1) — no long-lived credential.
  • Deploy roles read no customer data directly (G2) — no database destroy, copy or share, no Duplo database change, no customer-data bucket object, no database-login parameter.
  • No other environment (G3) — nothing tagged for a different environment.
  • No account controls (G4) — the organization, account settings, CloudTrail and KMS key deletion.
  • Guards fence what the service honours — a deny keyed on a tag the service never sets guards nothing.

Troubleshooting ​

🚫 AccessDenied on a deploy

The boundary allows everything except the guards, so an AccessDenied that names a permissions boundary is a guard. Match the denied action to a guard in boundary-policy.ts (the uploaded policy has no Sids, so match on action and resource) with the table below. If the message names a service control policy, a resource policy, or the service itself, the denier is the organization or the resource, not this boundary; in the non-prod account it is the organization's NonProd guard policy. Reproduce the denial in a guard test before changing anything.

🏷 CreateRole or PutRolePermissionsBoundary denied

G1: the role is not carrying one of this environment's boundaries (onramp-deploy-*boundary-<env>). A role a stack declares is stamped automatically (env-conventions.ts); a plane's runtime role must carry that plane's boundary by name. Fix the declaration, never the guard.

🔒 IAM write or PassRole on a deploy role

G1: anything tagged onramp:project=ci-identity is read-only to deploy roles. A role or policy that must change belongs in infra/ci-identity and is applied by an admin.

🗄 RDS, a customer bucket, or a database login

G2: a deploy never destroys, copies or shares a database, touches the Duplo app databases or AgentCore memory, reads a customer-data bucket's objects, or reads /onramp/*/app_db/* and /onramp/*/migrate/*. A role that must read data is the deployed workload's own role, created by the stack.

🌐 Action on another environment's resource

G3: the resource's environment tag names an environment other than this one or its migrate targets. Usually a stack pointed at the wrong environment, or a resource carrying the wrong tag. No pair crosses.

🌍 gdpr-prod region errors

The bootstrap guard fires if --env-tag gdpr-prod isn't paired with --region eu-west-1. The boundary is region-agnostic; the tier's region is region in envs.ts.

🧪 Guard tests red after an edit

A red test means the shipped policy changed behavior. Fix the policy, not the assertion, unless the behavior change is intended; then update the test in the same change.

🪪 A check is red after a merge

Declared deploy-role boundaries are applied is red, and blocks the merge, until an admin applies the stack from the branch (see What stops source from outrunning the apply).

Out of scope ​

Retiring deploy.py and migrating prod AgentCore (Knowledge Bases, OpenSearch Serverless, runtime roles) onto Pulumi is a separate effort. This deploy-role system is the safe-deploy enabler for it: once a prod role is stamped here, that migration deploys through it instead of with admin.

Internal documentation — gated behind Cloudflare Access.