Rotating the Flask SECRET_KEY

Provision and rotate the per-environment Flask signing key in SSM — and do it without discovering its blast radius by accident.

Owner: platformWhen to use: bringing up a new environment, or rotating a key you believe is exposed

Canonical sources. The resolver is app/api/utils/secret_key.py — it owns the SSM path, the fail-closed boot behavior, and the local-development escape. Every environment's parameter is created by infra/components/flask-secret-key.ts, composed per stack in infra/app/index.ts, and declared for the durable tiers in infra/app/provisioned-params.ts. This runbook is the HOWTO; none of those files is restated here.

Rotating logs everyone out. SECRET_KEY signs Flask session cookies, so a new value invalidates every active session in that environment — every logged-in user is bounced to the login screen. It also signs /gateway preview-login tokens (app/api/utils/preview_token.py), so every outstanding preview link dies immediately rather than at its normal 15-minute expiry. Neither effect is recoverable and neither is staged: it lands the moment the pods restart. Rotate deliberately, announced, in a low-traffic window — never as a side effect of a routine deploy.

The roll itself is not a clean cutover. Duplo replaces pods one at a time, so for the length of the roll old pods and new pods serve the same load balancer with different keys. A cookie signed by one side fails verification on the other: a user bounced to login on a new pod gets a new-key cookie, and the next request that lands on a surviving old pod bounces them again. Sessions flap until the last old pod is gone, then the one-time logout is complete. That is why the rotation steps below say restart the whole environment together, and why a rotation belongs in a low-traffic window. The first deploy of the SSM-backed key is a rotation in this sense too — off the retired committed literal — and it deliberately does not carry that literal as a SECRET_KEY_FALLBACKS entry to soften the roll: a fallback keeps the leaked key verifying for a whole release, which is the exposure the change exists to end.

What reads it

Consumer	What breaks on rotation
Flask session cookie (flask.session, Flask-Login remember cookie)	Every active session in that environment is logged out
/gateway preview-login + impersonation tokens (app/api/utils/preview_token.py)	Every outstanding link is rejected; new ones work immediately

The customer portal's orac cookie is not on this list. It carries the customer user's stored password hash (or an SSO token) and is matched against User.password (app/api/customer/helpers.py) — no signature, no SECRET_KEY, so it survives a rotation intact. Portal users still lose their Flask session like everyone else, because login_app_user uses Flask-Login; the surviving orac cookie is what lets the portal re-establish them without a new magic link.

The orac query parameter is a different thing that shares the name: on both the preview branch (/gateway?mode=preview&playbook=…) and the impersonation branch (/gateway?orac=…&project=…, no mode) it is a preview_token, and both die with the key.

The SSM path

One parameter per environment, SecureString:

/onramp/dev/flask/secret_key
/onramp/stage/flask/secret_key
/onramp/demo/flask/secret_key
/onramp/prod/flask/secret_key
/onramp/gdpr-prod/flask/secret_key

The env segment is ENVIRONMENT_NAME, lowercased. Per-PR previews follow the same convention (/onramp/preview-<N>/flask/secret_key).

gdpr-prod lives in eu-west-1; every other environment is in us-west-2. The app reads from the same per-environment region (app.api.utils.env_config.ssm_region_for_environment), so pass --region accordingly whenever you touch a parameter by hand.

Who creates it — and the check that will not let you forget

Nobody creates these by hand. Pulumi seeds every parameter with a random value on the stack's first apply:

Environment	Owning stack	Applied by
preview-<N>	pr-<N> (the preview core stack)	every preview deploy
dev, stage, prod, demo	the env core stack of the same name	a release to that tier (build-and-deploy.yml deploy-infra; dev on merge to main). A parameter already at the path makes the first apply fail ParameterAlreadyExists rather than silently replacing it — pulumi import it, or delete it and let the seed win
gdpr-prod	gdpr-prod-pipeline, beside the EU executor that reads it	the prod release

The app fails closed at boot without a readable parameter, so the ordering that matters is "the stack applies before the first pods that need it roll". Three things enforce it:

1. applied-params in infra-ci.yml runs infra/app/verify-provisioned-params.ts on every PR touching infra/** and fails while any durable tier lacks a declared parameter. A PR that introduces a new tier — or this key on a tier that has not applied yet — stays red until the owning stack has been applied. The job output names the exact stack and the dispatch command (gh workflow run "deploy-infra-<env>.yml"; deploy-infra-gdpr-prod-pipeline.yml with a pipeline_image_uri for the EU parameter). Run it from the branch that declares the parameter, then re-run the check; no new commit is needed.
2. Every duplo-deploy leg gates on it. Before the Flask roll, the leg runs the same verifier in --tier <env> mode under its own tenant credentials and refuses to roll while a parameter the pods cannot boot without is absent in that tier's home region. A red applied-params that was merged over lands here as a stopped deploy naming the stack to apply — the pods stay on the previous image — not as a crash-loop. Only boot-fatal members block; the ops token and internal-API secret keep their request-time failure mode.
3. The preview core stack creates the parameter and the Lambda's read grant in the same apply that creates the Lambda, so a preview can never boot ahead of its key.

Both checks prove existence, not readability. They use ssm:DescribeParameters, which needs neither the read grant nor KMS decrypt. Whether a serving pod's role can read the value is the check below, and it is still yours to run once per environment.

Prove the SERVING role can read it — the half no check can cover

This is the first unconditional, boot-fatal SSM read on the pod path. Every other one either self-skips on an unprovisioned environment (DocuSign, e-sign) or happens lazily on first use (the auth-server keypair). So nothing before it has proven that a serving pod's role can complete a GetParameter under /onramp/{env}/. If that grant turns out to be per-parameter rather than prefix-scoped, AccessDenied is deliberately uncaught and every pod in the environment crash-loops on the deploy that ships it.

The Pulumi-managed consumers need no such step: the preview Lambda and every tier's integrations-pipeline executor are granted the read by the stacks that build them (infra/app/index.ts, infra/components/integrations-pipeline.ts). The Duplo-managed serving pods are the one consumer nothing in this repo grants.

Run this from inside a pod in that environment — not from an operator laptop, which has a different identity — and in that pod's own region:

# exec into a running pod for the environment, then:
aws ssm get-parameter \
  --name "/onramp/${ENV_NAME}/flask/secret_key" \
  --with-decryption \
  --query 'Parameter.Name' --output text

It must print the parameter name. AccessDeniedException means the pod role's policy needs ssm:GetParameter on that ARN (and kms:Decrypt on the key that encrypted it) — fix that before the deploy, not after.

Repeat for all five environments. gdpr-prod runs in eu-west-1, so its pod resolves the parameter in eu-west-1; the other four are us-west-2.

Rotating an existing key

Pulumi seeds the value once and then leaves it alone (ignoreChanges: ["value"] in flask-secret-key.ts), so a rotation is an out-of-band write that survives every later apply.

1. Announce it. Every user of that environment is logged out; every outstanding preview link stops working. Say so first.

2. Overwrite the parameter:

   ENV_NAME=prod          # dev | stage | demo | prod | gdpr-prod
   REGION=us-west-2       # eu-west-1 for gdpr-prod
   
   aws ssm put-parameter \
     --region "$REGION" \
     --name "/onramp/${ENV_NAME}/flask/secret_key" \
     --type SecureString \
     --overwrite \
     --value "$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"

   Do not pass --description. Pulumi owns that field, and SSM keeps the previous description when an overwrite omits one.

3. Restart the pods. The value is memoized for the life of each process, so nothing picks up the new key until it restarts. This is deliberate: a half-rotated fleet would have workers rejecting each other's cookies, which presents as a random logout loop rather than a clean cutover. Restart the whole environment together, not one pod at a time.

4. Verify (below) before declaring it done.

Never reuse one environment's value in another: a key that verifies in two environments lets a token minted in the weaker one be replayed against the stronger one.

One way an apply can undo a rotation. ignoreChanges suppresses the diff on the value, not the value Pulumi sends when another input changes. A later change to the parameter's description or tags makes Pulumi re-put the parameter with the value it knows — the original seed — over the rotated one. That is why the description is a pinned constant and the tags are the stack's stable set. If everyone in an environment is logged out after an infra apply that touched flask-secret-key.ts, this is what happened: rotate again.

Verify

# The parameter exists, is a SecureString, and shows who last wrote it.
aws ssm describe-parameters --region "$REGION" \
  --parameter-filters "Key=Name,Values=/onramp/${ENV_NAME}/flask/secret_key" \
  --query 'Parameters[0].[Name,Type,Version,LastModifiedUser]' --output text

Then log in to the environment and confirm you get a session, and mint one fresh preview link and confirm it opens. A successful boot is itself evidence: the app refuses to start when the parameter is missing, unreadable, or empty.

Troubleshooting

Pods crash-loop with Flask SECRET_KEY is unavailable at SSM parameter …. The owning stack has not applied, or the pod's role cannot read the parameter. Check existence with describe-parameters (needs no KMS decrypt permission); if it exists, the role is missing ssm:GetParameter — or kms:Decrypt on the key that encrypted it. If it does not, applied-params on the PR that introduced it should have been red; apply the stack the check names.

Everyone got logged out and nobody rotated anything. The value changed under the fleet. Check the parameter's Version and LastModifiedUser in SSM: a Pulumi deploy role there, right after an infra apply, is the re-put described above.

Local dev reads a deployed environment's key, or refuses to boot. Three signals put a process on the local on-disk key (.flask_secret_key.local.key, gitignored): ENVIRONMENT_NAME names a local env, bun run dev exported AGENT_LOCAL_DEV=true (devtools/dev.mjs), or the process is serving a source checkout — a .git beside the repo root, which every clone and worktree has and no built image does. A workstation on the .env.example default ENVIRONMENT_NAME=dev therefore stays local under every launcher, including flask db upgrade and an IDE run configuration. A container that inherits a deployed env name with no AWS runtime at all (the migrate gate) falls back only where it says so with AWS_EC2_METADATA_DISABLED=true — the migrate gate already does.

Pods crash-loop with … is unreadable: this environment resolved no AWS credentials. The credential chain came up empty — an IRSA / instance-profile problem or an IMDS outage, not a missing parameter. It is deliberately fatal: botocore reports a transient metadata failure exactly as it reports never having had an identity, so treating it as "no AWS runtime here" would let one process sign with a key of its own invention and log out everyone it served. Fix the identity; do not set AWS_EC2_METADATA_DISABLED on a pod or a preview Lambda to get past it.

A preview's key. Do not touch it. Previews are provisioned by the pr-<N> stack and torn down with it.