Confirming HMAC Retirement

How to prove, from production telemetry, that every runtime → Flask internal callback authenticates on a forwarded Bearer JWT — and that the legacy X-Agent-Secret (HMAC) leg is cold — before Flask is flipped to JWT-only and the shared secret is deleted.

This is a temporary runbook tied to ONRAMP-4918. Retire it together with the secret leg.

Background

The HMAC → JWT cutover is staged. Dual-support (authenticate_internal_request accepts a JWT or the secret) is live everywhere, and every per-domain runtime already forwards a Bearer JWT. The remaining steps each wait for the prior to reach prod:

1. Flask stops accepting X-Agent-Secret (JWT-only contract).
2. Delete the live secrets + remove provisioning tooling.
3. Delete the one-time deleter.

Step 1 is only safe once the secret leg is proven unused in prod. This runbook is how you prove it.

Prerequisite: leg logging must be live in prod

The decision keys off a structured log line emitted at the single auth chokepoint (app/api/agent/helpers/internal_auth.py):

Line	Level	Meaning
internal_auth.authed leg=jwt path=…	info	callback authenticated on a Bearer JWT
internal_auth.authed leg=secret path=… ua=…	warning	callback fell back to the HMAC secret — must reach zero
internal_auth.denied path=…	info	neither leg authenticated (a 401)

Confirm the logging is deployed before trusting any "zero" result — a quiet query against a build that never emits the line looks identical to success:

message:"internal_auth.authed"   # must be NON-zero once logging is live

Plane 1 — Flask (OpenSearch / "Elasticsearch"): the receiver of truth

Flask is the side that actually decides the leg, so this is the primary gate. Logs land in OpenSearch (filebeat-7.11.1-*, service kubernetes.labels.app: flask). Cluster is VPN-only at http://10.220.61.14:9200 (no auth).

The Caddy access log redacts Authorization and does not reliably carry X-Agent-Secret, and a present Bearer does not prove it validated (the helper falls through to the secret on a bad Bearer). Only the app-level internal_auth.* line above is authoritative.

Prod tenants to sweep: prod-usw2 (the big one), prod-gdpr-ir (EU). Repeat each query per tenant — there is no cross-tenant rollup.

Gate query — secret-leg usage (must be zero)

GET filebeat-7.11.1-*/_count
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "tenant.name": "prod-usw2" } },
        { "term":  { "kubernetes.labels.app": "flask" } },
        { "range": { "@timestamp": { "gte": "now-7d" } } }
      ],
      "must": [
        { "match_phrase": { "message": "internal_auth.authed" } },
        { "match_phrase": { "message": "leg=secret" } }
      ]
    }
  }
}

Nonzero → there is still a straggler. Name it by listing the offending lines (path = endpoint → which runtime; ua = client):

GET filebeat-7.11.1-*/_search
{
  "size": 50,
  "sort": [{ "@timestamp": "desc" }],
  "_source": ["@timestamp", "message"],
  "query": {
    "bool": {
      "filter": [
        { "term":  { "tenant.name": "prod-usw2" } },
        { "range": { "@timestamp": { "gte": "now-7d" } } }
      ],
      "must": [{ "match_phrase": { "message": "internal_auth.authed leg=secret" } }]
    }
  }
}

Sanity query — JWT-leg volume (must track real traffic)

Confirms "zero secret" means "everyone moved", not "logging silently broke". leg=jwt should roughly equal total /api/internal/* callback volume (≈ 3k/day in prod-usw2 at time of writing):

GET filebeat-7.11.1-*/_count
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "tenant.name": "prod-usw2" } },
        { "term":  { "kubernetes.labels.app": "flask" } },
        { "range": { "@timestamp": { "gte": "now-24h" } } }
      ],
      "must": [{ "match_phrase": { "message": "internal_auth.authed leg=jwt" } }]
    }
  }
}

Plane 2 — AgentCore runtimes (CloudWatch): corroboration

CloudWatch confirms each runtime is deployed on JWT-forwarding code and is minting/forwarding without error. It is corroborating, not the gate — Plane 1 already records the outcome.

Account 111776553558. There are three AgentCore runtimes that call back into Flask — copilot, intel, portal. (The views and playbooks agents are domains inside the copilot runtime, not separate runtimes; the headless re-engagement flow runs in Flask, not AgentCore — measure it in Plane 1 by its service: JWT.)

Group names are /aws/bedrock-agentcore/runtimes/onramp_<env>_<runtime>-<suffix>-DEFAULT — note the env precedes the runtime, and the suffix changes on every redeploy, so resolve live rather than hardcoding. Prod spans two regions:

# prod-usw2 (us-west-2): onramp_PROD_USW2_{copilot,intel,portal}
aws logs describe-log-groups --profile readonly --region us-west-2 \
  --log-group-name-prefix /aws/bedrock-agentcore/runtimes/onramp_PROD_USW2_ \
  --query "logGroups[].logGroupName" --output text

# prod-gdpr-ir / EU (eu-west-1): onramp_prod_gdpr_ir_{copilot,intel,portal}
aws logs describe-log-groups --profile readonly --region eu-west-1 \
  --log-group-name-prefix /aws/bedrock-agentcore/runtimes/onramp_prod_gdpr_ir_ \
  --query "logGroups[].logGroupName" --output text

For each, confirm callbacks succeed and there are no "missing bearer" warnings from the outbound-bearer seam. Live agent traces are also visible in MLflow (see .claude/rules/mlflow.md).

The decision gate

Flip Flask to JWT-only only when all hold, across every prod tenant:

1. leg=secret count = 0 for ≥ 7 consecutive days, and
2. the window spans ≥ 1 full headless re-engagement cycle — it mints a short-lived per-vendor service JWT on a schedule, not on a user turn, so it can lag a deploy and is the slowest caller to prove itself, and
3. leg=jwt volume matches expected callback traffic (no silent drop), and
4. every runtime in Plane 2 confirmed on JWT-forwarding code.

After the flip — failure signature to watch

Once Flask is JWT-only, a caller that somehow still presents only the secret gets a 401, which surfaces as internal_auth.denied. Watch for a spike per environment immediately after each deploy:

GET filebeat-7.11.1-*/_count
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "tenant.name": "prod-usw2" } },
        { "term":  { "kubernetes.labels.app": "flask" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "must": [{ "match_phrase": { "message": "internal_auth.denied" } }]
    }
  }
}

A spike that correlates with the deploy → roll back (revert the JWT-only change; Flask returns to dual-support instantly — the secrets are still present on the runtimes). A flat line → safe to promote to the next environment.