Skip to content

Redacting secrets from the Caddy access log ​

Stop the webserver writing live credentials into OpenSearch, then rotate the ones it already wrote — in that order.

Owner: platformWhen to use: adding a secret-bearing header or URL, or remediating one already logged

Canonical sources. bun run logs.secret-audit (devtools/access-log-secret-audit.ts) owns the list of surfaces and is the only thing that decides whether a tenant is clean — this page is the HOWTO, not the inventory. The tenants that run a webserver are SERVICE_TENANTS.webserver in infra/duplo/pod-config.ts.

Why this exists ​

Caddy's JSON access log records the whole request header map and the whole URI. It redacts only its own built-in sensitive headers — Cookie, Set-Cookie and Authorization — so every other header, and every secret sitting in a path segment or query parameter, is written verbatim into filebeat-7.11.1-*. That index has no login beyond the corporate VPN and a 30-day retention, which makes it a credential store that nothing guards.

Nothing in the Flask app can prevent this. The fix is the filter log encoder on each tenant's Caddy site block, and that config is a Duplo-owned volume rather than anything in this repo.

TL;DR ​

🅰 Two shapes, two filters

A header needs replace; a URI needs regexp. A secret in a path segment or query string is invisible to a header filter, so a fix that only lists headers closes half the hole — most of this inventory is in the URI.

🅱 Redact, then rotate

Rotating first writes the new secret into the same unredacted log. Confirm the tenant is clean with the audit before you touch a single value.

🅲 Per tenant, every time

Each tenant carries its own site block and its own secrets. Four distinct EasyCron secrets across four tenants means four rotations, not one.

The filter ​

Splice this log block into each tenant's site block. The header list and the URI pattern both come from the surfaces logs.secret-audit checks; adding a surface there and not here is the failure this pairing exists to prevent.

log {
	format filter {
		wrap json
		fields {
			request>headers>X-Easycron-Token replace REDACTED
			request>headers>Or-Internal-Api-Auth-Secret replace REDACTED
			request>headers>X-Adobesign-Clientid replace REDACTED
			request>headers>X-Onramp-Ops-Token replace REDACTED
			request>headers>X-Guide-Me-Bridge-Token replace REDACTED
			request>headers>X-Embed-Token replace REDACTED
			request>uri regexp "(/integrations/adobe/webhook/|/integrations/call-recorders/[^/?]+/webhook/|/api/session/|/email-action/answer/|/sms/task-links/|[?&]t=|[?&]email=|[?&]sso_token=|[?&]code=|[?&]orac=|[?&]user_uuid=)[^/?&]*" "${1}REDACTED"
		}
	}
}

fields is a map, so one filter per field. Every URI surface shares a single alternating pattern; adding one means extending the alternation, never adding a second request>uri line.

/sms/task-links/ is deliberately not anchored on /api/. The link an SMS sends is the portal page /portal/sms/task-links/<token>; its SPA then calls /api/sms/task-links/<token>/resolve with the same token. A pattern anchored on either one leaves the other writing the token into the log.

[?&]t= covers two surfaces. The portal FastPass login (/api/portal/fast-pass?t=<token>) and the Aero daily digest's mute link (/org/aero-digest/mute?t=<token>) both carry their token in t, so one alternative redacts both.

[?&]user_uuid= covers the email unsubscribe page and its API call. The link is /portal/unsubscribe?user_uuid=<token>, and its SPA calls /api/email-opt-out/status?user_uuid=<token> with the same value.

replace, not delete, on the headers. A deleted header is indistinguishable in the log from a request that never carried one, so the audit could not tell a redacted tenant from an idle one and absence would read as proof. replace keeps the name and drops the value, which is exactly what Caddy does for Authorization itself.

${1} and not $1: Go reads $1REDACTED as a group named 1REDACTED and substitutes nothing, which silently deletes the prefix it was meant to keep.

Prove a change to the pattern before it reaches a tenant — this needs no Duplo access and no cluster:

bash
docker run --rm -i caddy:2.9.1 sh -c \
  'cat > /tmp/Caddyfile && caddy validate --config /tmp/Caddyfile --adapter caddyfile' <<'CFG'
:8080 {
	log { output stdout
		format filter { wrap json
			fields { request>uri regexp "(/integrations/adobe/webhook/|/integrations/call-recorders/[^/?]+/webhook/|/api/session/|/email-action/answer/|/sms/task-links/|[?&]t=|[?&]email=|[?&]sso_token=|[?&]code=|[?&]orac=|[?&]user_uuid=)[^/?&]*" "${1}REDACTED" } } }
	respond "ok"
}
CFG

validate only proves the config parses. To prove it redacts, run that same config with caddy run, curl the shapes you care about, and grep the log for the values you sent — the expected result is that a project=abc beside a redacted t= survives untouched.

The procedure ​

1
Baseline
local
2
Patch the site block
Duplo
3
Roll the webserver
Duplo
4
Rotate
SSM + EasyCron + DB

1 · Baseline the tenant ​

Record which surfaces are exposed and what their fingerprints are, so step 4 can prove a rotation actually changed the value.

bash
bun run logs.secret-audit --window 7d stage

Work one tenant at a time, lowest blast radius first: stage, prod-gdpr-ir, prod-usw2.

Verify ​

bash
bun run logs.secret-audit --window 24h

Exit 0 across every tenant. It prints a length and a SHA-256 prefix for anything still exposed and never the value itself, so its output is safe to paste into a ticket.

ExitMeansDo
0every surface was exercised, every value redactedrotate
1a credential is readableredact; do NOT rotate yet
2nothing exercised at least one surface — proves nothingwiden --window, or send a request through each
64bad arguments—

Exit 2 is not a pass. A short post-rollout window can contain no Adobe webhook and no FastPass login at all, and a run over one would otherwise print a clean bill for filters nothing ever tested.

Troubleshooting ​

🚫 Audit still EXPOSED after the rollout

Caddy kept the old config. The mounted volume is only read at start, so a service that was updated but not rolled serves the unfiltered site block. Confirm the pod actually restarted, then re-run with --window 1h.

⚠ Exit 2, surfaces marked `?`

Nothing exercised them in the window — not the same as redacted, which is why it is not exit 0. Widen the window, or drive one request through each, before treating it as clean.

🚫 Log lines stopped being JSON

The filter encoder replaces the format, so a fields block without wrap json demotes the line to console format and breaks every saved OpenSearch query against message.

⚠ EasyCron jobs 401 after rotation

SSM and the job headers disagree. Both sides flip together or not at all; bun run easycron.probe separates a dead route from a rejected token.

Internal documentation — gated behind Cloudflare Access.