Redacting secrets from the Caddy access log
Stop the webserver writing live credentials into OpenSearch, then rotate the ones it already wrote — in that order.
Canonical sources. bun run logs.secret-audit (devtools/access-log-secret-audit.ts) owns the list of surfaces and is the only thing that decides whether a tenant is clean — this page is the HOWTO, not the inventory. The tenants that run a webserver are SERVICE_TENANTS.webserver in infra/duplo/pod-config.ts.
Why this exists
Caddy's JSON access log records the whole request header map and the whole URI. It redacts only its own built-in sensitive headers — Cookie, Set-Cookie and Authorization — so every other header, and every secret sitting in a path segment or query parameter, is written verbatim into filebeat-7.11.1-*. That index has no login beyond the corporate VPN and a 30-day retention, which makes it a credential store that nothing guards.
Nothing in the Flask app can prevent this. The fix is the filter log encoder on each tenant's Caddy site block, and that config is a Duplo-owned volume rather than anything in this repo.
TL;DR
A header needs replace; a URI needs regexp. A secret in a path segment or query string is invisible to a header filter, so a fix that only lists headers closes half the hole — most of this inventory is in the URI.
Rotating first writes the new secret into the same unredacted log. Confirm the tenant is clean with the audit before you touch a single value.
Each tenant carries its own site block and its own secrets. Four distinct EasyCron secrets across four tenants means four rotations, not one.
The filter
Splice this log block into each tenant's site block. The header list and the URI pattern both come from the surfaces logs.secret-audit checks; adding a surface there and not here is the failure this pairing exists to prevent.
log {
format filter {
wrap json
fields {
request>headers>X-Easycron-Token replace REDACTED
request>headers>Or-Internal-Api-Auth-Secret replace REDACTED
request>headers>X-Adobesign-Clientid replace REDACTED
request>headers>X-Onramp-Ops-Token replace REDACTED
request>headers>X-Guide-Me-Bridge-Token replace REDACTED
request>headers>X-Embed-Token replace REDACTED
request>uri regexp "(/integrations/adobe/webhook/|/integrations/call-recorders/[^/?]+/webhook/|/api/session/|/email-action/answer/|/sms/task-links/|[?&]t=|[?&]email=|[?&]sso_token=|[?&]code=|[?&]orac=|[?&]user_uuid=)[^/?&]*" "${1}REDACTED"
}
}
}fields is a map, so one filter per field. Every URI surface shares a single alternating pattern; adding one means extending the alternation, never adding a second request>uri line.
/sms/task-links/ is deliberately not anchored on /api/. The link an SMS sends is the portal page /portal/sms/task-links/<token>; its SPA then calls /api/sms/task-links/<token>/resolve with the same token. A pattern anchored on either one leaves the other writing the token into the log.
[?&]t= covers two surfaces. The portal FastPass login (/api/portal/fast-pass?t=<token>) and the Aero daily digest's mute link (/org/aero-digest/mute?t=<token>) both carry their token in t, so one alternative redacts both.
[?&]user_uuid= covers the email unsubscribe page and its API call. The link is /portal/unsubscribe?user_uuid=<token>, and its SPA calls /api/email-opt-out/status?user_uuid=<token> with the same value.
replace, not delete, on the headers. A deleted header is indistinguishable in the log from a request that never carried one, so the audit could not tell a redacted tenant from an idle one and absence would read as proof. replace keeps the name and drops the value, which is exactly what Caddy does for Authorization itself.
${1} and not $1: Go reads $1REDACTED as a group named 1REDACTED and substitutes nothing, which silently deletes the prefix it was meant to keep.
Prove a change to the pattern before it reaches a tenant — this needs no Duplo access and no cluster:
docker run --rm -i caddy:2.9.1 sh -c \
'cat > /tmp/Caddyfile && caddy validate --config /tmp/Caddyfile --adapter caddyfile' <<'CFG'
:8080 {
log { output stdout
format filter { wrap json
fields { request>uri regexp "(/integrations/adobe/webhook/|/integrations/call-recorders/[^/?]+/webhook/|/api/session/|/email-action/answer/|/sms/task-links/|[?&]t=|[?&]email=|[?&]sso_token=|[?&]code=|[?&]orac=|[?&]user_uuid=)[^/?&]*" "${1}REDACTED" } } }
respond "ok"
}
CFGvalidate only proves the config parses. To prove it redacts, run that same config with caddy run, curl the shapes you care about, and grep the log for the values you sent — the expected result is that a project=abc beside a redacted t= survives untouched.
The procedure
1 · Baseline the tenant
Record which surfaces are exposed and what their fingerprints are, so step 4 can prove a rotation actually changed the value.
bun run logs.secret-audit --window 7d stageWork one tenant at a time, lowest blast radius first: stage, prod-gdpr-ir, prod-usw2.
Verify
bun run logs.secret-audit --window 24hExit 0 across every tenant. It prints a length and a SHA-256 prefix for anything still exposed and never the value itself, so its output is safe to paste into a ticket.
| Exit | Means | Do |
|---|---|---|
| 0 | every surface was exercised, every value redacted | rotate |
| 1 | a credential is readable | redact; do NOT rotate yet |
| 2 | nothing exercised at least one surface — proves nothing | widen --window, or send a request through each |
| 64 | bad arguments | — |
Exit 2 is not a pass. A short post-rollout window can contain no Adobe webhook and no FastPass login at all, and a run over one would otherwise print a clean bill for filters nothing ever tested.
Troubleshooting
Caddy kept the old config. The mounted volume is only read at start, so a service that was updated but not rolled serves the unfiltered site block. Confirm the pod actually restarted, then re-run with --window 1h.
Nothing exercised them in the window — not the same as redacted, which is why it is not exit 0. Widen the window, or drive one request through each, before treating it as clean.
The filter encoder replaces the format, so a fields block without wrap json demotes the line to console format and breaks every saved OpenSearch query against message.
SSM and the job headers disagree. Both sides flip together or not at all; bun run easycron.probe separates a dead route from a rejected token.