Web Lambda near its concurrency cap
What to do when the onramp-<env>-web-near-cap alarm fires: name the client in the burst, then pull the lever that fits.
Canonical sources. The cap, its derivation and the alarm: infra/components/web-lambda.ts (webLambdaReservedConcurrency, NEAR_CAP_ALARM_FRACTION). Measured database and region room per environment: capacity on each row of infra/app/envs.ts. Pool size and overflow: web_lambda_pool_size and web_lambda_max_overflow in app/config/environments.yaml. The per-session WAF rule: infra/components/edge-policy.ts.
Why it fires
The cap is the database's spare connections divided by the connections one instance may hold, so a cap hit means requests are refused (the throttles alarm), not that the database is full. The alarm fires at 75% so there is room to act first.
A burst costs more than its request count. Each new instance is a SnapStart restore of about 755 ms, so a client fanning out in parallel makes many cold instances at once instead of reusing a few warm ones. On Nov 18 2026 one user's 145-request fan-out reached 132 concurrent instances against a cap of 205.
1 · Identify the client
Open the saved Logs Insights query onramp-<env>/web/requests-per-second-by-client (CloudWatch, Logs Insights, Queries) on the web log group and set the window to the 10 minutes around the alarm. It counts authz.route lines per second by principal, vendor and endpoint, busiest first.
The top row names the principal (principal_uuid), the vendor (vendor_id) and the endpoint. A single principal with dozens of requests in one second on one endpoint is a fan-out; many principals each at a few is organic load, so skip to the cap levers.
Resolve the vendor to a customer with the onramp-db skill (dev and stage) or prod-db-investigation (US and EU), read-only.
2 · Levers
Ordered by how quickly each takes effect. The first three act on the cause; the rest buy room.
Contact the customer, or flip the per-session WAF rule from counting to blocking: add "rate-per-session" to blocking in the environment's edges({ blocking: [...] }) in infra/app/envs.ts and ship it (#18050).
Tradeoff: the rule answers 429 to any session past 300 API requests a minute, so a legitimate heavy session is cut off too. Check the rule's WAF counts first. Needs a release.
Replace the many per-item calls with one endpoint that returns them all, and have the page call it once. #18055 did this for Project Settings: about 84 email-template calls became one.
Tradeoff: a code change in app and UI, with the single-item endpoints kept for other callers. Slowest lever, but the only one that removes the load.
Re-measure the environment's capacity in infra/app/envs.ts (database max_connections, peak other connections, region unreserved concurrency and peak) and apply it; the cap is derived in webLambdaReservedConcurrency. #18056 shows the measurements.
Tradeoff: the cap can never exceed the database's spare connections. Past that, the database refuses connections instead of Lambda refusing requests. Needs a release.
Set web_lambda_max_overflow to 1 in app/config/environments.yaml. One instance then holds fewer connections, so the same database room supports a higher cap.
Tradeoff: a request needing a third connection waits the pool timeout and fails. Watch the pool-ceiling alarm after the change.
Keep warm instances so a burst reuses them instead of paying SnapStart restores.
Tradeoff: cost. Covering 130 instances at 3,538 MB is about $4.8k a month while it is on. Treat it as a stopgap and remove it once a fix lands; it does not raise the cap.
When the cap would exceed the region's room (regionUnreservedConcurrency less the other functions' peak and the floor AWS keeps), ask for a higher account concurrency quota in Service Quotas, us-west-2, then raise regionUnreservedConcurrency in capacity.
Tradeoff: the request takes days, and the database still bounds the cap.
Verify
After a lever lands, the ConcurrentExecutions Maximum for the function stays under 75% of the cap through the next busy hour, and the alarm returns to OK. Rerun the saved query to confirm the top client's per-second count dropped.