Lambda-readiness acceptance
bun run lambda-readiness.smoke <stage|prod|gdpr-prod> proves in a real environment what unit tests with a simulated S3 cannot: that the documents bucket, its grants and its lifecycle exist as the Pulumi program declares, that signed download links, the Postgres rate limits, the polled edit lock and the AI-runtime registry behave, and that the infra leg of a release did not fail silently. It prints one PASS, FAIL or SKIP row per check with evidence and writes the run as a JSON record (--out <file> names it).
Running it
| Command | What it does |
|---|---|
bun run lambda-readiness.smoke stage | every read check |
bun run lambda-readiness.smoke stage --verify | adds the write checks, which create and remove data in the acceptance org (stage only) |
bun run lambda-readiness.smoke prod | the read checks; --verify is refused on prod and gdpr-prod |
--plan | lists the checks and what each would do, and calls nothing |
--only <text> | the checks whose id contains <text>; a comma-separated list selects the union |
--aws-reads | lists the AWS reads and makes only anonymous read requests; it takes no --verify |
--origin https://stage-web.onrampapps.com | sends the behavior checks to another origin of the same environment: the parity check for the stage web Lambda. Only an origin listed for the environment is accepted: https://us-web.onrampapps.com for prod and https://eu-web.onrampapps.com for gdpr-prod, read checks only |
--strict | exits non-zero when a check that could run in this mode reports SKIP, so a run with no credentials is not read as green |
infra.* checks read AWS and derive what to expect from the Pulumi program, so a new prefix on the documents role or a new SSM parameter is checked without editing the suite. behavior.* checks call the app's own API as the acceptance users.
One writer at a time
Two runs that write to the acceptance org overlap badly: the rate-limit rows count per user and address, and the edit-lock, export and fixture rows share data. So --verify and the Playwright journeys each take the org's run lock before they write, and refuse to start while another run holds it, printing who does. The lock is a note named PWE2E_ACCEPT_RUN_LOCK on the standing fixture project that carries the run id, the host and the start time; the run deletes it when it ends, however it ends. The check is not atomic (the app has no compare-and-set), so two runs started within a couple of seconds of each other can both pass; the app's own recording order decides otherwise. A lock older than an hour belongs to a run that died and is taken over. Read-only runs take none. Before the standing project exists the first run goes without a lock and says so.
After every deploy
verify-infra-applied.yml runs --only infra. for each environment after its infra apply: from stage-post-apply-checks.yml for stage, and from release-post-apply-checks.yml for prod and gdpr-prod, which runs after the release workflow finishes. It reads AWS under the environment's deploy role and alerts to the release's Slack channel when a check fails or cannot run, so a leg whose pulumi up failed under best_effort is not hidden by a green release. It is alert-only: no job needs it, so it never fails a release or gates a promotion. It selects only infra. checks, so it needs no acceptance credentials and runs no --verify, and it omits --strict because an infra. check skips only where an environment declares no such resource. It covers what the infra. checks cover: a resource whose check is still a pending(...) stub reports SKIP and is not watched until its pull request replaces the stub. The run record is the lambda-readiness-<env> artifact on the run.
One-time setup
- An
ONRAMP_ADMINcreates an org for the acceptance users on stage through the admin API from the browser console (POST /api/admin/orgs, thenPOST /api/admin/users). Give its portal a hostname outside.onramp.us, and the org thefile-reviewfeature. Create twoOWNERusers on plus-addresses, withsend_invite_email: true, and set each password from its invite email. - Each user goes on its own line of
~/.onramp-acceptance-stage, asemail:password, the way~/.pgpass-stageholds a connection string. Never paste it into chat, an env var or a commit. Without the file, the authenticated checks reportSKIPwith that reason.
Reading AWS without the AWS CLI
AWS reads go through the AWS CLI where one is configured, such as a CI job under OIDC. With none, bun run lambda-readiness.smoke <env> --aws-reads prints the reads as JSON; an agent runs each through the AWS connector's run_script (call_boto3 takes the service, operation, region and params as given), saves { "<key>": <response> } (an error as {"__error": {"Code": "…"}}), and reruns with --aws-replay <file>. Keys come from the listing.
Adding a check when its pull request merges
Each unmerged pull request has a stub in infra-checks.ts or behavior-checks.ts made by pending(...); it reports SKIP with the pull request number. The pull request that merges replaces its stub with a real check, marked read or write. A write check removes what it creates, names it with the PWE2E_ACCEPT prefix and never runs on prod or gdpr-prod. A check that reads from the Postgres counters or the bucket proves it through the app's API or AWS, never the database.
The two Playwright journeys that need a browser live in playwright/lambda-readiness/: bun run e2e lambda-readiness --target https://stage.onramp.us.
What a run does to the environment
--verify creates and removes data in the acceptance org: playbooks and exports it deletes, and reference documents it removes. The file-review limit counts per author for an hour, so a run that reaches it locks the acceptance user out of file review until the window ends. An export takes up to six minutes, because the job queue picks it up on its own cadence. The large-file checks delete their file records afterwards but leave two 5.5 MB objects per run in the documents bucket, because deleting a file removes only its record.
Spot check after a run
The script proves each check it has. It does not prove the checks are the right ones, or that a PASS means what its title says. After the final run of a release candidate, give the saved run records (--out files) to an independent reviewer that has not seen how the checks were written, and ask it to attack them: a PASS whose evidence would also hold if the feature were broken, a SKIP hiding a real gap, an expectation copied from the code it checks, and any row whose evidence leaks a signed link or credential. The reviewer reads the records and the check source and reports findings; it does not rerun writes. Treat its findings as inputs to the release decision, and fix a confirmed one in the suite before trusting the run.
Reading a run for a release decision
Run stage --verify --strict on the cluster origin and again with --origin https://stage-web.onrampapps.com. A check that passes on one and fails on the other is the point of the second run: the cluster has a shared disk and the web Lambda does not, so a SKIP on the cluster origin can hide a failure that only the Lambda origin shows. Treat a FAIL on either as blocking what it blocks: the Lambda origin blocks moving traffic to the Lambda, the cluster origin blocks the release.
--strict names every check that could run and did not. Read that list before trusting a green run. The standing gaps, none covered by any check yet: a spreadsheet export near the 512 MB /tmp limit, a customer portal custom domain routed to the Lambda, a webhook delivered after its response, and a direct Function URL call without the edge secret being refused. The run record's awsSource names the replay file and its hash, so a reader can tell a replayed read from a live one.
Validating the archive and storage changes on stage
The archive and storage changes are ordinary rows of the suite. Select them all with --only behavior.files-,behavior.task-file,behavior.editor-video,behavior.loose-upload,behavior.archive-,infra.scheduled-job-lambdas,infra.shared-disk and --verify for the write rows. The object-store reads in those rows need the AWS CLI; under --aws-replay they report SKIP.
| Group | Rows | Needs |
|---|---|---|
| Project files download | behavior.files-download-* | the acceptance org's fixture project; fails when the Files route streams the zip instead of redirecting to the bucket |
| Archive API | behavior.files-archive-api | the same fixture project; a project with a file on the shared disk reports SKIP (unavailable), never PASS |
| Task files and editor videos | behavior.task-file-*, behavior.editor-video-*, behavior.loose-upload-* | the acceptance org |
| Scheduled-job Lambdas | infra.scheduled-job-lambdas-serve-release | AWS reads only |
| Documents bucket lifecycle | infra.documents-bucket-lifecycle | AWS reads only; every rule declared in infra/components/documents-bucket.ts must be live, including the tagged expiry of unconfirmed uploads under task-files/, imports/ and help-media/, and a live rule the program does not declare fails |
| Shared-disk comparison and backfills | infra.shared-disk-verify, behavior.task-file-after-backfill | always SKIP with the command: they need a web pod, which the suite cannot reach, so someone with pod access runs them and records the counts |
Read the Lambda row after a deploy has finished. A stage deploy rolls the job functions one after another for several minutes and a newer merge cancels a running deploy, so a read taken mid-roll sees stragglers on the old package; run it again when the deploy's Verify the web-app Lambdas serve this release step has finished.
The pod commands, all read-only except the backfill:
| Command | Effect |
|---|---|
flask shared-disk-verify | compares every shared-disk file with the buckets and reports counts; writes nothing |
flask task-files-backfill --dry-run | counts the task files it would copy |
flask task-files-backfill | copies disk-only task files to task-files/ in the documents bucket; a stage write, rerunnable |
The cutover gate
bun run lambda-readiness.gate counts every site in app/api that breaks when Flask runs on a Lambda, in five classes: thread after the response, uploaded file body read in a route, file streamed from a route, disk folder and streamed response. It fails while any unmarked site remains, whatever scripts/lambda-readiness.allowlist.json says, and prints the counts per class and per file. That allow-list only stops the count growing; the gate is the cutover precondition.
A site that is safe on a Lambda says so on its own line: # lambda-safe: <reason>, for example an upload capped below 4 MB before the read. A marker with no reason fails. The gate lists every marked site with its reason, so a reviewer audits the exceptions in one place.
What proves each failure kind on the Lambda origin
Run every row on --origin https://stage-web.onrampapps.com as well as the cluster origin. A row may fail on the Lambda origin until its fix lands; that failure is what the cutover gate means.
| Failure kind | Check |
|---|---|
| Upload over 5 MB through a signed link | behavior.task-file-direct-upload |
| Editor or loose upload | behavior.editor-video-mirrored, behavior.loose-upload-refuses-other-types |
| Resource upload, then its thumbnail read back by a later request | behavior.resource-upload-thumbnail-readback |
| Main file download over 4.5 MB as a signed redirect | behavior.large-download-signed-redirect |
| After-response side effect that completes | behavior.resource-direct-upload-thumbnail-readback and behavior.resource-upload-thumbnail-readback: the thumbnail is made by the after-response outbox worker, so a thumbnail that appears proves the worker ran on that origin |
| Guide Me or AI run held open too long | not driven: a request-bound AI call is cut at 90 seconds (its marker in the cutover gate says so) and longer work runs in the worker, so no start-then-poll route exists to check |
| Edit-lock polling | behavior.playbook-edit-lock-polling |
| Import, public image, help-center media straight to the bucket on a signed grant | behavior.import-direct-upload, behavior.public-image-direct-upload, behavior.public-video-direct-upload, behavior.public-loose-file-direct-upload, behavior.help-media-direct-upload; each reports SKIP while its route answers 404 or the acceptance user may not use it (help-center media needs a platform-staff login) |
| Library resource upload and file replacement over 4.5 MB straight to the bucket, thumbnail read back | behavior.resource-direct-upload-large-file, behavior.resource-direct-replace, and behavior.resource-direct-upload-thumbnail-readback for a small image, because the thumbnail service only makes thumbnails for files up to 1 MB; SKIP while /api/v2/resource_upload_grants answers 404 |
| File-review and playbook-step authoring uploads | not driven: the confirm starts an AI review the suite cannot run; the grant protocol is the one behavior.task-file-direct-upload already proves |
| A multipart body over 3 MB refused before the route reads it | behavior.multipart-over-cap-refused for the three capped upload routes, and behavior.custom-thumbnail-over-cap-refused for a resource's thumbnail |
A query parameter starting with $ reaching the app | behavior.dollar-query-names-reach-the-app, a read check that runs on every origin |
| The organization app and portal pages served by the origin | behavior.ui-pages-are-served, a read check: both pages load and the script each names is served |
| Dynamic text responses compressed when the client accepts gzip or br | behavior.edge-compresses-text-responses, a read check: the sign-in page, a script and, with a sign-in, a JSON answer over 1.5 KB must carry a content encoding |
| A route guarded by the EasyCron secret turning away a wrong token | behavior.cron-route-rejects-a-wrong-token-with-401, a read check: one wrong-token request to a dry-run job route, rejected before the job runs |
| The host Flask sees being the one the visitor called | behavior.flask-sees-the-host-the-visitor-called, a read check: a lookup by hostname must answer on a web-Lambda origin as it does on the environment's own host, and a visitor-sent viewer-host header must change nothing |
| A key under another organization's folder refused on confirm | behavior.upload-confirm-refuses-other-tenants-key: requires invalid_key, and that a missing object in the caller's own folder answers upload_missing instead |