Skip to content

Lambda-readiness acceptance ​

bun run lambda-readiness.smoke <stage|prod|gdpr-prod> proves in a real environment what unit tests with a simulated S3 cannot: that the documents bucket, its grants and its lifecycle exist as the Pulumi program declares, that signed download links, the Postgres rate limits, the polled edit lock and the AI-runtime registry behave, and that the infra leg of a release did not fail silently. It prints one PASS, FAIL or SKIP row per check with evidence and writes the run as a JSON record (--out <file> names it).

Running it ​

CommandWhat it does
bun run lambda-readiness.smoke stageevery read check
bun run lambda-readiness.smoke stage --verifyadds the write checks, which create and remove data in the acceptance org (stage only)
bun run lambda-readiness.smoke prodthe read checks; --verify is refused on prod and gdpr-prod
--planlists the checks and what each would do, and calls nothing
--only <text>the checks whose id contains <text>; a comma-separated list selects the union
--aws-readslists the AWS reads and makes only anonymous read requests; it takes no --verify
--origin https://stage-web.onrampapps.comsends the behavior checks to another origin of the same environment: the parity check for the stage web Lambda. Only an origin listed for the environment is accepted: https://us-web.onrampapps.com for prod and https://eu-web.onrampapps.com for gdpr-prod, read checks only
--strictexits non-zero when a check that could run in this mode reports SKIP, so a run with no credentials is not read as green

infra.* checks read AWS and derive what to expect from the Pulumi program, so a new prefix on the documents role or a new SSM parameter is checked without editing the suite. behavior.* checks call the app's own API as the acceptance users.

One writer at a time ​

Two runs that write to the acceptance org overlap badly: the rate-limit rows count per user and address, and the edit-lock, export and fixture rows share data. So --verify and the Playwright journeys each take the org's run lock before they write, and refuse to start while another run holds it, printing who does. The lock is a note named PWE2E_ACCEPT_RUN_LOCK on the standing fixture project that carries the run id, the host and the start time; the run deletes it when it ends, however it ends. The check is not atomic (the app has no compare-and-set), so two runs started within a couple of seconds of each other can both pass; the app's own recording order decides otherwise. A lock older than an hour belongs to a run that died and is taken over. Read-only runs take none. Before the standing project exists the first run goes without a lock and says so.

After every deploy ​

verify-infra-applied.yml runs --only infra. for each environment after its infra apply: from stage-post-apply-checks.yml for stage, and from release-post-apply-checks.yml for prod and gdpr-prod, which runs after the release workflow finishes. It reads AWS under the environment's deploy role and alerts to the release's Slack channel when a check fails or cannot run, so a leg whose pulumi up failed under best_effort is not hidden by a green release. It is alert-only: no job needs it, so it never fails a release or gates a promotion. It selects only infra. checks, so it needs no acceptance credentials and runs no --verify, and it omits --strict because an infra. check skips only where an environment declares no such resource. It covers what the infra. checks cover: a resource whose check is still a pending(...) stub reports SKIP and is not watched until its pull request replaces the stub. The run record is the lambda-readiness-<env> artifact on the run.

One-time setup ​

  1. An ONRAMP_ADMIN creates an org for the acceptance users on stage through the admin API from the browser console (POST /api/admin/orgs, then POST /api/admin/users). Give its portal a hostname outside .onramp.us, and the org the file-review feature. Create two OWNER users on plus-addresses, with send_invite_email: true, and set each password from its invite email.
  2. Each user goes on its own line of ~/.onramp-acceptance-stage, as email:password, the way ~/.pgpass-stage holds a connection string. Never paste it into chat, an env var or a commit. Without the file, the authenticated checks report SKIP with that reason.

Reading AWS without the AWS CLI ​

AWS reads go through the AWS CLI where one is configured, such as a CI job under OIDC. With none, bun run lambda-readiness.smoke <env> --aws-reads prints the reads as JSON; an agent runs each through the AWS connector's run_script (call_boto3 takes the service, operation, region and params as given), saves { "<key>": <response> } (an error as {"__error": {"Code": "…"}}), and reruns with --aws-replay <file>. Keys come from the listing.

Adding a check when its pull request merges ​

Each unmerged pull request has a stub in infra-checks.ts or behavior-checks.ts made by pending(...); it reports SKIP with the pull request number. The pull request that merges replaces its stub with a real check, marked read or write. A write check removes what it creates, names it with the PWE2E_ACCEPT prefix and never runs on prod or gdpr-prod. A check that reads from the Postgres counters or the bucket proves it through the app's API or AWS, never the database.

The two Playwright journeys that need a browser live in playwright/lambda-readiness/: bun run e2e lambda-readiness --target https://stage.onramp.us.

What a run does to the environment ​

--verify creates and removes data in the acceptance org: playbooks and exports it deletes, and reference documents it removes. The file-review limit counts per author for an hour, so a run that reaches it locks the acceptance user out of file review until the window ends. An export takes up to six minutes, because the job queue picks it up on its own cadence. The large-file checks delete their file records afterwards but leave two 5.5 MB objects per run in the documents bucket, because deleting a file removes only its record.

Spot check after a run ​

The script proves each check it has. It does not prove the checks are the right ones, or that a PASS means what its title says. After the final run of a release candidate, give the saved run records (--out files) to an independent reviewer that has not seen how the checks were written, and ask it to attack them: a PASS whose evidence would also hold if the feature were broken, a SKIP hiding a real gap, an expectation copied from the code it checks, and any row whose evidence leaks a signed link or credential. The reviewer reads the records and the check source and reports findings; it does not rerun writes. Treat its findings as inputs to the release decision, and fix a confirmed one in the suite before trusting the run.

Reading a run for a release decision ​

Run stage --verify --strict on the cluster origin and again with --origin https://stage-web.onrampapps.com. A check that passes on one and fails on the other is the point of the second run: the cluster has a shared disk and the web Lambda does not, so a SKIP on the cluster origin can hide a failure that only the Lambda origin shows. Treat a FAIL on either as blocking what it blocks: the Lambda origin blocks moving traffic to the Lambda, the cluster origin blocks the release.

--strict names every check that could run and did not. Read that list before trusting a green run. The standing gaps, none covered by any check yet: a spreadsheet export near the 512 MB /tmp limit, a customer portal custom domain routed to the Lambda, a webhook delivered after its response, and a direct Function URL call without the edge secret being refused. The run record's awsSource names the replay file and its hash, so a reader can tell a replayed read from a live one.

Validating the archive and storage changes on stage ​

The archive and storage changes are ordinary rows of the suite. Select them all with --only behavior.files-,behavior.task-file,behavior.editor-video,behavior.loose-upload,behavior.archive-,infra.scheduled-job-lambdas,infra.shared-disk and --verify for the write rows. The object-store reads in those rows need the AWS CLI; under --aws-replay they report SKIP.

GroupRowsNeeds
Project files downloadbehavior.files-download-*the acceptance org's fixture project; fails when the Files route streams the zip instead of redirecting to the bucket
Archive APIbehavior.files-archive-apithe same fixture project; a project with a file on the shared disk reports SKIP (unavailable), never PASS
Task files and editor videosbehavior.task-file-*, behavior.editor-video-*, behavior.loose-upload-*the acceptance org
Scheduled-job Lambdasinfra.scheduled-job-lambdas-serve-releaseAWS reads only
Documents bucket lifecycleinfra.documents-bucket-lifecycleAWS reads only; every rule declared in infra/components/documents-bucket.ts must be live, including the tagged expiry of unconfirmed uploads under task-files/, imports/ and help-media/, and a live rule the program does not declare fails
Shared-disk comparison and backfillsinfra.shared-disk-verify, behavior.task-file-after-backfillalways SKIP with the command: they need a web pod, which the suite cannot reach, so someone with pod access runs them and records the counts

Read the Lambda row after a deploy has finished. A stage deploy rolls the job functions one after another for several minutes and a newer merge cancels a running deploy, so a read taken mid-roll sees stragglers on the old package; run it again when the deploy's Verify the web-app Lambdas serve this release step has finished.

The pod commands, all read-only except the backfill:

CommandEffect
flask shared-disk-verifycompares every shared-disk file with the buckets and reports counts; writes nothing
flask task-files-backfill --dry-runcounts the task files it would copy
flask task-files-backfillcopies disk-only task files to task-files/ in the documents bucket; a stage write, rerunnable

The cutover gate ​

bun run lambda-readiness.gate counts every site in app/api that breaks when Flask runs on a Lambda, in five classes: thread after the response, uploaded file body read in a route, file streamed from a route, disk folder and streamed response. It fails while any unmarked site remains, whatever scripts/lambda-readiness.allowlist.json says, and prints the counts per class and per file. That allow-list only stops the count growing; the gate is the cutover precondition.

A site that is safe on a Lambda says so on its own line: # lambda-safe: <reason>, for example an upload capped below 4 MB before the read. A marker with no reason fails. The gate lists every marked site with its reason, so a reviewer audits the exceptions in one place.

What proves each failure kind on the Lambda origin ​

Run every row on --origin https://stage-web.onrampapps.com as well as the cluster origin. A row may fail on the Lambda origin until its fix lands; that failure is what the cutover gate means.

Failure kindCheck
Upload over 5 MB through a signed linkbehavior.task-file-direct-upload
Editor or loose uploadbehavior.editor-video-mirrored, behavior.loose-upload-refuses-other-types
Resource upload, then its thumbnail read back by a later requestbehavior.resource-upload-thumbnail-readback
Main file download over 4.5 MB as a signed redirectbehavior.large-download-signed-redirect
After-response side effect that completesbehavior.resource-direct-upload-thumbnail-readback and behavior.resource-upload-thumbnail-readback: the thumbnail is made by the after-response outbox worker, so a thumbnail that appears proves the worker ran on that origin
Guide Me or AI run held open too longnot driven: a request-bound AI call is cut at 90 seconds (its marker in the cutover gate says so) and longer work runs in the worker, so no start-then-poll route exists to check
Edit-lock pollingbehavior.playbook-edit-lock-polling
Import, public image, help-center media straight to the bucket on a signed grantbehavior.import-direct-upload, behavior.public-image-direct-upload, behavior.public-video-direct-upload, behavior.public-loose-file-direct-upload, behavior.help-media-direct-upload; each reports SKIP while its route answers 404 or the acceptance user may not use it (help-center media needs a platform-staff login)
Library resource upload and file replacement over 4.5 MB straight to the bucket, thumbnail read backbehavior.resource-direct-upload-large-file, behavior.resource-direct-replace, and behavior.resource-direct-upload-thumbnail-readback for a small image, because the thumbnail service only makes thumbnails for files up to 1 MB; SKIP while /api/v2/resource_upload_grants answers 404
File-review and playbook-step authoring uploadsnot driven: the confirm starts an AI review the suite cannot run; the grant protocol is the one behavior.task-file-direct-upload already proves
A multipart body over 3 MB refused before the route reads itbehavior.multipart-over-cap-refused for the three capped upload routes, and behavior.custom-thumbnail-over-cap-refused for a resource's thumbnail
A query parameter starting with $ reaching the appbehavior.dollar-query-names-reach-the-app, a read check that runs on every origin
The organization app and portal pages served by the originbehavior.ui-pages-are-served, a read check: both pages load and the script each names is served
Dynamic text responses compressed when the client accepts gzip or brbehavior.edge-compresses-text-responses, a read check: the sign-in page, a script and, with a sign-in, a JSON answer over 1.5 KB must carry a content encoding
A route guarded by the EasyCron secret turning away a wrong tokenbehavior.cron-route-rejects-a-wrong-token-with-401, a read check: one wrong-token request to a dry-run job route, rejected before the job runs
The host Flask sees being the one the visitor calledbehavior.flask-sees-the-host-the-visitor-called, a read check: a lookup by hostname must answer on a web-Lambda origin as it does on the environment's own host, and a visitor-sent viewer-host header must change nothing
A key under another organization's folder refused on confirmbehavior.upload-confirm-refuses-other-tenants-key: requires invalid_key, and that a missing object in the caller's own folder answers upload_missing instead

Internal documentation — gated behind Cloudflare Access.