Playwright E2E on GitHub Actions — Performance, Stability & Security Playbook

Why this lives here. A consolidated reference spec for keeping the Playwright E2E suite fast, stable, and secure in CI. Captured during the golden-seed E2E effort (PR #8980, see E2E Golden Seed) so future work — adding visual-regression tests, re-sharding, tuning the matrix — can spot-check against a known-good baseline instead of rediscovering tradeoffs.

How it maps to OnRamp today (verify before citing — these drift):

• Workflow: .github/workflows/e2e.yml — per-suite matrix + dorny path-filters, e2e-result is the required gate. (The playbook below shows shard-based matrices; OnRamp shards by suite/domain, not --shard=i/n — a deliberate choice, see "Deliberately NOT adopted" below.)
• CI serves a built bundle (PREVIEW_MODE=1 bun run dev), not the dev server — matches §4's "serve a built frontend" guidance.
• Seed: golden DB (devtools/db/, make db.seed) — OnRamp's answer to §3c "seed via API/fixtures, not UI". A reproducible secret-free Postgres seed, not network stubbing.
• Adopted (this PR): --only-shell install + browser-binary cache keyed on PW version (§1); storageState auth reuse via a setup project — one login per suite, fresh-context-per-test (§3a); permissions: contents: read least-privilege token (§5). The OnRamp e2e.yml uses bun, not the Node/npm ci shown below — translate commands.
• Deliberately NOT adopted (would regress this architecture — the playbook's own "apply selectively" caveat):
  • --shard=i/n (§2) — each matrix job's cost is dominated by uv sync + make db.seed
    • bun run build + stack boot, not test execution. Finer sharding multiplies that fixed setup cost. The per-suite/domain matrix already parallelizes; that IS our shard axis.
  • --only-changed (§3b) — redundant with, and riskier than, the dorny path-filter suite selection already in detect. Static-import analysis misses backend-driven UI; the path filters are the better fit here.
  • trace: 'on-first-retry' (§3d) — incompatible with our retries: 0: with no retry, it never captures. We use trace: 'retain-on-failure'.
  • blob + merge-reports (§7) — a reporting-DX convenience that mainly earns its complexity alongside --shard; without sharding, per-suite reports suffice.
• CI gotchas that override naive playbook application: backend qa_mode = environ.get("CI") hides the org login form, so CI starts the app with env -u CI; schema-baseline is pinned to the branch migration head. See memory project_ci_qa_mode_gotcha.
• OnRamp policy override — ZERO RETRIES. §4 and §6 below recommend retries: 2 to absorb transient flake. OnRamp deliberately overrides this to retries: 0 (playwright/shared/config.base.ts): a retry masks nondeterminism, and a nondeterministic test is a bug, not a transient. The gate must go red so the race gets fixed at the source. Flaky tests are not permissible — a known offender is quarantined (test.fixme + tracking issue) off the gate until rewritten to pass 100% on retries: 0 (--repeat-each=20 green). Policy lives in the playwright-testing skill ("No flaky tests — ZERO RETRIES"); first quarantine is P1-046 (#8995). Read §4's "retries masks flakiness" note as the rule, and its retries: 2 config as not OnRamp's setting.

Purpose: A consolidated, self-contained spec for making an existing Playwright E2E suite fast, stable, and secure when run in CI. Hand this to an implementing agent. Apply selectively — tradeoffs and caveats are noted inline so nothing here gets cargo-culted.

Assumptions / scope

• Runner: GitHub Actions, Linux (ubuntu-latest), headless.
• Stack under test: Vite/Vue frontend + an API backend over Postgres (backend-agnostic — none of this depends on the specific DB/host).
• Goal is the CI execution path, not local dev ergonomics and not agent/MCP token usage.
• Multi-worker, sharded execution is in scope.

---

TL;DR — do these first (highest leverage)

1. Reuse auth via storageState (setup project + dependencies). Usually the single biggest wall-clock win — turns per-test login (5–15s each) into a one-time cost. Treat the auth file as a secret (see Security).
2. Shard across runners with a GitHub Actions matrix + blob reporter + a merge-reports job. Combine with workers > 1 per shard and fullyParallel: true.
3. Stub the backend / seed via API instead of driving real network + UI for setup. This is the real attack on "browser I/O dominates" — the browser binary is a small slice; what the page does over the network is the mass.
4. Run changed-only on PRs, full suite on merge/nightly (--only-changed).
5. Record artifacts only on failure (trace: 'on-first-retry', video: 'retain-on-failure', screenshot: 'only-on-failure').
6. Cache browser binaries keyed on Playwright version, and install the headless shell only on CI (--only-shell).

Everything below expands these and adds the stability + security layers.

---

1. Browser choice (the "lighter Chrome" question, settled)

• Playwright already ships and auto-selects a lightweight chromium-headless-shell for headless runs (no UI, no X11/Wayland/D-Bus dependency, smaller footprint). When no channel is set, that shell is what you get on CI. That is the fast/light browser — there is nothing lighter worth switching to for full-fidelity E2E.
• The heavier option is opt-in: channel: 'chromium' | 'chrome' | 'msedge' uses the real Chrome "new headless," which is more faithful but slightly slower per Playwright's own testing. Only use it if you need real-Chrome rendering fidelity (visual regression, fingerprint-sensitive flows, Chrome-only APIs).
• Action: audit config for a stray channel override and remove it unless justified.
• CI install — shell only (skips downloading the full headed Chromium you never use):

  npx playwright install --with-deps --only-shell

  --only-shell and channel: 'chromium'/'chrome' are mutually exclusive — the latter needs the full browser. Pick one.

• Do not swap browser engines for speed: WebKit/Firefox are a coverage decision, not a perf one, and can be flakier. Lightpanda (ultralight Zig CDP browser) is genuinely lighter but is Beta, has incomplete Playwright feature-detection and network interception, and is built for scraping/agents — not suitable for high-fidelity app E2E today. Revisit later.

---

2. Parallelism & sharding

Workers (within one machine)

// playwright.config.ts
export default defineConfig({
  workers: process.env.CI ? 4 : undefined, // size to runner CPU; 4 is a starting point, not a law
  fullyParallel: true,                      // parallelize at the test level, not just the file level
});

• Each worker launches its own browser; over-provisioning causes CPU/memory contention and OOM. Match workers to the runner's core count.
• fullyParallel: true assumes strict test isolation. Tests that share state or depend on order will flake. Fix isolation first (see Stability).

Sharding (across machines)

• For large suites, split across runners with --shard=<i>/<n>. Rough scaling from real reports: a 100-test/30s suite ≈ 50 min on 1 worker → ~12–15 min at 4 workers → ~7–10 min across 4 shards. Some teams report 60 min → ~8 min.
• Caveat: Playwright's built-in sharding splits by lexical file-path order, so shards can be unbalanced and a slow shard becomes the bottleneck. For big suites, consider dynamic/timing-based sharding (compute shard count from test count, or balance by historical durations) rather than just adding more shards.
• Sharding requires the blob reporter + a downstream merge-reports job (see §7). Without blob, you only keep the last shard's results.

---

3. Cut redundant work (where the minutes actually are)

3a. Persistent auth — storageState

Log in once in a setup project, save state, and load it everywhere. This is typically the #1 win.

// auth.setup.ts
import { test as setup } from '@playwright/test';
const authFile = 'playwright/.auth/user.json';
setup('authenticate', async ({ page }) => {
  await page.goto(process.env.BASE_URL + '/login');
  await page.fill('#email', process.env.E2E_USER!);
  await page.fill('#password', process.env.E2E_PASSWORD!);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard');
  await page.context().storageState({ path: authFile });
});

// playwright.config.ts (excerpt)
projects: [
  { name: 'setup', testMatch: /.*\.setup\.ts/ },
  {
    name: 'chromium',
    use: { ...devices['Desktop Chrome'], storageState: 'playwright/.auth/user.json' },
    dependencies: ['setup'],
  },
],

• Supports multiple roles (separate setup files for admin/editor/viewer).
• sessionStorage is NOT captured by storageState — restore manually via addInitScript if the app needs it.
• Auth state expires; regenerate when auth-related failures appear.
• Keep a few dedicated login/permission tests so you still actually exercise the auth flow you're now bypassing.
• SECURITY: the auth file holds live session cookies/tokens. Gitignore playwright/.auth/, generate it at runtime in CI, never upload it as an artifact. (See Security.)

3b. Test-impact selection — --only-changed (Playwright ≥ 1.46)

# PRs: only tests affected by changes vs the base branch
npx playwright test --only-changed=origin/${{ github.base_ref }}

• Selection is based on static import analysis only — it does not track runtime/dynamic imports, env-driven behavior, config changes, or backend-driven UI. Monorepos and heavily abstracted setups will miss tests.
• Use the hybrid pattern: changed-only on PRs for fast feedback, full suite on merge + nightly to catch what static analysis misses.

3c. Reduce what the page does (the real "browser I/O" lever)

• Network stubbing — serve canned responses so tests don't wait on the real backend, which is faster and removes a class of flakiness:

  await page.route('**/api/**', route =>
    route.fulfill({ json: { /* fixture */ } }));
  // or replay a recorded session:
  await page.routeFromHAR('fixtures/api.har', { url: '**/api/**', update: false });

• Seed state via API, not UI — use the request fixture / apiRequestContext to set up data through direct API calls instead of clicking through forms.
• Block unneeded resources at the fixture/project level (not per test):

  await page.route('**/*', route => {
    const type = route.request().resourceType();
    if (['image', 'font', 'media'].includes(type)) return route.abort();
    return route.continue();
  });

  Also safe to block analytics/ads/tracking domains. Riskier: CSS (layout/visibility checks) — block conservatively. Disable resource blocking for visual-regression tests (run those as a separate project).
• Navigation: prefer waitUntil: 'domcontentloaded' over the default 'load' where you don't need every sub-resource; avoid redundant goto/navigations; in SPAs wait for a state signal (URL/element) rather than minimizing goto for its own sake.

3d. Artifact overhead & fail-fast

use: {
  trace: 'on-first-retry',     // not 'on' — full traces every test is a constant tax
  video: 'retain-on-failure',
  screenshot: 'only-on-failure',
},
maxFailures: process.env.CI ? 10 : undefined, // fast feedback on PRs; skip if suite is flaky

• maxFailures (fail-fast) is great for stable suites; risky with flaky tests (a transient failure aborts the run). Stabilize first.

---

4. Stability (a fast suite that flakes is worse than a slow one)

Config:

export default defineConfig({
  forbidOnly: !!process.env.CI,        // fail the build if a stray test.only is committed
  retries: process.env.CI ? 2 : 0,     // absorb transient flake — but see warning below
  fullyParallel: true,
  reporter: process.env.CI ? 'blob' : 'html',
  use: {
    actionTimeout: 15_000,
    navigationTimeout: 30_000,
    reducedMotion: 'reduce',           // kill animation-timing flakiness
    testIdAttribute: 'data-testid',
  },
  expect: { timeout: 7_000 },
  timeout: 60_000,
  // Start the app and WAIT for readiness so tests don't race the server:
  webServer: {
    command: 'npm run preview',        // serve a built frontend (or point baseURL at a deployed preview)
    url: process.env.BASE_URL,
    reuseExistingServer: !process.env.CI,
    timeout: 120_000,
  },
});

Practices:

• retries masks flakiness — it doesn't fix it. Track the flaky-test rate and fix chronic offenders; don't let retries hide a degrading suite.
• Never use waitForTimeout/hard sleeps. Rely on auto-waiting + web-first assertions (await expect(locator).toBeVisible(), toHaveText, etc.).
• Locators: prefer role-based (getByRole) and data-testid; avoid brittle CSS/XPath that breaks on DOM churn.
• Test isolation: each test gets a fresh context by default — keep it that way. No shared mutable state, no order dependence (this is the precondition for fullyParallel/sharding).
• Time-dependent tests: use page.clock to control time instead of waiting in real time.
• Pin the Playwright version; keep playwright.config.ts and the workflow in-repo and review them in PRs like any other code.
• Watch headless-shell vs headed rendering differences (fonts/GPU/animation timing). If a test only passes headed, that's a real behavioral difference — fix it (often via reducedMotion/fixed viewport), don't just flip modes.

---

5. Security (the part most "speed" guides skip)

• Secrets via GitHub Actions Secrets / Environments — never in the repo. Test-login creds, API keys → workflow secrets (e.g. secrets.E2E_USER, referenced with the ${...}-{...} Actions syntax), injected as env at run time.
• storageState auth files are secrets. They contain live session cookies/tokens. Gitignore playwright/.auth/, generate at runtime in CI, and do not upload them as artifacts.
• Traces / HAR / videos can leak data. They capture request/response headers, cookies, bearer tokens, and PII. Mitigate:
  • *-on-failure already limits volume.
  • Short artifact retention-days; restrict who can download.
  • Sanitize/scrub sensitive headers in fixtures/HARs; avoid recording traces against real sensitive data.
• Forked-PR / pull_request_target footgun: never expose secrets to untrusted forks. Use the pull_request event (secrets withheld from forks) for external contributions, and gate any secret-requiring E2E behind maintainer approval / environment protection rules / workflow_run. (Low risk for an internal private repo, but keep the guardrail if the repo ever opens up.)
• Least-privilege token: set a permissions: block (e.g. contents: read); don't run with the default broad GITHUB_TOKEN scope unless a step needs it.
• Test against a dedicated non-prod environment with synthetic/seeded data. Don't point E2E at production with real customer data.
• --no-sandbox is a conscious tradeoff. It disables the Chromium sandbox. Acceptable in ephemeral CI runners (the runner is the isolation boundary); don't blindly carry it into long-lived/shared environments. Prefer running as non-root in the official image where possible.
• Supply chain: npm ci with a committed lockfile (deterministic installs); pin action versions (ideally by commit SHA for third-party actions); pin Playwright + (if used) the Docker image by digest; enable Dependabot.
• Don't log secrets. GitHub masks registered secrets, but avoid console.log-ing env/credentials in tests or steps.

---

6. Recommended playwright.config.ts (consolidated)

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './e2e',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 4 : undefined,
  reporter: process.env.CI ? 'blob' : 'html',
  timeout: 60_000,
  expect: { timeout: 7_000 },
  use: {
    baseURL: process.env.BASE_URL,
    trace: 'on-first-retry',
    video: 'retain-on-failure',
    screenshot: 'only-on-failure',
    actionTimeout: 15_000,
    navigationTimeout: 30_000,
    reducedMotion: 'reduce',
    testIdAttribute: 'data-testid',
    // no `channel` → uses the fast headless shell
  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'], storageState: 'playwright/.auth/user.json' },
      dependencies: ['setup'],
    },
    // Optional separate project for visual tests (no resource blocking; consider channel:'chromium' for fidelity)
  ],
  webServer: {
    command: 'npm run preview',
    url: process.env.BASE_URL,
    reuseExistingServer: !process.env.CI,
    timeout: 120_000,
  },
});

---

7. GitHub Actions workflow (sharded, cached, merged) — current action versions

name: e2e

on:
  push:
    branches: [main]
  pull_request:

# Cancel superseded runs on the same ref → saves CI minutes
concurrency:
  group: e2e-${{ github.ref }}
  cancel-in-progress: true

# Least-privilege token
permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false            # one shard failing must NOT cancel the others — you want full results
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Cache Playwright browsers
        id: pw-cache
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          # Keyed on the lockfile; bumping Playwright re-resolves and invalidates automatically.
          # For tighter keying, hash the installed @playwright/test version instead.
          key: pw-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

      - name: Install browsers + OS deps (cache miss)
        if: steps.pw-cache.outputs.cache-hit != 'true'
        run: npx playwright install --with-deps --only-shell

      - name: Install OS deps only (cache hit — apt libs aren't cacheable)
        if: steps.pw-cache.outputs.cache-hit == 'true'
        run: npx playwright install-deps

      - name: Run E2E
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} \
              --only-changed=origin/${{ github.base_ref }}
          else
            npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
          fi
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          E2E_USER: ${{ secrets.E2E_USER }}
          E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}

      - name: Upload blob report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report/
          retention-days: 7         # keep short — blobs/traces can contain sensitive data

  merge-reports:
    if: ${{ !cancelled() }}
    needs: [test]
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Download all blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true       # `merge-multiple` requires download-artifact@v4
      - name: Merge into HTML report
        run: npx playwright merge-reports --reporter=html ./all-blob-reports
      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

Notes for the implementing agent:

• Alternative to caching: the official mcr.microsoft.com/playwright:<version>-noble Docker image ships browsers + OS libs preinstalled — pin it to your Playwright version (and digest) to skip the install/cache dance entirely. Trade: image pull vs. cache restore.
• Browser-binary caching typically saves ~2–3 min per job.
• --only-changed + --shard compose fine; some shards may end up empty on small change sets — that's handled gracefully.
• If you add cross-browser coverage, the matrix becomes shards × browsers (e.g. 4 × 3 = 12 jobs) — only do this if you actually need WebKit/Firefox coverage; it multiplies cost.
• Add a deploy job with needs: [merge-reports] (or [test]) and GitHub environment protection rules if you want E2E to gate deploys.

---

8. What to skip (myths / low-yield — don't let the agent waste time here)

• Bun instead of Node for execution. Browser I/O dominates, so the JS runtime barely moves test wall-clock; Playwright isn't natively Bun-compatible (Node-API browser launch, tracing/network-interception gaps, documented crashes), and the playwright CLI's node shebang means bun run silently delegates to Node anyway. (bun install as a faster package manager is fine, but that's it.)
• Swapping to a different browser engine for speed (WebKit/Firefox) — that's coverage, not performance.
• Lightpanda / ultralight browsers for E2E now — Beta, fidelity/feature gaps; fine to watch, not to bet CI on.
• --headless toggling as a perf lever — Playwright is already headless by default in CI; only relevant if someone overrode it, and the gain is ~10–30% at most.
• Chromium launch-flag soup — mostly placebo on the headless shell. The one worth setting in containers is --disable-dev-shm-usage (a stability fix for small /dev/shm, not a speed knob).
• Removing waitForTimeout "for speed" — do it, but it's a flakiness fix that happens to help speed, not a perf optimization per se.

---

9. Ongoing (so it doesn't silently rot)

Track over time and flag regressions before they compound:

• Average test duration (drift from 12s → 18s = the suite is gaining weight).
• Flakiness rate (reruns waste CI and erode trust).
• CI resource usage (per-test CPU/memory — multiplied by worker count, this is what slows pipelines).

Optimization is continuous, not one-time. Pick one lever, measure, repeat.

---

10. Open decisions for the implementing agent

1. Shard count — set to match the CI runner CPU and total suite size; start at 4 and tune by observed per-shard balance.
2. Visual tests — if any exist, split them into a project with resource-blocking off and decide whether they need channel: 'chromium' (real-Chrome fidelity) vs the shell.
3. Test environment — confirm the target: built static frontend + stubbed API (fastest/most isolated) vs. a deployed preview/staging env with seeded synthetic data. Drives webServer vs. external baseURL, and the network-stubbing strategy.
4. Cross-browser — confirm whether WebKit/Firefox coverage is actually required before multiplying the matrix.