Alert Autopilot

Turn a production error in Sentry into a draft PR, review-ready, with no human interaction until approval.

Owner: PlatformWhen to use: Production issues, Sentry first-seen

Canonical sources. The ingest resolver lives in infra/components/issue-autopilot/; the cap, settle gate, and sweeper in devtools/autopilot/; the workflow entry point in .github/workflows/autopilot.yml; and the dispatch job in .github/workflows/ai-review.yml. This runbook is the HOWTO — for cap removal and the six manual setup gaps this PR cannot fill.

TL;DR

🔗 The Pipeline

Sentry webhook fires → Lambda verifies and filters → GitHub issue created or updated → Autopilot workflow runs settle gate → Issue labeled and cap checked → Fixer drafts PR → Swarm reviews → Dispatch wakes autopilot → Three rounds of fixes, then human escalation.

📋 Labels

autopilot — drainable by the fixer. autopilot:queued — overflow, wait for the sweeper. autopilot:needs-human — fixer declined or spent three fix rounds. autopilot:noise — the fix stops an expected condition from being reported as an error. autopilot:decline:* — the reason category on a decline, named for the missing capability. env:* — environment labels: the tag names one of the prod tiers (prod, gdpr-prod, demo), matched case-insensitively; the label is the lowercase spelling.

⚙️ Daily Cap

Five PRs per day, UTC midnight boundary. Sixth issue gets autopilot:queued; the 30-minute sweeper drains it when room opens. Delete devtools/autopilot/cap.ts and remove the underCap call in devtools/autopilot/run.ts to remove the cap entirely.

The Pipeline

Sentry issue webhooks (created or unresolved) reach a Lambda at reserved concurrency one. That one slot is also what an unauthenticated request to the public Function URL occupies until the signature check rejects it, so a burst of junk traffic can make Sentry's delivery fail; a dropped delivery is not lost, because the daily reconcile pass re-reads the last 25 hours of issues from Sentry and files anything the webhook missed. The Lambda verifies the HMAC signature, resolves the Sentry project and environment, and checks whether the issue already links a GitHub issue. If it does, the issue is updated; otherwise, a new one is created with a pointer back to Sentry and labelled autopilot plus environment labels. No event payload leaves Sentry, no model is called, and AWS holds no state.

The GitHub issue then wakes the autopilot workflow. The settle gate reads the head's required checks and the review swarm's status and makes one of four decisions before any model call: unsettled (exit and wait), settled red (one fixer session), settled green with bot-approved (run promote), or already-handled (stale duplicate, exit). A fixer session reads the full Sentry event, performs triage (defect or decline), and either drafts a fix on a branch or labels the issue autopilot:needs-human with a decline reason as a comment. If a fix is drafted, it is pushed under the autopilot App's token so CI and the review swarm fire. When the swarm finishes, it emits a gating verdict and wakes the autopilot workflow again via a typed workflow_dispatch call. Up to three fix rounds are followed; on the fourth, if the PR has not settled green with bot-approved, it is labelled autopilot:needs-human and all spending stops. The workflow then waits for a human to approve, and promotes when they do.

Every autopilot run — a label wake, a dispatch wake, or the scheduled sweep — shares one concurrency group, so at most one runs at a time. A wake that arrives while another run is in flight queues behind it at no cost; but GitHub keeps at most one pending run per group, so a wake that arrives while one is already queued is dropped rather than piling up. That is why the scheduled sweep is not optional: every 30 minutes it enumerates every open autopilot issue that has neither a fix-PR marker comment nor autopilot:needs-human, oldest first, and drains as many as the daily cap allows. A dropped wake is harmless — the issue was never going to sit unfixed, only wait for the next sweep.

Labels and Meanings

autopilot

Applied by the ingest Lambda to an issue filed from a listed Sentry project with at least one prod-tier environment tag. This label is the wake event for the workflow. The listed projects are: api-onramp, ui-organization, ui-customer-portal. Unlisted projects and non-prod environments are dropped and counted. When customer-api moves into this repo, it will be a one-line addition to the list.

autopilot:queued

An issue labelled autopilot arrived but the daily cap was already full at that moment. This label is only the visible hint that a deferral happened — it is not what the sweep uses to decide what to drain. The 30-minute scheduled sweep (cron: "*/30 * * * *") re-reads the cap and enumerates every open autopilot issue that has neither a fix-PR marker comment nor autopilot:needs-human, oldest first by creation time, and drains as many as the cap allows — whether or not that issue ever carried autopilot:queued. The label is removed once its issue is drained, or re-applied on a later sweep if the cap is still full.

autopilot:needs-human

Applied when the fixer declines an issue (deeming it not a fixable defect in this repo), or after three fix rounds on a PR without reaching bot-approved + green. At this point, the workflow stops spending tokens and waits for a human approver. The label signals that no further automated fix attempts will happen.

autopilot:noise

Applied to a fix PR alongside autopilot, never in place of it, when the fixer's triage call was noise: the fix stops an expected condition from being reported as an error, rather than making broken code work. Ordinary review applies the same as any other autopilot PR — the label is a countable annotation, not a different code path. See "How the fixer decides" below for the call itself.

autopilot:decline: category labels

Applied alongside autopilot:needs-human whenever the fixer's own triage declines an issue — never for a round-cap hand-off, which carries no category. Each names the capability the session lacked, so the same five buckets count exactly what stands between the fleet and the "every Sentry defect fixed by the agent" goal. A session whose final message names no recognized decision or category at all is not a decline this file can invent a category for — that case retries as a failed drain (see parseTriageOutcome in devtools/autopilot/fixer.ts) up to the same three-attempt bound as any other failed drain, rather than landing here.

• autopilot:decline:needs-production-data — the fix needs tenant or production data the session cannot see. Closes when a human reproduces or verifies with that data and lands the fix, or gives the session a way to see it.
• autopilot:decline:needs-vendor-visibility — the fix needs visibility into a third-party API's behaviour, or an outage, the session cannot inspect. Closes when a human confirms the vendor-side cause, or the vendor issue resolves.
• autopilot:decline:needs-customer-action — the cause is a customer's own credential or configuration. What closes the category: a channel from the autopilot to CS or the customer, the notification block of the wider design.
• autopilot:decline:needs-dependency-fix — the error is entirely inside a dependency. What closes the category: a way for the fixer to upgrade a dependency or file the fix upstream.
• autopilot:decline:needs-product-decision — more than one plausible fix exists. What closes the category: the product-owner-in-the-loop block of the wider design.

Environment labels: env:prod, env:gdpr-prod, env:demo

Every Sentry issue that enters the ingest Lambda is tagged with all the production-tier environments it has been seen in. The environment tag names one of the prod tiers (prod, gdpr-prod, demo), matched case-insensitively; the label is the lowercase spelling. A regression (same defect in US and EU) carries both env:prod and env:gdpr-prod. Non-prod environments (dev, stage) are dropped and counted but not labelled.

The Daily Cap: Five PRs Per Day

The cap is defined as DAILY_CAP = 5 in devtools/autopilot/cap.ts. Every 30 minutes, and before a fixer session is started for an issue-wake, the cap gate runs a GitHub search for PRs created today (UTC calendar day) that carry the autopilot label. If the count is less than five, the issue is drained immediately; otherwise it is labelled autopilot:queued and the fixer does not run.

The UTC day boundary is YYYY-MM-DD from new Date().toISOString(), so midnight UTC is the exact moment the counter resets. The count is drawn from GitHub's own PR metadata (creation timestamp), not from a counter this tool maintains, so there is no state to reconcile and a transient search failure fails closed to "over cap" rather than guessing.

The 30-minute sweep (devtools/autopilot/schedule.ts) additionally never starts more than one drain of its own per sweep, even when the daily cap has headroom for more: the cap's GitHub search can lag a PR that same sweep's first drain just opened, so a re-read immediately after cannot be trusted to admit a second one. This is a separate limit from the daily cap itself — a sweep that stops after one drain reports one drain per sweep already spent, not under cap exhausted, and leaves the remaining issues unlabelled (autopilot:queued means specifically that the daily cap deferred them). In practice a second drain in one sweep is rare regardless: the sweep budgets itself to 25 minutes against the fix job's 30-minute timeout, and one drain alone can spend most of that.

Removing the Cap

To remove the five-a-day ceiling entirely, delete two things:

1. The file devtools/autopilot/cap.ts
2. The call to underCap(repo) in devtools/autopilot/run.ts (in the handleIssueWake function)

Nothing else in the codebase references the cap. The deletion is a one-line removal from one function and one file deleted entirely.

Tuning the Cap by PR

To change the cap from five to a different number without removing it altogether, edit devtools/autopilot/cap.ts and change the line:

export const DAILY_CAP = 5;

to your desired number. The change is immediate; no other code needs to change, and the next sweep will read the new value. Every deferred issue is drained in order when the next sweep runs (oldest first by creation time) — whether or not it ever carried autopilot:queued, since that label is a hint to a human, not the sweep's own selection key.

The Fixer's Triage Decision

Every issue labelled autopilot is drained by the fixer, which performs triage as its first step: is this a code defect in this repo that a bounded, minimal change can fix, an expected condition wearing an exception costume, or neither? The fixer reads the Sentry event and makes one of three calls:

• Defect: A code bug with an in-repo stack frame that a fix here can address. The fixer writes a minimal change plus a targeted test and opens a PR for review.
• Noise: Not a bug — an expected condition (a lookup that raises on "not found", a duplicate write, a handled failure logged at error level while the request still succeeds) reported as an error anyway. The fixer writes a minimal change that stops it being captured as an error, and opens a PR for review carrying autopilot:noise alongside autopilot.
• Decline: An upstream issue (vendor outage, timeout), no in-repo frame, or a fix that needs production data. The fixer adds the autopilot:needs-human label plus one autopilot:decline: category label (see Labels and Meanings above) and posts a comment led by that category, then the decline reason. No branch, no PR, no cap consumed.

The fixer's choice is its own verdict, not a flag in the ingest: every prod-tier issue from a listed project reaches the fixer unfiltered.

How the fixer decides

Noise is looked for first, ahead of a systemic defect: it is the smaller, safer PR — stop raising or logging one condition, rather than change the behaviour that produces it — and it is what reclaims ingest capacity, so it is worth more per PR than a rare real bug. Only once noise is ruled out does the fixer weigh a systemic defect (growing userCount, a regression, a release-correlated spike) against a decline.

The call is read off the same Sentry issue object every time, before the stack trace: count, userCount, firstSeen/lastSeen, level, substatus/isUnhandled, and the environments the issue has been seen in. A high, sustained count with userCount at or near zero is the noise signature; a non-zero, growing userCount with a recent or regressed firstSeen is the real-bug signature. The fixer traces the culprit to its real source before committing to either call — a wrong call costs a paid review cycle and a human's attention.

The call and the reasoning behind it are not left to the PR title: every fix PR's body carries a ## Verdict section — Call: noise or Call: defect, Why: <the signals>, Change: <what changed> — so a reviewer sees the fixer's own case before the diff.

The goal is every Sentry defect worked by the agent, not merely triaged by it — a decline is the measured gap between that goal and where the fleet actually stands. Because every decline carries a category, the gap is countable, one bucket at a time:

gh issue list --repo OnRampTech/main-web-application --label autopilot:decline:needs-production-data --state all

Swap needs-production-data for any of the other five category names to count that bucket instead. The noise call is counted the same way, on PRs rather than issues:

gh pr list --repo OnRampTech/main-web-application --label autopilot:noise --state all

Manual Setup: What This PR Cannot Fill

The following six steps must be completed by hand before the pipeline can run. They are marked as "holes" because no PR can execute them; they require admin access to external services, to the GitHub organization, or (for step 3) a follow-up PR of their own.

1. Create the Sentry Internal Integration

An internal integration in the Sentry organization is the credential that allows the Lambda to receive webhooks and read events, and to write back the GitHub issue link.

1. Go to https://sentry.io/settings/onramp/integrations/internal/ (Sentry org settings → Integrations → Internal Integrations).
2. Click Create Integration.
3. Name: onramp-autopilot (or similar).
4. Permissions — the client in infra/components/issue-autopilot/sentry.ts calls the issue-environment-tag, issue-detail, org-issue-list, and external-issue-link endpoints, which need:
   • Organization: Read (org:read) — every call is scoped under /organizations/{org}/...
   • Project: Read (project:read) — the reconcile poll's org-issue-list query is scoped to each project's slug
   • Issue & Event: Read (event:read) — the environment tag and the issue detail (?expand=sentryAppIssues) reads
   • Issue & Event: Write (event:write) — the external-issue-link write (linkExternalIssue, a best-effort POST to sentry-app-installations/{uuid}/external-issues/) that records the GitHub issue on the Sentry side
5. Webhook URL: the ingest Lambda's function URL — the autopilotFunctionUrl stack output on the prod stack (exported from infra/app/index.ts), obtained after pulumi up. It looks like:

   https://abcdef12345-789.lambda-url.us-west-2.on.aws/

6. Webhook events: Select Issue.
7. Create the integration.
8. Copy the integration's token (prefixed sentry_internal_integration_…) and the webhook secret. Also note the integration's installation UUID — you'll need it for step 3 below, and the Lambda needs it to call the external-issue-link endpoint.
9. Add fingerprint rules for the captures Sentry cannot group on its own. Several data loaders report errors as a freshly constructed BaseException with the tenant id in the message and no stacktrace, so Sentry falls back to grouping by message and opens one issue per tenant (eight issues in 90 days for the token-refresh case). One Sentry issue is one GitHub issue, so each split becomes a duplicate for the autopilot. Until the code-side fix lands (capture with a traceback or an explicit fingerprint), these rules belong in Project Settings → api-onramp → Issue Grouping → Fingerprint Rules. They were applied to api-onramp on 2026-09-02, so a fresh install only needs to confirm they are still present. Rules apply to new events only; merge the existing split issues by hand.

   error.type:BaseException error.value:"Failed to refresh token for salesforce tenant ID *" -> token-refresh-failure-salesforce
   error.type:BaseException error.value:"Failed to refresh token for hubspot tenant ID *" -> token-refresh-failure-hubspot
   error.type:BaseException error.value:"Failed to refresh token for adobe-acrobat-sign tenant ID *" -> token-refresh-failure-adobe-acrobat-sign
   error.type:BaseException error.value:"Failed to refresh token for *" -> token-refresh-failure

2. Create the GitHub Apps — onramp-autopilot and onramp-autopilot-ingest

Two separate Apps, not one shared between them, because their private keys carry different blast radii once they exist. The fixer's token is minted fresh inside a GitHub Actions job by actions/create-github-app-token and lives only for that job's run. The ingest Lambda's credential is different in kind: it is the App's own long-lived PEM, sitting in SSM behind a public Function URL for the life of the container — and a private key cannot be narrowed the way a minted token can. Whoever reads it can mint a token carrying the App's FULL install-wide grant, not just the one call the Lambda happens to make. Giving that App the fixer's contents/pull requests write scope would mean a leaked ingest key could push code or merge PRs, not just open issues — so the ingest Lambda gets its own App, registered with nothing beyond what it actually calls.

2a. onramp-autopilot — the fixer

The App provides the token that signs every issue comment, PR creation, branch push, and promotion made by the fixer (.github/workflows/autopilot.yml's settle and fix jobs). Its App ID and private key go to the repository secrets AUTOPILOT_APP_ID / AUTOPILOT_APP_PRIVATE_KEY — see step 4.

1. Go to https://github.com/organizations/onramp/settings/apps (Org settings → Developer settings → GitHub Apps).
2. Click New GitHub App.
3. Name: onramp-autopilot
4. Homepage URL: https://github.com/onramp/main-web-application
5. Webhook:
   • Uncheck Active (this app does not receive webhooks).
6. Permissions:
   • Repository permissions:
     • contents: read & write — push fix branches, create commits
     • issues: read & write — create and comment on issues
     • pull requests: read & write — open, comment on, and promote PRs
   • Account permissions: None.
7. Where can this integration be installed: Only on this account.
8. Create the app.
9. Copy the App ID and generate a new private key (the PEM-formatted key) — both go to the repository secrets in step 4.
10. Install the app on the repository:
    • Go to the App's settings → Install App.
    • Select the onramp organization and the main-web-application repository.

2b. onramp-autopilot-ingest — the ingest Lambda

The credential the ingest Lambda uses to open and update the GitHub pointer issue — nothing more. Its App ID is the githubAppId value the prod row's issueAutopilot({...}) member takes (step 3); its private key goes to the prodenvironment secret AUTOPILOT_GITHUB_APP_PRIVATE_KEY (step 4). No code change is needed for either — infra/components/issue-autopilot.ts already takes both as plain config inputs (IssueAutopilotArgs.githubAppId and the autopilotGithubAppPrivateKey Pulumi secret) — so this is App registration and secret wiring only.

1. Go to the same https://github.com/organizations/onramp/settings/apps page.
2. Click New GitHub App.
3. Name: onramp-autopilot-ingest
4. Homepage URL: https://github.com/onramp/main-web-application
5. Webhook:
   • Uncheck Active (this app does not receive webhooks).
6. Permissions:
   • Repository permissions:
     • issues: read & write — create, update, and comment on the pointer issue
     • metadata: read — the baseline permission every App carries; nothing else
   • Account permissions: None.
7. Where can this integration be installed: Only on this account.
8. Create the app.
9. Copy the App ID (for step 3) and generate a new private key (the PEM-formatted key, for step 4).
10. Install the app on the repository:
    • Go to the App's settings → Install App.
    • Select the onramp organization and the main-web-application repository.

3. Add the Prod Row's autopilot Member to infra/app/envs.ts

infra/components/issue-autopilot.ts writes two non-secret identifiers to SSM as plain String parameters — githubAppId and sentryInstallationUuid on IssueAutopilotArgs. Both are required, so there is no bare issueAutopilot() placeholder to fill in later: the prod row does not compose autopilot at all until this step, which only happens once steps 1 and 2 are done and both values are in hand.

1. Open infra/app/envs.ts and find the prod row (prod: buildEnv({ alarms: alarms(), ... })).

2. Add an autopilot member to that row, passing both values:

   • githubAppId: the numeric App ID of the onramp-autopilot-ingest App, from step 2b.9, as a string. This is the ingest App's own id — not the fixer App's (onramp-autopilot, step 2a), which the ingest Lambda never touches.
   • sentryInstallationUuid: the installation UUID from step 1.8.

   autopilot: issueAutopilot({
     githubAppId: "123456",
     sentryInstallationUuid: "00000000-0000-0000-0000-000000000000",
   }),

3. Add issueAutopilot back to the ../components/issue-autopilot import at the top of the file (removed when this member was pulled off the prod row, since it otherwise has no caller).

4. Open a PR with that row edit and merge it — this is a code change, not something applied by hand outside of git.

Until this step lands, the ingest Lambda, its Function URL, log group, scheduler and SSM parameters do not exist on prod at all — infra/app/index.ts's if (env.autopilot) branch has nothing to compose. There is no partial or fails-closed-at-cold-start deploy in between: the member is either fully composed (both identifiers required) or not composed (absent from the row).

4. Add the GitHub Secrets — Two Apps, Two Keys

There is one Sentry internal integration and two GitHub Apps (onramp-autopilot, onramp-autopilot-ingest), and their credentials are consumed from two different places, so each ends up stored as a GitHub secret in the scope that reads it:

• The ingest Lambda never reads a GitHub secret directly. .github/workflows/deploy-infra-reusable.yml reads three GitHub environment secrets on the prod environment and runs pulumi config set --secret with them; at deploy time, infra/components/issue-autopilot.ts reads that Pulumi config and creates the matching SSM SecureString parameters the Lambda reads at cold start.
• The Actions-side fixer (.github/workflows/autopilot.yml's settle and fix jobs) reads its own GitHub repository secrets directly as job env vars — no SSM, no Pulumi config.

The Sentry API token is the one credential needed by both sides, so it is stored under the same secret name in both scopes — a repository secret and an environment secret with the name AUTOPILOT_SENTRY_API_TOKEN are two distinct GitHub secrets that happen to share a name, and both must be set to the same value from step 1.8. The GitHub App credentials are NOT shared this way: the fixer's App (onramp-autopilot, step 2a) and the ingest App (onramp-autopilot-ingest, step 2b) are two separate registrations with two separate keys, each stored exactly once.

Secret name	Where it lives	Consumed by
AUTOPILOT_SENTRY_WEBHOOK_SECRET	Environment secret (prod)	deploy-infra-reusable.yml → Pulumi config autopilotSentryWebhookSecret → SSM → the ingest Lambda's webhook HMAC check
AUTOPILOT_SENTRY_API_TOKEN	Environment secret (prod)	deploy-infra-reusable.yml → Pulumi config autopilotSentryApiToken → SSM → the ingest Lambda's Sentry client
AUTOPILOT_SENTRY_API_TOKEN	Repository secret	autopilot.yml's settle/fix jobs, exposed as the SENTRY_API_TOKEN env var → devtools/autopilot/fixer.ts's direct Sentry read for the fixer's context
AUTOPILOT_GITHUB_APP_PRIVATE_KEY	Environment secret (prod)	deploy-infra-reusable.yml → Pulumi config autopilotGithubAppPrivateKey → SSM → the onramp-autopilot-ingest App mints a JWT to open/update the pointer issue
AUTOPILOT_APP_ID	Repository secret	autopilot.yml's settle/fix jobs → actions/create-github-app-token mints the onramp-autopilot fixer App's installation token
AUTOPILOT_APP_PRIVATE_KEY	Repository secret	autopilot.yml's settle/fix jobs → actions/create-github-app-token mints the onramp-autopilot fixer App's installation token

To set them:

• Environment secrets (three): Settings → Environments → prod → Environment secrets → Add secret. Add AUTOPILOT_SENTRY_WEBHOOK_SECRET and AUTOPILOT_SENTRY_API_TOKEN (the webhook secret and API token from step 1), and AUTOPILOT_GITHUB_APP_PRIVATE_KEY (the onramp-autopilot-ingest App's private key from step 2b.9). Set these BEFORE step 3's PR merges: once the prod row composes autopilot, an absent secret here fails that whole prod apply loudly (infra/components/issue-autopilot.ts's config.requireSecret throws) rather than skipping just this member — deliberately, so a secret that drops out on a later deploy can never silently delete the already-running Lambda, Function URL, log group, scheduler and SSM parameters.
• Repository secrets (three): Settings → Secrets and variables → Actions → New repository secret. Add AUTOPILOT_SENTRY_API_TOKEN (same token value as its environment-secret counterpart above), AUTOPILOT_APP_ID, and AUTOPILOT_APP_PRIVATE_KEY, using the onramp-autopilot fixer App's ID and PEM private key from step 2a.9.

Nobody runs aws ssm put-parameter by hand for any of this — the SSM SecureString parameters are created by Pulumi from the environment secrets above, at deploy time, every time deploy-infra-reusable.yml runs against prod.

One identity-tier apply IS needed, after this PR merges and before step 3 composes the member. The ingest Lambda attaches the AWS Parameters-and-Secrets extension layer, an ARN in AWS's own account, and creating a function with a layer requires lambda:GetLayerVersion on that layer; the prod deploy role's permissions boundary (infra/ci-identity/boundary-policy.ts) now grants it, and infra/ci-identity/deploy-role-policy.test.ts pins the grant. The boundary converges only by a manual apply of the identity-tier stack under an admin profile — see infra/ci-identity/README.md's Bootstrap section. Skipping it makes the follow-up's prod apply fail on CreateFunction with an AccessDenied, and because the infra leg is best-effort on the release path, that would leave prod's whole Pulumi delta unapplied until someone reads the alert.

5. Turn It On

The settle job in .github/workflows/autopilot.yml is gated on the GitHub Actions repository variable AUTOPILOT_ENABLED == 'true'. Until it is set, nothing runs — no scheduled sweep every 30 minutes, no label wake, no dispatch wake.

Set it only once steps 1–4 are all in place: the Sentry integration, both GitHub Apps, the six secrets, the identity-tier apply, and the prod row's autopilot member.

gh variable set AUTOPILOT_ENABLED --repo OnRampTech/main-web-application --body true

To turn it back off:

gh variable set AUTOPILOT_ENABLED --repo OnRampTech/main-web-application --body false

Turning it off stops every wake and sweep. It does not stop the ingest Lambda — that keeps filing and updating GitHub issues from Sentry webhooks regardless; only the workflow that reads and acts on them pauses.

6. Verify One Assumption on First Install

The external-issue link round-trip should be verified once the integration from step 1 exists, before relying on the pipeline:

The external-issue link round-trips. infra/components/issue-autopilot/sentry.ts writes the GitHub link with a POST to sentry-app-installations/{uuid}/external-issues/ and reads it back through the issue detail's expand=sentryAppIssues field. The write endpoint and the expand parameter are documented; the shape of each returned link item is not. Trigger one test issue, confirm the Sentry issue shows the GitHub link, and confirm the Lambda's next webhook for that issue logs a link match rather than a create. If the read shape differs, the GitHub body-marker search still prevents a duplicate, but fix parseExternalIssueLinks so the Sentry side is the record the charter names.

Once the six gaps are filled and this check passes, the pipeline is ready to receive webhooks from Sentry and begin draining issues.

Manual Re-Run

To manually re-run a specific round of the autopilot workflow (e.g., to retry a fixer after a transient failure, or to test a change), use the workflow_dispatch trigger in the GitHub UI or CLI.

Via GitHub UI

1. Go to Actions → Alert Autopilot 🤖 → Run workflow.
2. Select the branch (usually main).
3. Fill in the inputs:
   • reason: dispatch (for a PR re-run), issue (to re-wake from an issue), or schedule (to run the sweeper and cap drain)
   • issue: The GitHub issue number (required if reason=issue)
   • pr: PR number (required if reason=dispatch)
   • sha: The commit SHA the swarm reviewed (required if reason=dispatch)
   • gating: true or false (required if reason=dispatch; pass true if the swarm gated / draft → ready, else false)
   • round: The round number (required if reason=dispatch; use the PR's commit count at that time)
4. Click Run workflow.

Via CLI

gh workflow run autopilot.yml \
  -f reason=dispatch \
  -f pr=1234 \
  -f sha=abc123def456 \
  -f gating=true \
  -f round=1

To re-wake a specific issue:

gh workflow run autopilot.yml -f reason=issue -f issue=5678

For a scheduled sweep:

gh workflow run autopilot.yml -f reason=schedule

The dispatch always runs the settle gate first (a dry run that spends no tokens). If the head has already been handled, or if there is genuine work to do, the workflow prints its decision and acts accordingly. A stale dispatch (for a head that has moved on or been fixed already) exits in seconds without spending tokens.

What autopilot:needs-human Means

An issue is labelled autopilot:needs-human in two cases:

1. The fixer declined the issue. The issue also carries one autopilot:decline:* label naming the capability the session lacked, and a comment whose first line is that category followed by the reasoning. No branch was created, no tokens were spent beyond the single fixer session.

2. Three fix rounds on a PR without settling green and bot-approved. The fixer tried three times, and the PR still has blocking checks or review findings. The autopilot will not spend more model tokens.

What a human does next:

For a declined issue: review the fixer's reasoning. If you believe it can be fixed after all, remove both autopilot:needs-human and the autopilot:decline:* label, then remove and re-add autopilot to re-run the drain.

For a PR that spent three rounds: read the PR and the sticky comment from the review swarm. The sticky carries the findings and the settle classification. Fix any remaining issues, approve the PR, and the next wake of the autopilot workflow (the schedule, or a manually triggered re-run) reads the settled-green-approved state and runs pr.promote, which flips the PR to ready, requests a reviewer, and arms auto-merge.

Verify

The ingest Lambda's logs in CloudWatch Logs Insights should show structured log lines for every webhook:

fields @timestamp, @message
| filter @message like /autopilot.ingest.dropped/
| stats count() by reason

This confirms:

• How many issues are dropped due to unlisted-project (Sentry project not in the ingested list)
• How many are dropped due to non-prod-environment (no prod-tier environment tags)

A newly created issue should appear in the GitHub repository with the autopilot label within seconds of the Sentry event. The autopilot workflow run should complete (settle gate + cap check) within 10 seconds if no fixer is needed, or 15–20 minutes if a fixer session runs.

Troubleshooting

🚫 No GitHub issue created when I create a Sentry issue

Check:

1. Is the Sentry project slug one of the ingested ones? (Must be api-onramp, ui-organization, or ui-customer-portal.)
2. Is the environment tag one of the prod tiers? (The tag names one of prod, gdpr-prod, or demo, matched case-insensitively; the label is the lowercase spelling — env:prod, env:gdpr-prod, env:demo. dev and stage are dropped.)
3. Is the Sentry internal integration webhook delivery failing? Check Sentry's Issue Settings → Webhooks and look at the delivery logs.

🚫 Workflow runs but says "AUTOPILOT_APP_ID / AUTOPILOT_APP_PRIVATE_KEY are unavailable"

The GitHub App's secrets are not set in the repository, or they are set with the wrong names.

1. Go to Settings → Secrets → Actions.
2. Verify that AUTOPILOT_APP_ID and AUTOPILOT_APP_PRIVATE_KEY are present.
3. If they are missing, add them (copy the app ID and private key from the GitHub App settings).
4. If they are present, delete them and re-create them — the last update time may have stale metadata.

Also check that the App is installed on the repository (Settings → Integrations → Installed GitHub Apps).

⚠ PR opened but fixer never runs a second round after the swarm gating

The autopilot-dispatch job in ai-review.yml may not have run, or it may have failed silently.

1. Check the review workflow run in the PR's checks.
2. Look for the autopilot-dispatch job — it should appear after the main review job if the PR carries the autopilot label.
3. If it is missing, the review job may not have run (e.g., all review checks were skipped). Re-run the review workflow manually.
4. If autopilot-dispatch ran, check its logs. The gh workflow run call should succeed silently; if it errored, the error is in the logs.

🚫 Settle gate exits `unsettled` repeatedly, fixer never runs

A required check is still running, or the review workflow's in-flight marker is present.

1. Check the PR's required checks in the Checks tab. If any are still running (yellow spinner), wait for them to finish.
2. If all checks have concluded, check the sticky comment on the PR — look for the cr-meta block and its in_progress field. If it is true, the review swarm is still running; wait or re-run the review workflow.
3. If checks are concluded and the swarm is not in progress, the settle gate may be misclassifying. Open the workflow run log and look for the classifySettle debug output to see what it read.

⚠ Issue created but never enters the workflow (no `autopilot` label or PR)

1. Check if the issue has the autopilot label at all.
2. If it does — with or without autopilot:queued — a lost or dropped wake is not fatal. The next 30-minute sweep re-reads every open autopilot issue that has neither a fix-PR marker comment nor autopilot:needs-human and drains it regardless of whether it was ever labelled autopilot:queued. Wait for the next sweep, or trigger it now:

   gh workflow run autopilot.yml -f reason=schedule

3. If the issue has no autopilot label at all, the ingest side never picked it up — see the troubleshooting card "No GitHub issue created when I create a Sentry issue" above.