Skip to content

Code Review Risk Labels ​

Every PR gets the five-lens review swarm automatically. On top of that, CI tags each PR with a risk:low / risk:moderate / risk:high label, so you know when a change is worth a deeper read of your own before you approve it. The label is a signal; the decision is yours.

The high-risk path list does double duty. Adding an entry to it both re-weights this label and wakes the swarm's security reviewer on every file beneath that path, at its own model and timeout — so editing the list to tune a label also changes reviewer selection and per-round review cost.

Source of truth. The high-risk path list and the file-count threshold live in .github/code-review-heuristic.yml; the logic is in .github/scripts/classify-risk.ts. Numbers on this page are illustrative — when in doubt, read the YAML, not this doc.

The rule ​

Two independent conditions, evaluated over the PR diff:

  • cond_path — a non-test file of any status, added files included, falls under a high-risk path (the RBAC engine, sign-in, the external user API, ramps, playbook draft service, data fields, the workflow/playbook UI, portal security surfaces, CI identity, agent hooks and settings, …), or the diff touches a migration (in either Alembic tree), a route table or route policy, an IAM change, a sweeper-style tool, or deletes 20 or more test files.
  • cond_size — the number of modified source files exceeds the configured threshold.

The label is the XOR of the two:

cond_pathcond_sizeLabel
✗✗risk:low
✓✗risk:moderate
✗✓risk:moderate
✓✓risk:high

Neither → low. Exactly one → moderate. Both → high.

Why the two conditions count differently ​

cond_size counts modified source files only — files whose GitHub status is anything other than added, minus renames of 4 or fewer changed lines. A new file has no blast radius on existing call sites by definition, so a feature that adds 80 brand-new files is safe in a way that a 20-file refactor of existing code is not. A pure move is not an edit either.

cond_path counts added files too. New code in an authorization directory, a new migration or a new route table has no existing caller to modify, yet it is where the risk lives.

Tests and non-code (docs, config, lockfiles) never count toward cond_size, and a test-only edit to a high-risk path does not trip cond_path. The path list names a few config files, such as app/config/environments.yaml and infra/app/envs.ts, that count whatever their extension; the heuristic file's header says which.

Two triggers read names or diff text rather than a fixed path. An IAM change is an infra/ .ts file named like an IAM or boundary file, or one whose diff adds or removes a line with aws.iam., a policy statement keyword (Effect:, Action:, Resource:, Principal:, Condition:), a quoted service:Action string or an ARN, or one with edits but no diff because GitHub left the diff out for size. A sweeper-style tool is a file under devtools/, .github/scripts/, .github/workflows/ or infra/scripts/ whose name contains sweep, reap, retire, purge, destroy, delete or prune; a tool with another name for the same job is left to the model check.

The model check ​

When the rule above gives risk:low, a small model reads the PR title, the file list with change types and line counts, and the body. It answers three yes/no questions: does the PR change who can read or write what, is it safe only if a deploy-time condition holds, and does it delete or overwrite live resources or data. A yes to any of them makes the label risk:moderate. The model never sets risk:high. If the call fails, a PR that already has a risk: label keeps it, and a PR with none gets the label the rule gave it.

Try it ​

cond_size — modified source files: 3threshold = 15*
cond_path ✗cond_size ✗
risk:lowNeither condition tripped.

* Illustrative threshold for this widget only. The live value is source_file_modified_threshold in code-review-heuristic.yml.

Worked examples ​

Read these as the rule applied to diff shapes — the ✓/✗ columns are what matters, not exact counts.

Diff shapecond_pathcond_sizeLabel
New feature — all brand-new files (even 80 of them), anywhere✗✗risk:low
New files added under a risky dir, no existing file modified✓✗risk:moderate
A few-line edit to a high-risk file (e.g. an authz check)✓✗risk:moderate
Large refactor of existing code, no high-risk path touched✗✓risk:moderate
Large refactor that modifies a high-risk file✓✓risk:high
Pure rename of an existing module (4 or fewer changed lines per file)✓ if under a risky path, else ✗not countedrisk:low, or risk:moderate under a risky path
A new migration, route table, or IAM change✓✗risk:moderate
Docs/config-only PR, or test-only edits✗✗risk:low

What to do with the label ​

LabelSuggested action
risk:lowNothing — the automated SDK review is enough.
risk:moderateConsider running /code-review ultra <PR#> locally for a deeper pass, especially if you touched the risky area's behavior.
risk:highStrongly worth /code-review ultra <PR#> before merge — this PR both modifies a risky surface and is large.

The separate bot-approved label is the bot's read that it is asking for nothing more, not a human approval — a human review is still required to merge. REVIEW.md owns exactly when it is granted; don't restate the condition here, it has drifted before.

Where this lives ​

  • Heuristic config (paths + threshold): .github/code-review-heuristic.yml
  • Classifier: .github/scripts/classify-risk.ts
  • Workflow: .github/workflows/ai-review.yml
  • Label application: devtools/review-swarm/github/labels.ts
  • Security reviewer's path list (same high_risk_paths, via patternsFrom): devtools/review-swarm/reviewers/security/config.yml
  • Review rubric (what the reviewers flag): REVIEW.md

Internal documentation — gated behind Cloudflare Access.