Aero autonomy — rollout & operations
Turn on Aero's no-click actions for a workspace, watch what it does, and stop it.
What this is. Each night the re-engagement run writes recommendation cards: nudge an overdue assignee, reply to a comment, re-assign a task, update a playbook. Normally a person clicks each one. Under autonomy, a workspace's Super Admin can set each action to Always ask, Conditional (run unless a rule holds it) or Autonomous. The nightly run then acts on the cards itself, through the same agent-operation framework a click uses. That framework is what gives each change its "via Aero" activity entry and its undo.
Re-assign cards. The run suggests re-assigning only an overdue task whose assignee has not acted on the project for the workspace's inactivity threshold: 10 days unless an admin changes it in Settings → Aero → Recommendations. Flask drops any other re-assign card the agent sends (reassignment_eligible_task_ids in app/api/agent/services/re_engagement_data_service.py).
Canonical sources. The action registry, allowed modes and rule templates are in app/api/agent_autonomy/action_registry.py. The rule library is app/api/agent_autonomy/rules/library.py. The dispatcher and its refusals are app/api/agent_autonomy/services/dispatch_service.py. The caps are the aero_autonomy_* keys in app/config/environments.yaml. The metric lines come from app/api/agent_autonomy/metrics.py. This page is the how-to; those files are the truth.
TL;DR
Two things must both be true. OnRamp enables the aero_autonomy flag for the workspace, and its Super Admin moves an action off Always ask. Either one alone changes nothing.
The nudge is seeded. When OnRamp turns the agent (ai_agent) on for a workspace that has no autonomy settings yet, OnRamp seeds Nudge at Conditional with three guardrails (no nudge on the task in the last 7 days, at most 3 nudges per project per day, at most 2 nudges per task by Aero). It runs only where aero_autonomy is on, which is the flag's default. The workspace's admin sees it on Settings → Aero and can set it back to Always ask. A workspace that already had any setting, or had the agent before, is not changed; the global ai_agent toggle seeds nobody.
The flag is OnRamp's; the pause is the workspace's. Either stops every autonomous action at the next run. Undo keeps working under both.
Every change fires the workspace's own automations and emails, exactly like a click. The caps bound the volume: 50 operations per run and 100 per workspace per rolling day, shared by every action.
Facts the CSM and customer need
- Only the nightly batch acts on its own. A run started by hand from the project page only writes cards, whatever the settings say.
- The caps count operations, not emails. One nudge to three people is one operation. 50 per run and 100 per workspace per rolling 24 hours, shared across all four actions. Cards past a cap stay as cards for a person.
- Who Aero acts as. Each action runs as the Super Admin who last set that action's mode. If that person is deactivated, becomes a customer user, or is no longer an Owner or OnRamp admin, the run refuses the card (
refused/principal_*) and leaves it for a person. An autonomous nudge is sent in their name, with the project's internal owner as reply-to. - Under the seeded nudge policy, Aero acts as the project's internal owner. The nudge is sent in the owner's name, copied to them, with them as reply-to, and no admin is involved. A project with no active internal owner is refused (
principal_unknown) and keeps its card. - The comment template holds most comments. Its first rule holds any reply to a customer's comment, and any reply on a thread customers can see. Under the template, Conditional only runs replies on internal threads until an admin turns that rule off.
- Update playbook never publishes. It can be Conditional at most. A run writes the steps to a playbook draft and leaves the card for a person to review and publish. It holds when an earlier Aero draft of that playbook is still unpublished.
- Undo decays after 90 days. Operations purge at 90 days, which takes their undo and the "via Aero" link in the activity feed with them. The activity text itself stays. Nudge and comment reply can never be undone, because the email or reply has already gone out.
Before the first production flip
Each of these is tracked on ONRAMP-6987. Each one must be Done, or waived on that ticket by the epic owner.
Re-engagement reads are scoped to the workspace, and nudge recipients must be members of the task's project. Autonomy turns a cross-workspace read into a cross-workspace write.
Task and comment executors refuse archived and completed projects on every path, not only the autonomous one.
The nudge template carries the per-task repeat rule (task_nudged_within_days), so an overdue task is not nudged every night.
The prod and EU EasyCron jobs for /api/agent/operations/cron/sweep-stuck are live. Autonomous operations rely on the same sweep as Aero project editing (see that runbook).
Enable for a workspace
1 · Pre-flip audit
Know the first night's blast radius before the flag goes on. These are the automations and webhooks an autonomous change will fire, and how many cards each action would act on:
SELECT count(*) FROM or_project_rule WHERE vendor_id = :vendor_id;
SELECT count(*) FROM or_webhook WHERE vendor_id = :vendor_id;
SELECT action_type, count(*)
FROM or_agent_recommended_action
WHERE vendor_id = :vendor_id AND status = 'active' AND deleted_at IS NULL
GROUP BY action_type;action_type maps to actions as follows: nudge_overdue_users → nudge, prompt_unreplied_comments → comment, recommend_reassignment → re-assign, playbook_optimization → update playbook.
Share the counts with the CSM who asked for the flip. The customer must already have heard this sentence: "approved changes fire your configured automations and emails."
Kill switches
Delete the vendor_features row, or set feature_enabled = false. The next run records every card as gated / flag_off, and the Actions tab disappears from Settings → Aero.
The Super Admin uses Pause all actions on the Actions tab. It writes or_agent_autonomy_policy.paused = true. The next run records every card as gated / paused, and the levels are kept for when it resumes.
Applied changes stay undoable under both switches, because undo goes through the operation framework, which is gated only by ai-project-editing, and undo is exempt from that gate.
Decision records
Every card the dispatcher decides carries a JSON record in or_agent_recommended_action.autonomy_decision. Its keys are mode, outcome (asked / executed / gated / refused / failed / held_out), reason, operation_uuid, decided_at, and a rules block when rules ran.
SELECT action_type,
autonomy_decision->>'outcome' AS outcome,
autonomy_decision->>'reason' AS reason,
count(*)
FROM or_agent_recommended_action
WHERE vendor_id = :vendor_id
AND (autonomy_decision->>'decided_at')::timestamptz >= now() - interval '1 day'
GROUP BY 1, 2, 3
ORDER BY 4 DESC;What a person sees, and what Aero did without a click, is in or_agent_operation.approval_mode (human / autonomous / conditional):
SELECT approval_mode, kind, status, count(*)
FROM or_agent_operation
WHERE vendor_id = :vendor_id AND created_at >= now() - interval '1 day'
GROUP BY 1, 2, 3;Per-run cap check. No single run may start more than 50 autonomous operations. The metric lines carry no run id, so this check lives in SQL. It must always return no rows:
SELECT agent_execution_id, count(*)
FROM or_agent_operation
WHERE vendor_id = :vendor_id AND approval_mode <> 'human'
AND created_at >= now() - interval '7 days'
GROUP BY 1
HAVING count(*) > 50;Monitors
Each decision is logged as one line on the Flask logs: AGENT_AUTONOMY_METRIC {json}. The JSON keys are event, vendor_id, project_id, recommendation_id, action_key, mode, outcome, reason, operation_uuid, principal_user_id, matched_rule and rule_outcome. A second event, latch_lost, means a person acted on the card first. These lines are in filebeat-7.11.1-*. The JSON is part of the message text, not a set of fields, so each dimension is matched with match_phrase on its "key": value text. Read the opensearch-investigation skill before changing a query.
Four query-level monitors exist in OpenSearch alerting. Each one posts to #eng-alerts and repeats at most once an hour:
| Monitor | Runs | Fires when | id |
|---|---|---|---|
prod-usw2-flask-aero-autonomy-failed | every 15 min | any failed decision in the last hour | dhu0DKEBlaB-APakLHT5 |
prod-gdpr-ir-flask-aero-autonomy-failed | every 15 min | any failed decision in the last hour | iRu0DKEBlaB-APakLnSr |
prod-usw2-flask-aero-autonomy-daily-cap-invariant | hourly | more than 100 executed decisions in 24 hours | fBu0DKEBlaB-APakLXTM |
prod-gdpr-ir-flask-aero-autonomy-daily-cap-invariant | hourly | more than 100 executed decisions in 24 hours | jhu0DKEBlaB-APakL3SM |
failed means the executor ran and could not finish, for example a nudge that never left the queue. Investigate each one. The query:
{ "size": 0, "query": { "bool": { "filter": [
{ "term": { "tenant.name": "prod-usw2" } },
{ "term": { "kubernetes.labels.app": "flask" } },
{ "match_phrase": { "message": "AGENT_AUTONOMY_METRIC" } },
{ "match_phrase": { "message": "\"outcome\": \"failed\"" } },
{ "range": { "@timestamp": { "gte": "now-1h" } } } ] } } }This should never fire. The daily cap is enforced under a per-workspace lock before any write, and executed decisions are a subset of the operations the cap counts. If it fires, pause the workspace first, then investigate.
The threshold is 100 for the whole tenant, which matches the cap exactly while one workspace is flipped. Before a second workspace is flipped in the same tenant, add one copy per workspace with { "match_phrase": { "message": "\"vendor_id\": <id>" } } in the filter, and keep the tenant-wide copy as a backstop at 100 × the number of flipped workspaces.
{ "size": 0, "query": { "bool": { "filter": [
{ "term": { "tenant.name": "prod-usw2" } },
{ "term": { "kubernetes.labels.app": "flask" } },
{ "match_phrase": { "message": "AGENT_AUTONOMY_METRIC" } },
{ "match_phrase": { "message": "\"outcome\": \"executed\"" } },
{ "range": { "@timestamp": { "gte": "now-24h" } } } ] } } }Daily mix (dashboard, not an alert). This is one filters aggregation over the last 24 hours, with the same base filter and no outcome clause. It splits decisions into outcome buckets (executed, asked, gated, refused, failed), hold buckets ("reason": "rule_matched", "reason": "cap", "reason": "flag_off", "reason": "paused"), and one bucket per rule ("matched_rule": "<rule_key>"). Watch for two patterns: a climbing cap count, which means the workspace has more work than the cap allows, and refused with a principal_* reason, which means the admin who set the level has lost their role.
Success metrics
Run these on the read replica per workspace, weekly during a rollout.
-- Cards acted on by Aero vs by a person, by action (30 days)
SELECT action_type,
CASE WHEN autonomy_decision->>'outcome' = 'executed'
OR autonomy_decision->>'reason' = 'draft_only' THEN 'aero' ELSE 'person' END AS actor,
count(*)
FROM or_agent_recommended_action
WHERE vendor_id = :vendor_id AND acted_on_by IS NOT NULL
AND acted_on_at >= now() - interval '30 days'
GROUP BY 1, 2;
-- Time from card to execution, for cards Aero ran
SELECT action_type,
percentile_cont(0.5) WITHIN GROUP (ORDER BY acted_on_at - created_at) AS median
FROM or_agent_recommended_action
WHERE vendor_id = :vendor_id AND autonomy_decision->>'outcome' = 'executed'
AND created_at >= now() - interval '30 days'
GROUP BY 1;
-- Undo rate of what Aero did on its own
SELECT approval_mode,
count(*) FILTER (WHERE status IN ('undone', 'partially_undone'))::float / nullif(count(*), 0) AS undo_rate
FROM or_agent_operation
WHERE vendor_id = :vendor_id AND approval_mode <> 'human' AND kind <> 'undo'
AND created_at >= now() - interval '30 days'
GROUP BY 1;The first query reads the cards themselves, because a person's click on a nudge, comment or re-assign card acts directly and starts no agent operation. An Update playbook draft Aero wrote counts as Aero's, even though a person still publishes it.
Holdout
Aero leaves a fixed share of the nudges it would send on its own for a person instead, so the completion of nudged and not-nudged tasks can be compared. The share is aero_autonomy_holdout_fraction and the end date is aero_autonomy_holdout_until, both in app/config/environments.yaml. The defaults are 10% until 2026-12-10 (UTC).
- Which cards. Only nudge cards that Aero would otherwise send: the level is Autonomous or Conditional, the rules let the nudge run, and no cap held it. The bucket is the first eight hex digits of
sha1("aero-holdout:<card uuid>"), modulo 1000. A card is held out when its bucket is below the fraction × 1000. The same card always lands in the same bucket. - What happens. The card stays active for a person, Aero sends no email and starts no operation, and the decision is recorded as
held_out/holdout. The Aero settings page counts it in no counter. - How to end it. Set
aero_autonomy_holdout_fractionto0and release, or let the date pass. Either way every nudge after that runs as usual.
Compare the 14-day completion of the tasks the held-out and the sent nudge cards name, per workspace or across all of them:
SELECT a.autonomy_decision->>'outcome' AS arm,
count(*) AS cards,
count(*) FILTER (
WHERE t.completed_date IS NOT NULL
AND t.completed_date <= ((a.autonomy_decision->>'decided_at')::timestamptz + interval '14 days') AT TIME ZONE 'UTC'
) AS completed_in_14_days,
round(100.0 * count(*) FILTER (
WHERE t.completed_date IS NOT NULL
AND t.completed_date <= ((a.autonomy_decision->>'decided_at')::timestamptz + interval '14 days') AT TIME ZONE 'UTC'
) / nullif(count(*), 0), 1) AS completion_pct
FROM or_agent_recommended_action a
JOIN or_tasks t ON t.id = a.target_task_id AND t.project_id = a.project_id
WHERE a.action_type = 'nudge_overdue_users'
AND a.autonomy_decision->>'outcome' IN ('held_out', 'executed')
AND (a.autonomy_decision->>'decided_at')::timestamptz < now() - interval '14 days'
AND a.deleted_at IS NULL
GROUP BY 1;The last date filter keeps only cards whose 14 days are over. or_tasks.completed_date has no time zone and holds UTC. A person may still send a held-out card's nudge by hand: that is the comparison arm working as the workspace did before autonomy, not a leak.
First flip
One internal workspace, Re-assign at Conditional, for one week, before any customer workspace. Exit criteria before the next workspace or action:
- The failure monitor never fired, or every firing has a known cause that has been fixed.
- The daily-cap invariant and the per-run cap check stayed silent.
- Every change in the activity feed's Aero filter matches its card, and nothing landed outside the workspace.
Troubleshooting
The run was started by hand, or the dispatcher never ran for that execution. Only the nightly batch dispatches. Check that the card's agent_execution_id came from the nightly run.
The reason says which switch: flag_off means the flag row is missing or false, and paused means the workspace paused.
The admin who set the level can no longer act: deactivated, a customer user, demoted, or in another workspace. Another Super Admin saves the level again, which makes them the principal.
A Conditional rule held the card. This is working as intended. rules.matched_rule names the rule, and the save-time preview shows how often it holds.