Aero project editing — rollout & operations

Enable, watch, inspect, undo, and kill Aero-initiated project/task edits (agent operations) for a vendor.

Canonical sources. The kill switch is app/api/agent/services/agent_operation_service.py (AI_PROJECT_EDITING_FEATURE_CODE, gates on materialize/execute/retry, UNDO exempt). The editable surface is the executor registry — app/api/agent/services/task_change_executors.py registers it, and GET /internal/agent/operations/vocabulary reports it live. Metrics are the AGENT_OPERATION_METRIC lines on the app.api.agent.services.agent_operation_metrics logger. This runbook is the HOWTO; those are the truth.

TL;DR

• Default OFF everywhere. The feature row ships disabled; a vendor gets the capability only through a vendor_features override. Enabling is one row — but do the pre-flip audit first.
• The blast radius is the tenant's own automation config. Aero edits fire automations, notifications, emails and webhooks exactly like manual edits, per record — a 100-task change set is 100 of everything the tenant wired up.
• Undo survives the kill switch on purpose: disabling the feature never strands an applied change the tenant wants reverted.

Enable for a vendor

1. Pre-flip audit — before the row, not after the incident. Enumerate what the tenant has configured on the projects Aero will touch:

   -- automation rules that can fire on task/project changes for this vendor
   SELECT count(*) FROM or_project_rule WHERE vendor_id = :vendor_id;
   -- outbound webhooks registered for this vendor
   SELECT count(*) FROM or_webhook WHERE vendor_id = :vendor_id;

   Share the counts with the CSM who requested the flip; "approved changes will trigger your configured automations and emails" must be a sentence they have already said to the customer.

2. Flip:

   INSERT INTO vendor_features (feature_id, vendor_id, feature_enabled)
   SELECT id, :vendor_id, true FROM features WHERE feature_code = 'ai-project-editing';

   Flips land in the feature-flag audit log like any other VendorFeature write.

3. Verify: as a user of that vendor, ask Aero to change one task; the approval card must appear. For any other vendor a proposal must refuse with feature_disabled.

Kill switch

Delete the override row (or set feature_enabled = false). Semantics, all server-enforced and test-pinned:

• New proposals refuse (feature_disabled), including target-query resolution — the SQL never runs for a killed tenant.
• Pending proposals refuse execute/retry and expire on their 10-minute TTL.
• Already-applied operations remain undoable — UNDO is exempt at every gate.

Watch (metrics & alerts)

Every lifecycle event is one JSON log line marked AGENT_OPERATION_METRIC (logger app.api.agent.services.agent_operation_metrics), shipped with the Flask logs to OpenSearch. Events: proposed (source: explicit/target_query/undo), refused (stage + reason), executed (stage execute/retry, duration_ms, per-record status counts), stuck_recovered.

Suggested monitors (create in OpenSearch alerting on the flask log index):

• Per-record failure rate — event:executed where statuses.failed > 0; alert on sustained nonzero. A spike means an executor or a tenant's data shape is broken.
• Stale rate — event:refused AND reason:stale_fingerprint climbing means previews go stale faster than users approve: a product signal, not an error.
• Stuck recoveries — any event:stuck_recovered is a worker crash or kill mid-execution that nothing else reported. Investigate each.
• Refusal mix — feature_disabled for a vendor that should be enabled means the flip didn't land or the codes cache is stale.

Stuck-operation sweep + retention purge

POST /api/internal/agent/operations/sweep-stuck (internal JWT; service-principal tokens accepted — mutates only operation bookkeeping, never target records) does two things and returns {recovered: n, purged: m}:

• finalizes anything stranded in EXECUTING past the 15-minute heartbeat;
• purges operations older than the 90-day retention window (OPERATION_RETENTION). Change rows — the raw before-values — die with their operation via FK cascade; activity rows survive with operation_id nulled, so the audit text is permanent while the "via Aero" provenance chip decays with the purge. Undo and the "show me what you changed" listing are therefore bounded at 90 days by design (they decay much sooner in practice: drift excludes most undos within days).

Idempotent; healthy fleets sweep zero and purge only what aged out.

Deploy step (not yet done): create two EasyCron jobs — prod (app.onramp.us) and EU (eu.onramp.us) — each POSTing /api/agent/operations/cron/sweep-stuck every 15 minutes with the X-EasyCron-Token header. The cron route authenticates via the EASYCRON_SHARED_SECRET shared secret rather than a JWKS-validated JWT — EasyCron cannot mint one. Until the jobs exist, recovery still happens on the actor's next touch — the sweep is what removes the "until someone comes back" clause. Flip this paragraph to present tense in the change that confirms the jobs are live.

Verify the secret before relying on the cron (other endpoints already depend on it too — grep requires_easy_cron_token for the full set — but confirm per environment rather than trusting that): POST the cron URL with the environment's token; this actually runs the sweep and retention purge against that environment (idempotent, so safe to re-run, but it is a real invocation, not a dry run). A 500 "Cron auth not configured" means EASYCRON_SHARED_SECRET is missing from the pod env (Duplo tenant config); a 401 means the token is wrong; a 200 {recovered, purged} means the path is wired. demo and stage deliberately get no job until the ai-project-editing feature is enabled for a vendor there — add one when flipping.

The JWT-gated POST /api/internal/agent/operations/sweep-stuck remains in place for internal and manual callers.

Inspect an operation

SELECT o.id, o.uuid, o.status, o.intent, o.actor_user_id, o.executed_at,
       c.object_id, c.attribute, c.status, c.before_value, c.after_value, c.result_detail,
       c.observed_modified_at, c.applied_at, c.side_effects_at
FROM or_agent_operation o
JOIN or_agent_operation_change c ON c.operation_id = o.id
WHERE o.uuid = :uuid
ORDER BY c.id;

Activity rows carry operation_id, so "everything this operation touched" is SELECT * FROM or_activity_history WHERE operation_id = :id. The actor is always the real user; "via Aero" on the activity feed comes from the FK, not an actor swap.

The three timestamps are what answers "why was this record excluded". A record is refused when the target's live modified_at is past the greatest of observed_modified_at and, across every change in the operation that shares the same target, applied_at and side_effects_at — so compare against that maximum, not against the one row in front of you. side_effects_at is when that record's post-apply side effects finished; a target whose modified_at sits between applied_at and side_effects_at was moved by the operation's own cascade (a project automation firing on the update), not by a third party.

Undo on a customer's behalf

Undo is a normal operation: only the original actor can drive it through chat/card (actor-only enforcement). Support cannot undo as the customer — have the requesting user run "show me what you changed" → "undo it", or restore values manually using before_value from the inspect query above. Records changed by someone else since the original run are excluded from undo and reported, never overwritten.

Known interaction: Bedrock guardrail vs tool results

The copilot guardrail (default_guardrail_id in onramp-agents/ environments.yaml; the minimal guardrail carries PROMPT_ATTACK at input strength LOW with action BLOCK) can block a mid-turn model call after a tool result re-enters the conversation as user-role input — observed as the [E-01] blocked-input message truncating a review reply and blocking an undo turn. Isolated apply-guardrail repro of the same text passes, so the verdict depends on the full multi-step input. Diagnose live cases from the guardrail assessments in /aws/bedrock/all-model-invocations (CloudWatch, us-west-2):

fields @timestamp, modelId
| filter output.outputBodyJson.amazon-bedrock-guardrailAction = "INTERVENED"
| sort @timestamp desc

The EU tenant binds its own guardrail (environments.yaml EU block) — any policy change must be mirrored there.