Automation Action Failure Alerting — Runbook
Read the operator-side signal for project-automation action failures: what the normalized failure classes mean, which ones are the customer's to fix, and how to tell a rate change apart from the steady-state baseline.
Canonical sources. The failure taxonomy, its origins, and the severity mapping live in app/api/project_automation/services/automation_failure_signal.py — that file is the truth, and FAILURE_SIGNATURES is the list to extend. Emission is hooked at ProjectAutomationService.log_project_error, the one choke point every action failure funnels through. This runbook is the HOWTO only.
TL;DR
Every one of these failures is already reported to the customer — automation_errors_count plus a failed rule-history row, surfaced in the project's automation log and the projects-list badge. The signal exists for us.
~16 failures/day steady state, so Sentry gets at most one event per (action, failure class, vendor) per hour, carrying the count it stands in for. The exact rate lives on the AUTOMATION_ACTION_FAILURE log line, not in Sentry's event count.
info = the customer's CRM configuration or a transient CRM fault. warning = ours, or a class we don't recognize yet. Page on warning only; a new unknown issue means the taxonomy has a gap.
The failure classes
Measured over 30 days on prod-usw2, 490 updateSalesforceField failures (no updateHubspotField failures in the window). Roughly half are the customer's CRM setup — the other half were ours and had never been looked at.
| Class | Origin | Level | 30d | What it means |
|---|---|---|---|---|
restricted_picklist_value | customer | info | 141 | The mapped value isn't in the target restricted picklist. Their picklist, their fix. |
value_type_mismatch | onramp | warning | 83 | We sent a value the field's type can't hold ("No" → boolean, "not started" → percent). format_field_value_by_type didn't coerce it. |
value_too_long | onramp | warning | 66 | We sent a value longer than the field. Recurring case: a full project URL into a 50-char field. |
field_not_writable | customer | info | 61 | Field-level security — the connected profile can't write the field. |
non_json_response | onramp | warning | 57 | The integration path couldn't parse the response at all (usually an empty body). Never the customer's doing. |
customer_validation_rule | customer | info | 69 | Their own Apex / validation rule rejected the save, including the bare Can't Save Record. |
record_not_found | customer | info | 7 | The linked CRM record is gone, or the project never matched a unique entity. |
crm_limit_exceeded | transient | info | 4 | Salesforce governor limit. Retry is the answer. |
crm_auth_invalid | customer | info | 1 | The connection needs reconnecting. |
crm_unreachable | transient | info | 1 | Dropped connection or timeout. |
unknown | unknown | warning | — | Nothing matched. Add a signature. |
Order in FAILURE_SIGNATURES is load-bearing. Salesforce nests both restricted_picklist_value and value_too_long inside "There were custom validation error(s) encountered…", so customer_validation_rule must stay last among the validation classes or it swallows them. Add specific classes above it, not below.
The procedure
1 · Read the rate
The AUTOMATION_ACTION_FAILURE line carries one document per occurrence with no customer values in it. Daily counts split by origin — this is the "did the rate change" question, and onramp is the series that matters:
curl -s -H 'Content-Type: application/json' \
'http://10.220.61.14:9200/filebeat-7.11.1-*/_search?size=0' -d '{"query":{"bool":{"filter":[{"term":{"tenant.name":"prod-usw2"}},{"term":{"kubernetes.labels.app":"flask"}},{"match_phrase":{"message":"AUTOMATION_ACTION_FAILURE"}},{"range":{"@timestamp":{"gte":"now-14d"}}}]}},"aggs":{"per_day":{"date_histogram":{"field":"@timestamp","calendar_interval":"day"},"aggs":{"by_origin":{"filters":{"filters":{"onramp":{"match_phrase":{"message":"origin onramp"}},"customer_config":{"match_phrase":{"message":"origin customer_config"}},"transient":{"match_phrase":{"message":"origin transient"}},"unknown":{"match_phrase":{"message":"origin unknown"}}}}}}}}}'Swap the inner clauses for "failure_class <slug>", "action <slug>", or "vendor_id <id>" to split the same histogram on any other dimension.
There is no field to aggregate on — match_phrase is the only way in. message has no .keyword subfield, and extra= kwargs on a logger call reach OpenSearch as nothing at all: the filebeat pipeline ships the formatted line only, so a structured field never materializes (verified — zero docs cluster-wide carry extra.*). That is why the payload is JSON inside the message. The standard analyzer strips the JSON punctuation, so "origin": "onramp" tokenizes to [origin, onramp] and a two-token match_phrase matches it. Also: pass _source on any query that returns hits, and don't filter on log.level — it is empty in this cluster. The opensearch-investigation skill covers both traps.
Verify
A change to the taxonomy is verified by the unit suite, whose classification cases are real prod messages rather than invented ones:
uv run pytest app/api/project_automation/services/tests/test_automation_failure_signal.py -q --tb=short --no-headerTroubleshooting
The line is emitted at INFO, and only the app.api logger tree is raised to the level app_log_level resolves to (app/config/environments.yaml — INFO in every deployed environment); app/api/__init__.py leaves the root logger at ERROR. A metrics logger named outside that tree is silently dropped before any handler sees it, which is why this module uses __name__. Not hypothetical: agent_operations.metrics has zero docs in the cluster for exactly this reason, while app.api-tree INFO lines land fine.
By design — one event per (action, class, vendor) per hour per worker. Read occurrences_since_last_report on the event, or count the log lines. Never treat the Sentry event count as the failure rate.
Related: an event can arrive long after the failures it counts. When a fingerprint goes quiet mid-burst, its leftover count is flushed as its own event by the next failure in that worker — any fingerprint. So an info event for an org that stopped failing hours ago is the tail of that burst, not a new one; its occurrences_since_last_report is the count, and the log lines carry the real timestamps. A tail event's project_id and rule_uuid are null by design — the occurrence that would have named them is long gone by the time the window is reclaimed — so step 2's triage path doesn't apply to one. Pivot to the log lines via the tags (automation.vendor_id + automation.failure_class) to find the affected projects.
The fingerprint includes the action slug, and callers pass either updateSalesforceField or Update Salesforce Field for the same action. normalize_action collapses both — if a split appears, a new caller is passing a third spelling that doesn't normalize to the same slug.
Check the phrase you added is lowercase and appears in the provider string as a substring — matching is a plain in against a lowercased haystack of error_type, message and JSON-dumped details. No regex, no analyzer.