Automation Action Failure Alerting — Runbook
Read the operator-side signal for project-automation action failures: what the normalized failure classes mean, which ones are the customer's to fix, and how to tell a rate change apart from the steady-state baseline.
Canonical sources. The failure taxonomy, its origins, and the severity mapping live in app/api/project_automation/services/automation_failure_signal.py — that file is the truth, and FAILURE_SIGNATURES is the list to extend. Emission is hooked at ProjectAutomationService.log_project_error, the one choke point every action failure funnels through. This runbook is the HOWTO only.
TL;DR
Every one of these failures is already reported to the customer — automation_errors_count plus a failed rule-history row, surfaced in the project's automation log and the projects-list badge. The signal exists for us, so only the classes we can act on become Sentry issues.
~16 failures/day steady state, so Sentry gets at most one event per (action, failure class, vendor) per hour, carrying the count it stands in for. The exact rate lives on the AUTOMATION_ACTION_FAILURE log line, not in Sentry's event count.
Customer-config failures never reach Sentry; they stay on the log line and in the customer's automation log. info = a transient CRM fault. warning = ours, or a class we don't recognize yet. Page on warning only; a new unknown issue means the taxonomy has a gap.
The failure classes
Measured over 30 days on prod-usw2, 490 updateSalesforceField failures (no updateHubspotField failures in the window). Roughly half are the customer's CRM setup — the other half were ours and had never been looked at.
| Class | Origin | Sentry level | 30d | What it means |
|---|---|---|---|---|
restricted_picklist_value | customer | log only | 141 | The mapped value isn't in the target restricted picklist. Their picklist, their fix. |
value_type_mismatch | customer | log only | 83 | The rule maps a text data field's value to a typed Salesforce field and Salesforce rejects it ("No" → boolean, "not started" → percent). The rule's author has to map a source of the field's type, or fix the value entered. |
value_too_long | onramp | warning | 66 | We sent a value longer than the field. Recurring case: a full project URL into a 50-char field. |
field_not_writable | customer | log only | 61 | Field-level security — the connected profile can't write the field. |
non_json_response | onramp | warning | 57 | The integration path couldn't parse the response at all (usually an empty body). Never the customer's doing. |
customer_validation_rule | customer | log only | 69 | Their own Apex / validation rule rejected the save, including the bare Can't Save Record. |
record_not_found | customer | log only | 7 | The linked CRM record is gone, or the project never matched a unique entity. |
crm_limit_exceeded | transient | info | 4 | Salesforce governor limit. Retry is the answer. |
crm_auth_invalid | customer | log only | 1 | The connection needs reconnecting. |
crm_unreachable | transient | info | 1 | Dropped connection or timeout. |
webhook_unavailable | customer | log only | — | A callWebhook rule points at a webhook that was deleted or deactivated. |
webhook_user_missing | onramp | warning | — | A callWebhook rule fired with no resolvable triggering user, so no payload was sent. |
webhook_payload_unbuilt | onramp | warning | — | A callWebhook rule's trigger produced no payload to send. |
data_field_value_invalid | customer | log only | — | An updateDataField rule resolved a value its target field's type rejects (a dropdown option that doesn't exist, a non-ISO date, a non-number, an unknown user), so the field was left unchanged. The rule's author has to fix the mapping or the field's options. |
data_field_mapping_invalid | customer | log only | — | An updateDataField rule was saved with no target field, no mapped source, or a whole form step as its source. The action is skipped and the field is left unchanged; the rule's author has to finish the mapping. |
data_field_not_found | onramp | warning | — | An updateDataField rule's target field could not be saved for this project's vendor. |
data_field_user_missing | onramp | warning | — | An updateDataField rule fired with no user to attribute the write to. |
unknown | unknown | warning | — | Nothing matched. Add a signature. |
Order in FAILURE_SIGNATURES is load-bearing. Salesforce nests both restricted_picklist_value and value_too_long inside "There were custom validation error(s) encountered…", so customer_validation_rule must stay last among the validation classes or it swallows them. Add specific classes above it, not below.
The procedure
1 · Read the rate
The AUTOMATION_ACTION_FAILURE line carries one document per occurrence with no customer values in it. Daily counts split by origin — this is the "did the rate change" question, and onramp is the series that matters:
curl -s -H 'Content-Type: application/json' \
'http://10.220.61.14:9200/filebeat-7.11.1-*/_search?size=0' -d '{"query":{"bool":{"filter":[{"term":{"tenant.name":"prod-usw2"}},{"term":{"kubernetes.labels.app":"flask"}},{"match_phrase":{"message":"AUTOMATION_ACTION_FAILURE"}},{"range":{"@timestamp":{"gte":"now-14d"}}}]}},"aggs":{"per_day":{"date_histogram":{"field":"@timestamp","calendar_interval":"day"},"aggs":{"by_origin":{"filters":{"filters":{"onramp":{"match_phrase":{"message":"origin onramp"}},"customer_config":{"match_phrase":{"message":"origin customer_config"}},"transient":{"match_phrase":{"message":"origin transient"}},"unknown":{"match_phrase":{"message":"origin unknown"}}}}}}}}}'Swap the inner clauses for "failure_class <slug>", "action <slug>", or "vendor_id <id>" to split the same histogram on any other dimension.
There is no field to aggregate on — match_phrase is the only way in. message has no .keyword subfield, and extra= kwargs on a logger call reach OpenSearch as nothing at all: the filebeat pipeline ships the formatted line only, so a structured field never materializes (verified — zero docs cluster-wide carry extra.*). That is why the payload is JSON inside the message. The standard analyzer strips the JSON punctuation, so "origin": "onramp" tokenizes to [origin, onramp] and a two-token match_phrase matches it. Also: pass _source on any query that returns hits, and don't filter on log.level — it is empty in this cluster. The opensearch-investigation skill covers both traps.
Verify
A change to the taxonomy is verified by the unit suite, whose classification cases are real prod messages rather than invented ones:
uv run pytest app/api/project_automation/services/tests/test_automation_failure_signal.py -q --tb=short --no-headerTroubleshooting
The line is emitted at INFO, and only the app.api logger tree is raised to the level app_log_level resolves to (app/config/environments.yaml — INFO in every deployed environment); app/api/__init__.py leaves the root logger at ERROR. A metrics logger named outside that tree is silently dropped before any handler sees it, which is why this module uses __name__. Not hypothetical: agent_operations.metrics has zero docs in the cluster for exactly this reason, while app.api-tree INFO lines land fine.
By design — one event per (action, class, vendor) per hour per worker. Read occurrences_since_last_report on the event, or count the log lines. Never treat the Sentry event count as the failure rate.
Related: an event can arrive long after the failures it counts. When a fingerprint goes quiet mid-burst, its leftover count is flushed as its own event by the next failure in that worker — any fingerprint. So an info event (transient classes only) for an org that stopped failing hours ago is the tail of that burst, not a new one; its occurrences_since_last_report is the count, and the log lines carry the real timestamps. A tail event's project_id and rule_uuid are null by design — the occurrence that would have named them is long gone by the time the window is reclaimed — so step 2's triage path doesn't apply to one. Pivot to the log lines via the tags (automation.vendor_id + automation.failure_class) to find the affected projects.
The fingerprint includes the action slug, and callers pass either updateSalesforceField or Update Salesforce Field for the same action. normalize_action collapses both — if a split appears, a new caller is passing a third spelling that doesn't normalize to the same slug.
Check the phrase you added is lowercase and appears in the provider string as a substring — matching is a plain in against a lowercased haystack of error_type, message and JSON-dumped details. No regex, no analyzer.