ONRAMP-6068 Spike — D1 (inbound transport & vendor) + D5 (ingestion architecture)
Written 2026-08-10 against main
24864a73a5for ONRAMP-6068. Code line numbers are stale; current facts live in the ONRAMP-7078 stories.
Scope: D1 and D5 only. D2/D4 (token scheme + security) and D3 (parse benchmark) are owned by other agents — referenced here as "see D2" / "see D3", never duplicated. Verified at: repo HEAD 24864a73a5 (branch main), 2026-08-10. Every repo claim below carries a path + line I actually read at HEAD; drift from the planning docs is tabulated in §7. Evidence labels: [V] verified in repo/data I read · [I] inferred from verified facts · [H] needs a human with AWS/DNS/EasyCron console access.
0. Executive recommendation
| Decision | Recommendation | Confidence |
|---|---|---|
| D1 — transport | AWS SES inbound receipt rules (classic email receiving, us-west-2), conditional on a pre-registered D3 exit criterion. Not Mail Manager. Not an inbound-only vendor. | Medium-high, and deliberately falsifiable — see §1.7 |
| D5 — ingestion | Neither option in the brief. Copy DocuSign: SES → S3 (durable raw MIME) + notification → thin Lambda → HMAC-signed POST to Flask → verify → stage → process inline → mark processed, with an authenticated backstop cron drain. | High — this is a pattern already running in this repo |
The brief frames D5 as "synchronous POST vs S3/SQS cron-drained" and PLAN.md Q8 recommends synchronous. That dichotomy is false, and this repo already resolved it once: the DocuSign Connect receiver is synchronous and durable and has an authenticated cron backstop. Adopting it gets the latency of the sync option and the survivability of the cron option, and — because SES writes raw MIME to S3 before Flask is ever involved — the dead-letter requirement is satisfied by the transport itself rather than by application code.
1. D1 — Inbound transport & vendor
1.1 How sending actually works at HEAD — brief CONFIRMED, with one correction
The brief's claim ("SQS → an external Lambda → AWS SES, and the Lambda lives outside this repo") is correct. [V]
| Link in the chain | Evidence at HEAD |
|---|---|
| App never calls an ESP SDK | app/api/utils/email/Email.py:749-757 — send_email() does only EmailMessageBus(vendor_id=self.vendor_id).call_queue(json.dumps(email)) |
| Queue publish | app/api/utils/sqs_helper.py:177-182 — EmailMessageBus(SQSHelper), queue name from DB config key email-queue-name; call_queue → invoke_queue(..., always_send_to_sqs=True) (:162-174), non-FIFO |
| The payload is explicitly for a Lambda | Email.py:580 — the literal comment # package up a response that can be sent to the lambda |
Queue name is a DB row, not IaC and not environments.yaml | devtools/db/seed_data.sql:4491 → ('email-queue-name','duploservices-dev01-EmailQueue'). The duploservices- prefix is DuploCloud tenant naming → Duplo-provisioned |
| The repo does not even know the consumer's name | email_monitoring_service.py:233 _get_lambda_from_queue() discovers it at runtime via lambda:ListEventSourceMappings (docs/EMAIL-MONITORING-IAM-SETUP.md:103) |
| Provider is SES | email_monitoring_service.py:367,370 (get_send_quota, get_send_statistics); app/api/email_tracking/services/ses_vdm_insights_client.py:40 (sesv2.get_message_insights); app/api/api/__init__.py:397 BOTO3_SERVICES includes ses,sesv2 |
| No sending Lambda in this repo | Exhaustive grep for lambda_handler / exports.handler / def handler( → the only inline handler is the model-EOL canary (infra/components/model-eol-alerter.ts:346). Nothing sends email. [V] |
Correction to add: the outbound payload is asymmetric about Reply-To, and this matters to the epic.
Email.py:581-588 freshly-built branch → {From, To, Cc, Subject, RawMessage, transaction_id} ← NO top-level Reply-To
Email.py:592-600 pre-built-body branch → {From, To, Cc, Reply-To, Subject, RawMessage, ...} ← HAS top-level Reply-ToReply-To is set on the MIME envelope at Email.py:557, so it survives inside RawMessage in both branches — but it is a top-level key only in the deferred/pre-built branch. Whether a tokenized Reply-To actually reaches the recipient therefore depends on something not in this repo: if the Lambda calls SendRawEmail/SendEmail with RawMessage, the header survives and the outbound half genuinely needs no change; if it reads the top-level keys and reconstructs, the freshly-built branch's Reply-To is dropped — and the manual nudge, the one email that sets Reply-To today, goes through exactly that branch (task/services_old.py:3151-3153 → Email.py:581-588).
The brief calls the outbound half "cheap — no schema change needed." That is true of the schema and unverified end-to-end. It is corroborated by observed behavior (the brief and PLAN.md §3 both state that nudge replies land in the CSM's inbox today, which is only possible if the Lambda honors RawMessage) — so this is probably fine. But it is one RawMessage-vs-top-level-keys detail away from "we shipped tokenized Reply-To and no reply ever came back," and it costs one live send test on a deployed env to close. Do that before 6073 estimates are trusted. [I→H]
1.2 What IaC we own — and the two prerequisites nobody has listed
Pulumi (TypeScript) is the only IaC; no Terraform/CDK/CloudFormation/SAM/K8s anywhere. Two sanctioned projects (infra/app, infra/ci-identity); new Pulumi projects are frozen and file-gated (.claude/rules/infra-composition.md). App runtime for dev/stage/prod/demo/gdpr-prod is DuploCloud-managed and not in this repo. [V]
SES: nothing to extend. Exhaustive grep across infra/ .github/ docker/ hosting-conf/ scripts/ for ses:[A-Z], aws.ses, sesv2, EmailIdentity, ConfigurationSet, ReceiptRule, ReceiptRuleSet, RuleSet, DomainIdentity, MailFrom, DkimAttributes, verifiedIdentity → zero matches. SES identity/DKIM/configuration-set state is console- or Duplo-provisioned outside this repo. [V]
No S3→SNS/SQS pipeline exists to extend either. grep -rn "BucketNotification|bucketNotification" infra/ → 0. Every SNS subscription in the repo is protocol: "email" (human alerts, model-eol-alerter.ts:493-501) or Slack HTTPS (datawarehouse/alarms.ts:45). The only SQS queue in IaC is onramp-model-eol-canary-dlq. SES inbound is greenfield. [V]
Prerequisite #1 nobody has named: the CI deploy-role permissions boundary grants no
ses:action at all.infra/ci-identity/boundary-policy.tsenumerates allowed prefixes (s3:*,lambda:*,cloudfront:*,route53:ChangeResourceRecordSets,ssm:*,dynamodb:*, …);sesappears nowhere in it, nor inboundary-policy.legacy.json. Any Pulumi-declared SES resource is denied today. Fixing it means an admin-applied change toci-identity, a bootstrap project — not a normal PR. [V]
Prerequisite #2: the runtime IAM policy is sending-side read-only.
docs/EMAIL-MONITORING-IAM-SETUP.mdgrantsses:GetSendStatistics/GetSendQuota/GetAccountSendingEnabled, suppression-list read/delete,ses:GetMessageInsights. No receipt-rule permission of any kind. [V]
1.3 DNS — the GoDaddy fact is confirmed, and it does not favor a vendor
onramp.us is hosted at GoDaddy, verified from repo code rather than folklore: app/api/admin/services/domains/godaddy_service.py:18 (https://api.godaddy.com/v1) and :263 (f"{self.base_url}/domains/onramp.us/records"), with ADMIN_DOMAINS_SETUP.md:8-20 describing the API key/secret setup. That code path is a runtime customer-white-label-subdomain feature (CNAMEs for <tenant> portals) — not IaC, and it has no MX/TXT/DKIM capability wired. [V]
Route53 is used, but for a different domain: onrampapps.com (zone {redacted}), of which Pulumi owns exactly two records — the *.preview.onrampapps.com A/AAAA CloudFront aliases (infra/components/spa-edge.ts:470-482). The parent zone is explicitly manual: "Pulumi never touches it" (infra/README.md:505). No aws.route53.Zone resource is declared anywhere. Watch a stale-doc trap: infra/pulumi/README.md still says preview.onramp.us while the implemented code uses preview.onrampapps.com — do not read that file as evidence that onramp.us is in Route53. [V]
Mail DNS in IaC: repo-wide grep for \bMX\b, mxRecord, dmarc, _domainkey, \bspf\b → 0 hits in infra/. [V]
This is where I disagree with the brief's framing. The brief presents SES's DNS/infra burden as a cost that an inbound-only vendor avoids. It does not. Both options require the identical change: a new MX record on the receiving subdomain, created at GoDaddy by hand or via the GoDaddy API. SES points it at inbound-smtp.us-west-2.amazonaws.com; Postmark/SendGrid/Mailgun point it at their inbound host. Same zone, same owner, same manual step, same Pulumi-can't-manage-it constraint. The DNS axis is a wash and should be struck from the comparison. [I, from verified DNS topology + §1.4]
1.4 Region availability — resolved, and it was a real risk
SES email receiving is not available in every region, and all receiving-side resources (SNS topics, KMS keys, Lambdas) must sit in the same region as the SES endpoint — S3 buckets are the only exception (https://docs.aws.amazon.com/ses/latest/dg/regions.html). Our home region is us-west-2 (infra/README.md:48).
us-west-2 is a supported email-receiving region: the AWS General Reference "Email Receiving endpoints" table lists inbound-smtp.us-west-2.amazonaws.com (https://docs.aws.amazon.com/general/latest/gr/ses.html). us-east-1, us-east-2, us-west-1, eu-west-1, eu-central-1, eu-west-2 and others are also listed. [V — authoritative AWS docs]
This had the potential to kill the SES option outright and does not. It also means the EU/GDPR deployment can receive in-region (see §1.8).
1.5 Cost at real volume — measured, not assumed
AWS MCP call_aws and the aws CLI both had no usable credentials in this session (the shell aws is aliased to duploctl jit web --admin --interactive, which needs an interactive login). Per repo doctrine that is not an AWS outage and not a blocker — I went to the production read replica instead. So SES get_send_statistics is unread [H], but the send volume it would have shown is measured below from our own tracking table.
Source: prod read replica onrampdb_prod (pg_is_in_recovery() = t guard passed), read-only, aggregates only.
Table sizes: or_email_tracking 687,526 rows / 462 MB · email 3,089,002 rows / 16 GB · or_email_jobs 7 rows.
Monthly send volume (or_email_tracking, created 2026-03-06 by migration 2026_03_06_9d2c8a4f1b6e, so March is partial):
| Month | Emails |
|---|---|
| 2026-03 | 62,450 (partial) |
| 2026-04 | 138,142 |
| 2026-05 | 131,577 |
| 2026-06 | 147,045 |
| 2026-07 (last full month) | 165,909 |
| 2026-08 (through the 10th) | 47,424 → ~4,742/day → ~147K/month pace |
July 2026 by template, and the customer/internal split (recipient resolved by lower(email_address) + vendor, because recipient_id is unusable — see §1.6). Reply-eligible templates only; the per-template sum across all 28 types reconciles exactly to 165,909, so there are no unattributed rows.
| Template | To customer | To internal | Vendors |
|---|---|---|---|
weekly_summary | 51,618 | 725 | 112 |
overdue_tasks | 22,400 | 210 | 63 |
upcoming_tasks | 7,201 | 128 | 47 |
comment | 4,897 | 6,446 | 68 |
task_assigned | 4,679 | 2,657 | 53 |
task_nudge_reminder | 2,327 | 237 | 50 |
tasks_starting_today | 1,464 | 76 | 17 |
tasks_reminder | 663 | 2,466 | 53 |
| Total customer-facing, reply-eligible | 95,249 |
Caveat: prod us-west-2 only. gdpr-prod/EU is a separate database not measured here, and demo/stage are excluded. Real global volume is higher. [V + H for the EU delta]
Inbound volume = reply rate × exposed sends. Reply rates to transactional notification mail are low single-digit percentages; at 1 / 3 / 5% of 95,249 that is ~950 / ~2,860 / ~4,760 inbound emails per month.
SES inbound à la carte pricing (authoritative, https://aws.amazon.com/ses/pricing/): $0.10 per 1,000 emails received, plus $0.09 per 1,000 incoming email chunks (256 KB per chunk).
| Scenario | Inbound/mo | SES inbound cost/mo |
|---|---|---|
| Full reply-eligible surface @ 3% | 2,860 | ~$0.55 |
| Full reply-eligible surface @ 5% | 4,760 | ~$0.91 |
| PLAN.md §6 scope (nudge + task_assigned) @ 3% | 210 | ~$0.04 |
For scale: our outbound 165,909/month at à la carte $0.10/1,000 is $16.59/month.
An inbound-only vendor at these volumes sits in the low tens of dollars per month on a published entry plan (exact list price [H] — check the vendor's current pricing page; do not quote mine). The entire cost spread across the whole option space is under ~$50/month ≈ two engineer-hours. Cost is not a decision input. The brief is right that cost is not decisive; it is now right with numbers, and the numbers are more lopsided than "not decisive" implies.
⚠️ One pricing wrinkle needing a human: SES introduced tiered plans (Essentials/Pro/Enterprise) alongside à la carte, and "new SES accounts and account × region combinations with no metered SES activity since June 1, 2025 will start on the Essentials plan beginning July 21, 2026." On a plan, inbound is billed as Mail Manager Email Processing ($0.15/1,000, and an Open Ingress Endpoint is $50/month) rather than the classic à la carte Inbound email line. Our us-west-2 account plainly has metered activity, so à la carte classic receiving should still apply — but confirm which plan the account is on before quoting a number to finance. [H]
1.6 Two data findings that change the plan, not just the cost model
(a) or_email_tracking.recipient_id is NULL in 100% of production rows. 0 of 165,909 July rows populated (account_id: 151,903 populated). Root cause verified in code, not guessed: Email.py:526-533 calls EmailTrackingCreateService().create_or_ignore(...) and never passes recipient_id, which defaults to None (email_tracking_create_service.py:22). The only recipient_id write anywhere is on the different table OREmail (custom_email_template_service.py:1273). So or_email_tracking.recipient_id has no writer at all. [V]
This is also a caught measurement trap worth recording: my first customer/internal split query joined on recipient_id and returned to_customer = 0 for every template. That reads as "we send no customer email," which is false. Same class as the request_url trap in the prod-db skill — the column cannot hold the answer.
Consequence for the epic: PLAN.md §5.2 wants email_transaction_id on the token "for funnel correlation," and §5.4 wants via-email attribution telemetry. Correlating an inbound reply back to a contact through or_email_tracking does not work today. This independently corroborates D2's recommendation to bind user_id on the token row and treat the sender address as confirmation only — the token is currently the only reliable link. Either backfill/populate recipient_id (small, and worth its own ticket) or accept the token as the sole correlation key.
(b) Our outbound emails have no text/plain part. Email.py:560-563 builds MIMEMultipart("alternative") and attaches only an HTML child; repo-wide grep for text/plain / MIMEText(..., "plain") across app/api/utils/email/ and app/api/custom_email_templates/ → 0 hits. Every notification is HTML-only inside a degenerate single-child multipart/alternative. [V]
Three consequences, and the third is the important one:
- Mild deliverability/accessibility cost (HTML-only is a spam-scoring signal; plain-text readers get nothing). Worth its own ticket regardless of this epic.
- A one-part
multipart/alternativeis technically odd and some clients treat it unpredictably. - It raises parse risk on both D1 options. When a customer replies, their client quotes our 15 KB nested-table HTML (
main_email_template.html). Clients that synthesize atext/plainpart produce a machine-flattening of that table — historically the worst input for reply-stripping heuristics. Clients that reply HTML-only hit exactly the documented condition under which Postmark'sStrippedTextReplydoes not populate at all, which the brief already flags.
Actionable and independent of D1: add a text/plain alternative to the outbound templates. It improves the quoted-reply shape for every client, helps deliverability, and is the single cheapest thing that de-risks D3 — and it is not in PLAN.md anywhere. Recommend it as a W1 enabler alongside 6069.
1.7 The decisive axis is parse-quality ownership — handing it to D3 with a pre-registered threshold
The brief is right that this is the deciding axis, and right that the decision belongs to measurement rather than a feature matrix. Restating it precisely so D3 can settle it:
- SES gives raw MIME. We own 100% of quoted-history stripping, signature stripping, and OOO/autoresponder/bounce detection, forever.
- A vendor gives a hosted stripped-reply field plus the mailbox-hash pattern, at the cost of a second vendor, a split deliverability/observability surface, and a new dependency on a path that must not silently drop customer replies.
Three things narrow that gap more than the brief allows:
- We are writing raw-MIME storage and parse code either way. PLAN.md §5.4 already requires storing the raw inbound message "for audit + parse debugging," and §5.2 already specifies a fallback parser (
email_reply_parserlineage) in addition to the provider's stripped field. So the vendor does not remove the parser — it adds a preferred input to it. The marginal deliverable SES adds is "make the fallback good enough to be primary." - The vendor's stripped field has a documented hole that our own templates make more likely (§1.6b). A hosted field that silently doesn't populate is worse than a parser we control, because the failure is invisible to us.
- SPF/DKIM/DMARC verdicts are available on SES too — receipt rules surface them in the rule metadata, so D4's verdict-handling design is not vendor-dependent.
Explicit hand-off to D3. D1 cannot be closed on repo evidence; it needs one number. D3 must report, on ≥20 real replies to our actual notification emails across Gmail / Outlook desktop / Outlook mobile / Apple Mail:
- clean-extraction rate of an open-source parser alone (the SES path), and
- clean-extraction rate of a vendor stripped-reply field, on the same corpus, and
- autoresponder/OOO/bounce detection rate, and
- per-reply: was a
text/plainpart present at all? (the Postmark-hole rate — §1.6b makes this non-theoretical)Pre-registered exit criterion, agreed before the numbers land so this isn't rationalized after the fact: if the open-source parser reaches ≥90% clean extraction with ≥95% autoresponder detection, ship SES. If it lands below 80%, the vendor's parse quality is buying something real — switch to the vendor and accept the second-vendor cost. Between 80% and 90%, decide on the shape of the failures: systematic and fixable (one client's quote marker) → SES plus targeted fixes; diffuse and unpredictable → vendor.
1.8 Two arguments for SES the brief does not make
EU/GDPR data residency. app/config/environments.yaml defines a gdpr-prod environment (line 115) alongside prod (110). Inbound email means receiving and storing customer-authored message content — squarely in scope for the EU deployment's residency posture. SES receives in-region (eu-west-1/eu-central-1/eu-west-2 are all supported receiving regions, §1.4) inside the same account and boundary model we already operate. Routing EU customers' replies through a US inbound-parse vendor is a DPA/sub-processor conversation with legal, plus a second region's worth of vendor configuration. This is a structural advantage for SES that no amount of parse quality offsets, and it is absent from the brief, PLAN.md, and the epic. It also raises a scope question nobody has asked (§4, item 11): does reply-by-email ship to EU orgs in v1 at all?
Config/infra doctrine fit. The SES path is expressible entirely in mechanisms this repo already has and already governs: a Tier-1 Pulumi component member in infra/app/envs.ts, a per-env *_provisioned: false opt-in boolean in environments.yaml (exact precedent: docusign_connect_hmac_provisioned:61, esign_token_encryption_provisioned:88), and an SSM secret leaf. A vendor adds an API credential (also SSM — fine) plus a vendor account, a second reputation surface, and a second observability plane that the email-monitoring dashboard (email_monitoring_service.py) knows nothing about.
1.9 Considered and rejected: SES Mail Manager
AWS's newer inbound product appears in the pricing page and is not the right shape for us: Email Processing is $0.15/1,000 (vs $0.10/1,000 classic inbound), and receiving arbitrary internet mail on our own domain needs an Open Ingress Endpoint at $50/month per endpoint per region — "charges apply even when endpoint status is Closed." Classic receipt rules deliver exactly what we need (recipient-condition matching → S3 + SNS/Lambda) at lower cost and lower conceptual weight. Revisit only if we later want its add-on scanning marketplace (Trend Micro / Spamhaus / Abusix). [V]
2. D5 — Ingestion architecture
2.1 The in-repo model, read verbatim
app/api/integrations/docusign/routes.py:82-160 is the receiver; webhook_verification.py:23-56 is the pure verifier; :179-181 is the authenticated backstop drain. Read at HEAD. The disciplines the inbound-email receiver must copy, each tied to its line:
| # | Discipline | Where | Why it is load-bearing for inbound email |
|---|---|---|---|
| 1 | Raw body captured before any parse — raw_body = request.get_data() is the first statement | routes.py:89 | HMAC is over the exact bytes as sent; a re-serialization will not match. For MIME this is doubly true — re-encoding a MIME body changes it. |
| 2 | Reject on missing signature header BEFORE reading the secret | routes.py:91-108, and the comment at :92-94 says why | So anonymous traffic cannot drive SSM GetParameter load. GetParameter throughput is account-shared — an unauthenticated URL that reads a secret per request is a free throttling attack on every other SSM consumer. |
| 3 | Split scanner noise from real-but-unsigned traffic — _is_connect_shaped() cheap shape check; shaped-but-unsigned gets its own searchable log line, unshaped gets a generic one; both log-only, neither mints Sentry | routes.py:65-79 + :102-107 | An unauthenticated URL is dominated by scanners. Minting Sentry from unsigned traffic lets the internet drown the one signal we actually alert on. Inbound analogue: a body that parses as MIME with a Received: header and a recipient matching our inbound subdomain is "real-but-unsigned" (a misconfigured receipt rule / Lambda missing its signing key) — the one real misconfiguration class, and it must stay searchable. Alert on it via a log-based metric, not Sentry. |
| 4 | Sentry reserved strictly for our-side misconfiguration | _reject_unverified, routes.py:49-62 | Reached only after a signature header exists or a secret read completed, so strangers cannot mint Sentry events. Copy this exact gating. |
| 5 | Cap the number of signature headers examined | webhook_verification.py:29-37, DOCUSIGN_MAX_CONNECT_SIGNATURES | Prevents a public caller amplifying HMAC CPU by presenting many forged headers. This is the concrete answer to the brief's "no rate limiting exists app-wide" concern for the cheapest abuse vector — bound the work per request rather than reaching for Flask-Limiter. |
| 6 | Constant-time compare, accept-any-of-N-keys | webhook_verification.py:40-56 (hmac.compare_digest; digest computed once per key, not per key×header) | Zero-downtime secret rotation with two active keys. Empty key set or absent headers fail closed. |
| 7 | Signed but unparseable → non-2xx, so it stays in the sender's retry queue instead of being silently dropped | routes.py:125-131 | Directly serves "must not silently drop customer replies." |
| 8 | Staging failure → non-2xx + explicit capture_exception | routes.py:133-142 | A staging failure means guaranteed redelivery; it must always be in Sentry. |
| 9 | Inline-processing failure AFTER successful staging → swallow, keep 2xx, let the drain retry | routes.py:144-159 | The subtle one. Once the event is durably staged, turning the ack non-2xx causes redelivery of an event we already hold — duplicate answers written to a task. Get this backwards and one flaky parse becomes a storm of duplicate task answers. |
| 10 | Receive-side is NOT feature-flag gated; the DRAIN is | webhook_service.py:35-40 (docstring) | Because the sender does not redeliver after a 2xx, a receive-side flag check permanently drops events arriving while an org's flag is off mid-rollout. Given PLAN.md §5.0 puts this whole epic behind a per-org flag, this trap is live for 6073/6074: stage every verified inbound reply regardless of the org's flag state, and gate processing. |
Additional shapes to reuse rather than reinvent:
- Staging table.
IntegrationWebhookData(app/api/platform_integrations/or_models.py:304-317) already carries exactly the right columns:external_id(dedupe key, indexed),webhook_dataJSON,or_vendor_id,processed_at(indexed),attempts,last_error,last_attempt_at. Copy this shape foror_inbound_email, withexternal_id= the SESmessageId. - Bounded retries via the config map, not a magic number: precedent
docusign_webhook_max_process_attempts(app/config/environments.yaml:69) anddocusign_webhook_drain_batch_size(:68). - SSM secret + fail-closed boot check.
app/api/integrations/docusign/secrets.pywholesale:_ssm_parameter_namelowercasesENVIRONMENT_NAME(:36-42); a 60s in-process memo bounds reads to ~1GetParameter/min/worker so scanner traffic cannot throttle SSM (:56-60and the comment above it); the boot check derives enforcement from the config map and skips envs that have not opted in (:100-165). Read thedev-config-env-name-overloadtrap first:_NO_AWS_RUNTIME_ERRORS = (NoRegionError, NoCredentialsError)(:33) is what stops a local/CI stack that inheritsENVIRONMENT_NAME=devfrom crash-looping the app and the seed. A genuine deployed misconfiguration still fails closed, because there the SSM call reaches AWS and returnsParameterNotFound/AccessDenied. Recommended: secret at/onramp/{env}/inbound-email/webhook-hmac, gated by a newinbound_email_hmac_provisioned: falseboolean inenvironments.yamlfollowingdocusign_connect_hmac_provisioned:61. Note the existing guard rejects a non-boolean (e.g. YAML"true"as a string) loudly rather than silently skipping enforcement — keep that.
2.2 The recommended architecture
customer reply
→ MX (reply.onramp.us, at GoDaddy) → inbound-smtp.us-west-2.amazonaws.com
→ SES receipt rule (us-west-2, ACTIVE rule set)
├─ S3 action: raw MIME → s3://onramp-inbound-email-<env>/ ← DURABLE, pre-Flask
└─ SNS/Lambda action: notify
→ thin Lambda (us-west-2): read S3 key + SES verdicts, HMAC-sign, POST
→ POST /api/inbound-email (Flask, unauthenticated, HMAC-verified)
1. raw_body = request.get_data() ← before any parse
2. no signature header? → log-only 401 (shaped vs unshaped split)
3. read HMAC keys from SSM (memoized) ← only now
4. verify (constant-time, any-of-N keys) ← fail closed
5. STAGE row in or_inbound_email (dedupe on SES messageId)
6. process inline (resolve token → parse → answer/comment)
7. mark processed_at; inline failure → swallow, stay 2xx
→ POST /api/inbound_email_jobs/drain @requires_easy_cron_token ← backstopWhy this beats both options in the brief:
- vs. pure synchronous with provider retries. Provider-side retry is bounded (a vendor typically retries for hours, then gives up; DocuSign is unusually generous at 15 days —
routes.py:4-6). With SES→S3 the raw MIME is durable indefinitely and independent of Flask, the Lambda, and the retry window. The dead-letter is the transport, not application code. - vs. pure S3/SQS cron-drained. Latency matters here in a way it does not for e-sign status: a customer who answers a question by email expects the task to reflect it, and a CSM watching the project expects the same. Waiting for the next cron tick is a product regression for the happy path. Inline processing gives immediate effect; the drain exists only for the unhappy path.
- Both, on the hard requirement. "Must not silently drop customer replies" is met at three independent layers (§2.4), no one of which is application-logic-dependent.
2.3 The EasyCron burden is real but small — and it does not deepen the auth problem
I read the procedure rather than guessing: .agents/skills/backend-development/SKILL.md:94-130.
The registered list (:99-118) already contains the five endpoints this epic touches: /api/project_daily_jobs/{upcoming_tasks,overdue_tasks,tasks_starting_today}_bottom_up, /api/project_daily_jobs/customer_weekly_summary, and /api/email_jobs/send_pending_customer_emails, plus /dailynotifications.
Actual burden of adding one backstop drain endpoint:
- Add the URL to the list in
SKILL.md, marked# job not created yet(:130). - Label the PR
EASYCRON_URL(:122). - Warning block near the top of the PR description (
:123-127). - A human creates the cron jobs — one per deployment, each behind its own domain (
app.onramp.us,eu.onramp.us,demo.onramp.us,stage.onramp.us), naming any deliberate exclusions (:130). - Because the endpoint will be authenticated, the console jobs must carry
X-EasyCron-Tokenfrom creation or every run 401s.
That is one list edit, one label, one PR note, and a human creating ~4 console jobs with a header. A few person-hours of coordination — not an architectural cost. The brief and PLAN.md Q8 both treat "EasyCron PR-coordination burden" as a reason to prefer synchronous; on the evidence that weighting is too high. The backstop is worth it, and the recommendation in §2.2 keeps synchronous processing anyway — the drain is additive.
On the unauthenticated-cron concern — verified, and it cuts the other way. Confirmed exactly at HEAD:
| Endpoint | Auth |
|---|---|
daily_jobs/routes.py:29 tasks_starting_today_bottom_up | none |
daily_jobs/routes.py:115 upcoming_tasks_bottom_up | none |
daily_jobs/routes.py:247 overdue_tasks_bottom_up | none |
daily_jobs/routes.py:515 customer_weekly_summary | none |
email_jobs/routes.py:10 send_pending_customer_emails | none |
daily_jobs/routes.py:550-552 calculate_projects_metrics | @requires_easy_cron_token |
docusign/routes.py:163-165 refresh-keepalive | @requires_easy_cron_token |
docusign/routes.py:179-181 process-webhook-events | @requires_easy_cron_token |
The four digest writers and the drain are unauthenticated [V] — PLAN.md open question 11 is accurate. But the precedent for a new cron endpoint in this repo is an authenticated one: DocuSign added two, both decorated. cron_auth.py:5 is explicit that the unauthenticated set is legacy — "Existing OnRamp EasyCron endpoints have no auth and rely on URL obscurity."
So adding a backstop drain deepens nothing, provided it ships with @requires_easy_cron_token. The premise in my instructions — that another cron endpoint would deepen an existing problem — holds only for an unauthenticated one, and there is no reason to add one. I'd go further: the inbound-email drain is a natural, low-risk vehicle to demonstrate the decorator on an email-adjacent endpoint, strengthening the case for the standalone ticket that retrofits the other five.
⚠️ A doctrine violation in the auth primitive itself, noted in neither the brief nor PLAN.md.
app/api/utils/cron_auth.py:46readsenviron.get("EASYCRON_SHARED_SECRET")— and:37readsenviron.get("ENVIRONMENT_NAME", "local_dev"). Per the binding config rules inCLAUDE.md,ENVIRONMENT_NAMEis the one sanctioned env var; a shared secret belongs in AWS SSM. So the existing cron-auth primitive violates the repo's own configuration doctrine. [V] Recommendation: use it as-is for the inbound drain and file a separate ticket to move the secret to SSM. Changing it in this epic's PR would simultaneously touch every existing EasyCron job's configuration — exactly the lockstep-or-401 failure mode described above. Do not let this epic become the vehicle for that migration, and do not let a reviewer talk you intoenviron.get("X") or <default>, whichCLAUDE.mdbans outright.
2.4 Failure path — what happens when Flask is down
The hard requirement is "must not silently drop customer replies." Three independent layers, then detection.
Layer 1 — S3 is durable before Flask exists in the story. The SES receipt rule's S3 action writes raw MIME on receipt. If Flask is down, the Lambda's POST fails and the mail is already saved. Nothing is ever only-in-flight. Set a bucket lifecycle retention comfortably exceeding any plausible outage + investigation window (90 days suggested); this doubles as the raw-message audit store PLAN.md §5.4 already requires, so it is not extra infrastructure.
Layer 2 — bounded retry with accounting. Lambda retries its POST with backoff (and SNS carries its own retry policy + DLQ if the notification path is SNS-based). Beyond that, the staging row carries attempts / last_error / last_attempt_at / processed_at (or_models.py:314-317), with the max bounded by a config-map key, not a literal.
Layer 3 — authenticated backstop drain. Reconciles two ways: staged-but-unprocessed rows, and S3 objects with no staging row at all (the case where Flask was down for the entire Lambda retry budget). The second reconciliation is the one that actually covers a long outage, and it is the one most likely to be forgotten.
Detection — the part that decides whether a drop is silent. Alarms, not logs alone:
- Age of the oldest
processed_at IS NULLstaging row exceeding N minutes. This is the single most valuable alarm; it catches stuck parses, poison rows, and outages alike. - Count of rows at
attempts >= max(a poison queue). These must be surfaced for human review. - S3-object count vs staging-row count over a rolling window — the only thing that detects mail that never reached the app at all.
logger.exceptionis alertable in this stack (per the background-thread/cron review facts), so the DocuSign Sentry discipline (#4/#8 above) transfers directly.
Two hard "do nots", both grounded in verified in-repo bugs:
Do NOT copy
_claim_email_job.app/api/email_jobs/services.py:52-70claims a job withDELETE ... RETURNING— the row is gone before the send is attempted — and the caller'sexceptat:45-47only logs. A send failure loses that email permanently, with no requeue path. Verified at HEAD; it is defect (c) in PLAN.md §5.3. This is the exact anti-pattern the inbound path must avoid: stage, mark processed, never delete-on-claim. Claim with aprocessed_at/attemptsupdate orSELECT ... FOR UPDATE SKIP LOCKED, never by deleting the evidence.Do NOT let exhausted-attempt rows be deleted or pruned. A customer reply that failed to parse five times is a product bug report with a customer waiting on the other end.
One more inherited trap worth stating in the 6073 spec: vendor_id = 0 is a legitimate tenant (.claude/rules/python.md), so any if not vendor_id guard on the inbound path is a bug — use is None. Local flows run as a seeded customer org, so local testing will not catch it; a test must pin the fixture to 0 explicitly. The staging table's or_vendor_id resolution and every drain filter are the likely sites.
3. What must change OUTSIDE this repo — the epic's most under-scoped risk
The brief is right to call this out. Concrete inventory, with what I could verify from the repo vs what needs console access.
| # | Change | Where it lives | Plausible owner | Status |
|---|---|---|---|---|
| 1 | MX record for the inbound subdomain (e.g. reply.onramp.us → inbound-smtp.us-west-2.amazonaws.com) | GoDaddy zone for onramp.us | Holder of the GoDaddy account / API credential (ADMIN_DOMAINS_SETUP.md:8-20) | Zone location [V] (godaddy_service.py:263); the record itself [H]. Pulumi cannot manage this. Required identically for a vendor. |
| 2 | TXT domain-verification record for the receiving domain in SES | GoDaddy | same as #1 | [H] |
| 3 | SES receipt rule set + rule (recipient condition, S3 action, SNS/Lambda action), in us-west-2, and ACTIVE — only one rule set can be active per region | AWS console, or a new infra/components/ses-inbound.ts Tier-1 member | Infra owner | ABSENT from repo IaC by exhaustive grep [V] |
| 4 | S3 bucket for raw inbound MIME + AllowSESPuts bucket policy (+ KMS grant if encrypted) | new Pulumi | Infra owner | No BucketNotification pattern exists anywhere to extend [V] |
| 5 | The receiving Lambda (S3/SNS → HMAC-signed POST to Flask). Net-new; no in-repo lambda resembles it | New code; placement per docs/proposals/domain-lambdaliths.md | Backend + Infra | [V] no such lambda in repo |
| 6 | CI deploy-role permissions boundary must grant ses: — it currently grants none | infra/ci-identity/boundary-policy.ts (admin-applied bootstrap project) | Whoever can apply ci-identity | [V]. Hard blocker for any Pulumi-managed SES resource; listed in no planning doc. |
| 7 | Runtime IAM if the app ever calls SES receiving APIs — current policy is sending-side read-only | docs/EMAIL-MONITORING-IAM-SETUP.md + the Duplo task role | Infra owner | [V] no receipt-rule permission today |
| 8 | SSM parameter for the inbound HMAC secret (/onramp/{env}/inbound-email/webhook-hmac) | AWS SSM, per env | Backend + Infra | Pattern [V] (docusign/secrets.py); provisioning [H] |
| 9 | EasyCron jobs for the backstop drain — one per deployment, carrying X-EasyCron-Token | EasyCron console | Whoever manages EasyCron | Procedure [V] (SKILL.md:120-130); job creation [H] |
| 10 | The SES-sending Lambda — needed only if the tokenized Reply-To does not survive (§1.1) | Outside repo, Duplo-provisioned; name discoverable only at runtime | Owner of the Duplo email stack | Queue name duploservices-dev01-EmailQueue [V] (seed_data.sql:4491); consumer not in repo [V]; behavior [H] |
| 11 | EU/GDPR: a second receipt rule set + MX for the EU deployment, or an explicit decision that EU orgs get no reply-by-email in v1 | eu-* region + DNS | Product + Infra + Legal | gdpr-prod env exists [V] (environments.yaml:115); the decision [H]. Raised nowhere in the epic. |
| 12 | SES account posture: which pricing plan the account is on (§1.5), and whether it is out of the sandbox for receiving | AWS billing/SES console | Infra / Finance | [H] |
Placement guidance, so this lands correctly: per .claude/rules/infra-composition.md, Tier 3 is FROZEN and file-gated — do not create infra/ses/. The right shape is a new infra/components/ses-inbound.ts plus a presence-toggled member in infra/app/envs.ts composed by infra/app/index.ts, following the resolveTier derive-don't-configure model. Adopting console-created SES resources into Pulumi later has a runbook precedent: infra/app/CUTOVER.md.
4. Where I disagree with the going-in position, and what contradicts the plan
- The one-click surface is ~7% of the addressable audience, and nobody has priced that. (The most consequential finding in this document.) PLAN.md §6 confines 6070 one-click links to immediate emails — nudge + task_assigned. Measured against real July volume, that is 7,006 customer-facing sends/month out of 95,249 reply-eligible (7.4%). The four deferred digests excluded by that decision are 82,683/month (87%), and
weekly_summaryalone is 51,618/month across 112 vendors — the single largest email surface we operate.task_nudge_reminder, the epic's hero flow, is 2,327/month (1.4% of all mail). The §6 scope decision is technically well-reasoned (tokens minted at enqueue burn their TTL before delivery —deferred-send-timing.md). But it converts the epic's flagship mechanism into a feature reaching a low-single-digit share of email. Open question 12 (render digests at drain) is therefore load-bearing for whether this epic is worth the infra lift in §3, not a follow-up ticket. Recommend promoting the render-at-drain refactor to a W1 enabler, or consciously accepting a ~7% v1 and saying so in the epic. - The DNS burden does not differentiate the options. Both SES and any vendor need the same new MX record in the same GoDaddy zone that Pulumi cannot manage (§1.3). Strike it from the comparison.
- D5's dichotomy is false and the repo already disproves it (§2.1-2.2). Neither "sync with provider retries" nor "S3/SQS cron-drained" is the answer; DocuSign's stage-and-backstop hybrid is, and it is already in production here.
- The EasyCron burden is over-weighted in the brief and PLAN.md Q8 — one list edit, one label, one PR note, ~4 human-created console jobs (§2.3).
- Adding a cron endpoint need not deepen the unauthenticated-cron problem. The last two cron endpoints added (both DocuSign) carry
@requires_easy_cron_token. The unauthenticated five are legacy, ascron_auth.py:5admits (§2.3). requires_easy_cron_tokenitself violates the repo's config doctrine —EASYCRON_SHARED_SECRETviaenviron(cron_auth.py:46) where doctrine mandates SSM. Noted in neither the brief nor PLAN.md. Ticket it; do not fix it inside this epic (§2.3).or_email_tracking.recipient_idhas no writer — NULL in 100% of prod rows becauseEmail.py:526-533never passes it (§1.6a). PLAN.md §5.2'semail_transaction_idfunnel correlation and §5.4's attribution telemetry cannot resolve a reply to a contact through that column today.- Our emails have no
text/plainpart (Email.py:560-563, §1.6b). This is a live risk to D3 on both options and the cheapest independent mitigation available. Absent from PLAN.md. - The "outbound half is cheap" claim is schema-true but unverified end-to-end (§1.1). Observed CSM-reply behavior corroborates it; one live send test closes it.
- Two unlisted hard prerequisites: the
ses:-less CI permissions boundary (§1.2, #6 above) and the EU/GDPR inbound decision (§1.8, #11 above).
5. Answers to the assigned questions, in one line each
- SES inbound vs an inbound-only vendor? SES, conditional on D3's pre-registered threshold (§1.7). Decisive reasons are same-account/same-region availability, a strictly better EU residency story, and the fact that we write raw-MIME storage and a fallback parser either way — not cost.
- Is there SES/MX/Route53/S3/SNS IaC to extend? No. Zero SES resources, zero mail DNS, zero bucket notifications.
onramp.usDNS is at GoDaddy; Route53 holds onlyonrampapps.com. Greenfield, plus an admin-applied permissions-boundary change. - Cost at current volume? SES inbound ≈ $0.55–0.91/month at 3–5% reply on the full reply-eligible surface (95,249 customer-facing sends/mo, measured); ≈$0.04/month under the current §6 scope. Whole option space spans under $50/month. SES
get_send_statisticsunread (no credentials this session) — volume measured fromor_email_trackingon the prod replica instead. - Synchronous POST vs cron-drained? Both, in the DocuSign shape: sync HMAC-verified receive → durable stage → inline process → authenticated backstop drain, with S3 as the pre-Flask dead-letter (§2.2).
- What if Flask is down? The reply is already in S3 before Flask matters; Lambda/SNS retry with backoff; the staging row accounts attempts; the backstop drain reconciles both staged-but-unprocessed rows and S3 objects with no row. Detection is an alarm on oldest-unprocessed-row age plus an S3-vs-row count reconciliation — not logs alone (§2.4).
6. Open questions for humans
- Does the SES-sending Lambda send
RawMessage, or reconstruct from top-level keys? (§1.1) One live send test on a deployed env, or one read of the Lambda. Gates the "outbound half is free" assumption. - Does reply-by-email ship to EU/
gdpr-prodorgs in v1? (§1.8) Product + legal. Changes the infra footprint by a whole region. - Who owns the GoDaddy zone and will create the MX + TXT records? (§3, #1-2)
- Who can apply the
ci-identityboundary change to allowses:? (§3, #6) Hard blocker for Pulumi-managed SES. - Which SES pricing plan is the account on, and is it out of the sandbox for receiving? (§1.5, §3 #12)
- Does the epic accept a ~7% one-click surface in v1, or does render-at-drain get promoted into scope? (§4, item 1) Product + eng lead.
- Existing SES state: what identities, DKIM configs, configuration sets, and rule sets already exist in us-west-2? The repo cannot say; only the console can.
7. Line-number drift from the planning docs
Re-verified at 24864a73a5. Most citations are exact; the drift is minor and noted for the next reader.
| Planning-doc citation | At HEAD | Status |
|---|---|---|
sqs_helper.py:177-182 (EmailMessageBus) | 177-182 | exact |
Email.py:554 From, :557 Reply-To | 554, 557 | exact |
Email.py:749-758 send_email | 749-757 | exact enough |
daily_jobs/routes.py:29,115,247,515 unauthenticated | 29, 115, 247, 515 | exact |
daily_jobs/routes.py:551 @requires_easy_cron_token | 550 route / 551 decorator / 552 def | exact |
email_jobs/routes.py:10 drain | 10 | exact |
cron_auth.py:27-55 | 27-58 | slight extension |
docusign/routes.py:82-140 receiver | 82-160 (handler runs to 160) | understated — the inline-process block at 144-159 is part of the pattern and worth reading |
docusign/routes.py:66-80 _is_connect_shaped | 65-79 | drifted by 1 |
docusign/routes.py:83-107 ("read verbatim") | 84-108 | drifted by 1 |
docusign/secrets.py:36-46, :64-93, :114-165, :33 | 36-42 / 47-49, ~56-60 memo, 100-165, 33 | approximately correct; :33 (_NO_AWS_RUNTIME_ERRORS) exact |
platform_integrations/or_models.py IntegrationWebhookData | 304-320 | (newly cited here) |
email_jobs/services.py:52 _claim_email_job | 52-70 | exact |
environments.yaml:61,68,69,88,115 | as cited | exact |
Method note: every prod figure came from the read replica under the prod-db-investigation recipe — pg_is_in_recovery() = t and onrampdb_prod confirmed before any query, forced read-only transactions, aggregates only, no row-level or PII data leaving the database. The reader host required the documented hostaddr workaround (system resolver filters the private address; dig @1.1.1.1 answers 10.x) — a DNS-filtering issue, not a VPN or database outage.