Skip to content

ONRAMP-6068 Spike — D1 (inbound transport & vendor) + D5 (ingestion architecture) ​

Written 2026-08-10 against main 24864a73a5 for ONRAMP-6068. Code line numbers are stale; current facts live in the ONRAMP-7078 stories.

Scope: D1 and D5 only. D2/D4 (token scheme + security) and D3 (parse benchmark) are owned by other agents — referenced here as "see D2" / "see D3", never duplicated. Verified at: repo HEAD 24864a73a5 (branch main), 2026-08-10. Every repo claim below carries a path + line I actually read at HEAD; drift from the planning docs is tabulated in §7. Evidence labels: [V] verified in repo/data I read · [I] inferred from verified facts · [H] needs a human with AWS/DNS/EasyCron console access.


0. Executive recommendation ​

DecisionRecommendationConfidence
D1 — transportAWS SES inbound receipt rules (classic email receiving, us-west-2), conditional on a pre-registered D3 exit criterion. Not Mail Manager. Not an inbound-only vendor.Medium-high, and deliberately falsifiable — see §1.7
D5 — ingestionNeither option in the brief. Copy DocuSign: SES → S3 (durable raw MIME) + notification → thin Lambda → HMAC-signed POST to Flask → verify → stage → process inline → mark processed, with an authenticated backstop cron drain.High — this is a pattern already running in this repo

The brief frames D5 as "synchronous POST vs S3/SQS cron-drained" and PLAN.md Q8 recommends synchronous. That dichotomy is false, and this repo already resolved it once: the DocuSign Connect receiver is synchronous and durable and has an authenticated cron backstop. Adopting it gets the latency of the sync option and the survivability of the cron option, and — because SES writes raw MIME to S3 before Flask is ever involved — the dead-letter requirement is satisfied by the transport itself rather than by application code.


1. D1 — Inbound transport & vendor ​

1.1 How sending actually works at HEAD — brief CONFIRMED, with one correction ​

The brief's claim ("SQS → an external Lambda → AWS SES, and the Lambda lives outside this repo") is correct. [V]

Link in the chainEvidence at HEAD
App never calls an ESP SDKapp/api/utils/email/Email.py:749-757 — send_email() does only EmailMessageBus(vendor_id=self.vendor_id).call_queue(json.dumps(email))
Queue publishapp/api/utils/sqs_helper.py:177-182 — EmailMessageBus(SQSHelper), queue name from DB config key email-queue-name; call_queue → invoke_queue(..., always_send_to_sqs=True) (:162-174), non-FIFO
The payload is explicitly for a LambdaEmail.py:580 — the literal comment # package up a response that can be sent to the lambda
Queue name is a DB row, not IaC and not environments.yamldevtools/db/seed_data.sql:4491 → ('email-queue-name','duploservices-dev01-EmailQueue'). The duploservices- prefix is DuploCloud tenant naming → Duplo-provisioned
The repo does not even know the consumer's nameemail_monitoring_service.py:233 _get_lambda_from_queue() discovers it at runtime via lambda:ListEventSourceMappings (docs/EMAIL-MONITORING-IAM-SETUP.md:103)
Provider is SESemail_monitoring_service.py:367,370 (get_send_quota, get_send_statistics); app/api/email_tracking/services/ses_vdm_insights_client.py:40 (sesv2.get_message_insights); app/api/api/__init__.py:397 BOTO3_SERVICES includes ses,sesv2
No sending Lambda in this repoExhaustive grep for lambda_handler / exports.handler / def handler( → the only inline handler is the model-EOL canary (infra/components/model-eol-alerter.ts:346). Nothing sends email. [V]

Correction to add: the outbound payload is asymmetric about Reply-To, and this matters to the epic.

Email.py:581-588   freshly-built branch → {From, To, Cc, Subject, RawMessage, transaction_id}   ← NO top-level Reply-To
Email.py:592-600   pre-built-body branch → {From, To, Cc, Reply-To, Subject, RawMessage, ...}   ← HAS top-level Reply-To

Reply-To is set on the MIME envelope at Email.py:557, so it survives inside RawMessage in both branches — but it is a top-level key only in the deferred/pre-built branch. Whether a tokenized Reply-To actually reaches the recipient therefore depends on something not in this repo: if the Lambda calls SendRawEmail/SendEmail with RawMessage, the header survives and the outbound half genuinely needs no change; if it reads the top-level keys and reconstructs, the freshly-built branch's Reply-To is dropped — and the manual nudge, the one email that sets Reply-To today, goes through exactly that branch (task/services_old.py:3151-3153 → Email.py:581-588).

The brief calls the outbound half "cheap — no schema change needed." That is true of the schema and unverified end-to-end. It is corroborated by observed behavior (the brief and PLAN.md §3 both state that nudge replies land in the CSM's inbox today, which is only possible if the Lambda honors RawMessage) — so this is probably fine. But it is one RawMessage-vs-top-level-keys detail away from "we shipped tokenized Reply-To and no reply ever came back," and it costs one live send test on a deployed env to close. Do that before 6073 estimates are trusted. [I→H]

1.2 What IaC we own — and the two prerequisites nobody has listed ​

Pulumi (TypeScript) is the only IaC; no Terraform/CDK/CloudFormation/SAM/K8s anywhere. Two sanctioned projects (infra/app, infra/ci-identity); new Pulumi projects are frozen and file-gated (.claude/rules/infra-composition.md). App runtime for dev/stage/prod/demo/gdpr-prod is DuploCloud-managed and not in this repo. [V]

SES: nothing to extend. Exhaustive grep across infra/ .github/ docker/ hosting-conf/ scripts/ for ses:[A-Z], aws.ses, sesv2, EmailIdentity, ConfigurationSet, ReceiptRule, ReceiptRuleSet, RuleSet, DomainIdentity, MailFrom, DkimAttributes, verifiedIdentity → zero matches. SES identity/DKIM/configuration-set state is console- or Duplo-provisioned outside this repo. [V]

No S3→SNS/SQS pipeline exists to extend either. grep -rn "BucketNotification|bucketNotification" infra/ → 0. Every SNS subscription in the repo is protocol: "email" (human alerts, model-eol-alerter.ts:493-501) or Slack HTTPS (datawarehouse/alarms.ts:45). The only SQS queue in IaC is onramp-model-eol-canary-dlq. SES inbound is greenfield. [V]

Prerequisite #1 nobody has named: the CI deploy-role permissions boundary grants no ses: action at all.infra/ci-identity/boundary-policy.ts enumerates allowed prefixes (s3:*, lambda:*, cloudfront:*, route53:ChangeResourceRecordSets, ssm:*, dynamodb:*, …); ses appears nowhere in it, nor in boundary-policy.legacy.json. Any Pulumi-declared SES resource is denied today. Fixing it means an admin-applied change to ci-identity, a bootstrap project — not a normal PR. [V]

Prerequisite #2: the runtime IAM policy is sending-side read-only. docs/EMAIL-MONITORING-IAM-SETUP.md grants ses:GetSendStatistics/GetSendQuota/GetAccountSendingEnabled, suppression-list read/delete, ses:GetMessageInsights. No receipt-rule permission of any kind. [V]

1.3 DNS — the GoDaddy fact is confirmed, and it does not favor a vendor ​

onramp.us is hosted at GoDaddy, verified from repo code rather than folklore: app/api/admin/services/domains/godaddy_service.py:18 (https://api.godaddy.com/v1) and :263 (f"{self.base_url}/domains/onramp.us/records"), with ADMIN_DOMAINS_SETUP.md:8-20 describing the API key/secret setup. That code path is a runtime customer-white-label-subdomain feature (CNAMEs for <tenant> portals) — not IaC, and it has no MX/TXT/DKIM capability wired. [V]

Route53 is used, but for a different domain: onrampapps.com (zone {redacted}), of which Pulumi owns exactly two records — the *.preview.onrampapps.com A/AAAA CloudFront aliases (infra/components/spa-edge.ts:470-482). The parent zone is explicitly manual: "Pulumi never touches it" (infra/README.md:505). No aws.route53.Zone resource is declared anywhere. Watch a stale-doc trap: infra/pulumi/README.md still says preview.onramp.us while the implemented code uses preview.onrampapps.com — do not read that file as evidence that onramp.us is in Route53. [V]

Mail DNS in IaC: repo-wide grep for \bMX\b, mxRecord, dmarc, _domainkey, \bspf\b → 0 hits in infra/. [V]

This is where I disagree with the brief's framing. The brief presents SES's DNS/infra burden as a cost that an inbound-only vendor avoids. It does not. Both options require the identical change: a new MX record on the receiving subdomain, created at GoDaddy by hand or via the GoDaddy API. SES points it at inbound-smtp.us-west-2.amazonaws.com; Postmark/SendGrid/Mailgun point it at their inbound host. Same zone, same owner, same manual step, same Pulumi-can't-manage-it constraint. The DNS axis is a wash and should be struck from the comparison. [I, from verified DNS topology + §1.4]

1.4 Region availability — resolved, and it was a real risk ​

SES email receiving is not available in every region, and all receiving-side resources (SNS topics, KMS keys, Lambdas) must sit in the same region as the SES endpoint — S3 buckets are the only exception (https://docs.aws.amazon.com/ses/latest/dg/regions.html). Our home region is us-west-2 (infra/README.md:48).

us-west-2 is a supported email-receiving region: the AWS General Reference "Email Receiving endpoints" table lists inbound-smtp.us-west-2.amazonaws.com (https://docs.aws.amazon.com/general/latest/gr/ses.html). us-east-1, us-east-2, us-west-1, eu-west-1, eu-central-1, eu-west-2 and others are also listed. [V — authoritative AWS docs]

This had the potential to kill the SES option outright and does not. It also means the EU/GDPR deployment can receive in-region (see §1.8).

1.5 Cost at real volume — measured, not assumed ​

AWS MCP call_aws and the aws CLI both had no usable credentials in this session (the shell aws is aliased to duploctl jit web --admin --interactive, which needs an interactive login). Per repo doctrine that is not an AWS outage and not a blocker — I went to the production read replica instead. So SES get_send_statistics is unread [H], but the send volume it would have shown is measured below from our own tracking table.

Source: prod read replica onrampdb_prod (pg_is_in_recovery() = t guard passed), read-only, aggregates only.

Table sizes: or_email_tracking 687,526 rows / 462 MB · email 3,089,002 rows / 16 GB · or_email_jobs 7 rows.

Monthly send volume (or_email_tracking, created 2026-03-06 by migration 2026_03_06_9d2c8a4f1b6e, so March is partial):

MonthEmails
2026-0362,450 (partial)
2026-04138,142
2026-05131,577
2026-06147,045
2026-07 (last full month)165,909
2026-08 (through the 10th)47,424 → ~4,742/day → ~147K/month pace

July 2026 by template, and the customer/internal split (recipient resolved by lower(email_address) + vendor, because recipient_id is unusable — see §1.6). Reply-eligible templates only; the per-template sum across all 28 types reconciles exactly to 165,909, so there are no unattributed rows.

TemplateTo customerTo internalVendors
weekly_summary51,618725112
overdue_tasks22,40021063
upcoming_tasks7,20112847
comment4,8976,44668
task_assigned4,6792,65753
task_nudge_reminder2,32723750
tasks_starting_today1,4647617
tasks_reminder6632,46653
Total customer-facing, reply-eligible95,249

Caveat: prod us-west-2 only. gdpr-prod/EU is a separate database not measured here, and demo/stage are excluded. Real global volume is higher. [V + H for the EU delta]

Inbound volume = reply rate × exposed sends. Reply rates to transactional notification mail are low single-digit percentages; at 1 / 3 / 5% of 95,249 that is ~950 / ~2,860 / ~4,760 inbound emails per month.

SES inbound à la carte pricing (authoritative, https://aws.amazon.com/ses/pricing/): $0.10 per 1,000 emails received, plus $0.09 per 1,000 incoming email chunks (256 KB per chunk).

ScenarioInbound/moSES inbound cost/mo
Full reply-eligible surface @ 3%2,860~$0.55
Full reply-eligible surface @ 5%4,760~$0.91
PLAN.md §6 scope (nudge + task_assigned) @ 3%210~$0.04

For scale: our outbound 165,909/month at à la carte $0.10/1,000 is $16.59/month.

An inbound-only vendor at these volumes sits in the low tens of dollars per month on a published entry plan (exact list price [H] — check the vendor's current pricing page; do not quote mine). The entire cost spread across the whole option space is under ~$50/month ≈ two engineer-hours. Cost is not a decision input. The brief is right that cost is not decisive; it is now right with numbers, and the numbers are more lopsided than "not decisive" implies.

⚠️ One pricing wrinkle needing a human: SES introduced tiered plans (Essentials/Pro/Enterprise) alongside à la carte, and "new SES accounts and account × region combinations with no metered SES activity since June 1, 2025 will start on the Essentials plan beginning July 21, 2026." On a plan, inbound is billed as Mail Manager Email Processing ($0.15/1,000, and an Open Ingress Endpoint is $50/month) rather than the classic à la carte Inbound email line. Our us-west-2 account plainly has metered activity, so à la carte classic receiving should still apply — but confirm which plan the account is on before quoting a number to finance. [H]

1.6 Two data findings that change the plan, not just the cost model ​

(a) or_email_tracking.recipient_id is NULL in 100% of production rows. 0 of 165,909 July rows populated (account_id: 151,903 populated). Root cause verified in code, not guessed: Email.py:526-533 calls EmailTrackingCreateService().create_or_ignore(...) and never passes recipient_id, which defaults to None (email_tracking_create_service.py:22). The only recipient_id write anywhere is on the different table OREmail (custom_email_template_service.py:1273). So or_email_tracking.recipient_id has no writer at all. [V]

This is also a caught measurement trap worth recording: my first customer/internal split query joined on recipient_id and returned to_customer = 0 for every template. That reads as "we send no customer email," which is false. Same class as the request_url trap in the prod-db skill — the column cannot hold the answer.

Consequence for the epic: PLAN.md §5.2 wants email_transaction_id on the token "for funnel correlation," and §5.4 wants via-email attribution telemetry. Correlating an inbound reply back to a contact through or_email_tracking does not work today. This independently corroborates D2's recommendation to bind user_id on the token row and treat the sender address as confirmation only — the token is currently the only reliable link. Either backfill/populate recipient_id (small, and worth its own ticket) or accept the token as the sole correlation key.

(b) Our outbound emails have no text/plain part. Email.py:560-563 builds MIMEMultipart("alternative") and attaches only an HTML child; repo-wide grep for text/plain / MIMEText(..., "plain") across app/api/utils/email/ and app/api/custom_email_templates/ → 0 hits. Every notification is HTML-only inside a degenerate single-child multipart/alternative. [V]

Three consequences, and the third is the important one:

  1. Mild deliverability/accessibility cost (HTML-only is a spam-scoring signal; plain-text readers get nothing). Worth its own ticket regardless of this epic.
  2. A one-part multipart/alternative is technically odd and some clients treat it unpredictably.
  3. It raises parse risk on both D1 options. When a customer replies, their client quotes our 15 KB nested-table HTML (main_email_template.html). Clients that synthesize a text/plain part produce a machine-flattening of that table — historically the worst input for reply-stripping heuristics. Clients that reply HTML-only hit exactly the documented condition under which Postmark's StrippedTextReply does not populate at all, which the brief already flags.

Actionable and independent of D1: add a text/plain alternative to the outbound templates. It improves the quoted-reply shape for every client, helps deliverability, and is the single cheapest thing that de-risks D3 — and it is not in PLAN.md anywhere. Recommend it as a W1 enabler alongside 6069.

1.7 The decisive axis is parse-quality ownership — handing it to D3 with a pre-registered threshold ​

The brief is right that this is the deciding axis, and right that the decision belongs to measurement rather than a feature matrix. Restating it precisely so D3 can settle it:

  • SES gives raw MIME. We own 100% of quoted-history stripping, signature stripping, and OOO/autoresponder/bounce detection, forever.
  • A vendor gives a hosted stripped-reply field plus the mailbox-hash pattern, at the cost of a second vendor, a split deliverability/observability surface, and a new dependency on a path that must not silently drop customer replies.

Three things narrow that gap more than the brief allows:

  1. We are writing raw-MIME storage and parse code either way. PLAN.md §5.4 already requires storing the raw inbound message "for audit + parse debugging," and §5.2 already specifies a fallback parser (email_reply_parser lineage) in addition to the provider's stripped field. So the vendor does not remove the parser — it adds a preferred input to it. The marginal deliverable SES adds is "make the fallback good enough to be primary."
  2. The vendor's stripped field has a documented hole that our own templates make more likely (§1.6b). A hosted field that silently doesn't populate is worse than a parser we control, because the failure is invisible to us.
  3. SPF/DKIM/DMARC verdicts are available on SES too — receipt rules surface them in the rule metadata, so D4's verdict-handling design is not vendor-dependent.

Explicit hand-off to D3. D1 cannot be closed on repo evidence; it needs one number. D3 must report, on ≥20 real replies to our actual notification emails across Gmail / Outlook desktop / Outlook mobile / Apple Mail:

  • clean-extraction rate of an open-source parser alone (the SES path), and
  • clean-extraction rate of a vendor stripped-reply field, on the same corpus, and
  • autoresponder/OOO/bounce detection rate, and
  • per-reply: was a text/plain part present at all? (the Postmark-hole rate — §1.6b makes this non-theoretical)

Pre-registered exit criterion, agreed before the numbers land so this isn't rationalized after the fact: if the open-source parser reaches ≥90% clean extraction with ≥95% autoresponder detection, ship SES. If it lands below 80%, the vendor's parse quality is buying something real — switch to the vendor and accept the second-vendor cost. Between 80% and 90%, decide on the shape of the failures: systematic and fixable (one client's quote marker) → SES plus targeted fixes; diffuse and unpredictable → vendor.

1.8 Two arguments for SES the brief does not make ​

EU/GDPR data residency. app/config/environments.yaml defines a gdpr-prod environment (line 115) alongside prod (110). Inbound email means receiving and storing customer-authored message content — squarely in scope for the EU deployment's residency posture. SES receives in-region (eu-west-1/eu-central-1/eu-west-2 are all supported receiving regions, §1.4) inside the same account and boundary model we already operate. Routing EU customers' replies through a US inbound-parse vendor is a DPA/sub-processor conversation with legal, plus a second region's worth of vendor configuration. This is a structural advantage for SES that no amount of parse quality offsets, and it is absent from the brief, PLAN.md, and the epic. It also raises a scope question nobody has asked (§4, item 11): does reply-by-email ship to EU orgs in v1 at all?

Config/infra doctrine fit. The SES path is expressible entirely in mechanisms this repo already has and already governs: a Tier-1 Pulumi component member in infra/app/envs.ts, a per-env *_provisioned: false opt-in boolean in environments.yaml (exact precedent: docusign_connect_hmac_provisioned:61, esign_token_encryption_provisioned:88), and an SSM secret leaf. A vendor adds an API credential (also SSM — fine) plus a vendor account, a second reputation surface, and a second observability plane that the email-monitoring dashboard (email_monitoring_service.py) knows nothing about.

1.9 Considered and rejected: SES Mail Manager ​

AWS's newer inbound product appears in the pricing page and is not the right shape for us: Email Processing is $0.15/1,000 (vs $0.10/1,000 classic inbound), and receiving arbitrary internet mail on our own domain needs an Open Ingress Endpoint at $50/month per endpoint per region — "charges apply even when endpoint status is Closed." Classic receipt rules deliver exactly what we need (recipient-condition matching → S3 + SNS/Lambda) at lower cost and lower conceptual weight. Revisit only if we later want its add-on scanning marketplace (Trend Micro / Spamhaus / Abusix). [V]


2. D5 — Ingestion architecture ​

2.1 The in-repo model, read verbatim ​

app/api/integrations/docusign/routes.py:82-160 is the receiver; webhook_verification.py:23-56 is the pure verifier; :179-181 is the authenticated backstop drain. Read at HEAD. The disciplines the inbound-email receiver must copy, each tied to its line:

#DisciplineWhereWhy it is load-bearing for inbound email
1Raw body captured before any parse — raw_body = request.get_data() is the first statementroutes.py:89HMAC is over the exact bytes as sent; a re-serialization will not match. For MIME this is doubly true — re-encoding a MIME body changes it.
2Reject on missing signature header BEFORE reading the secretroutes.py:91-108, and the comment at :92-94 says whySo anonymous traffic cannot drive SSM GetParameter load. GetParameter throughput is account-shared — an unauthenticated URL that reads a secret per request is a free throttling attack on every other SSM consumer.
3Split scanner noise from real-but-unsigned traffic — _is_connect_shaped() cheap shape check; shaped-but-unsigned gets its own searchable log line, unshaped gets a generic one; both log-only, neither mints Sentryroutes.py:65-79 + :102-107An unauthenticated URL is dominated by scanners. Minting Sentry from unsigned traffic lets the internet drown the one signal we actually alert on. Inbound analogue: a body that parses as MIME with a Received: header and a recipient matching our inbound subdomain is "real-but-unsigned" (a misconfigured receipt rule / Lambda missing its signing key) — the one real misconfiguration class, and it must stay searchable. Alert on it via a log-based metric, not Sentry.
4Sentry reserved strictly for our-side misconfiguration_reject_unverified, routes.py:49-62Reached only after a signature header exists or a secret read completed, so strangers cannot mint Sentry events. Copy this exact gating.
5Cap the number of signature headers examinedwebhook_verification.py:29-37, DOCUSIGN_MAX_CONNECT_SIGNATURESPrevents a public caller amplifying HMAC CPU by presenting many forged headers. This is the concrete answer to the brief's "no rate limiting exists app-wide" concern for the cheapest abuse vector — bound the work per request rather than reaching for Flask-Limiter.
6Constant-time compare, accept-any-of-N-keyswebhook_verification.py:40-56 (hmac.compare_digest; digest computed once per key, not per key×header)Zero-downtime secret rotation with two active keys. Empty key set or absent headers fail closed.
7Signed but unparseable → non-2xx, so it stays in the sender's retry queue instead of being silently droppedroutes.py:125-131Directly serves "must not silently drop customer replies."
8Staging failure → non-2xx + explicit capture_exceptionroutes.py:133-142A staging failure means guaranteed redelivery; it must always be in Sentry.
9Inline-processing failure AFTER successful staging → swallow, keep 2xx, let the drain retryroutes.py:144-159The subtle one. Once the event is durably staged, turning the ack non-2xx causes redelivery of an event we already hold — duplicate answers written to a task. Get this backwards and one flaky parse becomes a storm of duplicate task answers.
10Receive-side is NOT feature-flag gated; the DRAIN iswebhook_service.py:35-40 (docstring)Because the sender does not redeliver after a 2xx, a receive-side flag check permanently drops events arriving while an org's flag is off mid-rollout. Given PLAN.md §5.0 puts this whole epic behind a per-org flag, this trap is live for 6073/6074: stage every verified inbound reply regardless of the org's flag state, and gate processing.

Additional shapes to reuse rather than reinvent:

  • Staging table. IntegrationWebhookData (app/api/platform_integrations/or_models.py:304-317) already carries exactly the right columns: external_id (dedupe key, indexed), webhook_data JSON, or_vendor_id, processed_at (indexed), attempts, last_error, last_attempt_at. Copy this shape for or_inbound_email, with external_id = the SES messageId.
  • Bounded retries via the config map, not a magic number: precedent docusign_webhook_max_process_attempts (app/config/environments.yaml:69) and docusign_webhook_drain_batch_size (:68).
  • SSM secret + fail-closed boot check. app/api/integrations/docusign/secrets.py wholesale: _ssm_parameter_name lowercases ENVIRONMENT_NAME (:36-42); a 60s in-process memo bounds reads to ~1 GetParameter/min/worker so scanner traffic cannot throttle SSM (:56-60 and the comment above it); the boot check derives enforcement from the config map and skips envs that have not opted in (:100-165). Read the dev-config-env-name-overload trap first: _NO_AWS_RUNTIME_ERRORS = (NoRegionError, NoCredentialsError) (:33) is what stops a local/CI stack that inherits ENVIRONMENT_NAME=dev from crash-looping the app and the seed. A genuine deployed misconfiguration still fails closed, because there the SSM call reaches AWS and returns ParameterNotFound/AccessDenied. Recommended: secret at /onramp/{env}/inbound-email/webhook-hmac, gated by a new inbound_email_hmac_provisioned: false boolean in environments.yaml following docusign_connect_hmac_provisioned:61. Note the existing guard rejects a non-boolean (e.g. YAML "true" as a string) loudly rather than silently skipping enforcement — keep that.
customer reply
   → MX (reply.onramp.us, at GoDaddy)  →  inbound-smtp.us-west-2.amazonaws.com
   → SES receipt rule (us-west-2, ACTIVE rule set)
        ├─ S3 action:  raw MIME  →  s3://onramp-inbound-email-<env>/     ← DURABLE, pre-Flask
        └─ SNS/Lambda action: notify
   → thin Lambda (us-west-2): read S3 key + SES verdicts, HMAC-sign, POST
   → POST /api/inbound-email  (Flask, unauthenticated, HMAC-verified)
        1. raw_body = request.get_data()            ← before any parse
        2. no signature header?  → log-only 401 (shaped vs unshaped split)
        3. read HMAC keys from SSM (memoized)       ← only now
        4. verify (constant-time, any-of-N keys)    ← fail closed
        5. STAGE row in or_inbound_email (dedupe on SES messageId)
        6. process inline (resolve token → parse → answer/comment)
        7. mark processed_at;  inline failure → swallow, stay 2xx
   → POST /api/inbound_email_jobs/drain   @requires_easy_cron_token   ← backstop

Why this beats both options in the brief:

  • vs. pure synchronous with provider retries. Provider-side retry is bounded (a vendor typically retries for hours, then gives up; DocuSign is unusually generous at 15 days — routes.py:4-6). With SES→S3 the raw MIME is durable indefinitely and independent of Flask, the Lambda, and the retry window. The dead-letter is the transport, not application code.
  • vs. pure S3/SQS cron-drained. Latency matters here in a way it does not for e-sign status: a customer who answers a question by email expects the task to reflect it, and a CSM watching the project expects the same. Waiting for the next cron tick is a product regression for the happy path. Inline processing gives immediate effect; the drain exists only for the unhappy path.
  • Both, on the hard requirement. "Must not silently drop customer replies" is met at three independent layers (§2.4), no one of which is application-logic-dependent.

2.3 The EasyCron burden is real but small — and it does not deepen the auth problem ​

I read the procedure rather than guessing: .agents/skills/backend-development/SKILL.md:94-130.

The registered list (:99-118) already contains the five endpoints this epic touches: /api/project_daily_jobs/{upcoming_tasks,overdue_tasks,tasks_starting_today}_bottom_up, /api/project_daily_jobs/customer_weekly_summary, and /api/email_jobs/send_pending_customer_emails, plus /dailynotifications.

Actual burden of adding one backstop drain endpoint:

  1. Add the URL to the list in SKILL.md, marked # job not created yet (:130).
  2. Label the PR EASYCRON_URL (:122).
  3. Warning block near the top of the PR description (:123-127).
  4. A human creates the cron jobs — one per deployment, each behind its own domain (app.onramp.us, eu.onramp.us, demo.onramp.us, stage.onramp.us), naming any deliberate exclusions (:130).
  5. Because the endpoint will be authenticated, the console jobs must carry X-EasyCron-Token from creation or every run 401s.

That is one list edit, one label, one PR note, and a human creating ~4 console jobs with a header. A few person-hours of coordination — not an architectural cost. The brief and PLAN.md Q8 both treat "EasyCron PR-coordination burden" as a reason to prefer synchronous; on the evidence that weighting is too high. The backstop is worth it, and the recommendation in §2.2 keeps synchronous processing anyway — the drain is additive.

On the unauthenticated-cron concern — verified, and it cuts the other way. Confirmed exactly at HEAD:

EndpointAuth
daily_jobs/routes.py:29 tasks_starting_today_bottom_upnone
daily_jobs/routes.py:115 upcoming_tasks_bottom_upnone
daily_jobs/routes.py:247 overdue_tasks_bottom_upnone
daily_jobs/routes.py:515 customer_weekly_summarynone
email_jobs/routes.py:10 send_pending_customer_emailsnone
daily_jobs/routes.py:550-552 calculate_projects_metrics@requires_easy_cron_token
docusign/routes.py:163-165 refresh-keepalive@requires_easy_cron_token
docusign/routes.py:179-181 process-webhook-events@requires_easy_cron_token

The four digest writers and the drain are unauthenticated [V] — PLAN.md open question 11 is accurate. But the precedent for a new cron endpoint in this repo is an authenticated one: DocuSign added two, both decorated. cron_auth.py:5 is explicit that the unauthenticated set is legacy — "Existing OnRamp EasyCron endpoints have no auth and rely on URL obscurity."

So adding a backstop drain deepens nothing, provided it ships with @requires_easy_cron_token. The premise in my instructions — that another cron endpoint would deepen an existing problem — holds only for an unauthenticated one, and there is no reason to add one. I'd go further: the inbound-email drain is a natural, low-risk vehicle to demonstrate the decorator on an email-adjacent endpoint, strengthening the case for the standalone ticket that retrofits the other five.

⚠️ A doctrine violation in the auth primitive itself, noted in neither the brief nor PLAN.md.app/api/utils/cron_auth.py:46 reads environ.get("EASYCRON_SHARED_SECRET") — and :37 reads environ.get("ENVIRONMENT_NAME", "local_dev"). Per the binding config rules in CLAUDE.md, ENVIRONMENT_NAME is the one sanctioned env var; a shared secret belongs in AWS SSM. So the existing cron-auth primitive violates the repo's own configuration doctrine. [V] Recommendation: use it as-is for the inbound drain and file a separate ticket to move the secret to SSM. Changing it in this epic's PR would simultaneously touch every existing EasyCron job's configuration — exactly the lockstep-or-401 failure mode described above. Do not let this epic become the vehicle for that migration, and do not let a reviewer talk you into environ.get("X") or <default>, which CLAUDE.md bans outright.

2.4 Failure path — what happens when Flask is down ​

The hard requirement is "must not silently drop customer replies." Three independent layers, then detection.

Layer 1 — S3 is durable before Flask exists in the story. The SES receipt rule's S3 action writes raw MIME on receipt. If Flask is down, the Lambda's POST fails and the mail is already saved. Nothing is ever only-in-flight. Set a bucket lifecycle retention comfortably exceeding any plausible outage + investigation window (90 days suggested); this doubles as the raw-message audit store PLAN.md §5.4 already requires, so it is not extra infrastructure.

Layer 2 — bounded retry with accounting. Lambda retries its POST with backoff (and SNS carries its own retry policy + DLQ if the notification path is SNS-based). Beyond that, the staging row carries attempts / last_error / last_attempt_at / processed_at (or_models.py:314-317), with the max bounded by a config-map key, not a literal.

Layer 3 — authenticated backstop drain. Reconciles two ways: staged-but-unprocessed rows, and S3 objects with no staging row at all (the case where Flask was down for the entire Lambda retry budget). The second reconciliation is the one that actually covers a long outage, and it is the one most likely to be forgotten.

Detection — the part that decides whether a drop is silent. Alarms, not logs alone:

  • Age of the oldest processed_at IS NULL staging row exceeding N minutes. This is the single most valuable alarm; it catches stuck parses, poison rows, and outages alike.
  • Count of rows at attempts >= max (a poison queue). These must be surfaced for human review.
  • S3-object count vs staging-row count over a rolling window — the only thing that detects mail that never reached the app at all.
  • logger.exception is alertable in this stack (per the background-thread/cron review facts), so the DocuSign Sentry discipline (#4/#8 above) transfers directly.

Two hard "do nots", both grounded in verified in-repo bugs:

Do NOT copy _claim_email_job. app/api/email_jobs/services.py:52-70 claims a job with DELETE ... RETURNING — the row is gone before the send is attempted — and the caller's except at :45-47 only logs. A send failure loses that email permanently, with no requeue path. Verified at HEAD; it is defect (c) in PLAN.md §5.3. This is the exact anti-pattern the inbound path must avoid: stage, mark processed, never delete-on-claim. Claim with a processed_at/attempts update or SELECT ... FOR UPDATE SKIP LOCKED, never by deleting the evidence.

Do NOT let exhausted-attempt rows be deleted or pruned. A customer reply that failed to parse five times is a product bug report with a customer waiting on the other end.

One more inherited trap worth stating in the 6073 spec: vendor_id = 0 is a legitimate tenant (.claude/rules/python.md), so any if not vendor_id guard on the inbound path is a bug — use is None. Local flows run as a seeded customer org, so local testing will not catch it; a test must pin the fixture to 0 explicitly. The staging table's or_vendor_id resolution and every drain filter are the likely sites.


3. What must change OUTSIDE this repo — the epic's most under-scoped risk ​

The brief is right to call this out. Concrete inventory, with what I could verify from the repo vs what needs console access.

#ChangeWhere it livesPlausible ownerStatus
1MX record for the inbound subdomain (e.g. reply.onramp.us → inbound-smtp.us-west-2.amazonaws.com)GoDaddy zone for onramp.usHolder of the GoDaddy account / API credential (ADMIN_DOMAINS_SETUP.md:8-20)Zone location [V] (godaddy_service.py:263); the record itself [H]. Pulumi cannot manage this. Required identically for a vendor.
2TXT domain-verification record for the receiving domain in SESGoDaddysame as #1[H]
3SES receipt rule set + rule (recipient condition, S3 action, SNS/Lambda action), in us-west-2, and ACTIVE — only one rule set can be active per regionAWS console, or a new infra/components/ses-inbound.ts Tier-1 memberInfra ownerABSENT from repo IaC by exhaustive grep [V]
4S3 bucket for raw inbound MIME + AllowSESPuts bucket policy (+ KMS grant if encrypted)new PulumiInfra ownerNo BucketNotification pattern exists anywhere to extend [V]
5The receiving Lambda (S3/SNS → HMAC-signed POST to Flask). Net-new; no in-repo lambda resembles itNew code; placement per docs/proposals/domain-lambdaliths.mdBackend + Infra[V] no such lambda in repo
6CI deploy-role permissions boundary must grant ses: — it currently grants noneinfra/ci-identity/boundary-policy.ts (admin-applied bootstrap project)Whoever can apply ci-identity[V]. Hard blocker for any Pulumi-managed SES resource; listed in no planning doc.
7Runtime IAM if the app ever calls SES receiving APIs — current policy is sending-side read-onlydocs/EMAIL-MONITORING-IAM-SETUP.md + the Duplo task roleInfra owner[V] no receipt-rule permission today
8SSM parameter for the inbound HMAC secret (/onramp/{env}/inbound-email/webhook-hmac)AWS SSM, per envBackend + InfraPattern [V] (docusign/secrets.py); provisioning [H]
9EasyCron jobs for the backstop drain — one per deployment, carrying X-EasyCron-TokenEasyCron consoleWhoever manages EasyCronProcedure [V] (SKILL.md:120-130); job creation [H]
10The SES-sending Lambda — needed only if the tokenized Reply-To does not survive (§1.1)Outside repo, Duplo-provisioned; name discoverable only at runtimeOwner of the Duplo email stackQueue name duploservices-dev01-EmailQueue [V] (seed_data.sql:4491); consumer not in repo [V]; behavior [H]
11EU/GDPR: a second receipt rule set + MX for the EU deployment, or an explicit decision that EU orgs get no reply-by-email in v1eu-* region + DNSProduct + Infra + Legalgdpr-prod env exists [V] (environments.yaml:115); the decision [H]. Raised nowhere in the epic.
12SES account posture: which pricing plan the account is on (§1.5), and whether it is out of the sandbox for receivingAWS billing/SES consoleInfra / Finance[H]

Placement guidance, so this lands correctly: per .claude/rules/infra-composition.md, Tier 3 is FROZEN and file-gated — do not create infra/ses/. The right shape is a new infra/components/ses-inbound.ts plus a presence-toggled member in infra/app/envs.ts composed by infra/app/index.ts, following the resolveTier derive-don't-configure model. Adopting console-created SES resources into Pulumi later has a runbook precedent: infra/app/CUTOVER.md.


4. Where I disagree with the going-in position, and what contradicts the plan ​

  1. The one-click surface is ~7% of the addressable audience, and nobody has priced that. (The most consequential finding in this document.) PLAN.md §6 confines 6070 one-click links to immediate emails — nudge + task_assigned. Measured against real July volume, that is 7,006 customer-facing sends/month out of 95,249 reply-eligible (7.4%). The four deferred digests excluded by that decision are 82,683/month (87%), and weekly_summary alone is 51,618/month across 112 vendors — the single largest email surface we operate. task_nudge_reminder, the epic's hero flow, is 2,327/month (1.4% of all mail). The §6 scope decision is technically well-reasoned (tokens minted at enqueue burn their TTL before delivery — deferred-send-timing.md). But it converts the epic's flagship mechanism into a feature reaching a low-single-digit share of email. Open question 12 (render digests at drain) is therefore load-bearing for whether this epic is worth the infra lift in §3, not a follow-up ticket. Recommend promoting the render-at-drain refactor to a W1 enabler, or consciously accepting a ~7% v1 and saying so in the epic.
  2. The DNS burden does not differentiate the options. Both SES and any vendor need the same new MX record in the same GoDaddy zone that Pulumi cannot manage (§1.3). Strike it from the comparison.
  3. D5's dichotomy is false and the repo already disproves it (§2.1-2.2). Neither "sync with provider retries" nor "S3/SQS cron-drained" is the answer; DocuSign's stage-and-backstop hybrid is, and it is already in production here.
  4. The EasyCron burden is over-weighted in the brief and PLAN.md Q8 — one list edit, one label, one PR note, ~4 human-created console jobs (§2.3).
  5. Adding a cron endpoint need not deepen the unauthenticated-cron problem. The last two cron endpoints added (both DocuSign) carry @requires_easy_cron_token. The unauthenticated five are legacy, as cron_auth.py:5 admits (§2.3).
  6. requires_easy_cron_token itself violates the repo's config doctrine — EASYCRON_SHARED_SECRET via environ (cron_auth.py:46) where doctrine mandates SSM. Noted in neither the brief nor PLAN.md. Ticket it; do not fix it inside this epic (§2.3).
  7. or_email_tracking.recipient_id has no writer — NULL in 100% of prod rows because Email.py:526-533 never passes it (§1.6a). PLAN.md §5.2's email_transaction_id funnel correlation and §5.4's attribution telemetry cannot resolve a reply to a contact through that column today.
  8. Our emails have no text/plain part (Email.py:560-563, §1.6b). This is a live risk to D3 on both options and the cheapest independent mitigation available. Absent from PLAN.md.
  9. The "outbound half is cheap" claim is schema-true but unverified end-to-end (§1.1). Observed CSM-reply behavior corroborates it; one live send test closes it.
  10. Two unlisted hard prerequisites: the ses:-less CI permissions boundary (§1.2, #6 above) and the EU/GDPR inbound decision (§1.8, #11 above).

5. Answers to the assigned questions, in one line each ​

  • SES inbound vs an inbound-only vendor? SES, conditional on D3's pre-registered threshold (§1.7). Decisive reasons are same-account/same-region availability, a strictly better EU residency story, and the fact that we write raw-MIME storage and a fallback parser either way — not cost.
  • Is there SES/MX/Route53/S3/SNS IaC to extend? No. Zero SES resources, zero mail DNS, zero bucket notifications. onramp.us DNS is at GoDaddy; Route53 holds only onrampapps.com. Greenfield, plus an admin-applied permissions-boundary change.
  • Cost at current volume? SES inbound ≈ $0.55–0.91/month at 3–5% reply on the full reply-eligible surface (95,249 customer-facing sends/mo, measured); ≈$0.04/month under the current §6 scope. Whole option space spans under $50/month. SES get_send_statistics unread (no credentials this session) — volume measured from or_email_tracking on the prod replica instead.
  • Synchronous POST vs cron-drained? Both, in the DocuSign shape: sync HMAC-verified receive → durable stage → inline process → authenticated backstop drain, with S3 as the pre-Flask dead-letter (§2.2).
  • What if Flask is down? The reply is already in S3 before Flask matters; Lambda/SNS retry with backoff; the staging row accounts attempts; the backstop drain reconciles both staged-but-unprocessed rows and S3 objects with no row. Detection is an alarm on oldest-unprocessed-row age plus an S3-vs-row count reconciliation — not logs alone (§2.4).

6. Open questions for humans ​

  1. Does the SES-sending Lambda send RawMessage, or reconstruct from top-level keys? (§1.1) One live send test on a deployed env, or one read of the Lambda. Gates the "outbound half is free" assumption.
  2. Does reply-by-email ship to EU/gdpr-prod orgs in v1? (§1.8) Product + legal. Changes the infra footprint by a whole region.
  3. Who owns the GoDaddy zone and will create the MX + TXT records? (§3, #1-2)
  4. Who can apply the ci-identity boundary change to allow ses:? (§3, #6) Hard blocker for Pulumi-managed SES.
  5. Which SES pricing plan is the account on, and is it out of the sandbox for receiving? (§1.5, §3 #12)
  6. Does the epic accept a ~7% one-click surface in v1, or does render-at-drain get promoted into scope? (§4, item 1) Product + eng lead.
  7. Existing SES state: what identities, DKIM configs, configuration sets, and rule sets already exist in us-west-2? The repo cannot say; only the console can.

7. Line-number drift from the planning docs ​

Re-verified at 24864a73a5. Most citations are exact; the drift is minor and noted for the next reader.

Planning-doc citationAt HEADStatus
sqs_helper.py:177-182 (EmailMessageBus)177-182exact
Email.py:554 From, :557 Reply-To554, 557exact
Email.py:749-758 send_email749-757exact enough
daily_jobs/routes.py:29,115,247,515 unauthenticated29, 115, 247, 515exact
daily_jobs/routes.py:551 @requires_easy_cron_token550 route / 551 decorator / 552 defexact
email_jobs/routes.py:10 drain10exact
cron_auth.py:27-5527-58slight extension
docusign/routes.py:82-140 receiver82-160 (handler runs to 160)understated — the inline-process block at 144-159 is part of the pattern and worth reading
docusign/routes.py:66-80 _is_connect_shaped65-79drifted by 1
docusign/routes.py:83-107 ("read verbatim")84-108drifted by 1
docusign/secrets.py:36-46, :64-93, :114-165, :3336-42 / 47-49, ~56-60 memo, 100-165, 33approximately correct; :33 (_NO_AWS_RUNTIME_ERRORS) exact
platform_integrations/or_models.py IntegrationWebhookData304-320(newly cited here)
email_jobs/services.py:52 _claim_email_job52-70exact
environments.yaml:61,68,69,88,115as citedexact

Method note: every prod figure came from the read replica under the prod-db-investigation recipe — pg_is_in_recovery() = t and onrampdb_prod confirmed before any query, forced read-only transactions, aggregates only, no row-level or PII data leaving the database. The reader host required the documented hostaddr workaround (system resolver filters the private address; dig @1.1.1.1 answers 10.x) — a DNS-filtering issue, not a VPN or database outage.

Internal documentation — gated behind Cloudflare Access.