Skip to content

ONRAMP-6068 / D3 — Reply-parse quality benchmark ​

Written 2026-08-10 against main 24864a73a5 for ONRAMP-6068. Code line numbers are stale; current facts live in the ONRAMP-7078 stories.

Owner: D3 only (D1/D5 and D2/D4 are separate deliverables). Repo state: all repo claims verified at main @ 24864a73a5. No repo file modified. Harness: a local scratch directory (not committed)

Label key used throughout: MEASURED = I ran it or read it at HEAD. INFERRED = reasoned from evidence, not observed. SYNTHETIC = produced by scaffolding I wrote, and therefore not evidence about the real world. UNRESOLVED = genuinely unknown, needs the real corpus.


0. The honest headline: the benchmark the brief asks for cannot be run yet ​

The brief's acceptance criterion is "Parse benchmark run on ≥20 real replies across the four clients, committed as a fixture corpus, with autoresponder-detection measured."

That criterion is not met by this deliverable, and could not have been. A real reply to an OnRamp notification email can only be produced by sending our actual template to a real mailbox on each of the four clients and replying from it. We have no inbound infrastructure (D1 is greenfield), and — more fundamentally — no amount of research substitutes for the artefact. Every quoted-history shape, attribution string, and MIME layout that the benchmark exists to measure is generated by the client, not by us, and cannot be authored from imagination with any fidelity.

So this deliverable is three things instead:

  1. A measured characterisation of our outbound MIME, which changes the vendor decision (§1).
  2. A parser evaluation grounded in capability facts, not invented rates (§2), plus a runnable harness and fixture format so the measurement is one command away the moment samples exist (§3).
  3. A specified autoresponder rule set with the tier structure that makes its error profile safe (§4), and a corpus-collection plan that does not wait on D1 (§5).

§5 is the load-bearing schedule finding: the corpus can be gathered before any inbound infrastructure is built. That is stated loudly there because it moves the epic's critical path.

I have not fabricated a corpus. corpus/ in the harness contains 14 hand-authored samples whose provenance field reads SYNTHETIC-SCAFFOLDING, whose only job is to prove the code executes, and which the loader refuses to count toward the ≥20 criterion. The runner prints a three-line banner saying not to cite its numbers. Where a number in this document comes from those samples it is marked SYNTHETIC and is a statement about the harness, never about parse quality.


1. Our outbound email is HTML-only. There is no text/plain part anywhere. ​

MEASURED — definitive. Two MIMEText(...) call sites exist in the entire application, and both pass "html":

  • app/api/utils/email/Email.py:562 — htmlpart = MIMEText(body.encode("utf-8"), "html", "utf-8")
  • app/api/custom_email_templates/services/custom_email_template_service.py:1253 — same, for the test-send path

A repo-wide grep for "plain" in a MIMEText construction returns zero non-test hits. The envelope built at Email.py:552-575 is:

multipart/mixed                      <- envelope, Email.py:552
└── multipart/alternative            <- Email.py:560
    └── text/html                    <- Email.py:562  (the ONLY child)

The irony worth naming: multipart/alternative is already constructed and has exactly one alternative. The code is shaped for a plain-text part and simply never got one. Email.py:590-601 has a second branch for pre-built bodies, but its only producers at HEAD are tests and app/api/project/daily_jobs/tests/snapshot.py:64 — no production sender reaches it.

The template being quoted is app/api/utils/email/main_email_template.html: MEASURED 15,448 bytes, 8 <table>, 14 <tr>, 14 <td>, 17 inline style= attributes, one <style> block, and — notably — zero mso conditional comments. The nudge body itself (task_nudge/task_nudge.html) is a simple <ol>, so the parse hazard is entirely the chrome wrapping it, not the content.

What follows from this ​

The brief overstates one consequence, and I want to correct it rather than repeat it. The brief says a plain-text part being absent is "the documented condition under which Postmark's stripped field doesn't populate at all." The Postmark condition is real but it applies to the inbound reply, not to what we sent. MEASURED from Postmark's own docs: StrippedTextReply populates only when the reply carries In-Reply-To or References, when the Postmark address is in To or Cc, and — quoting them — "we cannot parse HTML email parts of replies to populate this field — it will only be applicable when there is a plain text email part in the reply." It is also documented as English-only and best-effort.

Whether the reply has a text part is the replying client's decision. MEASURED: Gmail down-converts an HTML compose into a text/plain alternative, so Gmail replies will generally carry one. UNRESOLVED: Outlook's reply format tends to follow the original message's format, and whether each Outlook build emits a text/plain alternative when replying to an HTML-only original is version- and policy-dependent. I could not establish it from documentation and I am not going to guess a rate. It is exactly what the corpus measures.

So: sending HTML-only does not by itself void a vendor's stripped-reply field. The vendor advantage does not "largely evaporate". What HTML-only actually costs us is narrower and still real:

  1. We have no lever over the quoted text's shape. Every client's quote of our email is its own HTML→text down-conversion of an 8-nested-table layout. We control none of it, and it differs per client and per version.
  2. Any HTML-only reply is a total loss for text-only parsers — including a vendor's stripped field. SYNTHETIC demonstration: on the one HTML-only sample in the scaffolding, email_reply_parser returns None and the simulated vendor field returns nothing, while quotequail's HTML path returns the answer. This is a capability fact — a text-only parser structurally cannot read an HTML-only message — and it is asserted as a CI test (test_email_reply_parser_cannot_read_an_html_only_message), not a rate.
  3. Accessibility and deliverability, which are outside this spike but point the same way.

Recommendation R1 — add a real text/plain part (independent of the vendor choice) ​

Add a plain-text alternative at Email.py:560-563, and the same at custom_email_template_service.py:1252-1254. This is robust to the UNRESOLVED question above: a text part can only help. It gives every client a plain-text ancestor to quote, it makes the down-conversion ours instead of each client's, and it costs one part per message.

Two constraints, both real:

  • Part order is not cosmetic. In multipart/alternative, text/plain must come first and text/html last. The current code attaches HTML first, so a plain part naively appended lands in the wrong position and clients will prefer the plain text.
  • Generating the text part is not free. The system templates render through two Jinja passes (Email.py:172-207) and per-org custom bodies are injected raw (Email.py:46), so a plain-text render needs its own path rather than an HTML strip at the end. Treat R1 as its own ticket, sized properly, not a line-change rider on the inbound work.

This is a finding with teeth, but its teeth point at our own sending path rather than at the vendor decision. It does not by itself select SES-inbound over a vendor — that is D1's call — but D1 should know that the stripped-reply advantage is contingent on the replying client, not on us, and that R1 improves parse quality under either transport.


2. Parser evaluation ​

All figures MEASURED on 2026-08-10 from the GitHub and PyPI APIs, and by installing into a throwaway Python 3.12 venv in the scratchpad. Nothing was installed into the repo.

CandidateLicenseLast releaseRepo last pushPy 3.12HTML?Strips sigs?Deps
quotequail (closeio)MIT2026-05-282026-07-22yesyes (quote_html)notyping-extensions; lxml for HTML
email-reply-parser (Zapier)MIT2020-10-072024-07-22yesnoyesnone
talon (Mailgun)Apache-2.02017-08-242023-10-18NO — hard failpartialyes (ML)heavy
talon-v2Apache-2.02021-08-12—NO — hard failpartialyes (ML)heavy
flanker (Mailgun)Apache-2.02019-12-052026-04-08untestedMIME onlynoheavy

talon is disqualified, not merely stale ​

MEASURED: uv pip install talon-v2 on Python 3.12 fails to build. Its dependency cchardet==2.1.7 cannot compile against CPython 3.12 — longintrepr.h was removed in 3.12. It additionally pins scikit-learn==0.24.1, which predates 3.12 support. talon proper last shipped to PyPI in 2017, and mailgun/talon has not been pushed to since October 2023 with 71 open issues.

This matters because talon is the library the epic would reflexively reach for (it is the well-known one, and it has the ML signature stripping). It is not installable on our runtime. pyproject.toml:5 pins requires-python = ">=3.12,<3.13". Adopting talon would mean vendoring it and replacing its ML stack — which is a project, not a dependency. I recommend against it and I think the brief's framing of "GitHub email_reply_parser lineage" as the fallback is the right instinct for the wrong reason: it is right because talon is unavailable, not because the lineage is strong. Note also that github/email_reply_parser (708 stars, pushed 2025-11-24) is Ruby; the Python member of the lineage is Zapier's port, last released 2020.

The two live options solve different halves ​

MEASURED by direct assertion, and pinned as tests:

  • quotequail segments quoted from unquoted content and does not remove signatures (test_quotequail_does_not_strip_signatures). It is the only candidate that reads text/html directly.
  • email_reply_parser removes quotes and signatures (test_email_reply_parser_does_strip_signatures) but is text/plain-only.

Neither is sufficient alone. One API detail worth recording because it is a live footgun: quotequail.quote() places the attribution line ("On … wrote:") in the unquoted segment unless you pass quote_intro_line=True. A naive integration leaks the attribution line into every extracted answer.

Recommendation R2 — quotequail + email_reply_parser, layered, with the HTML path as fallback ​

reply has text/plain?  ──yes──> quotequail.quote(text, quote_intro_line=True)
                       ──no───> quotequail.quote_html(html, quote_intro_line=True) -> flatten
                                          │
                                          └─> EmailReplyParser.parse_reply(...)   # signature strip

Implemented as composite_qq_erp in harness/extractors.py. Rationale:

  • Dependency cost is zero net-new transitive packages. MEASURED: lxml (6.0.4) and typing-extensions are already in uv.lock. quotequail needs only those; email-reply-parser has no dependencies. So R2 adds two small pure-Python packages and nothing else. Compare talon's scipy + scikit-learn + cchardet.
  • quotequail is the only actively maintained option (released 2026-05-28), and it is the only one that degrades instead of returning nothing on an HTML-only reply.
  • Both are MIT.

The naive baseline earns its place in the harness and should not be dropped. harness/extractors.py::_naive_baseline is ~20 lines that cut at the first attribution line or quote marker. SYNTHETIC: it scored above both libraries on the scaffolding (7/9 vs 6/9 and 3/9). I do not present that as evidence a hand-rolled cut is better — the scaffolding is my own invention and it flatters a strategy tuned to markers I chose. I present it as the reason the baseline must stay in the comparison: if the libraries do not clearly beat 20 lines of regex on the real corpus, the dependency is not earning its place, and that is a genuine possible outcome. Deciding R2 for real requires the real corpus.

Scaffolding run — for harness verification only ​

SYNTHETIC. Do not cite. 14 samples, 9 scored for extraction, all hand-authored by me.

extractor                  clean    rate  trunc  dirty  miss   n/a  leak%
naive_baseline                 7    78%      0      1     1     0   11%
email_reply_parser             6    67%      0      2     0     1   22%
quotequail_text                3    33%      0      4     1     1   44%
quotequail_html                3    33%      0      5     1     0   56%
composite_qq_erp               5    56%      0      3     1     0   33%
vendor_stripped_reply          0     0%      0      0     0     9    0%

What this run legitimately establishes: every code path executes; the composite improves on quotequail alone exactly as designed (signature stripping is what moves 33%→56%); vendor_stripped_reply correctly reports NO_INPUT throughout because no sample has been round-tripped through a vendor sandbox yet; and a bottom-posted reply is a MISS for four of six strategies. That last one is a capability observation worth carrying forward — reply-below-the-quote is a known hard case for every top-post-assuming parser — but its frequency is entirely unknown and is one of the things the real corpus must measure.


3. The harness and fixture format ​

a local scratch directory (not committed)

FileRole
harness/corpus.pyfixture loading, strict sidecar validation, coverage_gaps()
harness/extractors.pysix strategies + HTML flattening
harness/automated.pyTier A / Tier B detection (§4)
harness/scoring.pyrubric, normalisation, leak markers
harness/report.pymetrics + rendering
run_benchmark.pyCLI (--corpus, --client, --extractors, --json)
test_parse_benchmark.pypytest form — MEASURED: 45 passed, 2 skipped
make_synthetic_corpus.pyscaffolding generator; read its docstring
README.mdrun instructions, fixture schema, repo-migration steps

Fixture format: one directory per client, one .eml + one .expected.json sidecar per sample. Raw .eml rather than a JSON blob of extracted parts, because headers are load-bearing for §4 and any pre-extraction bakes today's MIME-walking assumptions into the fixture. A sidecar per sample rather than a shared manifest, because samples are then atomic to add, review as self-contained diff hunks, and four people collecting replies in parallel never conflict on an index file.

json
{
  "client": "gmail", "platform": "web", "category": "answer",
  "expected_reply": "We need 25 seats to start, scaling to 40 in Q4.",
  "expected_is_automated": false,
  "provenance": "REAL — captured 2026-08-12 by a team member, Gmail web, nudge txn {redacted}",
  "notes": "top-posted, no signature",
  "vendor_stripped_text_reply": null
}

expected_reply is authored by hand, never by running a parser — otherwise the parser becomes its own ground truth. vendor_stripped_text_reply: null means "the field did not populate", which is a measurable result rather than missing data. provenance must start with REAL or SYNTHETIC; the harness keys its real-vs-synthetic accounting off it.

Scoring rubric ​

OutcomeMeaning
EXACTequals expected after NFC + strip
NORMALIZEDequals after whitespace / smart-quote / soft-hyphen normalisation
DIRTYwhole answer present plus extra text
TRUNCATEDextracted is a subset — part of the answer is gone
MISSneither contains the other
NO_INPUTextractor could not run (no text/plain part)

Clean extraction = EXACT + NORMALIZED, reported overall and per client. Exact-match alone is too brittle (Outlook injects soft hyphens and NBSPs; smart quotes vary by client), and contains-answer alone is too lenient (it would score a full quoted history as a success).

TRUNCATED is ranked below DIRTY on purpose, and the two are never collapsed into one accuracy number. Leakage is recoverable: a human or a downstream model can ignore a trailing signature. Truncation destroys the customer's words with no signal that anything was lost — a one-word answer like "25" sitting above "Sent from my iPhone" is exactly the case where an over-eager stripper returns empty. Any parser choice that buys less leakage with more truncation is the wrong trade, and a single blended metric would hide it. The harness therefore reports truncation_rate separately and the pytest threshold for it (test_truncation_rate_ceiling, floor 2%) is tighter than the one for clean extraction.

Metrics reported: clean-extraction rate overall and per client; truncation rate; signature/chrome leakage rate with a per-marker breakdown; NO_INPUT rate (the HTML-only loss); inbound MIME shape census; detection precision/recall/F1 for Tier A alone and Tier A+B; and coverage_gaps() against the brief's criteria.

Leak markers are two sets. Generic client artefacts (signature delimiter, "Sent from my", attribution lines, Outlook's ____ rule, disclaimers, phone numbers) and — the valuable half — ONRAMP_LEAK_MARKERS, drawn from our own chrome and verified at HEAD: notifications@onramp.us, sent you a message about N task/s, Go to Project, View in browser, Unsubscribe, Start Date:/Due Date:/Percent Complete:. Our own 15KB table layout leaking into an extracted answer is the specific failure this spike exists to quantify, so it gets its own markers rather than hiding inside a generic leak count.

Migration into the repo ​

Corpus → app/api/inbound_email/tests/fixtures/reply_corpus/, matching the existing app/api/<domain>/tests/fixtures/ convention. Scrub real names, addresses, and phone numbers from bodies and headers; keep quote markers, attribution lines, and MIME layout byte-identical, because that is the thing under test. harness/automated.py becomes a production module (it is pipeline logic, not test logic); scoring and reporting stay test-side. Keep the capability and rule tests as CI gates; enable the two skipped threshold tests only once the real corpus exists and set their floors from that first measured run. Keep the corpus out of template_playbooks/, project-incoming/, and module_library/ — those are bandit-scanned recursively and an assert in a test under them fails B101. Run as -q --tb=short --no-header.


4. Autoresponder / OOO / bounce detection ​

The brief asks for a number. A number requires real autoresponders, so what I can deliver is the rule set, its reliability split, and the property that makes its error profile acceptable. Signals below are MEASURED from RFC 3834, RFC 3462, Microsoft's X-Auto-Response-Suppress documentation, and two practitioner references (Cerb, multi_mail).

Tier A — header signals. Reliable. ​

A hand-typed human reply essentially never carries these, so Tier A can be treated as authoritative.

SignalRule
Auto-Submitted (RFC 3834)present and token before ; not no → automated
X-Auto-Response-Suppresscontains OOF, AutoReply, or All → automated (Exchange)
Precedencebulk | list | junk | auto_reply
X-Precedenceauto_reply | bulk
Preferenceauto_reply (cPanel/BoxTrapper ships this misspelling — keep both)
X-Autoreply, X-Autorespond, X-AutoReply-From, X-Mail-Autoreplypresence
X-AutogeneratedForward | Group | Letter | Mirror | Redirect | Reply
X-FC-MachineGeneratedtrue
X-POST-MessageClassstarts 9
Delivered-ToAutoresponder
List-Id / List-Post / List-Unsubscribepresence → mailing list, not a reply
Return-Pathpresent and <> or <MAILER-DAEMON> → bounce
Content-Typemultipart/report → DSN (RFC 3462)
X-Failed-Recipientspresence → bounce
From localpartmailer-daemon, postmaster, no-reply, noreply, donotreply, bounce(s), autoreply

A bug the scaffolding caught, worth recording because it is easy to reintroduce. My first implementation treated an absent Return-Path as a null reverse-path, because the empty string was in the null-path set. Result: every ordinary human reply was classified automated — precision 29%, ten destructive false positives. Fixed by requiring the header to be present (harness/automated.py, detect_tier_a), and pinned by test_absent_return_path_is_not_a_null_return_path. This is the single highest-severity bug shape on this path: it silently discards every customer answer while looking like a working filter. Whoever implements detection should write that test first.

Tier B — subject and body prose. Heuristic. ​

Fires on real prose and therefore has a real false-positive rate. Subject patterns: automatic reply, auto-?response, out of (the) office, on vacation|holiday|leave, (annual|parental|sick|maternity) leave, away from my desk, undeliverable, delivery status notification, returned mail, plus abwesenheit / absence du bureau / fuori sede for non-English OOO. Body patterns: i (a)?m (currently )?out of the office, i will be out of the office, on annual leave, on vacation until, limited access to email, will respond when i return, this is an automated reply, please contact … in my absence, plus bounce prose (could not be delivered, mailbox is full, 55x 5.x.x).

Body scanning is capped at the first 600 characters (BODY_SCAN_CHARS). Scanning the whole body would match quoted history and signature boilerplate — a footer reading "contact X in my absence" would flag every message from that sender — and inflate false positives. An OOO notice states its business in its opening lines.

The design property that makes this safe ​

The brief correctly identifies the asymmetry: "a false positive silently discards a real customer answer while a false negative records 'I'm on vacation' as an answer." My answer is to remove the destructive branch from the unreliable tier entirely:

EvidenceVerdictBehaviour
Any Tier A hitAUTOMATEDnever recorded as an answer; no auto-reply (RFC 3834 loop rule)
Tier B onlyHOLDnot recorded, not discarded — routed to a human via the post-as-comment disposition already specified in PLAN.md Q1 / 6074
NeitherHUMANcandidate answer

Only Tier A — the reliable tier — is allowed to be destructive. A Tier B false positive costs a CSM one comment to read, not a lost answer. That converts the fatal error class into a nuisance, and it means Tier B can be tuned aggressively for recall without the usual precision anxiety. test_no_destructive_false_positive_on_any_answer_sample gates exactly this property, and the harness reports Tier-A-destructive false positives separately from Tier-B holds so the two are never averaged together.

Tier B is not optional. SYNTHETIC demonstration, and this is a capability point rather than a rate: an Exchange OOO configured without any auto-headers is invisible to Tier A. On the scaffolding, Tier A alone gives recall 75% (one miss: outlookdesk-004-ooo-no-headers); Tier A+B gives 100% with zero false positives. The 75%/100% figures are artefacts of a corpus I authored and must not be quoted — but the structural fact that header-less Exchange OOO exists and only prose can catch it is real, and it is why the second tier has to be there. X-Auto-Response-Suppress is also, per Microsoft's own docs, a request to suppress rather than a declaration of automation; treating it as Tier A evidence is a pragmatic call that the real corpus should confirm.

One further signal, noted but not recommended as primary: our address token is bound to a recipient, so an OOO firing within seconds of send is weakly detectable by elapsed time. It is fragile (digest sends are drained on a separate cron, so the send moment is not the enqueue moment) and should at most contribute to a HOLD.


5. Getting the corpus — and it does not need inbound infrastructure ​

This is the finding that changes the epic's critical path, so it is stated plainly: the ≥20-real-reply benchmark can be collected and run before D1 builds anything.

The reasoning: the benchmark measures what the replying client produces — quoted-history shape, attribution format, signature placement, MIME structure. None of that depends on how the message reaches us. It depends only on (a) the recipient having received a genuine OnRamp notification email, and (b) someone capturing the raw reply. Neither requires an MX record, a receipt rule, or an ingestion endpoint. A human can reply and then save the message as .eml from their own client's Sent folder, or send the reply to an ordinary mailbox someone already owns and export it there.

The brief treats D3 as downstream of D1 (D1's recommendation is to be "backed by the parse-quality measurement (D3)", and "we own 100% of the parsing" is called the decisive downside). That ordering is right on substance — but it reads as though D3 must wait for transport, and it does not. D3 is only blocked on the vendor-specific half (§5.2), not on the parser and client-shape half (§5.1), which is the majority of the decision content.

5.1 No-infrastructure path — start now, ~2–4 hours of human time ​

Prerequisite: the ability to send a real OnRamp notification email to an address on each client. Already available — POST /tasks/nudge_email (app/api/task/routes.py:1927-1952, login_required only) sends the real TaskNudgeEmail through the real template, and the test-send endpoint (custom_email_template_routes.py, /send-test-email) is a second route in. Both go through the same HTML-only builder characterised in §1, so the captured quote shape is the real one.

  1. Set up five mailboxes across the four clients: Gmail (web + one mobile app), Outlook desktop (Windows Outlook 365 against Exchange Online), Outlook mobile (iOS or Android app), Apple Mail (macOS + iOS). Personal accounts are fine; a real corporate Exchange tenant is the most valuable single addition because gateway disclaimer appenders and header-less OOO both live there.
  2. Send a real nudge on a seeded project to each address, from a deployed environment so the chrome (org logo, brand colour, links) is production-shaped.
  3. Reply from each client, deliberately varying the shape — this variation is the experiment:
    • top-posted one-liner, no signature
    • top-posted with a delimiter signature
    • bottom-posted (answer below the quote) — known hard case, frequency unknown
    • inline/interleaved reply
    • single-token answer ("25") with a mobile signature — the truncation trap
    • a real human reply that is not an answer (a question back) → category: not_an_answer
    • rich text: a bulleted list or a bold word
  4. Capture the raw message. Gmail: Show original → Download Original. Apple Mail: drag the message to Finder, or Raw Source. Outlook desktop: File → Save As → .msg, then convert, or View Source. Outlook mobile: no export — reply to a mailbox that can export (e.g. a Gmail account), and capture there. Save into corpus/<client>/.
  5. Trigger real autoresponders: switch on each mailbox's own vacation responder and re-send. Exchange Online, Gmail Vacation Responder, and iCloud all emit different headers, and an on-prem Exchange tenant is the one most likely to emit none — that sample is disproportionately valuable because it is the only way to measure Tier B's contribution.
  6. Trigger a real bounce: send to a guaranteed-invalid address on a real domain, and to a full mailbox if one is available.
  7. Author each .expected.json by hand. Never by running a parser.
  8. python run_benchmark.py → real numbers. coverage_gaps() reports what is still short.

Target ≥20 answers (≥4 per client, ≥5 for whichever client the customer base skews to), plus ≥4 autoresponders across ≥3 mail systems, plus ≥2 bounces.

Scrub before committing. Real names, addresses, and phone numbers come out; quote markers, attribution lines, and MIME structure stay byte-identical.

5.2 What genuinely does need infrastructure ​

Only one column of the benchmark: vendor_stripped_text_reply. Measuring a vendor's stripped field requires routing a real message through that vendor. The harness already accommodates this without blocking — the field is a sidecar entry, so the corpus is collected once and the vendor column is backfilled later by replaying the same .eml files through a sandbox inbound address. null scores as NO_INPUT, which is the honest representation.

Consequence for D1: the parser-side and client-shape evidence lands early; the vendor-field comparison lands when a sandbox exists. Per the epic's own lesson (third-party-integration-lessons), that sandbox exercise should happen on a deployed environment early rather than against a stub, because a stub would encode our assumptions about the very field we are trying to measure.

  1. Now, unblocked: §5.1 collection. Owner: whoever has the five mailboxes. ~2–4 hours.
  2. Now, unblocked: R1 (text/plain part) as its own ticket — improves parse quality under either transport, and re-running the benchmark after R1 quantifies the gain.
  3. On the real corpus: confirm or overturn R2. Set the threshold floors.
  4. When a sandbox exists: backfill the vendor column, then let D1 decide.

6. Disagreements with the brief ​

  1. "Note honestly if a plain-text part is often absent (that's the documented condition under which Postmark's stripped field doesn't populate)." Two different things are conflated. We send HTML-only (MEASURED, §1), but Postmark's condition is about the inbound reply's text part, which the replying client controls. Our sending choice does not directly disable the vendor field. The real costs of HTML-only are the three in §1, and R1 is worth doing for those reasons rather than for the Postmark one.
  2. D3 is not as blocked on D1 as the brief's ordering implies. §5 — the majority of the benchmark is collectable today. Worth reflecting in the epic's critical path.
  3. "GitHub email_reply_parser lineage" as the fallback parser understates the gap. The Python member of that lineage last released in 2020 and cannot read HTML. Given we send HTML-only and some replies will be HTML-only, an HTML-capable parser is not a nice-to-have. quotequail is a candidate the brief does not mention and is the only actively maintained option; talon, which the brief does mention, is not installable on Python 3.12 (MEASURED).
  4. A single "clean extraction rate" is the wrong headline on its own. Truncation and leakage have opposite costs and must be reported separately (§3), or a parser that quietly loses one-word answers will look like an improvement.

7. Top risk ​

The benchmark stays unrun, and reply parsing ships on assumption. Everything here is scaffolding, capability facts, and a rule set — the actual quality number for our actual template does not exist, and it is the number the epic named its #1 quality risk. The mitigation is cheap and available today (§5.1, a few hours of one person's time with five mailboxes), which is precisely why it is at risk of being deferred behind the infrastructure work it does not depend on.

Second risk: the Return-Path bug class in §4 — a detection rule that silently discards every real answer while appearing to work. It cost me one debugging cycle on synthetic data; on production data it would look like customers simply not replying.

Internal documentation — gated behind Cloudflare Access.