RFC: Deploy-Synced Help Documentation
Make each help page a build artifact of the route it describes, so the pipeline that ships a UI change also proves, re-captures, and rewrites the page that documents it.
💬 Feedback wanted — this is a draft. Weigh in on the GitHub Discussion before we commit. Open questions are flagged in Risks & open questions.
TL;DR
Deploys say which pages went stale
A production deploy walks its changed files through the component dependency graph and route manifest we already fabricate, landing on the help pages that describe those routes. No semantic guessing from raw diffs.
Playwright is the witness, not the model
A doc script walks the real flow on the seeded CI stack, asserts the end state, and captures screenshots and a screencast in the same pass. The model writes prose from that recorded trace. A failing script means the page is stale, and it cannot hallucinate.
Pylon becomes a renderer we can drop
Canonical pages live in the repo as Markdown with frontmatter. Merge renders HTML to Pylon over its API and syncs the same Markdown to a Bedrock Knowledge Base for Aero. Leaving Pylon costs one re-render.
Motivation
Help documentation lives in Pylon and is maintained by hand. The product changes faster than a person can rewrite it, so the help center is chronically behind. The cost lands on customers first and on us second.
End users do not know what OnRamp can do today and have no trustworthy place to check. This suppresses adoption and feature discovery — the primary cost, ahead of any internal one.
CS routes questions to product and engineering that a current help center would answer without a human.
Prose can be spot-checked by reading. A stale screenshot looks fine until a customer follows it and the button is not there. Nothing tells us which images no longer match the product.
Reframing the problem
Four corrections to how the problem is usually stated. Each one changes the design.
The asset already half-exists, and it is not prose. What we want is a capability manifest: every route, what it is called, what it is for, which flag gates it, which components render it, and one executable flow that proves it works. onramp-agents/src/shared/utils/route-map.json already carries a human title, a one-line description, aliases, and an agentVisibility predicate such as flag:core-insights for every org-admin route. The generated dependency graph maps each of 119 shared components to the concrete page files that import it. A help page is that manifest entry plus prose plus media.
Ground truth must be executed, not inferred. The usual framing pits "LLM reads the diff" against "LLM drives the app." Neither should be the witness. Raw diffs are the input LLMs handle worst — the ReleaseEval benchmark found models still struggle on diff-to-summary while doing well on structured inputs, and no public precision or recall figures exist for "is this PR user-facing." Free-roaming browser agents are worse than their marketing: on the human-judged Online-Mind2Web set the best agents score 56–61% while the same agents self-report 86–89% on WebVoyager, and the dominant failure mode is hallucinated success. What is reliable is a deterministic Playwright script against seeded state, which is what our 13 E2E suites already are.
Trigger on rollout, not merge. Code ships dark. Flags are three booleans on Feature plus a per-vendor override row, with an audit log and no events. Docs should publish when a feature becomes globally enabled and not retired, and should be filtered per customer by the same enabled_feature_codes Aero's SQL tool already carries in caller state.
Pylon is a tenancy, not a home. Their 2026 changelog is dominated by Account Intelligence: project templates with task dependencies and relative due dates, milestone-completion triggers, Smart Projects recommended for "customer onboarding, technical implementations," Kanban and Gantt in the customer portal, customer-created tasks, a bulk task-import API, and Health Score Builder GA on 31 August. That is a playbook to project to milestone to portal model, backed by a $31M Series B. Every article we host there also trains their support agent on our vocabulary and our gaps. Their API is good enough to keep them as a renderer. It is not good enough to be a source of truth: HTML body only, no Markdown import, no knowledge-base webhooks, and no KB tools in their MCP server.
Landscape
What exists as of September 2026, by role in the pipeline. Claims marked reported come from third parties or vendor blogs rather than a vendor's own documentation.
| Tool | Role | Git/Markdown source? | Structured export for retrieval | Generates from product changes? | Pricing signal |
|---|---|---|---|---|---|
| Pylon KB | Renderer + support-agent knowledge | No — HTML via API, no import | REST CRUD, hosted llms.txt; no per-article Markdown, no KB webhooks or MCP | No — drafts from tickets only | $59–139/seat plus add-ons (reported) |
| Mintlify | Docs host + drift agent | Yes, GitHub App | Markdown, llms-full.txt, hosted MCP | Yes — merge-triggered doc PRs, Pro and up | Free tier; ~$450/mo Pro |
| GitBook | Docs host | Yes, bi-directional Git Sync | Markdown, llms-full.txt, REST | No | Free; $65/site plus $12/user |
| ReadMe | Docs host | Yes, bi-directional | MCP with an update-docs tool | DIY via MCP in CI | Free; $250/mo Pro |
| VitePress | Self-hosted renderer | Yes, we own it | Markdown; plugin for llms.txt | Only what we build | Free — already runs this docsite |
| Intercom / Zendesk | Support plus KB | No | Full article CRUD | From conversations only | $29–132 / $55–115 per seat |
| kapa.ai / Inkeep | Retrieval and answer layer | Ingests GitHub Markdown | Retrieval API plus read-only MCP | No — coverage-gap reports | Contact sales |
| Doc Detective | Docs-as-tests capture | Yes, steps in Markdown | n/a | Re-captures when UI drifts past a threshold | Open source |
| Playwright 1.63 | Screenshot and screencast engine | Yes, specs | Traces with aria snapshots | Yes, when scripted | Open source |
| Argos / Chromatic | Visual-diff review UI | n/a | n/a | Staleness signal only | $100 / $179 per month, 35k shots |
| Arcade / Storylane / Supademo | GIFs and interactive demos | No | Supademo exports Markdown | No re-record API on any of them | $50/mo entry |
| Pendo / Whatfix / Userpilot | In-app guidance | No | Pendo guide export | Prompt-generated; no selector self-heal | Pendo free under 500 MAU; $279–299/mo others |
| Promptless | Drift agent, end-user scope | PR-style | n/a | Yes — PRs, Slack, tickets | Contact sales |
| Bedrock Knowledge Bases | Aero retrieval | S3 sync of Markdown | Retrieve API, metadata filters, hierarchical chunking | n/a | Managed KB $5/GB/mo, $1 per 1k retrieves |
Three findings worth stating plainly:
- No vendor generates end-user docs from product changes. Intercom, Zendesk, Decagon, Sierra, and Pylon all draft articles from conversations. The merge-triggered doc agents are developer-docs shaped. Promptless is the only vendor whose stated scope includes support articles, with contact-sales pricing and vague destinations.
- Every GIF and interactive-demo tool is human-recorded. None offer a re-record API. Arcade's Enterprise API can ingest a video we made but cannot export one back. For regeneration, Playwright is the only engine.
llms.txtis theater. Ahrefs found 97% of published files received zero requests. Publish it because it is free; expect nothing. What agents actually consume is a search-then-fetch-then-cite retrieval shape, which Aero gets natively from a Bedrock KB.
Proposal
Canonical pages are Markdown in the product repo. A production deploy nominates stale pages, Playwright proves what is true and captures the media, a model writes the prose from that trace, and one bot PR per release goes to a single owner.
flowchart TD
D[Prod deploy or flag rollout] --> N[Nominate]
N -->|changed files| G[Dependency graph]
G -->|page files| R[Route manifest]
R -->|frontmatter routes| P[Candidate help pages]
P --> X[Prove: run doc scripts on seeded stack]
X -->|pass| M[Fresh screenshots and screencast plus aria trace]
X -->|fail| S[Mark page stale: flow broke]
M --> W[Write: rewrite prose from trace]
W --> J[Judge faithfulness and score confidence]
J --> Q[One help-bot PR, tiered review]
S --> Q
Q -->|merge| PY[Render HTML to Pylon]
Q -->|merge| KB[Sync Markdown to Bedrock KB for Aero]
style X fill:#e6f3ff,color:#000
style Q fill:#fff3e0,color:#000The page as a binding
A help page binds a route, a set of flags, a doc script, and its captures:
help/playbooks/create-a-playbook.md playwright/help/playbooks.create.doc.ts
--- authed fixture, seeded docs org
slug: create-a-playbook capture matrix light and dark
pylon_article_id: 7244650068 screencast with showActions
routes: [app.playbooks.new] aria snapshot per step
features: []
media: [create-01.png, create-flow.gif]
verified_sha: 7f9e0b0be3
---verified_sha is the load-bearing field. It records the product commit the page was last proven against, which makes freshness measurable and rollback a git revert plus a re-push.
Where media lives
The repository owns the image. A page's media: entries are repository-relative paths, stored through Git LFS so the bytes stay out of the packfile while the path and its version stay in the tree, resolved by the commit that produced them. Checking out a tag reproduces that release's pages, screenshots included, and the corpus an agent reads is self-contained rather than a set of links into somebody else's store.
Pylon is a render target for images exactly as it already is for prose. The push step uploads each referenced image to Pylon's asset store and rewrites the src before sending the body, the same way it already renders Markdown into Pylon's HTML subset. Pylon's copy is derived and disposable: delete it and the next push restores it.
Serving from Pylon rather than from a bucket of our own is settled by what Pylon already does. An upload returns a signed assets.usepylon.com URL under the workspace, expiring in the year 9999, and every image on a published article today is served from exactly that store in exactly that form. Nothing about a customer-facing bucket, its permanence, or its CDN needs deciding.
The cost is real and is accepted rather than argued away. LFS relocates the churn of regenerated captures, it does not remove it: every refresh is another stored version, storage and bandwidth are billed, and each clone and CI checkout pays for them unless it opts out. The alternative that keeps the repository authoritative more cheaply is a content hash in the page with the bytes in a bucket of our own, which buys a smaller repository at the price of needing our infrastructure to resolve an image at all.
This is the mechanism for pages the loop produces from here. Images uploaded directly in Pylon's editor, and those on articles the import skipped as internal or unlisted, stay where they are; bringing them under the repository is its own piece of work. The conflict guard already refuses to overwrite a page edited in Pylon, so the two can coexist until then.
Review by exception
The bot PR sorts its own changes into three tiers, so a release that only moves pixels produces no human work:
| Tier | Contents | Action |
|---|---|---|
| 0 | Media-only refresh, pixel variation under threshold | Auto-merge |
| 1 | Prose changed, faithfulness judge passed | Owner reads a redline |
| 2 | New page, proposed removal, doc script failed, or flag mismatch | Owner decides |
Current vs proposed
Pages are Markdown in the repo with route and flag frontmatter. Deploys nominate candidates through the dependency graph; doc scripts prove them on the seeded stack and regenerate the media; a model rewrites the prose from the trace. Merge pushes HTML to Pylon and syncs Markdown to a Bedrock KB that Aero retrieves from and cites. In-app links resolve route name to slug from the frontmatter instead of hard-coded URLs.
What already exists
Roughly 60% of the plumbing is in place, which is the main reason this is worth doing now rather than buying it.
| Capability | Where it lives today | Gap |
|---|---|---|
| Route manifest with titles, descriptions, gating flags | devtools/context-fabrication/ui/extract-routes.ts emits route-map.json | None — read it |
| Component to page mapping | Generated dependency-graph.json, 119 components | None — walk it |
| LLM classification in CI | .github/scripts/classify-pr-label.ts, Haiku, degrades to SKIP | Reuse the pattern for pages with no route match |
| Release range against running prod | release-analysis.yml and notes-baseline.js | None — extend it |
| Seeded stack plus authed Playwright fixtures | .github/actions/setup-seeded-stack, playwright/shared/auth.setup.ts | Needs a realistic docs org |
| Pylon API client | app/api/help_center/services/pylon.py | Read-only; needs a write path |
| Aero knowledge base registry | onramp-agents/src/shared/knowledge/registry.py | One entry, console-managed, nothing populates it |
| Visual regression baselines | Nothing | No toHaveScreenshot anywhere in playwright/ |
Cost envelope
| Item | Basis | Order of magnitude |
|---|---|---|
| Page rewrite plus faithfulness judge | Sonnet, roughly 45k in and 4k out, two calls | ~$0.15 per page |
| Nomination classifier | Haiku, 16-token output | under $0.01 per PR |
| Full corpus regeneration | Rare — only on a style-guide change | ~$30 |
| CI | Seeded stack plus doc scripts per prod deploy, ~15 min | tens of dollars per month |
| Bedrock Managed KB | Corpus well under 1 GB | under $20 per month |
| Visual-diff review UI (optional) | Argos Pro, 35k screenshots | $100 per month |
Rollout
Each phase is useful if we stop there.
Repo becomes the source of truth; deploys start nominating pages.
- Import. Pull every published article from the Pylon articles endpoint, convert HTML to Markdown, commit under
help/with frontmatter for slug, Pylon article id, collection, routes, features, and status. Seed the route bindings by matching article titles against the route manifest's titles and aliases; the owner fixes the misses once. - Push on merge. A publish workflow renders Markdown to HTML and patches changed articles, posting new ones as drafts. Turn on Pylon's enforced review so its editor cannot silently diverge.
- Nominate on deploy. Extend the release-analysis workflow: walk changed
app/ui-*files through the dependency graph to page files, then to routes, then to help pages. For PRs with no route match, classify from title, body, labels, and paths — never the diff. Open one PR labeledhelp-botwith a redline per nominated page and a fresh authed screenshot of each nominated route. - Kill the hard-coded links. Replace the pasted
help.onramp.usURLs in org-admin with a route-to-slug lookup generated from the frontmatter.
Unlocks: version-controlled docs with history, Pylon as a pure renderer, and a stale-page nomination within an hour of every production deploy.
Alternatives considered
- Release notes into Pylon drafts. Filter the release's PRs by the user-facing label, summarize each with the existing Bedrock job, and post drafts through Pylon's review flow. Ships in a week and reuses almost everything. Rejected as the destination because a changelog is not documentation: articles accrete per release and never revise the page that describes the feature, so the help center gets more out of date, faster, with more content. Its genuine value — a per-deploy nomination signal — is folded into Phase 1.
- Separate content repo with sync adapters. Keeps the product repo clean and makes the rendering surface a config row, which is the purest form of vendor optionality. Rejected for now because the proof step needs the product's Playwright fixtures, auth setup, and seeded stack; cross-repo checkout works but every fixture rename breaks docs silently, and nothing type-checks frontmatter references to routes and flags in the other repo. This is the fallback if bot PRs and media prove too noisy in the product repo.
- Headless CMS as the middle layer. Attractive for non-engineer editing, but it reintroduces the exact drift problem: two places to edit, one pipeline.
- Agent-native, no help site. Documentation exists only as a retrieval corpus plus generated in-product guidance. This is the honest end state of "the asset is a representation of the experience, not a site," and Aero already has navigation and entity-URL tools plus a read-only SQL kit that carries the caller's enabled features, so answers could cite the customer's real project rather than a generic example. Rejected as a starting point: one owner cannot audit a corpus with no pages, a bad chunk becomes a confidently wrong answer with no page to correct, it requires app instrumentation that does tax engineering velocity, and if it underdelivers we have neither docs nor a site. Phase 4 is its first defensible slice.
- Buy Mintlify Automations or Promptless. Mintlify triggers on merged PRs, reads the diff, and opens a docs PR, with hosted Markdown,
llms-full.txt, and an MCP endpoint included. Fastest path to a reviewed loop. It writes from the weakest input, captures no media, knows nothing about our flags or seeded stack, and publishes no accuracy metrics. Reasonable as the renderer if we leave Pylon; not a substitute for the proof step.
Scored comparison
Weights reflect the stated priority order: velocity and single-owner operability dominate, dual consumption ties with them because Aero is the second consumer, visual media is scored on regenerability rather than polish, and vendor optionality and reversibility round it out. Scores are 1–5.
| Constraint | Weight | Release notes | Spec-derived | Content repo | Agent-native | Buy |
|---|---|---|---|---|---|---|
| Zero engineering tax | 25 | 5 | 4 | 4 | 3 | 5 |
| One owner, review by exception | 20 | 3 | 5 | 4 | 2 | 3 |
| Dual consumption, human plus Aero | 20 | 2 | 5 | 4 | 4 | 4 |
| Visual media regenerable | 15 | 2 | 5 | 3 | 3 | 1 |
| Pylon optionality | 10 | 2 | 5 | 5 | 5 | 4 |
| Reversibility when wrong | 10 | 5 | 4 | 3 | 2 | 4 |
| Weighted score of 5 | 3.25 | 4.65 | 3.85 | 3.10 | 3.60 |
The recommended option loses a point on velocity only because bot PRs and media live in the product repo.
Risks & open questions
The golden seed is two vendors with example.com users. The generator that produces plausible account names runs at runtime, not in the CI seed. If captures look like a test rig, the visual half of this design has no engine.
Mitigation: this is the first prototype, before any prose is generated. Doc scripts assert the seeded state they need, so a thin seed shows up as a red page rather than a bad screenshot.
Pylon has no knowledge-base webhooks. A CS typo fix in their editor is silently overwritten on the next push, and trust in the pipeline dies in one Slack thread.
Mitigation: enforced review in Pylon, plus a nightly diff of Pylon's published HTML against our render that opens a PR when they disagree.
They are E2E tests. An engineer sees a red script blocking nothing and deletes it, and the page silently stops being proven.
Mitigation: doc scripts gate nothing — they only mark pages. Deleting one fails the docs run loudly and notifies the owner.
Tiering set too conservatively turns every release into tier-1 review work.
Mitigation: measure review minutes per release from week one, targeting under 30, and move the confidence thresholds rather than adding people.
Fonts, relative dates, avatars, animation timing, and the 14px-versus-16px root font density that org-admin flips per route.
Mitigation: Linux-pinned container, frozen clock, element masking, a capture stylesheet, density pinned per page, and a variation threshold rather than pixel equality.
Style drift across hundreds of pages is a slow failure that no single review catches.
Mitigation: the style guide as a cached system prefix, the faithfulness judge for accuracy, and a quarterly human sample read for voice.
Open questions
Can the seeded CI stack produce customer-grade screenshots? Run the demo-data generator in the seeded-stack action, capture five pages light and dark, and show them to CS blind against real customer screenshots. Two days. Everything else waits on a yes.
Answered 2026-09-15: not today. The golden seed is reference tables plus two empty tenants: zero projects, playbooks, tasks, or accounts, so every project and portal page in CI renders its empty state. The demo-data generator (four scenarios, 3 to 25 projects each) cannot fill the gap as wired: it refuses to run unless the vendor is flagged internal_demo_instance and already has an active playbook, neither of which the seed provides, and it is reachable only through an authenticated org-owner request that hands off to the background job service. What is already in place: CI boots the stack as a PR preview, so the portal's preview-login gives Playwright a customer session, and dark mode is a single stored preference. Prerequisites for a yes: the seed authors one active playbook and the demo flag for the customer tenant, the seeded-stack action calls the generator service directly after seeding, and a Playwright project captures light and dark against both sessions. Whether the generator's output reads as a real customer is still the open half; the blind CS comparison stands.
How good is graph-walk nomination? Replay the last 20 production releases through dependency graph to routes to pages and have the owner label each nomination stale or not. Target recall above 90%, with the classifier filling the gap.
Prose from trace versus prose from PR summary? Ten pages, both variants, blind rating by CS on accuracy and tone. This decides whether Phase 2's proof step earns its cost or Phase 1's redlines are already good enough.
How does Pylon behave on reconcile? Does patching with the publish flag preserve slug, SEO, and translations? Does enforced review actually block editor saves? Does their visibility config map onto per-account gating well enough to mirror our flags?
Screencast to GIF quality. Confirm a 20-second flow with action highlights lands under 3 MB after the palette pass and stays legible at the portal's phone width.
Does a help KB actually improve Aero? Build a 50-question set from real CS escalations and measure answer accuracy with and without the retrieve tool. This is also the baseline for the freshness metric's downstream effect.
Where does media live? Answered: the repository owns it through Git LFS and Pylon renders it, under "Where media lives" above. What is still open is the exploratory-agent option, if we ever want one walking preview environments: it runs on the Claude API with Playwright as the executor rather than through AgentCore, which is a second credential path to own.