Form Builder Architecture
How OnRamp's form builder stores, evolves, evaluates, and AI-authors a form — all in place, on the storage the platform already has, without breaking a single existing form.
📐 Scope — this is the definition architecture: the JSON document a FORM step persists, how it upgrades, how conditions and calculated values evaluate, and how an AI agent authors it. The field-by-field JSON Schema reference is its generated companion, Form-Document Schema. Answer storage is deliberately out of scope — it does not change (see Answers are untouched).
TL;DR
The form is a flat adjacency list
A FORM step's definition is one JSON document in step_configuration: a root order plus an elementsMap of id-keyed nodes. Flat, not nested — the shape an LLM emits reliably and a validator checks in one pass.
A versioned normalize(), never a migration
Every persisted document carries a schemaVersion. On read, normalize() upgrades legacy shapes leniently; on write it stamps the current version. Touching a form migrates it. No parallel system, no new tables, no data breakage.
One generated meta-schema, three jobs
The element-plugin registry emits a single JSON Schema for the whole document. That one artifact is the strict save validator, the AI structured-output target, and the reference documentation — so none of them can drift from the others.
What a FORM step is today
An OnRamp task is a tree of steps. One step type is FORM, and a FORM step is itself a tree of elements (text inputs, dropdowns, tables, signatures, headings, …). Two untyped JSON columns on the step row carry everything:
| Column | Holds | Lifecycle |
|---|---|---|
step_configuration | the form definition — which elements exist, their config, layout, and visibility conditions | design-time; edited in the builder |
action_response | the answers a respondent submits | run-time; shipped verbatim to Zapier / Prismatic / CRM via the task.subtask.action_response webhook keymap |
These are two separate stores with two separate lifecycles, and the architecture keeps them separate. Everything below is about the definition column. The answer column is an external contract and does not change.
The definition column already holds a flat structure: a root children id list, a flat elementsMap, and a derived dependencies index. The architecture is the disciplined evolution of exactly that shape — not a replacement for it.
The in-place principle
There is one form builder, and it is the one that ships today. The design brings rich, flexible, AI-authorable capability into it in place, under three hard constraints:
The step_configuration shape becomes a versioned, canonical envelope. There is no or_form table, no submission table, no answer-value table. The definition lives where it always has.
Every step_configuration already at rest in every tenant must keep rendering and answering. This is the top correctness bar — enforced by a versioned upgrade path and a corpus of real legacy fixtures.
The registry, the JSONLogic engine, the DAG/cycle layer, normalize(), and the meta-schema are portable code. A separate feature (public forms) that stores a form in a different column reuses that library without any shared table.
Why in-place costs the AI goal nothing. An agent's authoring target is the form-definition envelope — a flat, declarative, registry-validated JSON document. A meta-schema-constrained tool, a validation-feedback loop, and patch-by-id editing behave identically no matter which column the JSON lives in. What makes forms AI-ready is a clean declarative schema, a strict validator, and a stable reference grammar — never the table it sits in. All three are built here, in place.
The canonical envelope
One JSON document per form. A flat adjacency list: root order + an id-keyed node map.
{
"schemaVersion": 5, // format version normalize() upgrades every doc to
"children": ["el_a", "el_b"], // root order — ids into elementsMap
"elementsMap": {
"el_a": {
"id": "el_a", // stable, opaque id — never an array index, never a DB id
"type": "TEXT_INPUT", // registry key = the single source of truth
"key": "company_name", // author-stable SEMANTIC anchor (calc-value vars, AI, piping) — NOT the answer key
"label": "Company name",
"config": { /* type-specific; shape declared by the element plugin */ },
"validation": { /* declarative rule list */ },
"conditions": { /* visibility: a filterSet, compiled to JSONLogic at eval time */ },
"value": { /* OPTIONAL JSONLogic: a calculated/derived field. Presence ⇒ read-only */ },
"binding": { /* OPTIONAL symbolic external-data reference; resolved late */ },
"layout": { "width": "half" }, // full | half | third | quarter
"children": ["el_c"], // containers only; kind declared by the plugin
"parent": null // derived back-pointer, recomputed on save
}
},
"dependencies": { /* derived visibility edge index; recomputed on every normalize */ },
"settings": { /* OPTIONAL form-level overlay; empty ⇒ all defaults */ }
}Invariants
Enforced strictly on new/edited/AI-emitted input, leniently on legacy upgrade (see normalize()):
- Ids are stable and opaque. Reordering is a change to
children; an element's id never moves and is never a positional index or a numeric DB id. - Answers are id-keyed, permanently.
action_responseis keyed by element id at every level (flat answers, table cells by column id, list instances by child id), and the webhook ships that id-keyed blob verbatim. Answer keying therefore stays id-based forever — it is the external contract. The elementkeyis the semantic anchor for the new layers only (calculated-valuevariables, AI semantics, answer piping); theid ↔ keymap is derived fromelementsMapwhen those layers need it. - Referential integrity. Every id/key referenced in
children,conditions,value, orbindingresolves to a live element. A dangling reference is rejected on the authoring path and repaired with telemetry on the legacy-upgrade path. typeis in the registry. An unknown type is rejected — the registry is the closed set an author or agent is constrained to.- Dependency graphs are acyclic.
conditionsandvaluereference graphs are compiled and cycle-checked at save; a cycle is rejected loudly, at authoring time. dependenciesis derived — persisted but never trusted.canonicalize()drop-and-recomputes it on every normalize pass; the client visibility evaluator may read it, but no writer ever hand-authors it. Theparentback-pointer is treated the same way.
Canonicalization means idempotency, not byte-identity: normalize(load(doc)) is a fixed point. One canonical form — sorted keys, derived fields recomputed, legacy shape upgraded to the current schemaVersion. Save always emits it, the AI emits it, and the validator rejects non-canonical new input with a normalized suggestion.
Answers are untouched
The action_response column, its id-keyed shape, and its webhook keymap are an external contract consumed by live customer integrations. The architecture does not touch it. Queryable/typed answers — "every value for field X", trend analytics — are an optional future read-model projected from action_response on write, never a rewrite of it, and never required for AI authoring.
The element-plugin registry and its meta-schema
Each element type is one self-registering plugin — the evolution of the existing metadata registry. One plugin declares everything about a type:
interface ElementPlugin {
type: string; // single source of truth
kind: 'input' | 'content' | 'layout' | 'repeating' | 'object'; // containment + answer-shape rules
configSchema: JSONSchema; // drives the configurator UI generically + validates `config`
validationContract: ValidationRule[]; // typed rules; predicate slots are JSONLogic
answerType: null | 'text' | 'number' | 'date' | 'json' | 'signature'; // how it (de)serializes in action_response
renderer: Component; // editor + runtime, degrades to read-only
configurator?: Component; // OPTIONAL escape hatch; the generic panel is the default
}The registry does three things at once: it kills the hand-written per-type configurator panels (one generic configSchema-driven panel replaces them), it is the closed set the AI is constrained to, and it is the single authority for how each element serializes into an answer.
The registry emits the meta-schema. Composing every plugin's configSchema and validationContract under a discriminated union on type produces one JSON Schema (draft 2020-12) for the whole document. That single generated artifact is simultaneously:
- the strict save/authoring validator — one schema, not per-element imperative checks;
- the structured-output / tool schema an AI agent is constrained to;
- the generation-target reference (Form-Document Schema).
It is generated, never hand-written, so it cannot drift from the plugins that define each type. configSchema stays plain JSON Schema; validationContract is a list of typed rules whose predicate slots are JSONLogic, because cross-field rules ("end ≥ start") and conditional-required are not expressible as per-element JSON Schema — and riding the same evaluation engine means conditional-required falls out of visibility semantics for free.
normalize(): support both, migrate by usage
normalize(raw) → canonical is the single function that lets old and new document shapes coexist. It has two sharply separated modes:
Input: a legacy persisted document (older or absent schemaVersion). Behavior: repair and log telemetry, never throw — legacy data must never brick. Strip dangling references, synthesize missing keys deterministically, fold legacy top-level layout keys into settings. The read path runs this mode, so every consumer downstream sees only the canonical current shape.
The split is non-negotiable: a silent repairer between an agent and persistence means the agent never learns its output was malformed and can never self-correct. The lenient path exists only for data already at rest.
Mechanics. A versioned upgrade chain applies pure vK → vK+1 steps up to the current version (CURRENT_SCHEMA_VERSION, today 5), each unit-tested with real legacy fixtures — Alembic, but for the JSON envelope. The read path upgrades on load; the write path always emits canonical and stamps the current version, so touch = migrate. The backend normalize/validate is authoritative — the AI tool and the save path both go through it; any front-end mirror kept for offline editing is held honest by a shared fixture corpus.
Conditions are the one thing deliberately not migrated: per-element conditions stay in their existing filterSet shape and are compiled to JSONLogic at evaluation time (see below), so the shared condition-authoring surface and its many non-form consumers are untouched and there is zero condition-data risk.
flowchart TD
LOAD["Load step_configuration"] --> V{"schemaVersion?"}
V -- "absent / old" --> UP["normalize upgrade mode:<br/>lenient repair, log telemetry"]
V -- "current" --> CANON["already canonical"]
UP --> CANON
CANON --> USE["builder / renderer / evaluator / AI tool<br/>see ONLY canonical vN"]
USE -- save --> WRITE["write canonical + stamp schemaVersion = N"]
NEW["new / AI-emitted doc"] --> STRICT["normalize validate mode:<br/>fail loud, machine-readable errors"]
STRICT --> WRITE
BACKFILL["optional bounded backfill job"] -. "normalize + write over untouched rows" .-> WRITE
style STRICT fill:#ffe6e6,color:#000
style UP fill:#fff3e0,color:#000Backfill, then contract. A bounded, idempotent job (keyset paging, structurally bounded — never while True) runs normalize+write over untouched rows at leisure, backed by a functional index on the document's schemaVersion. Once telemetry shows effectively no documents below the current version, the oldest upgrade steps are deleted — the strangler-fig endgame that stops the dual-read tax from being paid forever. The seam (strict-validate + canonicalize + stamp) is permanent; only the upgrade rungs are disposable. The backfill entry point lives in app/api/base/services.py.
The conditional & calculated-value engine
Two rule slots per element, with different grammars chosen by need — both running on one compiled JSONLogic engine and one dependency graph:
| Slot | Grammar | Why |
|---|---|---|
conditions (visibility, boolean) | stored as filterSet, compiled to JSONLogic at eval | filterSet is the app-wide, AI-proven authoring grammar; no storage migration |
value (calculated/derived, any type) | JSONLogic-native | boolean filterSet can't express it; a new capability, no legacy |
flowchart TD
A["Per-element conditions: filterSet (stored, unchanged)"] --> A2["Compile filterSet to JSONLogic (in-memory, derived)"]
A2 --> B["Extract field / var refs per element"]
B --> C["Build dependency graph, keyed by element id"]
C --> D{"Topo-sort — cycle?"}
D -- cycle --> E["Reject at save / publish — clear error"]
D -- DAG --> F["Ordered element list"]
F --> G["Evaluate compiled rules once each, with memo"]
G --> H["Apply parent-to-child visibility cascade"]
H --> I["Visible set"]
I -. "client json-logic-js (live UX)" .-> UX["No server round-trip"]
I -. "server json-logic-py (on submit)" .-> TRUST["Re-validate — never trust the client"]
style E fill:#ffe6e6,color:#000The server is the authority; the client is UX only.
- Visibility. The client compiles and evaluates for live UX; the server compiles the same stored
filterSetand re-evaluates on submit. On mismatch the server's set wins — a server-hidden field's answer is dropped, a server-visible-but-required field that the client hid is rejected with a field error. - Calculated
value. The client may compute for display, but the server recomputes on submit and discards the client's value; a read-only calc field is never client-writable, or a crafted request could submit arbitrary totals. - Parity is three-way. The compiler-plus-evaluator must agree JS ↔ Python and match the legacy server evaluator on a corpus drawn from real legacy condition data. Operator vocabulary is canonicalized in the compiler and types are coerced there, so a
==divergence between JS and Python never gets a chance to matter. - Repeating scope. A rule referencing an element inside a repeating LIST/TABLE has no single value, so the strict validator rejects cross-boundary references into repeating containers (
{code:'REPEATING_SCOPE'}). Per-instance scoping is a deliberately-designed later capability, never an accident.
One edge derivation, three consumers: the same dependency graph feeds cycle detection, evaluation order, and the (roadmap) dependency-visualization views. Graph and cycle logic live in app/api/utils/form_graph.py; the compiler in app/api/utils/form_conditionals/; the validation engine in app/api/utils/form_validation/.
Partial-progress persistence
Moving visibility evaluation client-side removes the server round-trip that incidentally autosaved in-progress answers. That autosave is re-provided deliberately as a dedicated PATCH .../answers write (save partial answers, no evaluation), because losing it would silently regress a phone-heavy portal. The client mechanism is layered: a debounced autosave on change is what actually prevents loss, backed by a teardown flush on visibilitychange/pagehide via navigator.sendBeacon (async fetch can't complete on unload). Hidden-field answers are dropped once, at submit, where visibility authority already runs — not on every partial save.
AI authoring
The playbook agent authors a form by emitting the envelope, constrained to the generated meta-schema, persisting to step_configuration. Four pieces make it reliable, not merely possible:
The authoring tool's schema is the registry's emitted meta-schema. The agent cannot emit an unknown type or malformed config — the closed set is enforced at the tool boundary. Flat beats nested for LLM structured output.
Strict validation returns machine-readable, self-correctable errors (CYCLE, DANGLING_REF, UNKNOWN_TYPE). The agent retries against the error, not against prose. This is exactly why the lenient repairer must never sit on the authoring path.
The flat map's superpower: editing an existing form is an id-addressed patch (set elementsMap[id].label, insert id into children at k), not a full-doc rewrite — small, reviewable, cheap-to-validate diffs. Concurrent human + agent edits are guarded by an optimistic-concurrency revision check.
In a playbook or library context, no concrete project data exists yet. The agent writes a symbolic binding that resolves late — without it, every AI-authored form would be static, forfeiting the onboarding-context advantage.
Enforcement is in-tool validation against the generated meta-schema returning those machine-readable errors, plus enum constraints for what tool type hints can express. When AI authoring ships, the boundary line in onramp-agents/src/agents/playbooks/prompts/playbook_architect.md — today "cannot create, edit, or read task steps" — flips to a meta-schema-bounded authoring tool. The prompt is already primed for it.
Cross-context binding
Three envelope keys reserve external references, derived values, and structural markers. They are easy to conflate and must stay hard-separated — the meta-schema must never teach an agent to put one kind of reference in another's key:
| Key | Meaning | Example |
|---|---|---|
binding | symbolic external data reference (CRM, data field, project/account/contact attr, prior answer), resolved late | prefill industry from Account.Industry |
value | calculated/derived field: JSONLogic over other answers in the same submission | a total, a concatenation, a score |
facets | container role markers (assignment / completion / status) for the task-as-one-document horizon | reserved; not yet emitted |
A binding is { source, key, mode, when? }. source is one of data_field, crm_field, project_attr, account_attr, contact_attr, prior_answer; key is a stable identifier, never an environment-local numeric id; mode is prefill, writeback, or both. Following convention-over-configuration, when is optional and defaults from the source:
when | Meaning | Default for |
|---|---|---|
project_create | resolve once at project instantiation | crm_field, data_field, *_attr |
on_source_available | resolve when the upstream answer/task becomes available | prior_answer |
render | re-resolve at each open (live refresh) | none yet |
Unresolved is a normal state, not an error — a missing CRM value or an unanswered source task simply skips that field and never fails project creation. The resolver reuses the existing CRM/Workflows field-mapping machinery and the clone-time merge-field remap. Provenance of a machine-written answer is instance data and lives in a sidecar key inside action_response, never in facets.
Anchor files
| Concern | Path |
|---|---|
normalize, upgrade chain, reserved keys, CURRENT_SCHEMA_VERSION | app/api/utils/form_utils.py |
| Dependency graph, cycle detection, repeating-scope guard | app/api/utils/form_graph.py |
Conditionals compiler (filterSet → JSONLogic) | app/api/utils/form_conditionals/ |
| Declarative validation engine | app/api/utils/form_validation/ |
| Generated document meta-schema (artifact) | app/api/utils/form_schema/form-document-schema.json |
| Strict document validator | app/api/utils/form_schema/validator.py |
Element-plugin registry + configSchemas | app/onramp/components/OFormBuilder/utils/registry.js |
| Front-end meta-schema composer | app/onramp/components/OFormBuilder/engine/metaSchema.js |
Generic configSchema-driven configurator | app/onramp/components/OFormBuilder/components/GenericConfigurator.vue |
| Definition + answer columns on the step | app/api/task_step/or_models.py |
| Schema-version backfill | app/api/base/services.py |
| AI authoring boundary prompt | onramp-agents/src/agents/playbooks/prompts/playbook_architect.md |
Related
- Form-Document Schema — the generated, field-by-field JSON Schema reference for the envelope described here.
- Task as One Document — the horizon this architecture ladders toward: steps and forms as one substrate.
- Form Builder Feature Landscape — where the builder sits against best-of-breed tools, and the product roadmap this foundation unlocks.