Agent smoke-test gate
A CI check that proves every agent still builds and wires up before a PR merges — with zero per-agent configuration. Does each agent boot, are its tools registered, does the shared SQL kit fail safe. Built on AWS's strands-agents-evals SDK so we can grow richer checks later — but today it's a deterministic smoke test, no LLM-as-judge, runs only as a PR gate, and a new agent is covered the moment it exists.
What it is
We have seven agents and no other automated check that they actually wire up end-to-end. Plain unit tests mock the runtime, so an agent can pass every test and still be broken in practice (bad tool wiring, a tool dropped from the registry, the SQL tool throwing 500s). This gate, by convention, builds every agent for real, confirms its tools are registered, and exercises the shared SQL kit's fail-safe path. Deterministic, credential-free, and it runs on every PR that touches onramp-agents/. There is nothing to configure — add an agent and it's in the gate.
What "Strands Evals" gives us
It's an AWS SDK for testing agents. After an agent runs it can look at two things:
- Output — what the agent said (its final answer).
- Trajectory — what the agent did (the tools it called / has registered).
We assert against those. Every v1 check is a plain deterministic assertion — no model involved. "Did the SQL tool leak a raw error?" and "Did the agent register any tools?" have one right answer; you don't need a model to grade them.
Later, the same SDK supports LLM-as-judge: hand the answer to Claude and have it score against a plain-English rubric — for subjective questions like "was this answer actually helpful?". We are not doing that in v1 (see Later). That's the one layer that does need per-agent config — and the
evals/judges.pyseam is already in place for it.
v1 — the three checks
Derived by convention and run for every agent. All deterministic (assertions, no model grading, no AWS creds).
Liveness. Each agent builds via build_agent() and returns a non-error response to a trivial prompt. Catches broken tool wiring and import/manifest breakage that mocked unit tests miss.
Tool-registry sanity. The built agent registers at least one tool and the tool names are unique (no clobbering). Reads the agent's real tool registry. (The router is exempt — it's a structured-output classifier with no tools.)
SQL fail-safe. Drive the shared SQL read kit through a simulated backend failure (5xx / network / 403) and assert it degrades to a sanitized error code — never leaks a raw exception or DB string into the agent's context. Plus a happy-path parse check. Runs once (the kit is shared).
Why check 03 earns its keep long-term. It guards a deterministic, security-flavoured plumbing invariant — "a backend failure never leaks internals to the model/user" — on a different layer than the LLM-judge evals we may add later. Those judge semantic quality (helpful? faithful? routed right?) with a live model; they would never catch the SQL kit starting to return
str(exc). This check would, on every PR, for free.
Out of scope (by design): routing quality (does the router pick the right agent?) and answer quality both need a live model, so this deterministic gate does not test them — it proves agents build and their tools are wired, not that the LLM behaves. That's the job of the future LLM-judge layer (see Later).
How it actually works
File layout
The whole gate lives in one shared package. Agents carry no deterministic-eval config — only an inert seam file for the future LLM-judge layer.
onramp-agents/
├── src/
│ ├── shared/
│ │ └── evals/ # ← the entire gate lives here
│ │ ├── checks.py # deterministic evaluators + stubbed agent builders
│ │ └── runner.py # run_all(): build every agent, run the 3 checks
│ └── agents/
│ └── <domain>/
│ └── evals/
│ ├── __init__.py
│ └── judges.py # ← inert seam: future LLM-judge config (does nothing yet)
│ # portal also keeps evals/harness.py (its existing rollout gate)
└── tests/
└── unit/test_smoke_evals.py # ← the PR gate: asserts run_all() passesZero config (and the future seam)
There is nothing to write for the deterministic gate. run_all() discovers every src/agents/*/ directory and derives all three checks by convention — a new agent is covered the moment it exists, no file to add, no placeholder to fill, no coverage gate to satisfy.
The only per-agent eval file is the inert seam, scaffolded by /agent-create:
# src/agents/<domain>/evals/judges.py
"""LLM-as-judge config for the <domain> agent (future seam).
Deterministic checks run automatically — nothing to configure here for those.
When the LLM-judge layer lands, list the judges for this agent, e.g.:
LLM_JUDGES = ["helpfulness", "faithfulness"]
"""
LLM_JUDGES: list[str] = []It does nothing today. It exists so that which judge runs against which agent — the one thing convention can't decide — has an obvious, colocated home when the LLM-judge layer ships. A simple list of evaluator names, per agent.
Each check maps to a Strands Evals primitive
No LLM in any of these — built-ins / small custom evaluators, plus one plain structural assertion.
| Check | Source of truth | Evaluator |
|---|---|---|
| liveness | the agent's response text | RespondedEvaluator (custom — non-empty, no error sentinel) |
| tool-registry sanity | the agent's registered tool names | plain assertion (non-empty + unique) |
| SQL fail-safe | the kit's structured response | ToolHealthEvaluator (custom — strict success, or a sanitized code with no leaked DB string) |
Deterministic + credential-free
The PR-gate CI (onramp-agents-ci.yml) runs make agent.test with no AWS credentials, so the gate must never call Bedrock. It doesn't: the real build_agent() body runs (real import, tool-kit wiring, manifest, tool registration — the breakage mocks miss), but the leaf Agent + model loader + the SQL kit's httpx transport are stubbed (mirrors test_views_agent.py::_FakeAgent). Fully deterministic.
flowchart TD
PR["Open a PR that changes an agent"] --> CI["CI runs the smoke gate"]
CI --> BUILD["Build every agent (model stubbed)"]
BUILD --> L{"Each agent boots and answers?"}
L -->|no| FAIL["PR blocked"]
L -->|yes| T{"Each agent's tools registered and unique?"}
T -->|no| FAIL
T -->|yes| S{"Shared SQL kit fails safe (no leak)?"}
S -->|no| FAIL
S -->|yes| PASS["Safe to merge"]How to extend it (future contributors)
I'm adding a brand-new agent
Nothing to do for the smoke gate — it's covered automatically the moment the src/agents/<domain>/ dir exists. /agent-create also drops an inert evals/judges.py seam for you; leave it empty until the LLM-judge layer exists.
I want to pin a specific deterministic regression
The smoke gate is convention-wide (liveness / tool sanity / SQL fail-safe). For a one-off deterministic assertion specific to one agent, add a normal unit test under tests/unit/ — that's the right home for bespoke checks. The smoke gate stays uniform.
I want an LLM judge on this agent's answers
That's the future layer. When it ships, list the judge evaluator names in the agent's evals/judges.py (LLM_JUDGES = [...]). That's the one place per-agent eval config lives — because which judge fits which agent isn't knowable by convention.
Later (explicit non-goals for v1)
Each is a clean add-on once the smoke gate is trusted — none require rearchitecting:
- LLM-judge rubrics for subjective quality (helpfulness, faithfulness, routing correctness). Costs money per run; configured per agent via
evals/judges.py. - Targeted regression cases — for env-divergent classes (
vendor_id = 0, hardcodedor_object_type.id), which in practice surface as SQL failures the fail-safe check already exercises. - Live-model / environment runs — running against real Bedrock or a deployed env (dev/stage) is intentionally out of scope. v1 is a deterministic PR gate only.