I Didn't Write the Code, I Wrote the Grader

Building RouteCause, an autonomous Kubernetes inference-diagnostics agent, in a director-only run at Claude Build Day (ex-Fable 5 Build Day). Seven engineering lessons.

• Repo (built in-window): github.com/nsega/routecause
• Environment (brought in): github.com/nsega/inference-lab
• 1-min demo: youtu.be/XXl5l0c5GmM

LLM serving on Kubernetes rarely degrades because you ran out of capacity. It degrades because of routing and scheduling: a mis-weighted Endpoint Picker (EPP) scorer, an unhealthy endpoint left in rotation, prefix-cache-aware routing silently switched off by a rollout. When you get paged on an SLO breach, the truth is spread across four places (gateway/EPP metrics, the scheduler plugin config, live endpoint state, and recent manifest diffs), and you hand-correlate them under pressure. Worse, the ecosystem is mid-reorg (EPP consolidated from kubernetes-sigs/gateway-api-inference-extension into llm-d Router), so the docs lag the code you’re actually running.

At Claude Build Day (June 13, Ferry Building, SF; announced as Fable 5 Build Day, run as Claude Build Day), I built RouteCause: an agent that does that four-source correlation autonomously and hands back a root-cause report with a validated fix. The constraint was the interesting part. The event’s whole premise is director-only: brief the model well, then interact minimally and let it build. I ran Claude Code with Opus 4.8 in /effort ultracode, and the model wrote every line of routecause/ inside the build window. I wrote three documents and then mostly watched.

Here’s what held up under that constraint, at the level of code, not vibes.

The environment RouteCause diagnoses (nsega/inference-lab, brought in pre-event): a one-command kind cluster running llm-d Router (EPP) over three simulated vLLM backends, Prometheus, and an in-cluster load generator, plus the three reproducible fault scenarios (S1/S2/S3) it’s graded against. The lab only produces honest, queryable symptoms; it holds no diagnosis logic of its own.

1. ultracode + a machine-checkable rubric did the steering, not me

ultracode is the trigger that tells Claude Code to write and orchestrate its own multi-agent harness, a dynamic workflow, for the task at hand, instead of planning and executing everything in one context window. I paired it with Opus 4.8’s xhigh reasoning effort, so the model spun up and coordinated its own subagents in the background while my session stayed free. That only pays off if “done” is something the model can check without me.

So the human artifacts were a brief, a rubric, and a design draft:

• BRIEF.md: problem, target user, the exact fault_category enum the report must emit (scorer-weight-misconfig / unhealthy-endpoint-in-rotation / prefix-cache-routing-disabled / other), and hard rules.
• RUBRIC.md: MUSTs (A1–A4) and SHOULDs (A5+) phrased as checks a fresh agent can grade against, plus a deterministic grader in workflow/.
• DESIGN-DRAFT.md: the architecture below, with rubric check-IDs referenced inline.

The model hill-climbed against that target with /goal, read its own results, and decided the next move. My leverage was entirely upstream of the keyboard: a vague rubric line means a brilliant agent confidently sprinting in the wrong direction.

That dynamic-workflows guide (which the event handed out as a resource) is also the blueprint for everything below. It names the failure modes that long, single-context agent runs fall into (agentic laziness, self-preferential bias, and goal drift) and the patterns that structurally prevent them. One of those patterns, root-cause investigation, is almost exactly RouteCause: generate hypotheses from disjoint evidence, then make each face a panel of refuters. I didn’t invent the shape. I wired that pattern to Kubernetes inference and made “done” machine-checkable.

2. Three collectors that never share a context (E1/E2/E3)

The diagnose() pipeline (nsega/routecause, built in-window): three isolated-context collectors (E1/E2/E3) fan into a deterministic aggregate; an optional LLM layer hypothesizes, refutes, and reconciles; then a schema-enforced report ships with a fix validated by kubectl apply --dry-run=server. Everything left of the dashed line is strictly read-only: only the quarantined Actuator may mutate the cluster, and ground truth sits entirely off the diagnosis path.

The diagnostic core is deliberately not one smart agent with all the evidence. It’s three context-isolated collectors, each reading a disjoint source, each forming hypotheses from its source alone:

• E1, Metrics (prom.py): Prometheus only. 2-minute-window PromQL (symptoms fully develop ~3 min after a fault, so a short window is fine immediately). Notably, there is no vllm:request_errors_total series in this stack, so the S2 error signal is low request_success_total rps plus elevated per-backend TTFT on the degraded replica, not an error counter.
• E2, SchedConfig (collectors/): the epp-config ConfigMap only. Healthy EPP scorer weights are pinned (queue=2, kv=2, prefix=3); S1 and S3 live here.
• E3, ClusterState (kube.py): the Kubernetes API only, covering endpoint health, pool membership, and a Deployment-arg diff against baseline (this is where the --failure-injection-rate of an unhealthy backend shows up).

The point of disjoint sources isn’t tidiness. A single context that owns all the evidence quietly develops self-preferential bias toward whatever story it formed first, the same failure mode that makes a model a poor judge of its own output. Splitting the evidence is the guide’s fan-out-and-synthesize pattern: each collector runs in its own clean context, and the orchestrator reconciles their independent reads. The report contract enforces it: schema v1.0 requires fault_category plus ≥2 pieces of cross-source evidence before a diagnosis is valid.

POST /diagnose {pool}
  → E1 ∥ E2 ∥ E3   (isolated contexts, independent hypotheses)
  → merge + rank
  → refuters       (one per hypothesis, fresh context, "falsify this")
  → report + fix   (kubectl apply --dry-run=server gates the fix)

3. The verifier is a separate process, and it earned its keep

Every hypothesis that survives ranking faces an independent refuter subagent in a fresh context whose mandate is to falsify it, the guide’s adversarial verification pattern. The builder never grades its own work:

The builder may never grade its own work.

This is where the run produced the moment the event explicitly asks for. On S2 (unhealthy endpoint), a fresh-context verifier rejected the report on A3: it had falsely cited the two healthy backends (backend-0, backend-1) as “patched off-baseline.” The builder fixed the root cause, a fresh verifier re-graded, and S2 passed.

The bonus was sharper than the catch itself: the verifier also exposed a blind spot in my grader. grade_scenario.py had been re-validating only the Prometheus citations, not the k8s-api ones, so it had been passing the buggy report. I hardened the grader to re-read live Deployment args and fail A3 on any false “off-baseline” claim. An independent context didn’t just check the work; it checked the checker.

4. The intersection→majority baseline bug

The S2 rejection traced back to one bug with two symptoms, and it’s a clean distributed-systems lesson.

E3 (and the fix generator) computed the “healthy baseline” as the intersection of all backends’ args. But S2 also diverges backend-2’s latency args (prefill 2ms→10ms, itl 25ms→40ms). Once those diverge, the args that backend-0/backend-1 shared fall out of the intersection, so the healthy backends look off-baseline (false evidence), and fix-gen picks the wrong target (backend-0), whose “corrected” args equal its current args → empty diff → crash.

The fix: define baseline as the majority (an arg shared by ≥2 backends), and select the fix target by the backend carrying an active --failure-injection-rate (fallback: most-divergent).

a baseline is the majority, never the intersection: a fault on one member must not make the others look faulty.

“Normal” in a replica set is a statistical notion, not an absolute one. The moment one member fails, an intersection-based definition of normal makes its healthy peers look guilty. Letting the agent build it surfaced the bug as a written rule I can now reuse.

5. Read-only by construction; only the Actuator may mutate

The agent is a diagnostician first. That’s the guide’s quarantine pattern (agents that read evidence are barred from high-privilege actions, which are left to a separate acting agent), and it’s enforced structurally here, not by convention:

• kube.py exposes only read verbs. There is no patch/apply path in the diagnosis side at all.
• Mutation lives solely in actuator.py, behind an explicit ROUTECAUSE_ALLOW_APPLY=1 opt-in. The collectors and refuters can’t reach it.

The stretch goal (A6) closes the loop inside that quarantine: the Actuator applies the dry-run-validated fix, reloads the EPP, and watches the same metrics until recovery. On S1 it took P95 e2e 18.96s → 15.28s (back under the 16s SLO), max queue 88 → 0, idle backends back to ~3.5 rps, applied=true, slo_met=true.

There’s a second quarantine, too: ground truth. The injected fault label in lab/state/fault.json is chmod 600, the literal path appears nowhere in routecause/ (every verifier confirmed grep -clean), and diagnose() receives the pool name only. The only things allowed to read ground truth are the fresh-context verifiers and a runtime-parameterized grader. Put the answer key out of reach and the agent has no choice but to actually diagnose, exactly how you’d evaluate a human on-call engineer.

6. The live cluster is the only source of truth, so terminology discipline matters

The live cluster is the only source of truth at diagnosis time.

No web lookups, no upstream docs at diagnosis time. In a mid-reorg ecosystem that’s not a purity stance, it’s correctness: the lab pins scheduler v0.8.0 / GIE v1.5.0, and public docs lag those pins. The naming is an active trap: “Inference Scheduler” was renamed to llm-d Router, while “Request Scheduler” survives as a sub-component. Conflate them during config verification and a subagent will “confirm” the wrong thing. The reports say llm-d Router (EPP) on purpose.

Two more environment details that saved time:

• Standalone deployment mode (EPP + Envoy sidecar co-located in one pod) for the kind cluster, which removes a whole class of gateway-mode setup risk.
• Every kubectl pinned to --context kind-inference-lab: the same kubeconfig also holds other GKE contexts, and you do not want a diagnosis agent free-roaming those.

7. Memory as distilled rules: fail → investigate → verify → distill → consult

The single biggest reason a 6.5-hour autonomous run converged instead of diverging was a persistent NOTES.md running a five-stage loop: fail → investigate → verify → distill → consult. Every failure became a general rule, and the agent consulted the rules instead of re-deriving them. It’s the memory and rule-adherence pattern from the guide, run live: mine the run for the corrections you keep making, and distill the survivors into a rules file the agent rereads.

Turn mistakes into general rules; consult the rules instead of re-deriving them.

The distilled rules read like a senior engineer’s scar tissue: baseline is the majority, not the intersection; a grader must re-validate every evidence source, not just the convenient one; a diagnosis isn’t done when it names the cause: it must also enumerate the alternatives it falsified; don’t gate detection on a slow-moving metric window when a fast-moving one carries the same signal. Eighteen granular commits, 24 green tests, and a clean reject→fix→pass arc later, none of those lessons had to be learned twice.

What I’m taking away

Be honest about where the day was actually won: mostly before any product code, in environment boot, arm64 image validation on Apple Silicon first, and a commit-level provenance boundary between what was brought in (inference-lab) and what was built in-window (routecause). The flashy autonomous build sat on top of unglamorous prep and discipline.

And the deeper shift is hard to unsee. When a model can run for hours, grade its own output in an independent context, and surface a bug in your own grader, the scarce engineering skill stops being “writing the code” and becomes “designing the environment and the rubric the code will be judged against.” The brief is the steering wheel; the verifier is the brakes; the cluster is the source of truth.

Which leaves the question I keep circling back to: if the agent can grade itself, who grades the grader, and how much heavier does that make the responsibility of the person who writes the rubric?

RouteCause is open source. Repo: github.com/nsega/routecause · Environment: github.com/nsega/inference-lab · Demo: youtu.be/XXl5l0c5GmM

Event & further reading: Claude Build Day · A harness for every task: dynamic workflows in Claude Code (Anthropic), the patterns this build leans on.