Bringing the Orchestrator In-House
Replace the Claude API at the planner / router / decider slot of the agent harness with a self-hosted, fine-tuned Qwen-3.5-4B orchestrator. SFT distils ~80% of the lift; GRPO closes the rest.
Why bring the orchestrator in-house
The orchestrator is the model that plans, routes, and decides on every turn. Today that is the Claude API — which means four standing costs.
Success = six promotion gates
A surgical swap, not a rebuild
The orchestrator is only the LLM Gateway's primary slot. Everything the harness does around it — tools, skills, sandboxes, memory, billing, auth — stays exactly as it is. That is what de-risks the whole project.
The scoreboard — and its honest provenance
Every promotion decision is read off a frozen, checksum-guarded eval set. It is never trained on. But it is worth being precise about what it actually measures.
Training data that was engineered, not scraped
The first model stalled not on size but on data conventions. v2 fixes the label boundaries with an explicit policy, then teaches the hard cases directly.
From a stalled Gemma to a QLoRA'd Qwen
Gemma-3-4B plateaued — 5.4% argument exact-match, 24/35 skill cases misrouted. That was a data problem, not a size problem. Qwen-3.5-4B zero-shot already matched a full Gemma fine-tune.
Replay eval — a freeze-frame at every decision
We hand the model the exact context of a single recorded decision, grade one next action deterministically, then discard it. It's teacher-forced imitation, which is precisely why the strict numbers need the hand-judging that follows.
The numbers — strict and corrected, side by side
Strict exact-match scoring looks flat at ~37–38%. Hand-judged “true” accuracy tells a different story: v3 is the first version clearly above baseline. Flip the Strict ↔ True toggle to switch every headline below.
Strict pass rate
exact-matchExact-match against a mostly-unverified recording of production — it penalizes valid alternatives. See §10 for how much of the gap is a scoring artifact.
We didn't trust the scorer — so we re-judged
Strict exact-match penalizes valid alternatives against mostly-unverified gold. On the flipped cases, the scorer's verdict had its sign backwards.
Why is wrong_tool so high — golden or synthetic data?
The answer is both: the golden set inflates the count (~28% false via unverified gold + exact-match), while a synthetic tool-mix imbalance causes the real errors. Here is the evidence.
Case explorer — real false failures
click to inspectFour verified cases where v3's route was fine (or better) but differed from the recording. Open one to see the conversation and a gold-vs-v3 side-by-side.
What the numbers do and don't mean
This section is not optional — it is what makes the ~63% credible. The same model reads very differently depending on how strictly you score it.
- ~63% is per-step and lenient; errors compound, so real end-to-end vs the platform is much lower and unmeasured.
- The eval is imitation-of-production, not correctness — the gold is mostly unverified.
- Judge estimates carry ±3–4pp sampling error.
- Relative ranking: v3 > v1 ≈ baseline.
- format_invalid and premature_stop gains are code-verifiable.
- The wrong_tool root cause (tool-mix imbalance) holds regardless of scorer.
SFT is one gate on a four-phase path
Each phase has an explicit decision gate. We are at phase 2 — data + SFT. Next is reinforcement.
GRPO/DPO polish a policy that's already decent — they cannot manufacture competence out of a policy that's still routing badly.
- Skill-vs-tool labels are consistent, and the skill→tool_call confusion cell is shrinking (target: <10 of 35, from 24).
- Golden set is frozen and leak-free — RL needs a leak-free scoreboard.
- In-distribution routing is respectable: action-type ~70%+, tool-name meaningfully above 27%.
- Train/serve parity is re-confirmed against the live gateway.
Post-v3 SFT is better (a real +1.5pp gain) but still noisy enough — over-acting, Bash regressions, ~38% strict pass — that starting RL right now would reinforce inconsistency faster, not fix it.
A thin execution slice — not the whole platform
~80% of production is irrelevant to RL rollouts. We need the part the model actually drives, plus a reward harness that finally gives us an honest end-to-end number.
Glossary & data lineage
Definitions for the terms used throughout, and the frozen sources every number traces back to.
All figures frozen & checksum-guarded · SFT stage · Qwen-3.5-4B · agreement-with-production, hand-verified where it matters.