Going In-House
01 · Overview

Bringing the Orchestrator In-House

Replace the Claude API at the planner / router / decider slot of the agent harness with a self-hosted, fine-tuned Qwen-3.5-4B orchestrator. SFT distils ~80% of the lift; GRPO closes the rest.

Golden set
3,357

decision points · 94 sessions

1,711 scored in the 18k window

Synthetic corpus
9,488

decision points · 1,232 sessions

LLM-authored (Opus), never templated

Best-model accuracy (v3)
38.4%
+0.9ppvs base

the toggle up top flips this — try it

strict · exact-match

Strict pass rate
38.4%

exact-match vs mostly-unverified gold

Today

Claude API in the primary slot

Every orchestration turn calls an external model.

external dependencyper-turn costdata leavesadded latency
Goal

Self-hosted Qwen-3.5-4B orchestrator

Same harness, same 19 tools & 22 skills — only the brain changes.

data ownershiplower costlocal latencyno external dep
02 · Problem & Goal

Why bring the orchestrator in-house

The orchestrator is the model that plans, routes, and decides on every turn. Today that is the Claude API — which means four standing costs.

Cost

Every orchestration turn is a paid external API call.

Data ownership

User context leaves our infrastructure on every turn.

Latency

A network round-trip sits on the critical path of each decision.

External dependency

Availability and pricing are outside our control.

Success = six promotion gates

1routing accuracy
2argument quality
3end-task completion
4recovery rate
5stop precision
6cost / latency delta
end-task completion is the one that matters most — and the hardest to measure
03 · System Architecture

A surgical swap, not a rebuild

The orchestrator is only the LLM Gateway's primary slot. Everything the harness does around it — tools, skills, sandboxes, memory, billing, auth — stays exactly as it is. That is what de-risks the whole project.

What changes

One layer in the stack. Cutover is a config flip; Claude stays as automatic fallback.

Frontend (Amplify)unchanged
LLM Gateway — primary slotonly this changes
Agent loop (query_engine)unchanged
19 tools · 22 skillsunchanged
Executor + Docker sandboxesunchanged
Memory · billing · auth · S3unchanged
Runtime flows

The paths the orchestrator drives — none of them change with the model swap.

  1. 1Receive user turn → system prompt rebuilt fresh (git status + memory + library + skills)
  2. 2Model plans → emits a tool_call (the primary-slot decision)
  3. 3Executor runs the tool in the sandbox
  4. 4tool_result is appended; loop repeats
  5. 5Model decides to stop → final text to user
Executor ABCLocal / Docker implementations
Sandboxone per-user Docker container, lazy-created
Mount/home bind mount, label-discovered
Dispatchtool loop in query_engine.py
Schemaprovider-neutral tool layer
Persistenceaudit trail + S3 sync
The action space

What the orchestrator must learn to dispatch into: 19 tools and 22 skills (11 enabled).

File & shell
BashRun shell commands; cwd persists.
ReadRead a file with line numbers.
WriteCreate or overwrite a file.
EditExact-string targeted edits.
GlobFind files by glob pattern.
Grepripgrep content search.
Web
WebFetchFetch a URL as markdown, or web-search.
BrowserDrive a sandboxed Chromium session.
Generative
ImageGeneratorText-to-image (Gemini / DALL-E).
VideoGeneratorText/image-to-video with audio (Veo).
AudioGeneratorText-to-speech (ElevenLabs).
DocumentGeneratorStreamed HTML→PDF deliverables.
Orchestration
AgentSpawn an autonomous sub-agent.
skillLoad a named skill's full instructions.
skill_scriptRun an enabled skill's script.
tool_result_loadPage through a cached tool result.
Actions & memory
phone_callOutbound / conference calls (voice AI).
MemoryRewriteCreate / search / update long-term memory.
ShareFileSurface a deliverable file to the user.

04 · Data Engine · Golden Set

The scoreboard — and its honest provenance

Every promotion decision is read off a frozen, checksum-guarded eval set. It is never trained on. But it is worth being precise about what it actually measures.

Sessions
94

48 mined prod + 46 scripted

Turns
470

across all sessions

Decision points
3,357

the unit of scoring

Scored
1,711

fit the 18k window

Replay46 scripted scenarios re-run live on the platform.
ScrapePlaywright captures thinking, route, tool inputs & results turn-by-turn.
Correct201 policy violations fixed across 8 dimensions.
FreezeChecksum-guarded v2.0, zero training leakage.
frozen v2.0 checksum-guarded 0 training leakagecovers all 21 non-MCP toolshard cases over-represented
The 201 correction-policy violations
A judgment-review pass (91 sessions via a Workflow run, 13 more recovered via individual re-review after the first run crashed) found 201 policy violations across 8 dimensions in the mined production traces. All 201were resolved before freezing — every corrected value is grounded in the session's own trace (real file paths, real measured numbers, real call IDs), nothing invented.
DimensionCount
Other81
Untruthful final36
Skill misroute28
AskUser unnecessary23
ShareFile wrong14
AskUser missed9
Memory rule6
ShareFile missed4

Resolved in two passes: a mechanical pass fixed 62 decision points where a dedicated tool (Read/Grep/Glob/Edit/Write) should have been used instead of raw Bash — a full re-audit confirmed 0 remaining unflagged misroutes across all 720 original Bash calls. An authoring pass (61 high-confidence + 110 medium/low-confidence findings, via parallel sub-agent fan-out) wrote grounded gold corrections for the rest — 170 decision points touched total, using omit, rewrite_tool_call, rewrite_final_text and insert_before actions. 2 authoring bugs were caught by post-hoc verification and repaired before freezing.

What was actually wrong, and what the fix looks like
One real, high-confidence example per dimension: what the old (Claude-powered) production system did (observed), vs. what the correction pass changed the label to (gold). Every fix is grounded in that session's own trace — nothing invented.

User asked

hi

Observed (old system did this)

Observed final response to a greeting ('hi') is empty — a bad gold label for a greeting turn.

Gold (corrected to this)

A brief, friendly greeting reply asking what the user needs.

05 · Data Engine · Synthetic Corpus

Training data that was engineered, not scraped

The first model stalled not on size but on data conventions. v2 fixes the label boundaries with an explicit policy, then teaches the hard cases directly.

Sessions
1,232

100 held out

Decision points
9,488

8,559 clean after repair

Policy repairs
49+330

relabels + ShareFile removals

Leakage audit
0 hits

0 discarded

Why the old data capped accuracy

The same “video” intent was labelled three different ways — the model can't learn a boundary that isn't there.

Write
×22
VideoGenerator
×14
skill
×1
0/8MemoryRewrite proactive triggers hit
24/35skill cases misrouted to a direct tool
A sample SFT record

Response-only loss masking: only the model's reasoning and tool call are trained (highlighted). The prompt is masked.

systemmasked
You are the orchestrator. Route each turn to the right tool…
usermasked
Make a 20-second promo video from the footage I uploaded.
thinkingtrained
The user wants video from existing footage → this is the video-from-video skill, not a raw Write. Load the skill first.
tool_calltrained
skill(name="video-from-video")
The v2 fix

A 27-row intent→route policy with tie-breakers first, then four Tier-C teaching waves (+332 sessions).

W1contrastive skill-vs-tool (double as DPO pairs)
W2stop / AskUser discipline
W3MemoryRewrite proactive triggers
W4tool_result_load / Bash-vs-Write
06 · Models & Training

From a stalled Gemma to a QLoRA'd Qwen

Gemma-3-4B plateaued — 5.4% argument exact-match, 24/35 skill cases misrouted. That was a data problem, not a size problem. Qwen-3.5-4B zero-shot already matched a full Gemma fine-tune.

Gemma-3-4B — where it stalled

Trained on 1,000 synthetic sessions / 8,899 dp. 24/35 skill cases misrouted → a data problem, not model size.

Gemma-3-4BQwen-3.5-4Bbase constraint relaxed 2026-07-13; zero-shot ≈ a full Gemma fine-tune
QLoRA recipe — v2 vs v3

Same config file (qlora_qwen35.yaml), edited in place between runs — same frozen v2 corpus, same everything except the 3 highlighted rows. Served via vLLM.

Fieldqwen35-qlora-v2qwen35-qlora-v3
Run tagchangedqwen35-qlora-v2qwen35-qlora-v3
LoRA rank / αchangedr=16 · α=32r=128 · α=256
Dropout00
Target modulesq / k / v / o / gate / up / down_projq / k / v / o / gate / up / down_proj
Quantizationload_in_4bitload_in_4bit
Max sequence16,38416,384
Learning rate2.0e-4 · cosine, warmup 0.12.0e-4 · cosine, warmup 0.1
Effective batch16 (2×8 grad-accum)16 (2×8 grad-accum)
Epochs33
Optimizerchangedpaged_adamw_8bitadamw_8bit
Gradient checkpointingchangedunslothtrue
Precisionbf16bf16
Checkpoint selectionbest by eval_lossbest by eval_loss
Loss / packingresponse-only · packing offresponse-only · packing off
Results — v2 vs v3

Replay eval on the common 1,711-case golden set. Baseline is Qwen-3.5-4B zero-shot (no fine-tune); v2 is r=16/α=32 (labeled “v1” in the eval scoreboard); v3 is r=128/α=256, the first version to beat the zero-shot baseline on true accuracy.

MetricBaselinev2v3
Strict pass rate37.5%36.8%38.4%
True accuracy (hand-judged)~59.4%~58.6%~63.4%
Argument quality55.4%57.3%58.1%
wrong_tool (strict)20.3%26.2%27.6%
format_invalid6.5%2.4%3.1%
premature_stop8%3%5.2%
07 · Evaluation Methodology

Replay eval — a freeze-frame at every decision

We hand the model the exact context of a single recorded decision, grade one next action deterministically, then discard it. It's teacher-forced imitation, which is precisely why the strict numbers need the hand-judging that follows.

1
Freeze a framePick one decision point from the recorded trajectory.
2
Rebuild the contextExact system prompt, real dir/date/files, all prior messages + tool results.
3
Ask for one stepThe model emits a single next action — nothing more.
4
Grade by codeDeterministic scorer compares to the gold action.
5
Advance the reelThe recording moves on, never the model — one mistake can't cascade.
temperature 0 checksum-guardedno judge model in the loop1,711 of 3,357 points fit the 18k window
08 · Results & Scoreboard

The numbers — strict and corrected, side by side

Strict exact-match scoring looks flat at ~37–38%. Hand-judged “true” accuracy tells a different story: v3 is the first version clearly above baseline. Flip the Strict ↔ True toggle to switch every headline below.

Strict pass rate

exact-match
Baseline

Qwen-3.5-4B zero-shot

37.5%
v1

QLoRA (corpus v1)

36.8%-0.7pp
v3
best

QLoRA (corpus v2)

38.4%+0.9pp

Exact-match against a mostly-unverified recording of production — it penalizes valid alternatives. See §10 for how much of the gap is a scoring artifact.

Strict verdict distribution

Share of the common 1,711 scored decision points, by scorer verdict.

Argument quality

Field-level argument correctness, trending up across versions.

Baseline55.4%
v157.3%
v358.1%
Per-tool routing — baseline vs v2 vs v3

The base model already knows the externally-obvious tools; the gap is in the convention-heavy ones. All 20scored tools shown (excludes the “correctly made no call” <omit> case). Band reflects the baseline classification.

Band
AudioGenerator(4 cases)100%25%75%already knows
WebFetch92%75%49%already knows
ShareFile85%92%89%already knows
SetReminder81%86%90%already knows
phone_call80%85%80%already knows
VideoGenerator78%72%69%already knows
ImageGenerator77%73%66%already knows
Browser69%64%55%mixed
Bash63%75%64%mixed
Read58%46%50%mixed
Write50%60%54%mixed
DocumentGenerator50%50%65%mixed
Edit48%83%78%mixed
Grep(3 cases)33%33%33%mixed
AskUser27%26%34%convention gap
MemoryRewrite23%29%27%convention gap
Glob22%33%33%convention gap
skill16%37%30%convention gap
tool_result_load13%21%25%convention gap
Agent(2 cases)0%0%50%convention gap
09 · Deep Analysis · The Corrected Verdict

We didn't trust the scorer — so we re-judged

Strict exact-match penalizes valid alternatives against mostly-unverified gold. On the flipped cases, the scorer's verdict had its sign backwards.

Scorer says
-0.7pp

v1 looks like a regression under strict exact-match.

Judges say
net +25

for v1 on the 342 flipped cases (≈ +1.5pp corrected).

Multi-agent judge panels

8 judges pairwise-rated all 342 flipped cases; 5 judges absolute-rated a 200-case sample per model, extrapolated with 95% CIs.

156
55
131
v1 better156
tie55
baseline better131
Strict vs true accuracy, per model

True (hand-judged) accuracy carries ±4pp sampling error. v3 is the first version clearly above baseline on both.

Baselinestrict 37.5% · true 59.4%
v1strict 36.8% · true 58.6%
v3strict 38.4% · true 63.4%
strict true (hand-judged)
~25% of flagged flips were mis-scored or tiestrue: baseline ~59% → v3 ~63%
10 · wrong_tool Root-Cause Investigation

Why is wrong_tool so high — golden or synthetic data?

The answer is both: the golden set inflates the count (~28% false via unverified gold + exact-match), while a synthetic tool-mix imbalance causes the real errors. Here is the evidence.

Strict wrong_tool (v3)
27.6%

473 cases — the alarming headline.

Corrected estimate
~20%

real routing error after removing false failures.

False failures
27.5%

were valid-alternative routes, not mistakes.

What wrong_tool actually is

120 wrong_tool cases hand-judged: 27.5% good · 48% partial · 24% bad.

28%
49%
23%
Scorer / gold artifact · 28%equivalent_tool + gold_weak — the model's route was fine, it just differed from the (unverified) recording.
Real error · 49%wrong_route + over_action + malformed — a genuine routing mistake.
Mixed · 23%partially defensible; graders split.
Root-cause tags:wrong_route 33%other 23%over_action 16%gold_weak 16%equivalent_tool 12%malformed 2%
Confusion — top substitutions

Gold tool → what v3 used instead (counts).

WriteBash
25
BashShareFile
21
tool_result_loadWebFetch
15
BashWrite
12
WriteShareFile
12
ReadBash
11
ImageGeneratorVideoGenerator
11
MemoryRewriteDocumentGenerator
11
The smoking gun — training mix vs eval need vs prediction

Where the corpus over-taught a tool, v3 over-uses it; where it under-taught (Browser), v3 can't reach for it. Shares are % of tool calls.

Case explorer — real false failures

click to inspect

Four verified cases where v3's route was fine (or better) but differed from the recording. Open one to see the conversation and a gold-vs-v3 side-by-side.

11 · Honesty & Limitations

What the numbers do and don't mean

This section is not optional — it is what makes the ~63% credible. The same model reads very differently depending on how strictly you score it.

The honesty ladder

One model, four framings — from the harshest scorer to the metric we actually care about but can't yet measure.

Strict exact-match~38%
too harsh — exact-match vs unverified gold
Honest per-step~47.5%
~45–50%
Lenient per-step~63%
hand-judged reasonableness
End-to-end vs platformunknown, lower
unmeasured, materially lower (errors compound)
12 · Roadmap & Next Phases

SFT is one gate on a four-phase path

Each phase has an explicit decision gate. We are at phase 2 — data + SFT. Next is reinforcement.

Phase 1

Pipeline rig

Gate: e2e run
Phase 2we are here

Data + SFT

Gate: format + routing
Phase 3

GRPO + DAgger

Gate: recovery + completion lift
Phase 4

Shadow → Cutover

Gate: parity at lower cost
RL stage — GRPO

SFT teaches “imitate this one demonstrated action.” It can't teach “of these several plausible actions, which is actually best” — there's no comparative signal in plain behavioral cloning.

  1. 1For a given state, generate G ≈ 8–16 completions from the current policy — not just one.
  2. 2Score each completion: verifiable parts as code (right tool? valid schema? did it contradict an observed tool result? clean termination?) per the “never judge the verifiable” rule; unverifiable parts (tone, reasonable judgment calls) go to the Nemotron Super 120B judge — pairwise/rubric scoring across the G completions, not a 1–10 rating.
  3. 3Push the policy toward the higher-scored completions relative to its peers in the same group — no separate reward/value model needed, which is what makes GRPO cheaper than classic PPO.
Advantage
 = (r − mean) / std → clipped PG + KL to SFT ref
Dataset
prompts only (state half of an SFT record)

DAgger loop: model drives in the sandbox → hits a derailed state → Claude re-labels → add to data → retrain. Fixes distribution shift, not just disagreement (full breakdown in §13).

DPO — the lighter fallback

Instead of a full live-harness RL loop, take the contrastive pair_id examples already authored (one skill-load side, one direct-tool side, near-identical prompt) as ready-made chosen/rejected pairs and run a cheaper preference-tuning pass on just that boundary.

Natural bridge, not throwaway work: the contrastive skill-vs-tool pairs are exactly DPO training data, and the DAgger loop keeps feeding this same boundary fix as it relabels derailed states.

What the GRPO stage needs
  • Rollout accessthe thin execution slice (agent loop + Docker sandbox + the 19 tools + skills) so the policy can actually generate G completions per state.
  • Self-hosted judge computea Nemotron-family judge via vLLM — the full Nemotron Super 120B (bipardai02, 4× H100 NVL) is Step 2's judge; Step 1 runs a judge scaled to the 4B pipeline.
  • Reward / rollout harnesstrace-comparison rewards — the honest end-to-end metric, not just per-step imitation.
  • DAgger relabeling loopClaude access to re-label derailed states as the policy drives — feeds both the SFT store and the DPO boundary-fix pairs.
Reward hygiene
  • Never judge the verifiable — code checks what code can.
  • Rubric, not vibes — discrete per-dimension rubrics.
  • Pairwise > absolute across the G completions.
  • Self-hosted Nemotron — budget + decorrelated from Claude.
Two-step plan — prove it cheap, then reuse it
Step 1 — prove it cheap (current)current
4B base (Qwen-3.5-4B)
  • QLoRA SFT
  • DAgger in shadow mode
  • Trace-comparison rewards
Step 2 — reuse it at the real targetfuture
Gemma-4 26B-A4B primary (Nemotron-family mid as fallback)
  • GRPO against the live harness
  • Nemotron Super 120B as RL judge — self-hosted via vLLM on bipardai02, 4× H100 NVL

What you learn getting DAgger/GRPO working on Qwen-3.5-4B now is meant to transfer directly, not be thrown away.

13 · What We Need to Build Next

A thin execution slice — not the whole platform

~80% of production is irrelevant to RL rollouts. We need the part the model actually drives, plus a reward harness that finally gives us an honest end-to-end number.

Reuse
  • Agent loop (query_engine.py)
  • Executor + Docker sandbox (executor_docker, container_manager)
  • The 19 tools + sandbox image
  • 22 skills
  • Local tools (Bash/Read/Write/Edit/Glob/Grep/…) run as-is
Build
  • Rollout / reward harness = the honest end-to-end metric
  • SFT round-2 tool-mix rebalance
  • Acceptable-alternatives set in the scorer
  • Calibrate Nemotron before it drives reward
Ignore (~80% of prod)
  • Amplify frontend
  • LLM Gateway
  • Postgres
  • S3 sync
  • Port-preview
  • Billing & auth
  • Multi-EC2
DAgger — fixing distribution shift

SFT only ever learned from Claude's trajectories — states Claude visited, decisions Claude made. Once the fine-tuned model runs live, its own small mistakes push it into states Claude never actually visited during data collection: a wrong tool call → a weird tool_result → now the model is in a derailed state nobody ever demonstrated how to recover from. Errors compound because there is zero training signal for "how do I get back on track from here." The classic imitation-learning failure mode.

  1. 1Let the learner (the fine-tuned model) drive a real turn inside the isolated sandbox — not Claude.
  2. 2Watch until it hits a derailed / bad state.
  3. 3Go back to Claude and ask: given this exact state, what's the correct next action? Claude re-labels it.
  4. 4Add that (state → Claude's correct action) pair into the training store.
  5. 5Retrain.

Repeat. Each round, the training data grows to cover the states the model's own policy actually wanders into — not just the states Claude wandered into.

no whole-system dependencyruns in existing per-user containersreward harness = honest e2e metricDAgger closes the recovery gap
14 · Appendix / Methods

Glossary & data lineage

Definitions for the terms used throughout, and the frozen sources every number traces back to.

Glossary

One assistant action to be scored — a single tool_call or a stop. The unit of both training records and eval.

Sources & lineage
ArtifactPath
Corrected eval verdictpre_eval/out/CORRECTED_EVAL_VERDICT.md
Strict reports (baseline / v1 / v3)pre_eval/out/{pre_qwen35_baseline,post_sft_qwen35,post_sft_qwen35_v3}/
Judge verdicts (v1 flips + absolute)pre_eval/out/{judging,abs_judge,v3_judge}/
QLoRA configfine-tuning-scripts_run/configs/qlora_qwen35.yaml
Frozen corpussynthetic_data/training/sft_dataset_v2.jsonl
Canonical narrative deckIn-House_Orchestrator_Training_Plan_v6.pptx

All figures frozen & checksum-guarded · SFT stage · Qwen-3.5-4B · agreement-with-production, hand-verified where it matters.