Generalised Plan
Canonical plan · v6

Training an In-House Orchestrator

Replacing the Claude API at the centre of our agent harness with a self-hosted, fine-tuned orchestrator — the model that plans, routes, and decides across every turn.

20 sections
Today: Claude APIorchestrates every step
Goal: Fine-tuned orchestratorself-hosted on bipardai02
Distil from Claudereinforce against the live harnessjudge with Nemotron Super
01 · Progress update · Recap

The plan so far — and where we stand

This deck lays out the plan to bring the orchestrator in-house. Here is that plan in brief — and the ground covered since.

The plan, in brief

  • Distil Claude's orchestration with SFT (~80% of the lift), then reinforce with GRPO — Nemotron Super is the self-hosted RL judge.
  • One trace store — mined production + synthetic — feeds SFT, GRPO and DPO alike.
  • The Golden set is the frozen, leak-free scoreboard every promotion decision reads off.
  • Cutover is a gateway config flip, with Claude kept as the automatic fallback.

Where we are — Phase 2: data + SFT

  • Root cause found — a data-conventions problem, not model size.
  • Base model switched: Gemma-3-4B → Qwen-3.5-4B.
  • Golden eval set built & frozen (v2.0 — 94 sessions, 3,357 decision points).
  • Qwen zero-shot baseline on Golden: 37.5% overall pass.
  • Old synthetic data repaired + new Tier-C waves authored.
  • Next: QLoRA Qwen-3.5-4B on the v2 corpus, then re-score against the baseline.
02 · Progress · Base model

From Gemma-3-4B to Qwen-3.5-4B

The first fine-tune plateaued at low routing accuracy. Diagnosis pointed at the data, not the model — but we also moved to a stronger, better-fit base.

Gemma-3-4B: where it stalled

  • Action-type accuracy — 48%
  • Tool-name match — 27% · argument exact-match — 5.4%
  • Stop precision — 30%
  • Fine-tuned on 1,000 synthetic sessions / 8,899 decision points — and still collapsed on arguments.

The failure signature

  • 24 of 35 skill cases were routed to a direct tool call instead.
  • Systematic bias plus a total argument-match collapse.
  • Points at a labeling / convention problem — not a capacity limit.

The switch: Qwen-3.5-4B

  • Base constraint relaxed 2026-07-13 (Chinese-origin models allowed).
  • Size floor ~7–14B; a 4B base proves the pipeline cheaply first.
  • Served via vLLM in the text tool-format, matching the training stack.
03 · Progress · Eval data

The Golden set — scraped from the live platform

Our single frozen source of truth for every promotion decision: 94 sessions, 470 turns, 3,357 decision points — never trained on.

Two provenances

  • 48 sessions mined from real production traffic.
  • 46 scripted business scenarios, run live on myemployee.ai and scraped turn-by-turn — thinking, route, tool inputs, results.

What it covers

  • All 21 non-MCP tools present — Bash, Browser, Write, WebFetch, ShareFile, ImageGenerator … down to Glob / Grep / Agent.
  • Skills & system parts: docx, pdf-reading, web-search, voice-calling, video-from-video, reminders, platform-help.
  • MCP tools deliberately deferred (need live accounts).

Gold-label correction pipeline

  • Source was observed behavior — including the old system's own mistakes.
  • 201 policy violations found across 8 dimensions (misroutes, untruthful finals, needless AskUser, ShareFile & memory errors).
  • All 201 corrected before the freeze.

Deliberate choices

  • Every correction grounded in the session's own trace — nothing invented.
  • Hard cases over-represented: recovery, ambiguous routing, stop/continue.
  • De-duplicated 104 → 94; frozen & checksum-guarded; zero leakage.
04 · Pre-eval · Method

How we measured it — replay-based eval

Each golden session is replayed like a film reel: the model is quizzed at every decision the original assistant faced, then scored by deterministic code.

1

Freeze-frame

  • Pause before every tool call and every turn's final reply.
  • Each pause is one scored case — 3,357 across the 94 sessions.
2

Hand over context

  • The model gets the conversation exactly as it stood: rebuilt system prompt (real dir, date, files), all prior messages, every earlier tool call and its recorded result.
  • Input ends at the model's turn — no 'what next?' hint.
3

Grade & discard

  • Its one next action is graded against the verified gold action by code — then thrown away.
  • The recording, never the model, advances the story — so one early mistake can't cascade.
05 · Pre-eval · Results

Zero-shot baseline: where 1,711 decisions landed

Untrained Qwen-3.5-4B on the frozen Golden set — 37.5% overall pass (strict 32.6%). 1,711 of 3,357 decision points fit the 18k window and were scored.

Argument quality55.4%args right when the tool was right
Recovery rate43.8%right move after an error (80 cases)
Stop precision62.2%stopped & answered when done
Format validity93.5%6.5% malformed tool-call JSON
The 1,711 decisions, by verdict
pass37.5%
right tool · wrong args20.6%
wrong tool20.3%
premature stop8%
failed to stop7%
malformed6.5%
known-bad repeated0.2%
06 · Pre-eval · The headline finding

Generic tools work; platform conventions don't

The baseline splits cleanly: the model already knows universal tools, but fails the platform's own conventions — precisely what fine-tuning must teach.

Already knows (≥75% routing)

  • WebFetch 92% · ShareFile 85% — universal, intuitive
  • SetReminder 81% · phone_call 80%
  • VideoGenerator 78% · ImageGenerator 77%
  • Omit-compliance 82% — avoids most known-bad calls

Platform conventions it can't know yet (≤27%)

  • tool_result_load 13% — never retrieves cached output
  • skill_script 14% / skill 16% — doesn't load skills first
  • Glob 22% — shells out instead of dedicated search
  • MemoryRewrite 23% — misses the memory rules
  • AskUser 27% — free-texts instead of the structured tool
07 · Pre-eval → Post-eval

The scoreboard: baseline vs post-SFT

Same frozen cases, same deterministic scoring, same serving config — the report diff is the training effect, isolated. The post-SFT column is a placeholder until the v2 QLoRA run lands.

Measured — zero-shot baseline

  • Overall pass — 37.5% (strict 32.6%)
  • Argument quality — 55.4%
  • Recovery 43.8% · stop precision 62.2%
  • Format validity — 93.5% (6.5% malformed JSON)
  • 1,711 of 3,357 decision points fit the 18k window

Post-SFT — targets (placeholder)

  • Overall pass — TBD (run pending)
  • Argument quality ≥ 85%
  • Stop precision ≥ 90% · malformed ≈ 0%
  • Platform-convention tools ≥ 80% routing
  • No regression on WebFetch 92% / ShareFile 85%

Where the lift must come from

  • Two-thirds of errors are conventions: wrong tool 20.3% + wrong args 20.6%.
  • tool_result_load 13% · skill_script 14% · skill 16%
  • Glob 22% · MemoryRewrite 23% · AskUser 27%
  • Exactly the boundary the v2 corpus attacks.
08 · Progress · Synthetic data (before)

Why the old synthetic data capped accuracy

The Tier A + B corpus (1,000 sessions / 8,899 decision points) was structurally valid — but taught the hardest decision inconsistently.

Inconsistent skill-vs-tool boundary

Root cause — same intent, contradictory labels:

  • 'video' → Write ×22 / VideoGenerator ×14 / skill ×1
  • 'logo' → ImageGenerator ×19 / a video skill ×4
  • 'pdf' split four different ways.

Whole behaviors never captured

  • MemoryRewrite proactive triggers: 0 / 8 in eval.
  • Stop / chat discipline under-taught — 30% stop precision.
  • The 'AskUser at hi' reflex: asking when it should just answer.

Content, not just routing

  • Where the tool was right, arguments were often wrong.
  • Structure-match ran ~2× exact-match.
  • The model learned the tool interfaces — not the content that fills them.
09 · Progress · Synthetic data (v2)

Repaired Tier A/B + a new Tier-C wave

One written routing policy, the old corpus repaired against it, then new boundary-focused data — all serialized for the Qwen stack.

Tier A/B — repaired

  • Routing policy written down first: a 27-row intent→route table with tie-breakers.
  • 49 targeted relabels + 330 mechanical ShareFile removals; 0 records discarded.
  • Rebuilt to 8,559 clean decision points, all invariants green.

Tier C — new waves

  • W1 — contrastive skill-vs-tool pairs (double as future DPO pairs).
  • W2 — stop / chat + AskUser discipline.
  • W3 — MemoryRewrite proactive triggers (mined from prod).
  • W4 — tool_result_load, Bash-vs-Write, weak-tool top-ups.
  • +332 sessions → frozen v2: 1,232 sessions / 9,488 dp (100 held out).

Why it moves the needle

  • Labels now consistent → attacks the 24/35 confusion directly.
  • Authored results are LLM-written but plausible; train/serve prompt is byte-identical (no skew).
  • A held-out synthetic slice separates a convention gap from a capacity limit.
10 · Progress · SFT data anatomy

What one SFT datapoint looks like

One JSONL record = one full multi-turn session. 1,232 records carry 9,488 assistant decision points — the model is trained only on the assistant's spans.

One record = one session

  • {id, source, messages, meta}
  • system: the Orvis orchestrator prompt — tool rules, ShareFile & memory conventions.
  • Then alternating user / assistant / tool turns, several user turns deep.
  • Long-prefix sessions reach ~12k tokens (16k training window).

Inside an assistant turn

  • <thinking>why this route…</thinking> — LLM-authored reasoning, never templated.
  • <tool_call>{"name": …, "arguments": …}</tool_call> — Qwen-native key.
  • Tool output returns as a bare <tool_response>…</tool_response> turn.
  • Or a final user-facing reply when done — that's the stop-discipline signal.

Loss masking — the crucial bit

  • Loss only on assistant spans: thinking + tool calls + final replies.
  • User text and tool observations masked to −100 — the model never learns to hallucinate observations.
  • Every <tool_call> validated against the 19-tool / 22-skill catalog before training.
  • Serialized prompt is byte-identical to serving (no train/serve skew).
11 · Progress · Data schemas

Two datasets, two schemas — and why

The same session data underneath, but a training target and an eval spec do opposite jobs — so their shapes differ.

Synthetic SFT record — a training target

{
  "id": "…", "source": "synthetic",
  "messages": [
    { role: system     loss: false }  # rendered prompt
    { role: user       loss: false }
    { role: assistant  loss: TRUE  }  # <thinking> +
                       #   <tool_call>{name,input}</…>
    { role: tool       loss: false }  # authored result
    { role: assistant  loss: TRUE  }  # final answer
  ],
  "meta": { category, num_turns,
            n_tool_calls, n_assistant_spans }
}

loss:true marks the trained spans · prompt byte-identical to serving · results LLM-authored

Golden record — an answer key

{
  "schema_version": "2.0", "session_id": "…",
  "source": "raw_prod | golden_scrape",
  "turns": [ {
    "user_message": "…",
    "trace": { thinking, text,
               route_sequence[], tool_calls[] },
    "decision_points": [ {
      dp_id, seq, kind,
      "observed": { tool, input, tool_use_id },
      "gold":     { tool, input, verified }  # answer key
      "scoring":  { score, reason }
    } ] } ]
}

no loss masks · scored per decision_point vs gold · observed is real, then corrected · frozen

12 · Next · SFT round 2

Post-eval → aimed fire: error-driven SFT round 2

The post-eval report is a failure map, not training data. Every failing verdict names a tool, a convention and an argument schema — round 2 authors fresh data at exactly those coordinates.

1

Read the failure map

  • Cluster post-eval misses by verdict × tool.
  • Wrong-args clusters name the exact schema field (path vs file_path).
  • Stop failures split into premature-stop vs failed-to-stop session shapes.
  • Golden itself stays frozen — statistics out, never content.
2

Author the targeted wave

  • Fresh synthetic sessions per cluster — the Tier-C playbook: contrastive pairs, proactive triggers, stop discipline.
  • Mixed with a replay slice of the v2 corpus so already-strong tools don't regress.
  • Same leakage audit (0.6 similarity threshold) gates the wave before anything trains.
3

Why the boost is cheap

  • Two-thirds of baseline errors are convention failures, not reasoning — and conventions are what SFT learns fastest.
  • Re-score on the identical frozen pipeline: the diff is fully attributable.
  • Clusters that stay low on the held-out synthetic slice too are capacity limits — those go to GRPO, not more SFT.
13 · RL · GRPO mechanics

Inside a GRPO step — how it actually learns

GRPO needs no critic model: it samples a group of answers to the same state, scores them, and pushes the policy toward the better-than-average ones.

1

Take a state

  • A prompt = messages up to one decision point — the state half of an SFT record.
2

Sample a group

  • Draw G completions (G ≈ 8–16) from the current policy at temp > 0 — diverse candidate next-actions.
3

Score each

  • Reward rᵢ per completion: code for verifiable dims + Nemotron pairwise for the unverifiable.
4

Group-relative advantage

  • Âᵢ = (rᵢ − mean) / std. The group's own mean is the baseline — no value/critic network.
5

Update the policy

  • Clipped policy-gradient toward high-Âᵢ, with a KL penalty back to the frozen SFT reference.

The key trick — Âᵢ = (rᵢ − mean) / std

  • No critic to train — the group is its own baseline (unlike PPO).
  • KL to the SFT reference keeps it from drifting or reward-hacking.
  • Dataset = prompts only — no stored target action.
14 · Stage 2 · GRPO

The dataset is trivial — the reward is the work

Sample G completions per orchestration state (G ≈ 8–16), score each, optimise toward the relatively better ones. Dataset = prompts only.

Verify in code (free, deterministic)

  • Schema validity — dispatch parses & matches signature
  • Correct subagent / tool selected for the intent
  • Execution success — sandbox runs without throwing
  • Format & clean termination — no runaway loops
  • Efficiency — fewer steps / sub-calls for same outcome

Judge with Nemotron (unverifiable only)

  • Was the decomposition sensible & complete?
  • Was this the right subagent for the subtask?
  • Was stopping here correct vs continuing?
  • Terminal task success vs ground truth
15 · Dataset structures · RL

What the RL stages actually consume

Unlike SFT, neither RL record stores a target action. GRPO scores live completions; DPO compares two stored ones.

GRPO — prompts only

{
  prompt: [ messages up to the
            decision point ],
  meta: {
    task_id, available_tools,
    ground_truth  // for reward
  }
}

DPO — preference pairs

{
  prompt:   [ ... ],
  chosen:   { reasoning, action },
  rejected: { reasoning, action }
}

chosen = Claude's action or top-scored sample. rejected = a pre-RL mistake or lowest-scored sample. Built offline.

16 · The reward judge

Nemotron Super as the self-hosted RL judge

GRPO runs G rollouts × many steps × many prompts — judge volume is enormous. On the Claude API that's a budget bonfire; on Nemotron Super (bipardai02, already deployed via vLLM) it's compute we own. Bonus: it decorrelates from Claude, which wrote the SFT data.

Never judge the verifiable

  • If a check can be code, make it code. Letting the judge score what a parser could adds cost and noise for nothing.

Rubric, not vibes

  • Give discrete scores per named dimension, not 'rate 1–10'. Vague judges get reward-hacked within a few hundred steps.

Pairwise > absolute

  • Score which of the G completions routed better. Relative scoring is more stable and matches what GRPO normalises anyway.
17 · Closing the loop

Train on the model's own mistake states

The single biggest cause of agent failure: the model errs at step 3, lands in a state no expert trace ever visited, and has no idea what to do.

Shadow mode is not enough

  • If the teacher always executes the real action, the trajectory only ever follows the expert path. The model's wrong step never propagates, so you never see the derailed state. That is disagreement mining — not distribution-shift correction.

Let the learner drive — in the sandbox

  • Periodically execute the model's own action so the trajectory enters states it actually reaches, then have Claude re-label the right move FROM that derailed state. Safe because rollouts run in our existing per-user isolated containers — never production.
1Model drives in sandbox
2Hits a derailed state
3Claude re-labels best action
4Add to dataset
5Retrain
18 · Evaluation

Define success before we spend GPU time

Every promotion decision — keep training, swap the base, ship — is gated on these. Without them, 'is it good enough?' has no answer.

Routing accuracy

  • Does the model dispatch to the same subagent Claude did?

Argument quality

  • Are dispatch arguments correct, not just the target name?

End-task completion

  • Does the full trajectory finish the task — the metric that matters most.

Recovery rate

  • After a wrong step, does it get back on track? Tests the DAgger loop directly.

Stop precision

  • Terminates at the right moment vs looping or quitting early.

Cost & latency delta

  • Measured off the credits ledger — savings vs the Claude baseline, already metered per call.
19 · Cutover · De-risking

The swap is a config change, not a rebuild

The LLM Gateway already routes each logical slot to a concrete provider + model via DB config, with a fallback chain and per-vendor adapters. The orchestrator becomes one more provider behind that same interface.

01

Register a provider

  • Wrap the self-hosted model on bipardai02 in an adapter that translates our canonical tool schema to its tool-calling format.
02

Repoint the primary slot

  • Flip the primary slot from Claude to the new provider in DB config — no redeploy, editable by admins.
03

Fallback stays armed

  • During shadow & cutover the chain auto-reverts to Claude on error, rate-limit, or context overflow. Rollback is instant.
20 · Roadmap

Phased delivery with a decision gate each stage

Phase 1

Pipeline rig

Step 1 — build the model/dataset-agnostic SFT+RL+eval loop; validate on 4B + QLoRA + a toy task.

Gate: End-to-end run, metrics move
Phase 2

Data + SFT

Step 2 begins — build the data engine, mine the event log, fine-tune the capable base on real traces.

Gate: Format + routing accuracy
Phase 3

GRPO + DAgger

Reinforce against the harness; add sandbox rollouts & re-labelling for recovery.

Gate: Recovery + completion lift
Phase 4

Shadow & cutover

Flip the gateway primary slot beside Claude on live traffic, then cut over with automatic rollback.

Gate: Parity at lower cost