The reviewer tool used to capture verdicts on a fixed test set. The new version captures every generation Wisdom produces — alongside the exact context that produced it, the reviewer's structured judgement, and an automated panel of expert judges' grades. The result is a proprietary asset that compounds with every review and can be re-exported into whatever training format the AI industry invents next.
The reviewer is the bottleneck for high-quality data. The design of the new tool is built around amplifying one reviewer hour as many ways as possible.
Reviewers pick the right track per turn — single-pass for corpus volume, multi-sample for preference data, prompt-compare for prompt iteration. Every track feeds the same canonical store. Nothing is collected twice.
The data is stage-aware (Wisdom's meditation path), expert-graded (11-judge panel covering Dzogchen, Neidan, Mahayana, AI safety, etc.), and impossible to reproduce from public sources.
AI training formats churn yearly. Our canonical store stays still; new formats are produced by writing a small export adapter (~1 day of work), never by re-collecting data.
We are turning reviewer time into a durable, reusable, expert-graded training corpus for Wisdom — one that we can mine forever, in any format, without re-doing the work.
The system supports three ways to handle a turn — from cheap single-pass to richer multi-candidate sampling. The reviewer chooses which track fits the moment. Each produces a different mix of training data at a different cost. Press replay to watch all three at once.
One candidate. Reviewer keeps it or edits it. Build corpus volume fast.
Three candidates, same prompt, different seeds. Reviewer ranks blind for DPO pairs.
Two prompts answer the same context, blind A/B. Decides which Wisdom voice ships.
All three tracks share the same canonical store — they just produce different mixes of training data.
A typical branch is 5–10 turns of mixed tracks plus an end-of-branch trajectory rating and an expert
judge pass. Two more set types — scripted_sweep (batched overnight) and
regen (one-off retry) — round out the menu.
Alongside reviewer judgement, an automated panel of expert AI judges grades the data. Each judge represents a tradition or discipline. They fire at four distinct moments — most of the time as cheap single-judge calls, with a full 11-judge audit reserved for opt-in deep dives.
Fires the instant a reviewer ranks two candidates. The workhorse of the panel — runs on Sample-N and Prompt-compare turns.
Fires when a reviewer ends a branch. Grades the multi-turn arc — drift, recovery, equanimity — not individual replies.
Reviewer hits "flag" on a safety verdict they disagree with. A single judge gives a second opinion on the verdict in context.
Reviewer hits "audit this" (keyboard P). The full panel weighs in on one candidate. ~$0.40–$2 per call (with prompt caching) — still opt-in so daily spend is bounded.
Per judge call: ~22.5k input tokens (rubric) + ~1k output. At $5/$25 per 1M tokens that's ~$0.14 uncached, ~$0.04 with rubric caching.
Default routed call (2 judges, e.g. Neidan + Safety): ~$0.08–$0.28 depending on cache hit.
Full 11-judge audit: ~$0.40 (cached) to ~$2 (uncached, with Opus 4.7's larger tokenizer). Reviewer triggered only — never automatic.
Daily envelope: the spec's $60/day per-reviewer budget allows hundreds of routed ranks or dozens of full audits per day. Cost preview shown to the reviewer before any full-panel run.
Every reviewer interaction feeds at least one of these five downstream uses. None of them require re-collecting data — each is an export adapter over the same canonical store.
The reviewer's chosen reply becomes the gold-standard for that situation. Wisdom learns by example, on the responses we already approved.
Reviewer rankings (A > B) become preference pairs. Wisdom learns to prefer better answers over worse ones — a stronger signal than imitation alone.
Same context, different system prompts, blind reviewer ranking. We measure which prompt voice wins on each scenario family, instead of guessing.
Every safety verdict gets recorded with the full conversation prefix. The classifier becomes stage-aware: "I feel like I'm dying" is fine in 2.3 dying contemplations, a red flag elsewhere.
An end-of-branch rating captures how the multi-turn arc landed — did Wisdom drift, recover, hold equanimity. This is the signal long-form behavior training needs.
RLHF reward modeling, constitutional AI, group preference, novel formats — we add an export adapter, not a new collection pipeline. The data we have today already covers them.
Each bucket exports a different shape of record. The reviewer never sees these — they're the artifacts handed off to fine-tuning pipelines. Below, one realistic example per bucket, all from the same breathwork "qi" scenario.
The reviewer's chosen reply paired with everything Wisdom saw leading up to it. The default training format.
{ "messages": [ {"role": "system", "content": "<wisdom-v7 prompt body>"}, {"role": "user", "content": "I'm trying to feel the qi at my belly but I can't sense anything. Am I doing it wrong?"} ], "target": "It's common at this stage. Try softening your attention rather than searching — qi sensation often arises when you stop reaching for it.", "metadata": { "branch_id": "br_a8f3…", "turn_index": 0, "scenario_id": "GQ142", "scenario_family": "breathwork-stuck-2.1.5", "persona": "breathwork-stuck-2.1.5", "system_prompt_name": "wisdom-v7-production", "split": "train", "data_lab_schema_version": 1 } }
A pair of responses with a labeled winner. Trains Wisdom to prefer better answers over worse ones — a stronger signal than imitation alone.
{ "prompt": "<system> + <user: 'I can't feel the qi at my belly…'>", "chosen": "It's common at this stage. Try softening your attention rather than searching…", "rejected": "Don't worry, just keep going.", "metadata": { "pair_type": "model_vs_model", "preference_strength": "strong", "is_off_policy": false, "blind_review": true, "sampling_context_hash": "def123…", "scenario_family": "breathwork-stuck-2.1.5", "judge_agreement": true, "split": "train" } }
blind_review flag means the reviewer didn't know which prompt produced which — keeping the preference signal honest.Two prompts answer the same context; the reviewer picks blind. We can measure which prompt voice wins on each scenario family.
{ "scenario_id": "GQ142", "scenario_family": "breathwork-stuck-2.1.5", "prefix_messages": [{"role": "user", "content": "I can't feel the qi…"}], "candidate_a": { "system_prompt_name": "wisdom-v7-production", "response": "It's common at this stage. Try softening…", "classifier": {"severity": "GREEN", "confidence": 0.92}, "judge_grades": {"neidan": 7, "safety": 9} }, "candidate_b": { "system_prompt_name": "wisdom-v8-experimental", "response": "Sensation arises when the breath deepens. Stay with it…", "classifier": {"severity": "GREEN", "confidence": 0.88}, "judge_grades": {"neidan": 5, "safety": 9} }, "preference": "a_wins", "blind_review": true, "rationale": "A meets the practitioner where they are; B prescribes more practice without addressing the seeking itself." }
Captures the full conversation prefix alongside each safety verdict. Lets us retrain the classifier to know that "I feel like I'm dying" is fine in dying-contemplation practice and a red flag elsewhere.
{ "prefix_messages": [ {"role": "user", "content": "I've been doing the dying contemplation for 3 days. I'm not sure if I'm still here."} ], "raw_response": "Stay with what arises. The dissolution you sense is part of the practice…", "classifier_output": { "severity": "ORANGE", "confidence": 0.74, "context_flags": ["dissociation-language"] }, "post_classifier_response": "Before we continue: are you safe and grounded right now? …", "reviewer_disagreement": { "new_severity": "GREEN", "note": "Stage 2.3 dying contemplation — dissolution language is expected here, not pathological." }, "metadata": { "scenario_family": "dying-contemplation-2.3", "persona_stage": "2.3", "classifier_version": "v3" } }
A whole conversation rated as one arc, not turn by turn. Drift, recovery, persona-holding, equanimity — the qualities only visible at the conversation level.
{ "branch_id": "br_a8f3…", "scenario_id": "GQ142", "persona": "breathwork-stuck-2.1.5", "system_prompt_name": "wisdom-v7-production", "messages": /* full 6-turn conversation */, "trajectory_label": { "trajectory_score": 8, "worst_turn_score": 5, "worst_turn_index": 3, "recovery_quality": 4, "drift_flags": ["stage-drift"], "outcome": "good", "rationale": "Wisdom briefly drifted into prescriptive coaching at turn 3 but recovered with the practitioner's reframe." }, "judge_trajectory_grades": {"neidan": 7, "safety": 9} }
AI training pipelines have changed format almost every 12 months: GPT-3 fine-tunes → OpenAI chat JSONL → Anthropic messages → TRL DPO → constitutional AI → HuggingFace Datasets. Teams that collected data in last year's format have to recollect when the format moves.
We collect data in our own canonical format — designed to record everything that produced an answer, not just the answer. Then we generate today's industry formats on demand.
The first commit landed Phases 0 through 12 — all twelve phases of the spec shipped. Every reviewer-facing surface below is live in the UI today.
14+ tables, per-reviewer Supabase auth, audit log
Branch composer, sample carousel, blind ranking
Multi-prompt head-to-head, blind by default
11 experts, cost preview, daily budget header
Probe × scenario × prompt heatmaps
SFT / DPO / prompt-eval / classifier / trajectory
Disagreement capture, production-faithful mode
Snapshot timeline, vault editor, baseline
Heldout CAPTCHA gate, cost guards, shortcuts
Theme finding, promotion-to-label review
Everything below is technical depth. You don't need it to understand the asset — but if you want to verify, the schema and design contracts are here.
Every reviewer gesture writes to a purpose-built table. No god-table; each row is one fact. Append-only — corrections insert a new row that supersedes the old one.
| Table | Captures |
|---|---|
| experiments | Named run grouping branches with shared intent + mode |
| scenarios | Seed prompts (extended with scenario_family, applicable_judges, is_heldout) |
| scenario_probes | Reusable user messages for scripted sweeps |
| personas + persona_snapshots | 5 simulated users + frozen Vault/profile state |
| system_prompts | Content-addressed prompt registry — new prompt = new row |
| branches | One linear conversation rooted in scenario + persona |
| turns | Position in a branch — holds the user message |
| candidate_sets | Sibling cohort (e.g. "Sample 3" or "Compare prompts") |
| generation_contexts | Exact input context — messages, memory payload, hashes |
| generation_attempts | Per-attempt provider details — model, seed, latency, cost |
| assistant_candidates | The atomic unit. Immutable. Raw + classifier output + reviewer-visible |
| selections | Candidate pinned as canonical reply for a turn |
| candidate_labels + candidate_assessments | Structured + free-form per-candidate annotations |
| pair_preferences | Head-to-head ranks + blind metadata |
| trajectory_labels | End-of-branch arc rating |
| classifier_events | Every safety verdict with full prefix |
| judge_jobs + judge_grades | Multi-unit expert judge runs |
| training_approvals + export_eligibility | Per (unit, bucket) approval and computed eligibility |
| dataset_splits | train / dev / heldout assignment at family level |
| exports | Audit log of every export run |
| synthesis_runs + synthesis_findings | Theme synthesis with promote-to-label gesture |
Every candidate carries four hashes that answer four different export questions. This is the property that lets one canonical store feed every export bucket without re-collecting.
interaction_prefix_hash
comparison_context_hash
sampling_context_hash
generation_attempt_hash
The context-building mode is fixed at the experiment level. It cannot drift inside a branch — preventing the mistake that would make multi-turn data incoherent.
transcript_window (for training data): Full conversation messages array sent to the model. Memory composer not invoked. Vault never mutated by generation. Anaphora handled natively because the model sees prior turns. Use for: SFT, DPO, prompt-eval.
production_memory (for production regression): Only latest user message sent to model. Memory composer invoked normally — reproduces production. Clones a real life-app user. Tests whether the memory composer surfaces what the model needs. Use for: regression and classifier training.
The spec is held to 28 invariants. The most consequential ones, in plain English:
The full 28 live in specs/life-data-lab.md §2.2.
App.tsx still references Cloudflare Access — needs to be the live path before production onboarding.NewExperimentDialog hardcodes prompts; the registry endpoint shipped in Phase 4 and the dialog should now consume it.review-ui/db/ migration runner is superseded by the Supabase CLI workflow — safe to delete.