LIFE Data Lab · Audit

Output-Format & Training-Readiness Audit

Is the Data Lab outputting data in the ideal format for future use and ML training? A static + live-data audit across 5 output surfaces and 4 interaction modalities, adversarially verified.
2026-06-06 · review-ui@a4a01f0 · life-app@126688c · live DB qsswasrphtkltbexvtfh · 42 agents · 50 findings → 38 confirmed
D+
Training-readiness
No — not today. The Data Lab is architected to produce ideal training data, but in its current state it cannot: (a) the approval→eligibility→export pipeline has an unwired middle stage (computeEligibility has zero callers), so every export returns 0 rows regardless of how much reviewed data exists; and (b) the formats it would emit are not conformant — both SFT flavors produce an identical non-standard {messages,target} shape that neither OpenAI nor Anthropic can load, the conditioning system prompt is stripped from every SFT/DPO record (making targets mis-attributed and irreproducible), and the predictive-profile product is doubly unreachable (missing enum value + a PostgREST numeric-as-string bug that nulls every prediction). Decisive caveats: (1) Everything is LATENT — the export/eligibility/approval tables are all empty (greenfield dev DB), so no bad data has shipped; these are first-use defects, not active corruption. (2) The non-conformant SFT shape is actually what spec §14.2 mandates, so fixing it is a spec/contract decision, not a pure bug-fix. (3) The quarantine invariants (the highest-stakes property for a safety dataset) genuinely hold — verified in code, tests, and live RLS. The good news: the lineage, provenance plumbing, and quarantine are sound, so the gap is finite and mostly additive — wire eligibility, fix the two SFT shapes, inject the system prompt, add the missing enum value, and coerce the numeric, and the data becomes train-ready.
Executive summary

The architecture is right; the boundary contract and the wiring are not

The Data Lab is built on a genuinely strong foundation — clean lineage (0 orphans, 1:1 candidate↔attempt, perfect sample_n DPO cohort shape), correct content-variant selection per format, complete human-edit provenance in the DB, and a quarantine layer (steering / judge-panel / reaction / disposition) that holds in code, tests, and live RLS (deny-all on every quarantined table). BUT it is not yet outputting training-ready data, for two compounding reasons. (1) The pipeline is broken end-to-end: computeEligibility() — the sole writer of export_eligibility — has ZERO production callers, so no approval ever becomes exportable and every export returns an empty file (A2-1, CRITICAL). (2) Where exports CAN be produced, the format is wrong: both SFT formats emit a non-standard {messages,target} shape loadable by neither OpenAI nor Anthropic and route to one identical adapter (A1-1, HIGH), the system prompt the model was conditioned on is absent from every SFT/DPO record (A1-2, CRITICAL), provider provenance (cost/stop_reason/request_id) is 100% null (B1-1, HIGH), and Data Product 3 (profiler labels) cannot even be requested because profiler_labels_jsonl is missing from the Postgres export_format enum (A1-4 = A4-1, HIGH) and its loader nulls every prediction via a numeric-as-string bug (A4-2, HIGH). The entire training/export tail is empty live (exports/eligibility/approvals = 0), so all of these are latent — nothing corrupt has shipped — but they are guaranteed-on-first-use defects, several masked by tests that mock the broken shapes. Net: the architecture is right; the boundary contract and the wiring are not. After the verification pass, of ~40 raw findings 1 was refuted, multiple HIGHs were tempered to MEDIUM/LOW, and two HIGH findings (the profiler_labels_jsonl enum gap) collapse into one cross-axis duplicate.
Scorecard

Five dimensions of training-readiness

FAILING
Format conformance
FAILING. Both SFT formats (sft_openai_jsonl, sft_anthropic_messages_jsonl) emit an identical {messages,target} record loadable by neither provider and route to one adapter (A1-1). System prompt absent from every SFT/DPO/prompt-eval record (A1-2). DPO prompt is a hand-rolled [system]/[user]/[assistant] text blob, not a TRL conversational list or chat-template (A1-6). profiler_labels_jsonl rejected at DB insert (A1-4=A4-1) and its rows null out via numeric-as-string (A4-2). huggingface_dataset_parquet/research_dump_jsonl 500 instead of 422 (A1-5). UTF-8/NDJSON serialization itself is correct. Note: the SFT shape conforms to spec §14.2 — the spec, not just the code, contradicts the 'standards at the boundary' promise.
MIXED
Provenance & lineage
MIXED. Lineage integrity is excellent: 0 orphans, 1:1 candidate↔attempt, correct parent/branch/turn links, perfect sample_n cohort hashing, system_prompt_hash matches registry on 35/35. But export-surface provenance leaks: provider cost/stop_reason/request_id 100% null on all model attempts (B1-1); seed never forwarded to the API so units are not bit-reproducible (B1-3); trajectory export hardcodes app_git_sha='unknown'+schema=1+created_at=now() and filters dates against now() not branch.created_at (B2-1, B2-2 HIGH); steered-vs-inline + generating model (assist_model) dropped at export (B3-1); no deterministic ordering → non-idempotent export bytes (A1-7).
INCOMPLETE
Label completeness
INCOMPLETE. candidate_assessments (spec's 'highest-value prompt-iteration signal') is write-only — never read by any exporter or the eligibility computer (A3-1). judge_grades.parsed_jsonb has no schema validation and a different shape per unit-type (A3-2). training_approval_audit is never written — approval-transition history empty by construction (A2-3). No explicit reviewer-approval route exists for raw model candidates (A2-2). safety_sft bucket has eligibility logic but no exporter/format (B4-1). Human-edit provenance (edit_distance/reviewer_id/was_steered) captured in DB but dropped from SFT/DPO metadata (B4-3).
STRONG
Quarantine safety
STRONG (the standout dimension). All four P0 invariants hold in canonical code AND live RLS: steering, judge-panel, reaction, and disposition tables are all RLS deny-all (0 policies, service-role-only); no export loader/adapter/worker joins any quarantined table; reaction promotion is id-gated and revocable with an 8-vector adversarial test + positive control; the disposition answer-key is isolated on a separate deny-all table and compile-time + runtime leak-guarded out of the profiler path. The one gap is a missing judge-panel regression test (A5-1) — structurally safe today but unguarded against a future refactor.
PARTIAL
Interaction modalities
PARTIAL. Single + sample_n generation, human_edit, and steering-promote modalities have clean, verified lineage and content selection. Multi-turn trajectory is the weakest surface — provenance hardcoded, date filter broken, system-prompt-derivation unreliable, no interior-selection completeness guard, system body omitted. Fork/clone and scenario-seed lineage code is entirely unexercised live (0 forks/clones/committed turns) so its correctness is unproven end-to-end. The canonical judge path (judge_jobs/judge_grades) has never run — all 128 judge artifacts are on the quarantined judge_panel path.
Where the pipeline breaks

End-to-end data flow

generate (single / sample-N / compare / steer / edit)→ candidates + provenance→ reviewer approves→ computeEligibility() ✗ no caller→ export_eligibility (always empty)→ export = 0 rows
The keystone break (A2-1): nothing a reviewer approves ever becomes eligible, so every export returns an empty file. Downstream, even a populated export would emit a non-conformant SFT/DPO shape with the system prompt stripped (A1-1, A1-2). Quarantine, by contrast, holds end-to-end — steering / judge-panel / reaction / disposition are RLS deny-all and never joined by any loader.
Severity distribution

38 confirmed findings (1 refuted, several tempered in verification)

2
6
14
16
2 Critical 6 High 14 Medium 16 Low
Critical & High

Must-fix before any training export (8)

CRITICAL A2-1 safe

computeEligibility() has zero production callers → export_eligibility can never be populated → no unit is ever exportable end-to-end

life-app data-lab/exports/eligibility-computer.ts + canonical-loaders.ts
Evidence
eligibility-computer.ts is the SOLE writer of export_eligibility, and computeEligibility is referenced only by its own definition + tests (no route/worker/CLI/DB-trigger imports it). canonical-loaders.ts:82-109 gates every bucket on export_eligibility WHERE eligible=true AND is_current=true and early-returns [] when empty. Approval writers (turns-edit.ts:638-701, pair-preferences.ts:897) write training_approvals then return without recomputing. Spec §B.1 line 3189 / Phase-6 plan line 2862 require the missing recompute wiring. Live: export_eligibility=0, exports=0, training_approvals=0, selections=26.
Why it matters for training
The entire training-data product is unreachable: every SFT/DPO/prompt_eval/classifier_train/trajectory export returns an empty file no matter how much reviewed data exists. There is no path by which a reviewer's approval becomes an exportable unit.
Recommended fix
Call computeEligibility after every training_approvals write (pair-preferences, turns-edit, product-tagging, promote-reaction, selections) for the affected (unit_type,target_id,bucket), AND ship a /data-lab/eligibility/recompute route or worker pass. Add an integration test asserting approval→eligibility=true→non-empty export.
Verification: Confirmed across code (zero non-test callers), DB (no trigger populates the table), approval writers (none recompute), and live data (all tail tables empty). Severity CRITICAL upheld.
CRITICAL A1-2 safe

System prompt is absent from every SFT/DPO/prompt-eval record — model_seen_messages_jsonb has no system message and the loader never injects the registry body

life-app canonical-loaders.ts (SFT/DPO/prompt-eval prefix derivation)
Evidence
loadSftCanonicalRows builds prefix_messages from ctx.model_seen_messages_jsonb verbatim (canonical-loaders.ts:449-475) and never injects the resolved system prompt; system_prompts is loaded only for {id,name} (select 'id, name' at :182). DPO/prompt-eval loaders identical (:646-651, :857-862). Live: 32/32 generation_contexts have 0 system-role messages; system_prompts.body is fully populated (11/11) and recoverable but never joined. Spec §14.2 (life-data-lab.md:2322) mandates messages[0]={role:'system',content:<prompt body>}.
Why it matters for training
SFT/DPO trains the model on user→assistant pairs with the actual conditioning system prompt stripped out — the learned policy is mis-attributed and irreproducible, and the exact prompt that produced the target is unrecoverable from the export (only an id/name in metadata, no resolver wired in).
Recommended fix
Resolve system_prompts.body in the loaders and prepend it as messages[0] (OpenAI/spec shape) or a top-level system field (Anthropic). Add a leak-test asserting messages[0].role==='system' (or non-empty system) on real loader output.
Verification: Confirmed on all three legs (code, live DB, spec). The only passing system-message test hand-seeds the fixture (sft-adapter.test.ts), masking the production gap. CRITICAL upheld.
HIGH A1-1 investigate

SFT output shape conforms to NO real downstream spec ({messages,target}); OpenAI and Anthropic flavors are byte-identical and both wrong

life-app sft-adapter.ts + export-worker.ts
Evidence
sft-adapter.ts:53-78 emits {messages,target,metadata} with target as a separate top-level key (assistant turn NOT appended to messages). export-worker.ts:111-116 routes BOTH sft_openai_jsonl and sft_anthropic_messages_jsonl to the same sftAdapter → byte-identical output. Verified live against official docs: OpenAI fine-tune JSONL needs the assistant target as the LAST message with no 'target' key; Anthropic forbids role:system in messages and uses a top-level system param. Loadable by neither.
Why it matters for training
A file labeled sft_openai_jsonl is rejected by the OpenAI file validator; the Anthropic flavor is identical and equally invalid. Every SFT export needs a post-hoc transform before it can train anything, defeating the 'standards at the boundary' promise (Invariant 17).
Recommended fix
Split the two formats. For OpenAI fold target into messages as the final assistant message (drop the 'target' key). For Anthropic hoist system to a top-level field, keep messages user/assistant only. Stop routing both enum values to one adapter. NOTE: this changes the spec §14.2 output contract — treat as a spec decision, not a silent bug-fix.
Verification: All mechanical claims TRUE, but downgraded CRITICAL→HIGH: the shape is exactly what spec §14.2 mandates (code is spec-faithful), exports are empty (0 rows, no data-loss), and the fix is a contract change — hence fix_class investigate, not safe.
HIGH A1-4 safe

profiler_labels_jsonl (Data Product 3) missing from Postgres export_format enum — export 500s at DB insert (unreachable end-to-end) [DEDUP: A1-4 ≡ A4-1]

review-ui supabase export_format enum vs life-app exports route/worker
Evidence
types.ts:53 lists profiler_labels_jsonl in EXPORT_FORMATS; exports.ts:186 accepts it then inserts into exports.format (enum column) at :276-291 BEFORE the worker runs. Live: export_format enum = exactly 8 values, profiler_labels_jsonl absent; `select 'profiler_labels_jsonl'::export_format` → ERROR 22P02 invalid enum input. No migration adds it (the reaction-promotion migration 20260605131000 only ALTERs training_unit_type). So the INSERT throws → HTTP 500 before the worker (which fully implements the format) is reached.
Why it matters for training
The only export path that turns promoted reaction predictions/labels into a training/eval artifact is dead-on-arrival; the entire predictive-profile data product is undeliverable downstream. Classic TS/DB enum drift.
Recommended fix
ALTER TYPE export_format ADD VALUE IF NOT EXISTS 'profiler_labels_jsonl' (cannot run in a txn block — match the reaction-promotion migration pattern), apply via supabase db push. Add a parity test asserting every TS EXPORT_FORMATS member exists in the live enum.
Verification: Confirmed by both A1 and A4 finders and reproduced live (cast error). HIGH upheld — fails loud (500), not silent corruption. Deduplicated: A4-1 is the same defect, counted once.
HIGH A4-2 safe

Profiler export loader nulls predicted_confidence (PostgREST returns numeric as a STRING) — cascades to a fully null prediction object in Data Product 3

life-app reaction-export-product.ts loader + transform
Evidence
reaction-export-product.ts:172-175 keeps predicted_confidence only when typeof === 'number'. predicted_confidence is a Postgres numeric (live data_type=numeric) and PostgREST/supabase-js serialize numeric as a JSON STRING — proven live: dlq/PostgREST return "predicted_confidence":"0.65". So typeof is 'string' → null → transformReactionRow (:347-364) builds prediction only when !== null → the ENTIRE prediction object (distribution, argmax, calibration, model) drops to null. The reveal path (reaction-prediction.ts:1264) already uses Number() coercion; the export path does not. Test mocks it as a JS number, masking it.
Why it matters for training
Every promoted prediction in the calibration export would ship with prediction:null, losing distribution/argmax/confidence/calibration — making the profiler-labels/calibration training set worthless.
Recommended fix
Coerce defensively: predicted_confidence = p.predicted_confidence == null ? null : Number(p.predicted_confidence) (mirror reaction-prediction.ts:1264), and guard transformReactionRow on Number.isFinite. Add a test feeding the string '0.7' (real PostgREST shape) asserting prediction.confidence===0.7 and prediction non-null.
Verification: Confirmed with a decisive live PostgREST probe (numeric serialized as string). Guaranteed-on-first-use; latent only because training_approvals/exports are empty. HIGH upheld.
HIGH B1-1 safe

Provider provenance cost_usd / stop_reason / provider_request_id are 100% NULL on every model attempt (the ChatGenerationOutput abstraction drops them)

life-app generation-service.ts + AIProvider
Evidence
generation-service.ts hardcodes providerRequestId=null (:1199), stop_reason:null (:1242), cost_usd:null (:1246). ChatGenerationOutput (ai-provider.ts:22-27) exposes only {content,modelProvider,modelName,tokenUsage}. Both providers HAVE the data and discard it: anthropic-provider.ts:339 reads response.stop_reason (+response.id); openrouter-provider.ts:228 reads choice.finish_reason (+response.id, +cost via /generation). Live: 35/35 model attempts have all three NULL. Spec §105/§252 require provider_request_id, stop_reason, cost as per-attempt provenance.
Why it matters for training
Without stop_reason, truncated/length-capped generations are indistinguishable from natural stops — corrupts SFT target quality and trajectory-reward signal. Without cost no per-unit economics. Without request_id no exact replay/debug.
Recommended fix
Extend ChatGenerationOutput to surface stopReason/providerRequestId/costUsd, populate both providers, thread into runOneAttempt's generation_attempts insert (columns already exist — additive, no schema change). For OpenRouter optionally backfill cost from /generation by id.
Verification: Confirmed in code + live (35/35 null). One rationale clause ('replay impossible') mildly overstated — replay is degraded not impossible (hash uniqueness intact). HIGH upheld: spec MUST-level, near-free to capture, real SFT/trajectory impact.
HIGH B2-2 safe

Trajectory date-range filter applied against now() instead of branch.created_at — silently misfilters trajectory exports

life-app canonical-loaders.ts (trajectory loader) + applyFilterSpec
Evidence
canonical-loaders.ts:1248 passes createdAt: new Date().toISOString() to applyFilterSpec with the (false) comment 'branches lack a uniform created_at filter target'. applyFilterSpec (:311-316) compares that value to f.created_after/before. All other loaders pass the real row created_at. Live: branches.created_at IS populated 15/15 (2026-05-20..06-06). UI exposes the date pickers (ExportFilterBuilder.tsx:355/382) and the route forwards them. So created_before=<past> excludes ALL branches; created_after=<past> includes ALL.
Why it matters for training
Date-bounded/incremental trajectory dataset builds are non-deterministic and wrong, breaking idempotent slice-based training-set construction for the trajectory_reward format.
Recommended fix
Add created_at to loadBranches select and pass branch.created_at (or trajectory_label.created_at) to applyFilterSpec; delete the incorrect comment.
Verification: Confirmed: code, live data (branches.created_at populated), UI exposure all verified; logic direction confirmed. HIGH upheld — silent correctness bug; latent only because trajectory_labels is empty.
HIGH A2-2 safe

No approval/eligibility HTTP route exists; code defers to a 'Phase 8 approval endpoint' that was never built

life-app routes/data-lab/index.ts + pair-preferences.ts/turns-edit.ts
Evidence
routes/data-lab/index.ts mounts ~25 sub-routers but NONE for /approvals or /eligibility. pair-preferences.ts:847-848 and turns-edit.ts:674-675 both defer to a non-existent 'Phase 8 approval endpoint'. Spec B.1 step 7 (life-data-lab.md:3188) says approving a raw model candidate for SFT 'requires explicit reviewer action'. The only training_approvals writers are auto-defaults; none can set approved on a raw model candidate. Additionally computeEligibility and createProductTaggingService both have ZERO callers/routes.
Why it matters for training
Reviewers cannot perform the core curation action (explicitly approve/exclude a unit for a bucket), so the explicit-approval class of training data (e.g. SFT on raw model outputs) cannot be created at all — only auto-defaults are captured.
Recommended fix
Build a training-approvals endpoint (POST per (unit_type,target_id,bucket) with supersedes/is_current append-only semantics) that (a) writes training_approvals, (b) writes a training_approval_audit transition row, (c) calls computeEligibility. Expose it in the Assessment/Export UI.
Verification: Confirmed; the gap is broader than stated (eligibility materialization is also orphaned). HIGH upheld.
Medium

Correctness & completeness (14)

MEDIUM A1-6 investigate

DPO 'prompt' is a bespoke '[system]/[user]/[assistant]' text blob, not a clean TRL prompt; missing system block

life-app dpo-adapter.ts renderPrompt
Evidence
dpo-adapter.ts:86-100 renderPrompt joins '[system]\n'/'[user]\n'/'[assistant]\n' with blank lines into the TRL prompt field (:141). Verified against official TRL docs: DPO/Preference accepts plain-text strings (consumed as-is) OR conversational {role,content} lists (chat-templated) — not hand-rolled bracket tags, which get tokenized verbatim. Compounded by A1-2: live 15/15 prefixes have 0 system messages so no [system] block is emitted at all. SFT keeps the same prefix as a structured messages list; DPO flattens it.
Why it matters for training
DPO gradient is computed on chosen vs rejected given prompt; fake role tags the model never sees at inference contaminate the preference signal and make the format non-portable across TRL/tokenizers.
Recommended fix
Emit prompt as a conversational message list (preferred for current TRL) or apply the target tokenizer's chat template; include the resolved system prompt. Update the DPO test which currently locks in the bracket format.
Verification: Confirmed against TRL docs + live (no system in prefixes). MEDIUM upheld.
MEDIUM A1-8 safe

SFT adapter test hardcodes a system message into prefix_messages, masking the live 'no system prompt' defect

life-app exports/__tests__/sft-adapter.test.ts
Evidence
sft-adapter.test.ts:30-33,118-131 inject {role:'system'} into prefix_messages fixtures and assert rec.messages===prefix_messages. The adapter and loader pass prefix through verbatim with zero system injection (sft-adapter.ts:54, canonical-loaders.ts:449-475), so the green test only passes because the fixture is hand-seeded. Live: 32/32 contexts contain no system message; generation-service types model_seen_messages_jsonb as user|assistant only (system not even representable).
Why it matters for training
CI asserts a property (system-prompt presence) the production data path never satisfies, so the most important conformance regression (A1-2) is invisible to tests; future format work is 'verified' against a fiction.
Recommended fix
Add a loader-level integration test using realistic user/assistant-only model_seen_messages_jsonb that asserts the loader injects the resolved system prompt; stop pre-seeding system into the adapter fixture.
Verification: Confirmed; MEDIUM upheld (test-quality issue; the deeper data defect is A1-2). Sibling of the dpo-adapter test which similarly masks A1-6.
MEDIUM A3-1 investigate

candidate_assessments — spec's 'highest-value prompt-iteration signal' — is never read by any exporter or the eligibility computer (write-only)

life-app canonical export pipeline / candidate_assessments
Evidence
candidate_assessments is written by its own route but read by no export loader, worker, route, or eligibility-computer (only a doc comment in product-tagging.ts:15). eligibility-computer gates exclusively on training_approvals.approval_state and never references candidate_assessments/failure_tags/training_suitability/confidence. Live: candidate_assessments=0 (greenfield).
Why it matters for training
The richest human per-candidate annotation (accept_state, failure_tags, training_suitability, confidence) cannot influence what gets exported nor attach to exported rows — training/eval datasets lose the most actionable reviewer signal and failure-mode taxonomy.
Recommended fix
Attach the current candidate_assessments row to SFT/DPO/prompt-eval export metadata (failure_tags/confidence/training_suitability). The eligibility-gating half (honor training_suitability=exclude/eval_only) is a product decision — the route deliberately separates 'assessment (signal)' from 'approval (gate)'.
Verification: Downgraded HIGH→MEDIUM (partially_confirmed): write-only is confirmed, but the eligibility-gating recommendation is plausibly by-design; metadata-projection gap is the undisputed part, latent at 0 rows.
MEDIUM A3-2 safe

judge_grades.parsed_jsonb has no schema validation and heterogeneous untyped shapes across unit-types

life-app judge fanout / canonical loaders / DPO+prompt-eval+trajectory exports
Evidence
judge-fanout parseJudgeJson explicitly does NOT zod-validate; CanonicalJudgeWriter.insertGrade inserts parsed_jsonb verbatim; types type it as Record<string,Record<string,unknown>>; dpo/prompt-eval/trajectory adapters pass it through; canonical-loaders.ts only guards parsed_jsonb!==null. Shapes diverge per unit-type (pairwise {preferred,score_a,score_b,...} vs candidate {score,turn_scores,drift_flags,...}). No DB CHECK on parsed_jsonb. PairwiseJudgeOutput/CandidateJudgeOutput interfaces exist but never validate runtime output. Live: judge_grades=0.
Why it matters for training
Exported judge consensus/grades are LLM-emitted JSON with no enforced score range, no required fields, and a different shape per unit-type; consumers cannot rely on a stable score key/scale and a malformed/hallucinated grade flows straight into the dataset.
Recommended fix
Add per-unit-type zod schemas (score range, required keys) validated in judge-fanout before insertGrade or in loadCompleteGradesByTarget; drop/flag grades that fail rather than exporting raw.
Verification: Confirmed; MEDIUM upheld (latent — canonical judge path unused, judge_grades=0). Cross-repo chain real (writer in review-ui, reader in life-app).
MEDIUM B2-1 safe

Trajectory export hardcodes app_git_sha='unknown' + schema_version=1 + tag_taxonomy='v1', discarding provenance available on branches/trajectory_labels

life-app canonical-loaders.ts (trajectory loader)
Evidence
canonical-loaders.ts:1288-1291 hardcodes data_lab_schema_version:1, app_git_sha:'unknown', tag_taxonomy_version:'v1' into trajectory_reward_jsonl metadata (via trajectory-adapter.ts:65-67). loadBranches (:162) doesn't select app_git_sha/data_lab_schema_version/created_at though branches carries them NOT NULL. Sibling loaders propagate the real row app_git_sha.
Why it matters for training
Trajectory records carry false provenance so trajectories cannot be traced to the model/build/schema that produced them, defeating reproducibility for trajectory_reward training.
Recommended fix
Populate from real rows: app_git_sha + data_lab_schema_version from branch (add to loadBranches select), tag_taxonomy_version from trajectory_labels. Remove the 'unknown'/literal-1 fallbacks.
Verification: Downgraded HIGH→MEDIUM (partially_confirmed): defect real and correctly located, but in the current dev DB every app_git_sha is already 'unknown' (env unset) so impact is latent; the exported record has no created_at field (uses exported_at), and tag_taxonomy 'v1' is hardcoded by the candidate loader too. trajectory_labels empty.
MEDIUM B3-1 safe

Steered-vs-inline provenance (was_steered) and the generating model (assist_model, plus model/provider) are dropped at export

life-app sft-adapter.ts/dpo-adapter.ts + canonical-loaders.ts
Evidence
sft-adapter.ts:53-77 and dpo-adapter.ts:117-137 metadata carry source/chosen_source but no model/provider/assist_model/producer_type/was_steered; loaders never select them. Live: 4 promoted steered candidates have generation_attempts.model=NULL, assist_model='claude-opus-4-7', was_steered=true — none appear in any export. Broader: the 35 raw model generations also carry model='anthropic/claude-opus-4-7'+provider='openrouter' that is dropped too. Migration 20260519160000 created was_steered specifically so downstream wouldn't need to join steering_sessions.
Why it matters for training
A steered human_edit (drafted by a model then accepted) is indistinguishable from a reviewer-typed inline edit and the producing model is lost; model/provider provenance is lost for ALL candidates. Downstream cannot filter/weight/attribute steered or model-generated rows.
Recommended fix
Add model, provider, producer_type, assist_model, assist_provider, was_steered to SFT/DPO loaders + adapter metadata (StandardRecord metadata has an index signature — additive, no schema change). Mirror the chosen_source/rejected_source pattern.
Verification: Downgraded HIGH→MEDIUM: confirmed and broadened (model provenance lost for all rows, not just steered), but provenance-completeness gap (not a leak/format break), exports empty, fix purely additive.
MEDIUM B4-1 investigate

safety_sft bucket has eligibility logic but NO export format/loader/adapter — classifier-modified safety-correction content is unreachable

life-app export-worker / canonical-loaders / export_format enum vs eligibility-computer
Evidence
eligibility-computer fully implements safety_sft eligibility (evalSafetySft:201-210, loadSafetySftFacts:697-718, unit_type=candidate). But export-worker.ts runAdapterForFormat has no safety_sft case, sftAdapter EXCLUDES classifier_modified rows, loadSftCanonicalRows reads only bucket='sft', and EXPORT_FORMATS has no safety_sft format so the route rejects it. classifier_train emits a different (classifier-training) shape, not an SFT target. Spec designates safety_sft as the ONLY home for classifier-modified SFT content. Live: 9 legacy classifier_modified=true rows; no current writer sets the flag.
Why it matters for training
The eligibility computer can mark rows bucket=safety_sft that no exporter can ever drain — the highest-signal safety-correction data class is structurally unreachable, defeating spec §14.2's safety_sft purpose.
Recommended fix
Either add a safety_sft export_format + loader + worker case, OR (if classifier-substitution is permanently retired) remove the safety_sft bucket/eligibility code and update the spec — don't leave a promised-but-dead bucket.
Verification: Downgraded HIGH→MEDIUM: confirmed dead-end, but contingent on a writer that doesn't yet exist (all writers hardcode classifier_modified=false); impact latent.
MEDIUM A2-3 safe

training_approval_audit is never written — append-only approval-transition history is empty by construction

life-app all training_approvals writers + review-ui migration table
Evidence
grep for training_approval_audit across both trees returns only the migration (create table + RLS). All 4 training_approvals writers set supersedes/is_current but none inserts a from_state→to_state audit row; no DB trigger populates it (live triggers = []). product-tagging/promote-reaction even compute priorRow.approval_state for the supersede flip then discard it. Migration §4.11b comment promises 'Append-only history of every approval state transition, for forensics'.
Why it matters for training
Provenance/forensics is incomplete: no normalized record of who changed an approval X→Y or when, weakening auditability of which units were train-eligible at export time.
Recommended fix
In every approvals writer (and the future approvals endpoint) insert a training_approval_audit row (from_state, to_state, reviewer_id, reason) in the same operation. Add a test asserting one audit row per approval write.
Verification: Downgraded HIGH→MEDIUM (partially_confirmed): dead table confirmed, but training_approvals itself is an append-only superseding chain (supersedes/is_current/reviewer_id/reason/created_at) so the audit row is largely redundant; greenfield-empty.
MEDIUM A2-4 investigate

Live dataset_splits are orphaned from the only data-bearing scenario_family; the real family has NO split row and silently defaults to 'train' (fail-open)

life-app canonical-loaders.ts resolveSplit + review-ui dataset_splits data
Evidence
resolveSplit (canonical-loaders.ts:211-215) returns 'train' when a family has no dataset_splits row and is_heldout=false (fail-open). Live: the 9 dataset_splits families (incl. 2 heldout) have 0 scenarios; the only family with scenarios/branches (stage-general-foundations, 15 branches, is_heldout=false) has 0 split rows → resolves to 'train'. No FK or integrity check requires real families to have a split row.
Why it matters for training
A future export dumps 100% of real data into train with no held-out eval set — the leakage/over-fit risk the split system exists to prevent.
Recommended fix
Make resolveSplit fail-closed for missing split coverage on release-candidate/heldout-sensitive exports (don't default unknown families to 'train'); add a data-integrity check that every family with scenarios has exactly one dataset_splits row. Spec §15.2 mandates most-restrictive-wins, contradicting the current fail-open default.
Verification: Confirmed (fail-open default + total orphaning + no integrity guard); MEDIUM — latent because exports/eligibility empty.
MEDIUM A2-5 safe

computeEligibility demote→insert is non-transactional with a TOCTOU computation_version probe loop — concurrent recomputes can violate the partial unique index

life-app data-lab/exports/eligibility-computer.ts
Evidence
eligibility-computer.ts:804-860: demote UPDATE, then a read-then-increment loop probing computation_version for a free slot, then INSERT is_current=true — three separate Supabase calls, no transaction/lock/RPC/upsert. Two concurrent calls for the same (unit_type,target_id,bucket) both demote, probe the same free version, both INSERT → violates the export_eligibility_current partial unique index (throws). computation_version is overloaded as a retry counter despite being a constant=1. No concurrency test.
Why it matters for training
Once A2-1 wires recompute-on-event, concurrent same-tuple recompute (e.g. a pair approval + a selection on the same candidate) crashes recomputes or makes exportability nondeterministic — a reproducibility hazard for training snapshots.
Recommended fix
Make demote+insert one atomic Postgres RPC/transaction (or delete+insert with ON CONFLICT on the partial-current index — note the project's known PostgREST partial-index ON CONFLICT gotcha); stop overloading computation_version. Add a concurrency test.
Verification: Confirmed; MEDIUM upheld — latent (path uninvoked, table empty); the partial unique index is a real DB backstop against the worst (double-current) outcome, so worst case is a thrown error not silent corruption.
MEDIUM A2-6 investigate

Trajectory/canonical split-leak defense is not defense-in-depth — heldout exclusion depends on the (un-run) eligibility computer + complete dataset_splits coverage

life-app canonical-loaders.ts applyFilterSpec/resolveSplit + adapters
Evidence
applyFilterSpec defaults the split filter to ['train'] and resolveSplit fails OPEN (unknown family → 'train'). The deep heldout EXCLUSION lives in the eligibility computer which never runs (A2-1). Adapter-level guards exist (sft/dpo/prompt-eval/trajectory: split==='heldout' return false) but classifier-train-adapter has NO heldout guard, and all guards key off the fail-open resolveSplit. Spec §15.2 mandates fail-closed most-restrictive split; code defaults fail-open.
Why it matters for training
The no-leakage-across-splits invariant (heldout never contaminates train) is not defended in depth; combined with A2-4, a mis-resolved heldout family could be exported into train.
Recommended fix
Fail-closed on missing split coverage for release-candidate/heldout-sensitive exports; enforce heldout exclusion at BOTH eligibility-compute and load time (add a classifier-train heldout guard). Add a leak test: a heldout family never appears in a default train export even with stale/missing eligibility rows.
Verification: Partially_confirmed: the 'only guard' framing omits the adapter-level guards (so 4/5 buckets are doubly defended), but the fail-open resolveSplit gap and the classifier-train guard absence are real; MEDIUM.
MEDIUM B3-2 safe

Promote endpoint is not idempotent — re-promoting a steering draft creates a duplicate training candidate and orphans the first; no DB guard

life-app steering-promote-service.ts + route
Evidence
steering-promote-service.ts:263-554 selects message.promoted_candidate_id but never checks it before inserting a new assistant_candidates + generation_attempts row, then overwrites the link (step 8) and treats a failed link as a soft error (with a comment acknowledging the duplicate risk). No route dedupe; promoted_candidate_id has FK only, no UNIQUE. UI guards the happy double-click but the server contract is unguarded (lost-response retry / second tab / direct API / soft-error path).
Why it matters for training
A retry silently injects a duplicate steered training unit and orphans a was_steered candidate no message points to — inflating training data and breaking the message↔candidate lineage that export attribution relies on.
Recommended fix
In promote(), if message.promoted_candidate_id is already set, return the existing candidate (idempotent). The partial-UNIQUE-on-promoted_candidate_id half is mis-targeted (re-promote moves the link) — the code-level idempotency check is the real fix.
Verification: Confirmed (authors aware per code comment); MEDIUM upheld — happy path UI-mitigated, live data clean, requires an uncommon failure mode to fire.
MEDIUM B4-2 safe

DPO adapter lacks the last-mile classifier_modified guard that SFT has (single-layer vs two-layer defense)

life-app dpo-adapter.ts vs sft-adapter.ts
Evidence
sft-adapter.ts:39 sftRowEligible returns false for classifier_modified rows (re-checked in transform); dpoRowEligible (dpo-adapter.ts:67-78) checks preference/strength/cross_prompt but NOT classifier_modified, though DpoCanonicalCandidate carries the field. DPO relies solely on eligibility-computer evalDpo. dpo-adapter.test never asserts a classifier_modified pair is dropped. Spec §14.2 forbids classifier-modified content in normal SFT/DPO.
Why it matters for training
If a stale/buggy export_eligibility row marked a classifier-modified pair eligible, DPO would emit post-classifier (safety-patched) text as chosen/rejected, training the model to imitate classifier patches — SFT is protected by two layers, DPO by one.
Recommended fix
Add `if (row.chosen.classifier_modified || row.rejected.classifier_modified) return false` to dpoRowEligible plus a regression test, mirroring sftRowEligible.
Verification: Partially_confirmed: code defect real; the finding's 'no writer sets classifier_modified=true' sub-claim is false (9/39 rows have it), but no active leak because pair_preferences=0 and export_eligibility=0. MEDIUM.
MEDIUM A5-1 safe

Judge-panel quarantine has no regression/leak test, unlike steering & reaction

life-app exports/__tests__/ (missing judge-panel guard)
Evidence
grep 'judge_panel' over life-app server src = zero (source AND tests). steering-leak.test.ts and reaction-quarantine-leak.test.ts both hardcode table-specific lists and dynamically assert loaders never .from() them; there is no equivalent guard for judge_panel_runs/judge_panel_grades. canonical-loaders joins only judge_jobs/judge_grades. Live: judge_panel_grades=128 (the only populated judge path), RLS deny-all. Export loaders use the service-role client (bypasses RLS), so the test — not RLS — is the real defense.
Why it matters for training
An unguarded refactor wiring judge_panel into a loader would silently leak quarantined eval grades (128 live rows) into training labels.
Recommended fix
Add a judge-panel leak guard mirroring the other two: assert no canonical loader's fromCalls contains judge_panel_runs/judge_panel_grades, plus a static-source grep that canonical-loaders.ts has no 'judge_panel' substring.
Verification: Confirmed; MEDIUM upheld — regression-prevention gap (no active leak), but it's the one judge path with 128 live quarantined rows.
Low

Hardening & hygiene (16)

LOW B1-3 investigate

Seed is never forwarded to the provider API; model units are not bit-reproducible from stored provenance

life-app generation-service.ts + providers
Evidence
The token 'seed' is absent from all three provider files; generation-service.ts comments that the seed exists only for hash-uniqueness/replay tooling and is not forwarded. Single-shot path stores seed=null (31/35 model attempts null; only the 4 sample_n siblings carry a seed). Live: 33/35 attempts ran at temp>=0.7.
Why it matters for training
generation_attempt_hash is documented as the 'exact replay' key, but with no forwarded seed and non-deterministic sampling the stored provenance cannot regenerate the exact bytes — replay/audit and seed-conditioned dedup are aspirational.
Recommended fix
When the AIProvider grows seed support, forward params.seed; until then document that generation_attempt_hash is a uniqueness token not a reproducibility guarantee, and consider populating the single-shot seed for column uniformity.
Verification: Spec-acknowledged limitation; LOW. No adversarial verdict requested for this finding; severity left as finder-assigned.
LOW A1-5 investigate

huggingface_dataset_parquet / research_dump_jsonl are enum-claimed but throw not-implemented (500 instead of 422)

life-app export-worker.ts runAdapterForFormat + route
Evidence
export-worker.ts:147-151 throws for both formats; the route validates them as acceptable then inserts a 'running' row → worker throws → row failed + 500. uploadOne content-type hardcoded application/x-ndjson would mislabel future Parquet. But the UI ExportDialog offers only 5 formats (neither is selectable), and exports=0.
Why it matters for training
Spec advertises HF Parquet/research-dump as boundary formats; a direct API caller gets a hard failure rather than a clean 422.
Recommended fix
Reject the two unimplemented formats at the route with 422 'not yet supported' until adapters exist (or implement them with correct content-types).
Verification: Downgraded MEDIUM→LOW: unreachable from UI, never exercised, explicitly deferred, fails cleanly.
LOW A1-7 safe

No deterministic ordering before serialize → non-idempotent export bytes for identical filter_spec

life-app adapters + canonical-loaders.ts
Evidence
Loaders read ids with no ORDER BY (Postgres arbitrary order); adapters iterate in loader order; only 2 incidental sorts exist; serializeToJsonl joins in array order; no output-byte hash is stored. Live probe: IN(...) without ORDER BY returned a different order than the id list. filter_spec_hash dedups requests, not output bytes.
Why it matters for training
Identical filter_spec can yield different line orderings → different file hashes, complicating dataset versioning/diffing (though trainers are order-insensitive and re-exports are content-identical).
Recommended fix
Add .order('id') in loaders + a stable record sort before serialize; optionally store an output byte-hash for idempotency checks.
Verification: Downgraded MEDIUM→LOW (partially_confirmed): mechanism proven, but the spec basis was overstated — row-level reproducibility (version fields/hashes) IS satisfied; no invariant mandates byte-identical ordering.
LOW A3-3 safe

QuarantinedJudgeWriter relevance & failure_modes columns are permanently null due to a dual-schema conflict in the candidate-panel template

review-ui evaluator judge-writers.ts + candidate-quality-judge template
Evidence
judge-writers.ts:294-298 extracts parsed['relevance']/['failure_modes'] into dedicated columns; live 124/124 done grades are null on both. The rubric BODIES (coaching.md:481-484 + 10 siblings, injected via {{rubricBody}}) DO request these fields, but the candidate-panel template appends a final 'Output JSON schema' that omits them, so the model follows the last schema and drops them. judge_panel_grades is a quarantined non-export table.
Why it matters for training
Two columns the rubric requests are permanently null, masked silently by best-effort extraction; the per-grade relevance signal the UI 'By dimension' view appears to promise is never persisted (that view is actually fed by a separate config table).
Recommended fix
Reconcile the panel template's final schema with the rubric (or drop the dead columns/extraction). Note the finding's central 'no rubric emits these' claim is false — the rubric bodies do request them.
Verification: Downgraded MEDIUM→LOW (partially_confirmed): mechanics true but the thesis (no rubric emits the fields) is wrong; quarantined non-export table, so a schema-hygiene gap not a training defect.
LOW A3-4 safe

judge_grades migration comment + writer docstring document a parsed_jsonb shape that drifts from actual judge output

review-ui judge_grades migration comment + writer docstring
Evidence
Migration 20260503204523:540 documents parsed_jsonb as '{score, relevance, failure_modes, rationale, key_observations}' but real rubric outputs contain neither relevance nor failure_modes, and pairwise uses score_a/score_b not score.
Why it matters for training
A future consumer trusting the documented JSONB shape looks for keys that never exist and may mis-handle pairwise (no score key), degrading reliable consumption of the quality layer.
Recommended fix
Correct the migration comment + writer docstring to the actual per-unit-type shapes and document that the shape is unit-type-dependent.
Verification: Finder-assigned LOW; no separate adversarial verdict (closely related to A3-2/A3-3 which were verified). Docs-only fix.
LOW B1-2 safe

Candidate_sets stuck in status='generating' when the SSE client disconnects after the candidate is persisted

life-app routes/data-lab/turns.ts SSE single-generate path
Evidence
turns.ts:933-944 returns early on client disconnect, skipping the markCandidateSetStatus('complete') call — an intentional, tested behavior (test asserts status stays 'generating'). Live: 4 sets stuck 10-11 days with actual==requested_n==1.
Why it matters for training
Leftover 'generating' rows are a lifecycle/observability blemish; the original 'training data dropped' claim is FALSE — no export loader/eligibility-computer references candidate_sets.status, and all 5 zombie sets have 0 selections so are independently non-exportable.
Recommended fix
Optional housekeeping: flip to a terminal status on disconnect or add a TTL reaper. Not required for training-readiness.
Verification: Downgraded MEDIUM→LOW (partially_confirmed): code mechanic real but the training-impact claim refuted — exports never read candidate_sets.status; cosmetic.
LOW B2-3 safe

Trajectory loader derives system_prompt_id from generation_attempts instead of the frozen branches.system_prompt_id

life-app canonical-loaders.ts (trajectory loader)
Evidence
canonical-loaders.ts:1195-1273 cross-walks generation_attempts for the first non-null system_prompt_id rather than reading the immutable branches.system_prompt_id (frozen by trigger). loadBranches doesn't select it. Live: branches.system_prompt_id 15/15; generation_attempts.system_prompt_id null 4/39.
Why it matters for training
Trajectory records could report a null or non-canonical system prompt id, mis-attributing which prompt produced the trajectory.
Recommended fix
Use branches.system_prompt_id (add to loadBranches select) as the authoritative source; drop the generation_attempts cross-walk for trajectory.
Verification: Downgraded MEDIUM→LOW (confirmed): real fragile defect, but all 15 live branches currently recover the correct prompt (0 mismatches) and trajectory export never run — zero current manifestation.
LOW B2-4 safe

No completeness guard: a branch can be 'ended' and trajectory-exported with an interior turn missing a selection, producing two consecutive user messages

life-app eligibility-computer.evalTrajectory + trajectory loader
Evidence
evalTrajectory gates only branch_status=ended/turn_count>=2/has_label/!heldout — no per-turn selection check; the message builder pushes assistant only if a selection exists, so an interior gap yields back-to-back user turns. Live: 6 selection-less turns, all terminal, 0 interior. No DELETE/clear path exists; the continuation guard requires the latest turn selected before appending, so the normal/fork workflow cannot create an interior gap.
Why it matters for training
Multi-turn formats require strict user/assistant alternation; an interior gap would be malformed — but reaching it requires an out-of-band mutation that no code path produces.
Recommended fix
Add an eligibility/label-time invariant that every turn_index<max has a selection, OR have the loader skip-or-flag a record with a missing interior selection. Add an interior-gap test fixture.
Verification: Downgraded MEDIUM→LOW (partially_confirmed): code facts accurate but no in-code path can create the gap; defense-in-depth hardening, 0 live instances.
LOW B2-5 investigate

Trajectory export omits the system-prompt body from the messages array though the record type and format support a system role

life-app trajectory loader + adapter
Evidence
TrajectoryCanonicalBranch.messages allows role:'system' but the loader pushes only user/assistant and never resolves system_prompts.body (loadSystemPrompts selects id,name only). Spec §14.2 shows messages beginning with a system message.
Why it matters for training
A trajectory_reward sample omitting the system prompt is not self-contained/replayable — reward/return estimation can't reconstruct the conditioning.
Recommended fix
Resolve branches.system_prompt_id→system_prompts.body and prepend {role:'system',content:body} (consistent with the A1-2 SFT/DPO fix).
Verification: Downgraded MEDIUM→LOW (partially_confirmed): no loader anywhere resolves the body (SFT/DPO also omit it via A1-2 — not trajectory-specific); trajectory_labels empty so never exercised.
LOW B4-3 safe

Human-edit provenance (edit_distance, reviewer_id, was_steered) captured in DB but dropped from SFT/DPO export metadata

life-app sft-adapter.ts/dpo-adapter.ts metadata
Evidence
generation_attempts holds edit_distance/reviewer_id/edited_from_attempt_id (live 4/4 human_edit populated, distances 374-1569) and assistant_candidates.was_steered=true, but SFT/DPO metadata carry only source/chosen_source/is_off_policy. Spec §14.8 only requires chosen_source='human_edit'+is_off_policy=true for off-policy distinction, which the code DOES emit.
Why it matters for training
Off-policy pairs lose the finer 'how far the human moved' and 'steered-vs-scratch' signal — additive enrichment, not a missing required field.
Recommended fix
Add edit_distance/reviewer_id/was_steered (and optionally a raw_model_response hash) to SFT/DPO metadata (additive, no schema change).
Verification: Downgraded MEDIUM→LOW (confirmed): omission real, but core off-policy signal already preserved per spec; export tail empty.
LOW B1-4 investigate

turns.ts bypasses the candidate-set service's status-transition guards (raw UPDATE on the hot path)

life-app routes/data-lab/turns.ts vs candidate-set-service.ts
Evidence
candidate-set-service.ts declares itself the single source of truth with guarded markComplete/markPartialFailed, but turns.ts inserts candidate_sets inline and flips status via its own unguarded raw UPDATE, leaving the service guards as dead code on the production path.
Why it matters for training
Latent: a future bug could mint an illegal status transition without tripping a guard, eroding the status field's trustworthiness for export filtering.
Recommended fix
Route candidate_set create + status flips through the service (or delete the unused guarded methods to keep one honest path).
Verification: Finder-assigned LOW; no separate adversarial verdict. Latent, not currently realized.
LOW B2-7 breaking

branches.status is plain TEXT+CHECK while turns.status and selections.commit_status are proper enum types

review-ui branches table schema
Evidence
Live: branches.status data_type=text with CHECK(draft|ended|archived); turns.status=turn_status enum; selections.commit_status=commit_status_type enum. The trajectory adapter + eligibility-computer gate on status='ended' as a string.
Why it matters for training
Schema-hygiene inconsistency on the one lifecycle value that gates whether a multi-turn branch becomes trajectory training data; an enum would be refactor-safe.
Recommended fix
Optionally migrate branches.status to a branch_status enum for parity (low priority — the CHECK already enforces the value set).
Verification: Finder-assigned LOW; no separate adversarial verdict. Breaking (alters existing column type) — decision item.
LOW A4-3 safe

TrainingUnitType TS type omits reaction_prediction/reaction_label though the DB enum and promote-reaction.ts use them

life-app exports/types.ts vs training_unit_type enum
Evidence
types.ts:77-81 defines TrainingUnitType as candidate|pair|branch|classifier_event only; promote-reaction.ts defines a parallel REACTION_PROMOTION_UNIT_TYPES tuple and migration 20260605131000 adds the two values to the Postgres training_unit_type enum (live = 6 values).
Why it matters for training
Type drift between the canonical TS unit-type and the DB enum invites a future writer to use the narrower type and silently exclude/mis-tag reaction artifacts.
Recommended fix
Extend TrainingUnitType to include reaction_prediction|reaction_label and derive promote-reaction's tuple from it.
Verification: Finder-assigned LOW; no separate adversarial verdict. Additive type fix.
LOW A4-4 investigate

Profiler loader does not filter is_stale predictions — a promoted-then-staled prediction would still export

life-app reaction-export-product.ts loadPromotedPredictions
Evidence
loadPromotedPredictions (reaction-export-product.ts:257-274) selects promoted ids with no is_stale=false filter, unlike the reveal path. Live: 0 stale rows, promotion table empty.
Why it matters for training
A calibration/eval set could include predictions made against a context that no longer matches the labeled interaction, subtly corrupting calibration fit.
Recommended fix
Filter .eq('is_stale',false) in loadPromotedPredictions (and log dropped ids), or auto-retract promotion when staled. Confirm with product whether promotion pins a row regardless of staleness.
Verification: Finder-assigned LOW; no separate adversarial verdict. Latent (0 stale rows).
LOW A4-5 safe

No DB-level CHECK that reaction label_class / predicted_argmax belong to the target's class space (app-validated only)

review-ui migration 20260605104502 reaction tables
Evidence
Migration 20260605104502 has CHECKs on condition/mode/family/source/confidence but label_class and predicted_argmax are free TEXT, validated only in app code; classes is a per-row text[] so only a trigger could enforce it. Live: 100% in-space (90 predictions + 9 labels).
Why it matters for training
A non-app writer (raw SQL backfill, future endpoint) could insert an out-of-space class that breaks argmax/distribution alignment — currently prevented only by convention.
Recommended fix
Add a validation trigger asserting label_class/predicted_argmax = ANY(reaction_targets.classes) as defense-in-depth. Low priority given clean live data.
Verification: Finder-assigned LOW; no separate adversarial verdict. Defense-in-depth, clean live data.
LOW A1-9 safe

Empty export buckets produce no file rather than an observable zero-row entry

life-app types.ts serializeToJsonl + export-worker.ts
Evidence
serializeToJsonl is correct UTF-8/NDJSON, but materializeOutputs drops empty sub-buckets entirely, so a legitimately-empty split produces NO file; a consumer cannot distinguish 'not run' from 'ran, zero rows' (row_count=0 is on the exports row but omitted from the per-file list).
Why it matters for training
Downstream pipeline can't tell an intentionally-empty split from a missing export without reading audit metadata — affects dataset-completeness accounting.
Recommended fix
Emit a zero-byte/header-only file or record an explicit {filename,row_count:0} entry so empty buckets are observable in the files list.
Verification: Finder-assigned LOW; no separate adversarial verdict. Minor completeness-accounting nit.
Needs your sign-off

Decisions & breaking changes (7)

A1-1
Why it needs your call
Splitting sft_openai_jsonl and sft_anthropic_messages_jsonl into two correct transforms (fold target into messages as the final assistant turn / hoist system to a top-level field) CHANGES the output contract that spec §14.2 explicitly mandates ({messages,target} with system inside messages). The code is spec-faithful, so this is a spec/product decision, not a silent bug-fix — the spec must be amended in lockstep.
Options
(a) Amend spec §14.2 to the real OpenAI/Anthropic shapes and split the adapter (recommended — restores 'standards at the boundary'); (b) keep §14.2 as an internal canonical shape and add an explicit documented post-transform step consumers must run; (c) status quo (rejected — neither file loads).
A1-6
Why it needs your call
Re-rendering the DPO prompt as a conversational message list or via the target tokenizer's chat template changes the dpo_trl_jsonl output shape (the current bracket-tag blob is locked in by a passing test) and depends on which downstream tokenizer/TRL convention is targeted — a format-contract + tooling decision.
Options
(a) Emit conversational {role,content} lists (most portable for current TRL); (b) render via a specific target tokenizer chat template (pin the tokenizer); (c) document the exact bracket template consumers must reverse (weakest). Include the system prompt in all cases.
A2-4
Why it needs your call
Making resolveSplit fail-closed on missing dataset_splits coverage changes export behavior (families with no split row would be refused rather than dumped into train) and requires a product decision on split-assignment policy + possibly seeding splits for real families. Spec §15.2 favors fail-closed most-restrictive, but flipping it affects what currently 'works'.
Options
(a) Fail-closed for release-candidate/heldout-sensitive exports + add an integrity check requiring one split row per real family (recommended); (b) keep fail-open but loudly warn; (c) auto-seed missing families to a default split (decide which).
B4-1
Why it needs your call
safety_sft is either an unfinished feature (needs a new export_format + loader + worker case) or dead code to remove (the bucket + eligibility logic) — and removal contradicts spec §14.2 which designates safety_sft as the home for classifier-modified content. Requires a product decision on whether classifier-substitution is a live data class.
Options
(a) Implement the safety_sft format/loader/adapter (if classifier-substitution is a real future flow); (b) remove the safety_sft bucket + eligibility code and update the spec (if permanently retired — no current writer sets classifier_modified=true).
B2-7
Why it needs your call
Migrating branches.status from TEXT+CHECK to a branch_status enum is a schema migration altering an existing column type, with code touchpoints that compare status as a string. Low value (the CHECK already enforces the value set).
Options
(a) Migrate to a branch_status enum for parity with turns/selections (refactor-safe but breaking); (b) leave as TEXT+CHECK (recommended — adequate).
A4-4
Why it needs your call
Whether to exclude is_stale promoted predictions from the profiler export is a product semantics decision: does an explicit reviewer promotion 'pin' a specific prediction row regardless of later context edits, or should staleness retract it?
Options
(a) Filter .eq('is_stale',false) in the loader (treat staleness as auto-retract); (b) auto-retract the promotion when a prediction is staled; (c) keep pinning by row id and document that promotion overrides staleness.
B1-3
Why it needs your call
Making generation_attempt_hash a true replay key requires forwarding seed to the provider API — but provider seed support/determinism varies and most attempts run at temp>=0.7, so true bit-reproducibility may be unattainable. The decision is whether to invest in seed forwarding or formally redefine the hash as a uniqueness-only token.
Options
(a) Forward params.seed when the AIProvider gains seed support and pin temperature for reproducible runs; (b) formally document generation_attempt_hash as a uniqueness token (not a reproducibility guarantee) and stop implying exact replay.
Ready to ship

Safe-fix plan (21 non-breaking)

IDRepoFileChange
A2-1 life-app packages/server/src/data-lab/exports/eligibility-computer.ts + routes/data-lab/index.ts + approval writers Wire computeEligibility() into every training_approvals write (pair-preferences, turns-edit, product-tagging, promote-reaction, selections) and add a /data-lab/eligibility/recompute route or worker pass. This is the keystone fix — without it nothing exports.
A1-2 life-app packages/server/src/data-lab/exports/canonical-loaders.ts Resolve system_prompts.body in the SFT/DPO/prompt-eval loaders and inject it as messages[0]={role:'system'} (OpenAI/spec) or a top-level system field (Anthropic).
A1-4 review-ui supabase/migrations/ (new) export_format enum ALTER TYPE export_format ADD VALUE IF NOT EXISTS 'profiler_labels_jsonl' (outside a txn block, matching the reaction-promotion migration pattern); apply via supabase db push.
A4-2 life-app packages/server/src/data-lab/exports/reaction-export-product.ts Coerce predicted_confidence = p.predicted_confidence == null ? null : Number(p.predicted_confidence) (mirror reaction-prediction.ts:1264); guard transformReactionRow on Number.isFinite instead of typeof.
B1-1 life-app packages/server/src/data-lab/services/ai-provider.ts + anthropic-provider.ts + openrouter-provider.ts + generation-service.ts Extend ChatGenerationOutput with stopReason/providerRequestId/costUsd, populate both providers (stop_reason/finish_reason + response.id; OpenRouter cost via /generation), thread into the generation_attempts insert. Columns already exist — additive.
B2-2 life-app packages/server/src/data-lab/exports/canonical-loaders.ts Add created_at to loadBranches select and pass branch.created_at (or trajectory_label.created_at) to applyFilterSpec; delete the incorrect 'branches lack a created_at' comment.
A2-2 life-app packages/server/src/routes/data-lab/ (new approvals route) + index.ts Add a training-approvals POST endpoint (per unit_type/target_id/bucket, append-only supersedes/is_current) that writes training_approvals + training_approval_audit and calls computeEligibility; expose it in the UI.
A2-3 life-app packages/server/src/data-lab/ (all training_approvals writers) Insert a training_approval_audit row (from_state,to_state,reviewer_id,reason) in the same operation as every training_approvals write.
A2-5 life-app packages/server/src/data-lab/exports/eligibility-computer.ts Make demote+insert atomic via a single Postgres RPC/transaction (or delete+insert with ON CONFLICT on the partial-current index — heed the project's PostgREST partial-index ON CONFLICT gotcha); stop overloading computation_version as a retry counter.
A3-2 life-app packages/server/src/data-lab/exports/canonical-loaders.ts (or judge fanout in review-ui) Add per-unit-type zod schemas (score range, required keys) validated before surfacing judge grades; drop/flag grades that fail validation instead of exporting raw.
B3-1 life-app packages/server/src/data-lab/exports/sft-adapter.ts + dpo-adapter.ts + canonical-loaders.ts Select and surface model, provider, producer_type, assist_model, assist_provider, was_steered in SFT/DPO loaders + adapter metadata (StandardRecord metadata has an index signature — additive).
B4-2 life-app packages/server/src/data-lab/exports/dpo-adapter.ts Add `if (row.chosen.classifier_modified || row.rejected.classifier_modified) return false` to dpoRowEligible, mirroring sftRowEligible (defense-in-depth).
A5-1 life-app packages/server/src/data-lab/exports/__tests__/ (new judge-panel-leak.test.ts) Add a judge-panel leak guard: assert no canonical loader's fromCalls contains judge_panel_runs/judge_panel_grades, plus a static-source grep that canonical-loaders.ts contains no 'judge_panel' substring.
B3-2 life-app packages/server/src/data-lab/services/steering-promote-service.ts In promote(), if message.promoted_candidate_id is already set, return the existing candidate (idempotent) instead of inserting a duplicate.
B2-3 life-app packages/server/src/data-lab/exports/canonical-loaders.ts Use branches.system_prompt_id (add to loadBranches select) as the authoritative trajectory system_prompt_id; drop the generation_attempts cross-walk.
B4-3 life-app packages/server/src/data-lab/exports/sft-adapter.ts + dpo-adapter.ts Add edit_distance, reviewer_id, was_steered (and optionally a raw_model_response hash) to SFT/DPO export metadata (additive, no schema change).
A1-7 life-app packages/server/src/data-lab/exports/canonical-loaders.ts + adapters Add .order('id') in loaders and a stable per-record sort before serialize so identical filter_spec yields byte-identical exports.
A1-8 life-app packages/server/src/data-lab/exports/__tests__/sft-adapter.test.ts (+ new loader test) Add a loader-level test with user/assistant-only model_seen_messages_jsonb asserting the loader injects the resolved system prompt; stop pre-seeding system into the adapter fixture as if the loader produced it.
A4-3 life-app packages/server/src/data-lab/exports/types.ts Extend TrainingUnitType to include 'reaction_prediction'|'reaction_label' and derive promote-reaction's tuple from it (mirror the DB enum).
A3-4 review-ui supabase/migrations/20260503204523_data_lab_aux.sql (comment) + judge-writers docstring Correct the parsed_jsonb shape documentation to the actual per-unit-type shapes (pairwise: preferred/score_a/score_b/...; candidate/trajectory: score/turn_scores/...); note the shape is unit-type-dependent.
A1-9 life-app packages/server/src/data-lab/worker/export-worker.ts For a legitimately-empty bucket, record an explicit {filename,row_count:0} entry (or emit a header-only file) so empty splits are observable in the files list.
What's already strong

Verified strengths (10)

  1. Lineage integrity is excellent and verified live: 0 orphans across turns/candidates/selections, strict 1:1 candidate↔generation_attempt (no missing/duplicate/multi-attempt), 0 dangling parent/branch/FK links, contiguous turn_index from 0 with no gaps/dupes, and a complete selections_history audit log (53 rows/26 turns, no gaps).
  2. All four P0 quarantine invariants hold in code AND live RLS — the single most important property for a safety dataset. Steering, judge-panel, reaction, and disposition tables are all RLS deny-all (0 policies, service-role-only); no export loader/adapter/worker joins any quarantined table; the disposition answer-key is isolated on a separate deny-all table with compile-time + runtime leak guards.
  3. Reaction quarantine is protected by a genuinely adversarial test suite (8 bypass vectors — no-approval, eval_only, wrong bucket, unit-type/id collision, prediction/label confusion, revoke-via-supersede, stale is_current — plus a positive control proving the gate isn't trivially empty), not a smoke test.
  4. sample_n cohorts have textbook DPO shape: 1 comparison_context_hash / 1 interaction_prefix_hash / 1 sampling_context_hash / N distinct generation_attempt_hash / 1 shared generation_context — exactly what preference-pair training needs.
  5. Each implemented export format selects the CORRECT content variant: SFT/DPO/prompt_eval emit assistant_candidates.content (= reviewer_visible_response, which is raw model output for model rows and the corrected text for human_edit rows); classifier_train emits the raw classified response + post-classifier + verdict. No format emits a wrong variant (the suspected selected_content bug, A1-3, was refuted).
  6. Model attribution is fully consistent across columns and JSONB: model/provider/system_prompt_id agree across the row, api_request, and api_response_metadata on 35/35 model attempts; system_prompt_hash matches the registry body_hash on 35/35; 0 dangling system-prompt FKs.
  7. Predictive-profile data quality is high where populated: all 90 reaction_predictions have valid probability distributions whose class keys exactly match reaction_targets.classes, sum to ~1.0, with argmax==mode and predicted_confidence==max(prob); calibration is consistent; condition/mode enums clean; donor columns never exposed; all 9 labels in-class-space.
  8. NDJSON serialization is correct — proper UTF-8 and JSON escaping via JSON.stringify with trailing newlines.
  9. Human-edit and steering-promote provenance is captured correctly in the DB: parent_candidate_id + shared generation_context_id (Invariant 24) + edit_distance + edited_from_attempt_id + assist_model + reviewer_id all populated; was_steered is 100% consistent and backed by a CHECK constraint (was_steered ⇒ source=human_edit).
  10. Schema versioning, content-addressed hashing (interaction/comparison/sampling/prior_selection_chain hashes), and frozen-param enforcement (branches frozen-params trigger) are in place — the provenance scaffolding the formats need is present, even where the export layer doesn't yet surface it.
Generated by a 42-agent adversarial audit workflow · findings verified against canonical code, live DB, and specs · 2026-06-06