Data Lab · Quality Pass

18 rounds on the Data Lab

An adversarial × thermo-nuclear × live-browser verification loop over review.life-labs.dev — ~105 bugs fixed across 18 rounds, then stopped at diminishing returns (a deliberate call, not a clean-room finish).

baseline 6e033ed → final af1caf7  ·  2026-05-29  ·  20 PRs

At a glance

The headline numbers

~105
Confirmed bugs fixed
18
Verification rounds
20
PRs shipped + deployed
+17
New test files
1
CRITICAL · ~6 HIGH
0
Deferred (all found = fixed)

Honest final state on af1caf7  THERMO 0 VERIFY-UI CLEAN ADVERSARIAL — stopped at diminishing returns

This was not a formally-clean round. Thermo and the live-browser sweep came back clean, but the adversarial finder kept surfacing something every round — by rounds 14-18, only rare-path edges (several of them edges of the previous round's fix). Every finding was fixed and shipped (nothing deferred), but "adversarial finds zero" was never reached, because an agent instructed to find bugs essentially never returns empty. The run was stopped at round 18 by a deliberate diminishing-returns call.

What's solid: thermo clean, verify-ui clean, every high-frequency path works, ~105 real bugs fixed, 742 frontend + 117 Functions tests green, build + typecheck green, and every invariant held (test suite, the /api/data-lab/* proxy + auth contract, auto-deploy).

Effort curve

Lines changed per round

Strongly front-loaded. Rounds 1-8 reshaped the codebase (bug fixes + a structural simplification); rounds 9-18 are a thin tail of edge-case guards — each fix small, decaying toward noise. The amber divider marks where the loop dropped to adversarial-only (thermo + verify-ui were already clean).

Setup #74
+156 / −137
R1 #75
+2171 / −142
R1▸ #77
+1596 / −1334
R2 #78
+2472 / −208
R3 #79
+1637 / −117
R4 #80
+1011 / −208
R5 #81
+896 / −176
R6 #82
+741 / −97
R7 #83
+1044 / −102
R8 #84
+906 / −408
R9 #85
+71 / −0
R10 #86
+208 / −21
R11 #87
+268 / −54
R12 #88
+227 / −8
R13 #89
+140 / −6
↓ adversarial-only tail (thermo + verify-ui already clean)
R14 #90
+89 / −7
R15 #91
+77 / −13
R16 #92
+83 / −6
R17 #93
+26 / −9
R18 #94
+186 / −7
insertionsdeletions  — bars scaled to the 2,472-line peak (R2)
Discovery curve

Findings per round

Confirmed findings (adversarial + thermo) per round. ~80% of all bugs were caught in the first 8 rounds. The amber tail (14-18) is the diminishing-returns zone: real, but rare-path — and several were edges of the prior round's own fix.

R1
15
R2
17 + 3
R3
9
R4
12 + 2
R5
8
R6
8 + 3
R7
9
R8
5 + simp
R9
1
R10
3
R11
4
R12
3
R13
2
R14
1
R15
2
R16
2
R17
1
R18
1
Round by round

What each round found & fixed

CRITHIGHSECURITYSIMPLIFYSELF-REGRESSION — tags flag the standout findings. Tinted rows = the adversarial-only diminishing-returns tail.

RoundFocus & key findings+ / −files
Setup
49b1d67
De-flake the suite — kill cross-file isolation leaks, never hit the network in tests.+156
−137
5
R1
298707d
15 correctness / race / auth bugs + a new judge-panel history endpoint. The first deep sweep.+2171
−142
30
R1▸
8f121d6
SIMPLIFYStructural thermo cleanup — SSE transport extracted, server layer boundary fixed, the 2,083-line BranchView decomposed.+1596
−1334
17
R2
fd549c5
17 adversarial findings + 3 thermo cleanups — the single biggest haul.+2472
−208
30
R3
1e7e0d6
9 findings — steering SSE stream, judge retry, composer N/temperature, sweep aggregation.+1637
−117
20
R4
ce67eeb
12 adversarial findings + 2 thermo DRY consolidations.+1011
−208
26
R5
6d8ead5
SECURITYElevenLabs API key bundled into the browser (VITE_ leak) → moved behind a gated proxy. + sweep heatmap dead-404, +6 more.+896
−176
21
R6
e8c2cab
8 adversarial findings + 3 thermo nits.+741
−97
20
R7
8eee3df
HIGHSELF-REG9 findings incl. a HIGH retry-SSE regression from R6, + 2 HIGH dead-route / contract mismatches.+1044
−102
18
R8
4b60baf
CRITSELF-REGSIMPLIFYCRITICAL: R5's sweep-winrate supersession filter was inverted (kept corrected-away verdicts). + 5 dialogs → ModalShell, +4 more.+906
−408
24
R9
1a451b9
Bounded the judge-panel SSE reconnect — a bad service token (→ proxy 502) used to spin "reconnecting…" forever.+71
−0
2
R10
dec2ce3
SELF-REGR9's terminal message was dead on its target panel. + addComment-vs-commit race, + first-run tour hijacking the scenario deep-link.+208
−21
7
R11
2a254ac
SELF-REGInline CommentPanel double-submit (reopened by R10). + failed comment blanked the conversation, + stale cross-branch error, + sweep prompt UUIDs.+268
−54
6
R12
2e77d53
SELF-REGCommentsRail double-submit (sibling of R11). + reconnect cap defeated by 200-then-close, + AssessmentPanel superseded the wrong row.+227
−8
6
R13
4c95e60
ClassifierDisagreementDialog double-submit (3rd composer) + branch-tag add/remove flashing "Loading…".+140
−6
5
R14
26a2421
Disagreement dialog stale-closure auto-close silently swallowed a safety-judge-enqueue-failed warning. (Deferred when first found pre-stop, then shipped.)+89
−7
3
R15
f144e5a
HIGHSELF-REGR14's warning fix defeated by a refresh() flash that unmounted the dialog; + resubmit duplicated the label.+77
−13
3
R16
5ffbb91
SELF-REGEdited severity/note discarded on retry (lock fields); + disagreement chip not shown in rail until reload.+83
−6
3
R17
fb67ef4
HIGHJudge/Steer on an unselected single candidate was a silent no-op (cohort-aware context lookup). New area.+26
−9
1
R18
af1caf7
SELF-REGR17 only half-fixed it — the SteeringPanel still never mounted for an unselected candidate (gave it its own split).+186
−7
3
Before & after

Total code: initial branch vs. final

Tracked source across src/ · functions/ · server/src/ · supabase/ (TS/TSX/SQL). The net +10.9k lines (+15.1%) is dominated by tests and safety guards; R1's structural pass and R8's ModalShell consolidation account for most of the ~2.6k deletions.

Baseline · 6e033ed

72,493 lines
Source files (TS/TSX/SQL)72,493
Test files89

Final · af1caf7

83,433 lines
Source lines+10,940
Test files106 (+17)
+13,545
Total insertions
−2,600
Total deletions
+10,945
Net lines
113
Files touched
What's interesting

Patterns worth noting

🎯 Three independent verifiers

Each round cross-checked the same commit from three angles: an adversarial bug-hunt (fan-out finders → 2 independent skeptics must refute each finding before it's confirmed), a thermo-nuclear maintainability audit, and verify-life (headless Chromium driving the real CF-Access-gated site).

📉 Textbook convergence

Findings per round: 15 → 17 → 9 → 12 → 8 → 8 → 9 → 5 → 1 → 3 → 4 → 3 → 2 → 1 → 2 → 2 → 1 → 1. The first 8 rounds caught ~80% of all bugs; the rest was a long, shallow tail.

♻️ Fixes that bred edges

The tail's length had one main cause: a fix often created (or revealed) an adjacent gap, caught the next round. Eight instances:

  • R5 sweep filter → R8 CRITICAL inversion
  • R6 retry → R7 HIGH SSE regression
  • R9 reconnect msg → R10 dead-on-surface
  • R10 mutating decouple → R11 CommentPanel
  • R11 → R12 CommentsRail · R12 → R13 ClassifierDialog
  • R14 warning → R15 refresh-flash · R17 → R18 steer panel

🔁 The double-submit trilogy

One bug class — a same-tick double-click firing two writes — surfaced across three sibling composers (CommentPanel, CommentsRail, ClassifierDisagreementDialog), one per round (R11→R13). All three now share the same synchronous postingRef guard.

🧪 Tests grew faster than features

Test files went 89 → 106 (+17) and the live suite ended at 742 frontend + 117 Functions = 859 tests, all green. Every fix landed test-first (TDD), so each round is a permanent regression net.

🛡️ Severity mix

1 CRITICAL (training-data corruption — corrected-away verdicts kept in the sweep) and ~6 HIGH (a bundled API-key leak, an SSE regression, dead-route/contract mismatches, an assessment unique-index violation, two unselected-candidate panel no-ops). The rest were MEDIUM/LOW races, error-handling, and UX-integrity issues.

The takeaway

Why it never "finished" — and the fix for next time

The stop condition was "loop until the adversarial finds nothing." But an adversarial agent instructed to find bugs is asymmetric — it almost never returns empty. It will always surface something: a rare-path edge, a theoretical race, or (rounds 15/16/18) an edge of the fix it just suggested. So "zero findings" is effectively unreachable, and the loop runs until a human stops it. The verifiers that can hit zero (thermo, verify-ui) did; the one that structurally can't defined "done."

The fix: define done by an impact bar + a convergence signal, not by "zero of anything." Findings below the bar go to a backlog, not into the fix loop — which makes termination reachable (the loop ends when no above-bar findings remain, which actually happens).

Don't make "the agent finds zero bugs" the finish line. Make "no bug worth shipping a fix for, twice in a row" the finish line — and send everything below that to a backlog. That stops at diminishing returns, but not before: a genuine HIGH on a common path always clears the bar and keeps it going.

Held throughout

Guardrails that never broke

Across all 20 PRs, every deploy kept these green: the test suite + typecheck + Cloudflare Pages build; the /api/data-lab/* proxy & its CF-Access / service-token auth contract (untouched); the auto-deploy pipeline; and the Data Lab quarantine invariants (steering & judge-panel tables never join exports). Each PR shipped only after a local green gate, then auto-deployed and was re-verified live.