# Codex Ultra Review — Waypoint Estimator Design

_Model: gpt-5.6-sol (OpenAI Codex CLI v0.144.0) — reasoning effort: ultra — generated 2026-07-16._

_Target docs: estimator-design-memo.md, fable-max-bayesian-profiling-eval.md, waypoint-practitioner-stage-assessment.md._

---

## Overall assessment: needs revision

The design has good engineering bones—typed evidence, deterministic arithmetic, provenance, abstention, proper-score ambitions, and estimate-blind extraction—but it is not yet a coherent Bayesian state-space model. It is an elicited, tempered scorecard combining several Bayesian-looking components with posterior-dependent heuristics.

At \(N=5\)–10, M4 can establish implementation fidelity, gross discrimination failures, and within-rater concordance. It cannot establish population calibration, learn partial pooling, validate dynamics, or separate stage from rater/extractor error.

## 1. Unsound or incoherent update rules

### 1. The pillar coupling is not “exact” longitudinal inference

The model declares \(\delta_k\) static, but independently diffuses each pillar before recoupling it. Those cannot both be true:

- If \(\theta_k(t)=\theta_o(t)+\delta_k\) with static \(\delta_k\), pillar movement must be induced by \(\theta_o\).
- If each \(\theta_k\) gets its own \(Q\), then \(\delta_k\) is dynamic.
- A static \(\delta_k\) shared across time makes the longitudinal graph more than the one-slice tree claimed in [§4.5](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:173).
- After coupling, each redistributed pillar marginal contains other pillars’ evidence. Treating that marginal as local evidence at the next coupling double-counts unless the implementation preserves cavity messages/local likelihood factors.

The clamp also breaks the displayed equation: at a boundary, several offset values map to the same clamped \(\theta_k\), but \(P(\delta_k=\theta_k-\theta_o)\) counts only one.

**Fix:** either maintain a joint longitudinal posterior over \((\theta_o,\delta_{1:4})\), propagating only \(\theta_o\), or drop static offsets and specify a multivariate pillar transition. Impose \(\sum_k\delta_k=0\) if “overall” is meant to be the pillar center.

### 2. Overall stage is only prior-identified

Pillar evidence depends on \(\theta_k=\theta_o+\delta_k\). Therefore

\[
\theta_o'=\theta_o+c,\qquad \delta_k'=\delta_k-c
\]

leaves the pillar likelihood unchanged except at boundaries. Unless cross-pillar markers are abundant, the centered offset prior—not the dossier—locates overall stage.

The claimed archetype neutrality is also false operationally: if breathwork shifts the sensations offset prior by \(+3\), observing sensations at 40 implies overall near 37 rather than 40. Archetype changes overall inference even when \(p(\theta_o)\) is unchanged.

**Fix:** constrain offsets to sum to zero, require direct cross-pillar anchors, and report the proportion of overall information coming from direct evidence versus offset priors. For v1, deriving overall transparently from reported pillars may be safer.

### 3. Phase gating uses evidence twice and can become self-sealing

The algorithm updates phase first and then uses that updated phase to choose the weight of stage evidence from the same session. The correct joint update is based on

\[
p(e_t\mid\theta_t,h_{t-1})
 =\sum_\phi p(e_t\mid\theta_t,\phi_t)\,p(\phi_t\mid h_{t-1}),
\]

with phase and stage updated simultaneously. It is not generally

\[
L(e_t\mid\theta_t)^{E[\operatorname{gate}(\phi_t)]}.
\]

A regression-like session can raise \(P(\text{dip})\), which then discounts the same down-band evidence to 0.3. Sustained true regression can consequently keep classifying itself as “dip” and muting its own evidence.

**Fix:** use a joint factorial HMM/hidden semi-Markov likelihood \(P(e\mid\theta,\phi)\). For M4, the safer choice is to report phase separately and not let it modify stage likelihoods.

### 4. KL caps and persistent-run overrides are data-dependent power likelihoods

The [KL caps](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:165) select a tempering exponent after seeing the evidence. This makes results depend on arbitrary batching:

- Three short sessions can unlock more total movement than identical text in one session.
- Reordering sessions can change the result.
- “Same direction” is undefined for bump-shaped likelihoods or multimodal posteriors.
- Contradictory strong evidence can produce little net KL and escape the cap.

This is not ordinary Bayes. It is not even the standard coherent generalized-Bayes construction, which uses a prespecified additive loss and learning rate; see [Bissiri, Holmes & Walker](https://arxiv.org/abs/1306.6430).

**Fix:** replace caps with an explicit contamination model:

\[
p(e\mid\theta)=(1-\epsilon)L_{\text{signal}}(e\mid\theta)
+\epsilon L_{\text{noise}}(e),
\]

plus episode/source random effects. If caps remain as safety engineering, call the output a “tempered evidence score,” not a calibrated Bayesian posterior.

### 5. The presence-only likelihood lacks an opportunity model

The memo elicits \(f_m(g)=P(\text{marker appears in a typical month}\mid g)\), applies it to selected positive events, ignores non-fires, and applies fractional powers to repeats [§3.3](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:111).

That is only a valid partial likelihood under strong, unstated conditions about opportunity, selection, detection, and conditional independence. Presently it confounds stage with:

- conversation volume and window length;
- prompt/probe policy;
- practitioner expressiveness;
- extractor recall;
- the number of markers in the bank.

Merging repeated narratives does not address different markers caused by the same latent episode.

**Fix:** define observable opportunities. For invited probes, model every response category, including non-response. For passive evidence, use a count/point-process model such as

\[
N_{mw}\sim\operatorname{Poisson}(a_w d_m\lambda_m(\theta)),
\]

where \(a_w\) is opportunity and \(d_m\) extractor detection. If opportunity is unreliable, condition on the total emitted count and use a marked-multinomial likelihood. Cluster evidence by episode/marker family and permit one robust contribution per cluster.

### 6. The movement prior advances practitioners without evidence

With the proposed rates,

\[
E[\Delta\theta]/30\text{ days}
=(0.03\iota-0.005)\times30.
\]

That is:

- \(+0.75\) points/month at ordinary practice;
- \(+1.65\) at \(\iota=2\);
- still \(+0.12\) during “silence” at \(\iota=0.3\).

Under natural reflecting boundaries, the stationary adjacent-state ratio is \(0.03/0.005=6\), eventually concentrating roughly \(5/6\) of mass at 70. Silence also has lower absolute variance than active practice because only the forward rate is reduced.

Moreover, \(\exp(Q\Delta t)\) assigns nonzero probability to every distant state for any \(\Delta t>0\); it makes teleporting very unlikely, not impossible.

**Fix:** separate drift from volatility, use near-zero or state-dependent drift unless prospectively supported, and specify boundary behavior. Long plateaus and rare changes are better represented by an explicit-duration or change-point model than a permanently upward-biased random walk.

The toy consolidation also orders operations incorrectly. Events on days 2, 9, and 24 require

\[
K_2L_1K_7L_2K_{15}L_4K_6,
\]

not applying \(L_1L_2L_4\) followed by one \(K_{30}\). Transition and emission matrices do not commute.

### 7. Milestones and claims reuse observations

The same claim/telemetry observation can currently:

1. update stage through a guard marker;
2. update milestone probability;
3. increment claim reliability;
4. later feed the milestone back into stage.

That is multiple likelihood factors for one datum. Proper milestone messaging is

\[
L_{\text{milestone}}(\theta)
=\sum_m P(y\mid m)P(m\mid\theta),
\]

applied once. Crossing the 0.8 display threshold is not a new inference event.

The elicitation question “among claimants, how many truly reached it?” gives \(P(m=1\mid\text{claim})\), not the BKT quantities \(P(\text{claim}\mid m=0)\) and \(P(\text{no claim}\mid m=1)\).

Likewise, using \(L(\theta)^{E[r]}\) is not Bayesian marginalization. A coherent reliability model is a mixture such as

\[
P(e\mid\theta,r)
=rL_{\text{valid}}(e\mid\theta)
+(1-r)L_{\text{contamination}}(e).
\]

**Fix:** route every underlying datum into one joint factor. For v1, remove adaptive per-person reliability entirely; use a fixed conservative claim likelihood and an explicit `disputed` state.

### 8. “Evidence bits” are not absorbed information

The sufficiency rule sums each marker’s maximum adjacent-band log-likelihood ratio [§7](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:216). That is potential discriminative capacity, not realized information.

The toy example calls three events **3.97 bits**, yet the actual KL change from the carried prior to the evidence-only posterior is approximately **0.19 bits**, falling to **0.13 bits** after the kernel. The memo’s own example therefore passes the evidence threshold by a quantity about twenty times the realized posterior information.

Two opposite two-bit markers can also sum to four “evidence bits” while returning the posterior exactly to its prior.

**Fix:** require:

- a minimum number of independent episodes/sessions/channels;
- \(D_{\mathrm{KL}}(p_{\text{full}}\Vert p_{\text{no-data predictive}})\);
- leave-one-episode-out stability;
- a posterior-predictive conflict check.

Do not count guard contradictions as positive evidence mass.

## 2. Identifiability at \(N=5\)–10

At this sample size, the following are not empirically separable:

- stage versus phase;
- overall stage versus common pillar offset;
- movement rate versus emission error;
- archetype DIF versus genuine stage difference;
- practitioner expressiveness versus marker prevalence;
- extractor error versus rater error;
- rater bias versus latent truth.

The 56-point grid worsens the appearance of identification. Five coarse elicitation anchors and 5–10 narrow-range subjects cannot support point-level claims such as 36 versus 38. Narrow credible intervals would be produced by interpolation, fixed parameters, and transition assumptions—not by empirical resolution.

**Concrete changes:**

- Use three or four ordinal regions within the observed range for M4; do not report sub-band point estimates.
- Hold stage static within the M4 dossier window.
- Treat phase and milestones as separate outputs.
- Require at least three informative pillars or cross-pillar anchors before reporting overall.
- Scope every M4 conclusion to “map-exposed team members in the observed range.” It provides no validation of naive columns, high stages, rare milestones, or high-stage safety flags.

## 3. Where partial pooling fails

The memo is candid that nothing hierarchical is fitted. Consequently, v1 does not perform empirical partial pooling; it applies fixed expert priors and totally pooled likelihood tables.

Specific failures:

- Eight path-by-exposure cells with 5–10 people means most cells contain zero or one person.
- Everyone being map-exposed makes the naive/exposed contrast unidentified.
- Shared likelihood rows assume measurement invariance across practitioners.
- Path-conditioned likelihoods destroy cross-path comparability unless invariant anchor markers exist.
- `KL(individual || archetype prior)` mainly measures evidence volume and likelihood strength, not valid individuation.
- A wrong-archetype swap must differ because the model was hard-coded to differ; that is sensitivity, not proof against stereotyping.

Measurement-invariance work requires anchor items precisely because DIF otherwise changes the latent scale itself; see the primary study by [Belzak and Bauer](https://pmc.ncbi.nlm.nih.gov/articles/PMC7343596/).

**Fix:** remove archetype shifts from the primary v1 estimator. Run them as sensitivity analyses only. Later promote an archetype adjustment only if it improves locked, held-out prediction over an archetype-agnostic model using invariant anchors.

## 4. Miscalibration risks

The reported posterior propagates state uncertainty while treating hundreds of elicited values as known: marker curves, \(Q\), offsets, phase gates, milestone curves, confidence temperatures, and extractor behavior. The resulting 80% interval is therefore conditional on the whole elicited system being correct.

Independent ±20/40/60% sensitivity perturbations do not capture correlated expert error. Surya may systematically place every “insight” marker too high, or overestimate all telemetry discriminators together.

Other important risks:

- Confidence shrinkage appears in both compilation and the event exponent, creating a possible double-tempering bug.
- Adjacent-band caps permit much larger cumulative Bayes factors across several bands.
- A stochastic matrix raised to fractional \(\Delta t/7\) need not remain real and stochastic; use a continuous-time phase generator.
- The posterior-aware consolidation pass can add missed markers while seeing prior snapshots, so the canonical evidence channel is not actually estimate-blind.
- Different LLM families do not remove common-method correlation when they share the marker bank, dossier, and cultural cues.

**Fix:** represent elicited log-likelihood ratios as constrained distributions with marker-family-level correlated uncertainty, then integrate or ensemble over global parameter draws. Split consolidation into an estimate-blind evidence pass and a posterior-aware rationale pass that cannot add, remove, or reweight evidence.

## 5. What breaks under one expert rater

### Direct target leakage

The parent spec says the gold interview is “recorded and itself ingested as evidence” [§7](/Users/fionnenglish/LIFE/specs/waypoint-practitioner-stage-assessment.md:95), while the estimator is run after the interview. Unless the evaluated dossier is explicitly frozen beforehand, the estimator consumes the interaction that generated its reference label.

**Fix:** hash and freeze the dossier and estimator forecast before the interview. The interview transcript may enter only a separately named post-evaluation condition.

### The rater defines both ruler and answer key

Surya sets the priors, marker curves, dynamics, gates, milestone parameters, gold tolerance, and gold labels. Agreement can therefore establish consistency with Surya’s encoded theory, not external validity.

Freeze all tables before labels are collected. Never revise on M4 residuals and rescore M4 as if it remained validation.

### Interview-versus-dossier is not an information ceiling

Both ratings share the same expert, ontology, practitioner familiarity, thresholds, and memory. Their difference mixes extra information with intra-rater noise, mode effects, carryover, interviewer adaptation, and possible temporal change. An estimator can even agree with the interview better than the same rater’s dossier rating through regularization, so it is not a mathematical ceiling.

Call it a **within-rater mode discrepancy**.

### This is not an embryonic Hui–Walter design

Hui–Walter identification relies on conditions including populations with different prevalences and conditionally independent tests. Here the methods share a rater, evidence, taxonomy, and self-report channel; the target is also ordinal and multidimensional. Four readings do not create four independent tests. Those assumptions are central to the original framework and later identifiability work; see [Hui & Walter](https://doi.org/10.2307/2530508) and this [latent-class identifiability analysis](https://pmc.ncbi.nlm.nih.gov/articles/PMC7036323/).

### Gold smearing can make calibration look better

Applying a generic adjacent-band tolerance after a point rating changes the target to a constructed blur, favors diffuse forecasts, and can conceal systematic one-band bias. It remains a score of agreement with that blur, not calibration to latent stage.

**Fix:** have raters directly allocate probabilities across bands at rating time. With multiple raters, use rater-specific ordinal thresholds and noise; ordinal rater models themselves require identifying restrictions and repeated/multiple ratings, as illustrated in this [Bayesian ordinal rater-bias model](https://pmc.ncbi.nlm.nih.gov/articles/PMC8277718/).

If only Surya is available, collect randomized repeated ratings and hidden anchor vignettes, and report sensitivity under plausible rater confusion matrices. No statistical model can identify systematic single-rater bias from these data alone.

## 6. Validation claims that \(N=5\)–10 cannot support

Pillars, milestones, and rounds do not turn ten practitioners into 40–80 independent calibration cases. They share the practitioner, dossier, extractor, latent stage, and rater.

Even granting independent practitioners:

- \(8/10\) interval coverage has an exact 95% binomial interval of approximately **44%–97%**.
- \(4/5\) has approximately **28%–99%**.

Therefore:

- Drop the three-bin reliability analysis at M4.
- Treat coverage counts as descriptive only.
- Do not rely on paired bootstrap precision; show every practitioner’s paired score difference.
- Replace weighted \(\kappa\) with case-level RPS, signed ordinal error, and complete case displays.
- Make the practitioner—not pillars or events—the uncertainty/resampling unit.

Synthetic vignettes also cannot demonstrate 80% coverage unless latent states and observations are sampled from the actual stated generative model. Hand-authored personas are scenario tests, not calibration samples. Use formal simulation-based calibration for implementation checking, following [Talts et al.](https://arxiv.org/abs/1804.06788), and keep adversarial vignettes as separate behavioral unit tests.

H1 is the most promising validation channel, but forecasts must be immutable and pre-outcome. Both estimator and null must receive the same selected probe and context. Randomize a small audit fraction of eligible probes so posterior-driven item selection cannot manufacture favorable apparent skill.

## 7. Better model classes

### Defensible v1 at this sample size

Use a static, coarse Bayesian ordinal scorecard per pillar:

1. Three or four observable-range states.
2. Standardized probes with complete categorical likelihoods and known opportunity.
3. Passive markers treated as weak, episode-clustered robust Bayes factors.
4. Expert log-Bayes factors represented by distributions, not fixed points.
5. No stage dynamics, archetype shifts, adaptive claim reliability, or milestone-to-stage feedback.
6. Phase and milestones reported separately.
7. Overall either omitted or derived under an explicit sum-to-zero/weighted aggregation rule.
8. Report robustness envelopes across parameter draws, not nominally calibrated 80% intervals.

For tiny-\(N\) learning, a Beta-binomial or Dirichlet-multinomial population-to-individual model over repeated probe outcomes is more identifiable. Stage should earn its place by improving future predictive log score over this direct behavioral model.

### Later model, once data justify it

Use a joint dynamic ordinal multi-trait/multi-method model containing:

- stage and pillar latents;
- rater-specific ordinal thresholds;
- extractor false-positive/false-negative channels;
- explicit opportunity and probe-selection models;
- invariant anchor probes plus path/exposure DIF;
- episode-level dependence;
- rare change points or explicit-duration phase/stage states;
- irreversible first-passage milestone states.

A hidden semi-Markov model is a better structural match for long plateaus and explicit phase durations than a birth–death CTMC plus KL overrides; see [Yu’s HSMM formulation](https://www.sciencedirect.com/science/article/pii/S0004370209001416). But no richer model cures absent raters, missing anchors, or an unrepresentative cohort.

## The three changes with highest expected value

1. **Eliminate evaluation circularity:** freeze the pre-interview dossier and predictions, prohibit gold-interview evidence from the evaluated run, add repeated probabilistic ratings and at least one genuinely independent blinded rater, and redefine M4 as a feasibility/reliability case series.

2. **Replace the heuristic updater with one coherent observation model:** known opportunity or count-based emissions, episode clustering, joint phase handling, robust contamination mixtures, and one likelihood contribution per underlying datum; remove KL/run caps and adaptive Beta claim weighting.

3. **Radically simplify v1 for \(N=5\)–10:** coarse static pillar states, uncertain expert Bayes factors, no archetype effects or stage dynamics, and an identified or explicitly derived overall score; add hierarchical dynamics only after locked held-out prediction demonstrates incremental value.
tokens used
109,843
## Overall assessment: needs revision

The design has good engineering bones—typed evidence, deterministic arithmetic, provenance, abstention, proper-score ambitions, and estimate-blind extraction—but it is not yet a coherent Bayesian state-space model. It is an elicited, tempered scorecard combining several Bayesian-looking components with posterior-dependent heuristics.

At \(N=5\)–10, M4 can establish implementation fidelity, gross discrimination failures, and within-rater concordance. It cannot establish population calibration, learn partial pooling, validate dynamics, or separate stage from rater/extractor error.

## 1. Unsound or incoherent update rules

### 1. The pillar coupling is not “exact” longitudinal inference

The model declares \(\delta_k\) static, but independently diffuses each pillar before recoupling it. Those cannot both be true:

- If \(\theta_k(t)=\theta_o(t)+\delta_k\) with static \(\delta_k\), pillar movement must be induced by \(\theta_o\).
- If each \(\theta_k\) gets its own \(Q\), then \(\delta_k\) is dynamic.
- A static \(\delta_k\) shared across time makes the longitudinal graph more than the one-slice tree claimed in [§4.5](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:173).
- After coupling, each redistributed pillar marginal contains other pillars’ evidence. Treating that marginal as local evidence at the next coupling double-counts unless the implementation preserves cavity messages/local likelihood factors.

The clamp also breaks the displayed equation: at a boundary, several offset values map to the same clamped \(\theta_k\), but \(P(\delta_k=\theta_k-\theta_o)\) counts only one.

**Fix:** either maintain a joint longitudinal posterior over \((\theta_o,\delta_{1:4})\), propagating only \(\theta_o\), or drop static offsets and specify a multivariate pillar transition. Impose \(\sum_k\delta_k=0\) if “overall” is meant to be the pillar center.

### 2. Overall stage is only prior-identified

Pillar evidence depends on \(\theta_k=\theta_o+\delta_k\). Therefore

\[
\theta_o'=\theta_o+c,\qquad \delta_k'=\delta_k-c
\]

leaves the pillar likelihood unchanged except at boundaries. Unless cross-pillar markers are abundant, the centered offset prior—not the dossier—locates overall stage.

The claimed archetype neutrality is also false operationally: if breathwork shifts the sensations offset prior by \(+3\), observing sensations at 40 implies overall near 37 rather than 40. Archetype changes overall inference even when \(p(\theta_o)\) is unchanged.

**Fix:** constrain offsets to sum to zero, require direct cross-pillar anchors, and report the proportion of overall information coming from direct evidence versus offset priors. For v1, deriving overall transparently from reported pillars may be safer.

### 3. Phase gating uses evidence twice and can become self-sealing

The algorithm updates phase first and then uses that updated phase to choose the weight of stage evidence from the same session. The correct joint update is based on

\[
p(e_t\mid\theta_t,h_{t-1})
 =\sum_\phi p(e_t\mid\theta_t,\phi_t)\,p(\phi_t\mid h_{t-1}),
\]

with phase and stage updated simultaneously. It is not generally

\[
L(e_t\mid\theta_t)^{E[\operatorname{gate}(\phi_t)]}.
\]

A regression-like session can raise \(P(\text{dip})\), which then discounts the same down-band evidence to 0.3. Sustained true regression can consequently keep classifying itself as “dip” and muting its own evidence.

**Fix:** use a joint factorial HMM/hidden semi-Markov likelihood \(P(e\mid\theta,\phi)\). For M4, the safer choice is to report phase separately and not let it modify stage likelihoods.

### 4. KL caps and persistent-run overrides are data-dependent power likelihoods

The [KL caps](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:165) select a tempering exponent after seeing the evidence. This makes results depend on arbitrary batching:

- Three short sessions can unlock more total movement than identical text in one session.
- Reordering sessions can change the result.
- “Same direction” is undefined for bump-shaped likelihoods or multimodal posteriors.
- Contradictory strong evidence can produce little net KL and escape the cap.

This is not ordinary Bayes. It is not even the standard coherent generalized-Bayes construction, which uses a prespecified additive loss and learning rate; see [Bissiri, Holmes & Walker](https://arxiv.org/abs/1306.6430).

**Fix:** replace caps with an explicit contamination model:

\[
p(e\mid\theta)=(1-\epsilon)L_{\text{signal}}(e\mid\theta)
+\epsilon L_{\text{noise}}(e),
\]

plus episode/source random effects. If caps remain as safety engineering, call the output a “tempered evidence score,” not a calibrated Bayesian posterior.

### 5. The presence-only likelihood lacks an opportunity model

The memo elicits \(f_m(g)=P(\text{marker appears in a typical month}\mid g)\), applies it to selected positive events, ignores non-fires, and applies fractional powers to repeats [§3.3](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:111).

That is only a valid partial likelihood under strong, unstated conditions about opportunity, selection, detection, and conditional independence. Presently it confounds stage with:

- conversation volume and window length;
- prompt/probe policy;
- practitioner expressiveness;
- extractor recall;
- the number of markers in the bank.

Merging repeated narratives does not address different markers caused by the same latent episode.

**Fix:** define observable opportunities. For invited probes, model every response category, including non-response. For passive evidence, use a count/point-process model such as

\[
N_{mw}\sim\operatorname{Poisson}(a_w d_m\lambda_m(\theta)),
\]

where \(a_w\) is opportunity and \(d_m\) extractor detection. If opportunity is unreliable, condition on the total emitted count and use a marked-multinomial likelihood. Cluster evidence by episode/marker family and permit one robust contribution per cluster.

### 6. The movement prior advances practitioners without evidence

With the proposed rates,

\[
E[\Delta\theta]/30\text{ days}
=(0.03\iota-0.005)\times30.
\]

That is:

- \(+0.75\) points/month at ordinary practice;
- \(+1.65\) at \(\iota=2\);
- still \(+0.12\) during “silence” at \(\iota=0.3\).

Under natural reflecting boundaries, the stationary adjacent-state ratio is \(0.03/0.005=6\), eventually concentrating roughly \(5/6\) of mass at 70. Silence also has lower absolute variance than active practice because only the forward rate is reduced.

Moreover, \(\exp(Q\Delta t)\) assigns nonzero probability to every distant state for any \(\Delta t>0\); it makes teleporting very unlikely, not impossible.

**Fix:** separate drift from volatility, use near-zero or state-dependent drift unless prospectively supported, and specify boundary behavior. Long plateaus and rare changes are better represented by an explicit-duration or change-point model than a permanently upward-biased random walk.

The toy consolidation also orders operations incorrectly. Events on days 2, 9, and 24 require

\[
K_2L_1K_7L_2K_{15}L_4K_6,
\]

not applying \(L_1L_2L_4\) followed by one \(K_{30}\). Transition and emission matrices do not commute.

### 7. Milestones and claims reuse observations

The same claim/telemetry observation can currently:

1. update stage through a guard marker;
2. update milestone probability;
3. increment claim reliability;
4. later feed the milestone back into stage.

That is multiple likelihood factors for one datum. Proper milestone messaging is

\[
L_{\text{milestone}}(\theta)
=\sum_m P(y\mid m)P(m\mid\theta),
\]

applied once. Crossing the 0.8 display threshold is not a new inference event.

The elicitation question “among claimants, how many truly reached it?” gives \(P(m=1\mid\text{claim})\), not the BKT quantities \(P(\text{claim}\mid m=0)\) and \(P(\text{no claim}\mid m=1)\).

Likewise, using \(L(\theta)^{E[r]}\) is not Bayesian marginalization. A coherent reliability model is a mixture such as

\[
P(e\mid\theta,r)
=rL_{\text{valid}}(e\mid\theta)
+(1-r)L_{\text{contamination}}(e).
\]

**Fix:** route every underlying datum into one joint factor. For v1, remove adaptive per-person reliability entirely; use a fixed conservative claim likelihood and an explicit `disputed` state.

### 8. “Evidence bits” are not absorbed information

The sufficiency rule sums each marker’s maximum adjacent-band log-likelihood ratio [§7](/Users/fionnenglish/LIFE/specs/waypoint-research/deliverables/estimator-design-memo.md:216). That is potential discriminative capacity, not realized information.

The toy example calls three events **3.97 bits**, yet the actual KL change from the carried prior to the evidence-only posterior is approximately **0.19 bits**, falling to **0.13 bits** after the kernel. The memo’s own example therefore passes the evidence threshold by a quantity about twenty times the realized posterior information.

Two opposite two-bit markers can also sum to four “evidence bits” while returning the posterior exactly to its prior.

**Fix:** require:

- a minimum number of independent episodes/sessions/channels;
- \(D_{\mathrm{KL}}(p_{\text{full}}\Vert p_{\text{no-data predictive}})\);
- leave-one-episode-out stability;
- a posterior-predictive conflict check.

Do not count guard contradictions as positive evidence mass.

## 2. Identifiability at \(N=5\)–10

At this sample size, the following are not empirically separable:

- stage versus phase;
- overall stage versus common pillar offset;
- movement rate versus emission error;
- archetype DIF versus genuine stage difference;
- practitioner expressiveness versus marker prevalence;
- extractor error versus rater error;
- rater bias versus latent truth.

The 56-point grid worsens the appearance of identification. Five coarse elicitation anchors and 5–10 narrow-range subjects cannot support point-level claims such as 36 versus 38. Narrow credible intervals would be produced by interpolation, fixed parameters, and transition assumptions—not by empirical resolution.

**Concrete changes:**

- Use three or four ordinal regions within the observed range for M4; do not report sub-band point estimates.
- Hold stage static within the M4 dossier window.
- Treat phase and milestones as separate outputs.
- Require at least three informative pillars or cross-pillar anchors before reporting overall.
- Scope every M4 conclusion to “map-exposed team members in the observed range.” It provides no validation of naive columns, high stages, rare milestones, or high-stage safety flags.

## 3. Where partial pooling fails

The memo is candid that nothing hierarchical is fitted. Consequently, v1 does not perform empirical partial pooling; it applies fixed expert priors and totally pooled likelihood tables.

Specific failures:

- Eight path-by-exposure cells with 5–10 people means most cells contain zero or one person.
- Everyone being map-exposed makes the naive/exposed contrast unidentified.
- Shared likelihood rows assume measurement invariance across practitioners.
- Path-conditioned likelihoods destroy cross-path comparability unless invariant anchor markers exist.
- `KL(individual || archetype prior)` mainly measures evidence volume and likelihood strength, not valid individuation.
- A wrong-archetype swap must differ because the model was hard-coded to differ; that is sensitivity, not proof against stereotyping.

Measurement-invariance work requires anchor items precisely because DIF otherwise changes the latent scale itself; see the primary study by [Belzak and Bauer](https://pmc.ncbi.nlm.nih.gov/articles/PMC7343596/).

**Fix:** remove archetype shifts from the primary v1 estimator. Run them as sensitivity analyses only. Later promote an archetype adjustment only if it improves locked, held-out prediction over an archetype-agnostic model using invariant anchors.

## 4. Miscalibration risks

The reported posterior propagates state uncertainty while treating hundreds of elicited values as known: marker curves, \(Q\), offsets, phase gates, milestone curves, confidence temperatures, and extractor behavior. The resulting 80% interval is therefore conditional on the whole elicited system being correct.

Independent ±20/40/60% sensitivity perturbations do not capture correlated expert error. Surya may systematically place every “insight” marker too high, or overestimate all telemetry discriminators together.

Other important risks:

- Confidence shrinkage appears in both compilation and the event exponent, creating a possible double-tempering bug.
- Adjacent-band caps permit much larger cumulative Bayes factors across several bands.
- A stochastic matrix raised to fractional \(\Delta t/7\) need not remain real and stochastic; use a continuous-time phase generator.
- The posterior-aware consolidation pass can add missed markers while seeing prior snapshots, so the canonical evidence channel is not actually estimate-blind.
- Different LLM families do not remove common-method correlation when they share the marker bank, dossier, and cultural cues.

**Fix:** represent elicited log-likelihood ratios as constrained distributions with marker-family-level correlated uncertainty, then integrate or ensemble over global parameter draws. Split consolidation into an estimate-blind evidence pass and a posterior-aware rationale pass that cannot add, remove, or reweight evidence.

## 5. What breaks under one expert rater

### Direct target leakage

The parent spec says the gold interview is “recorded and itself ingested as evidence” [§7](/Users/fionnenglish/LIFE/specs/waypoint-practitioner-stage-assessment.md:95), while the estimator is run after the interview. Unless the evaluated dossier is explicitly frozen beforehand, the estimator consumes the interaction that generated its reference label.

**Fix:** hash and freeze the dossier and estimator forecast before the interview. The interview transcript may enter only a separately named post-evaluation condition.

### The rater defines both ruler and answer key

Surya sets the priors, marker curves, dynamics, gates, milestone parameters, gold tolerance, and gold labels. Agreement can therefore establish consistency with Surya’s encoded theory, not external validity.

Freeze all tables before labels are collected. Never revise on M4 residuals and rescore M4 as if it remained validation.

### Interview-versus-dossier is not an information ceiling

Both ratings share the same expert, ontology, practitioner familiarity, thresholds, and memory. Their difference mixes extra information with intra-rater noise, mode effects, carryover, interviewer adaptation, and possible temporal change. An estimator can even agree with the interview better than the same rater’s dossier rating through regularization, so it is not a mathematical ceiling.

Call it a **within-rater mode discrepancy**.

### This is not an embryonic Hui–Walter design

Hui–Walter identification relies on conditions including populations with different prevalences and conditionally independent tests. Here the methods share a rater, evidence, taxonomy, and self-report channel; the target is also ordinal and multidimensional. Four readings do not create four independent tests. Those assumptions are central to the original framework and later identifiability work; see [Hui & Walter](https://doi.org/10.2307/2530508) and this [latent-class identifiability analysis](https://pmc.ncbi.nlm.nih.gov/articles/PMC7036323/).

### Gold smearing can make calibration look better

Applying a generic adjacent-band tolerance after a point rating changes the target to a constructed blur, favors diffuse forecasts, and can conceal systematic one-band bias. It remains a score of agreement with that blur, not calibration to latent stage.

**Fix:** have raters directly allocate probabilities across bands at rating time. With multiple raters, use rater-specific ordinal thresholds and noise; ordinal rater models themselves require identifying restrictions and repeated/multiple ratings, as illustrated in this [Bayesian ordinal rater-bias model](https://pmc.ncbi.nlm.nih.gov/articles/PMC8277718/).

If only Surya is available, collect randomized repeated ratings and hidden anchor vignettes, and report sensitivity under plausible rater confusion matrices. No statistical model can identify systematic single-rater bias from these data alone.

## 6. Validation claims that \(N=5\)–10 cannot support

Pillars, milestones, and rounds do not turn ten practitioners into 40–80 independent calibration cases. They share the practitioner, dossier, extractor, latent stage, and rater.

Even granting independent practitioners:

- \(8/10\) interval coverage has an exact 95% binomial interval of approximately **44%–97%**.
- \(4/5\) has approximately **28%–99%**.

Therefore:

- Drop the three-bin reliability analysis at M4.
- Treat coverage counts as descriptive only.
- Do not rely on paired bootstrap precision; show every practitioner’s paired score difference.
- Replace weighted \(\kappa\) with case-level RPS, signed ordinal error, and complete case displays.
- Make the practitioner—not pillars or events—the uncertainty/resampling unit.

Synthetic vignettes also cannot demonstrate 80% coverage unless latent states and observations are sampled from the actual stated generative model. Hand-authored personas are scenario tests, not calibration samples. Use formal simulation-based calibration for implementation checking, following [Talts et al.](https://arxiv.org/abs/1804.06788), and keep adversarial vignettes as separate behavioral unit tests.

H1 is the most promising validation channel, but forecasts must be immutable and pre-outcome. Both estimator and null must receive the same selected probe and context. Randomize a small audit fraction of eligible probes so posterior-driven item selection cannot manufacture favorable apparent skill.

## 7. Better model classes

### Defensible v1 at this sample size

Use a static, coarse Bayesian ordinal scorecard per pillar:

1. Three or four observable-range states.
2. Standardized probes with complete categorical likelihoods and known opportunity.
3. Passive markers treated as weak, episode-clustered robust Bayes factors.
4. Expert log-Bayes factors represented by distributions, not fixed points.
5. No stage dynamics, archetype shifts, adaptive claim reliability, or milestone-to-stage feedback.
6. Phase and milestones reported separately.
7. Overall either omitted or derived under an explicit sum-to-zero/weighted aggregation rule.
8. Report robustness envelopes across parameter draws, not nominally calibrated 80% intervals.

For tiny-\(N\) learning, a Beta-binomial or Dirichlet-multinomial population-to-individual model over repeated probe outcomes is more identifiable. Stage should earn its place by improving future predictive log score over this direct behavioral model.

### Later model, once data justify it

Use a joint dynamic ordinal multi-trait/multi-method model containing:

- stage and pillar latents;
- rater-specific ordinal thresholds;
- extractor false-positive/false-negative channels;
- explicit opportunity and probe-selection models;
- invariant anchor probes plus path/exposure DIF;
- episode-level dependence;
- rare change points or explicit-duration phase/stage states;
- irreversible first-passage milestone states.

A hidden semi-Markov model is a better structural match for long plateaus and explicit phase durations than a birth–death CTMC plus KL overrides; see [Yu’s HSMM formulation](https://www.sciencedirect.com/science/article/pii/S0004370209001416). But no richer model cures absent raters, missing anchors, or an unrepresentative cohort.

## The three changes with highest expected value

1. **Eliminate evaluation circularity:** freeze the pre-interview dossier and predictions, prohibit gold-interview evidence from the evaluated run, add repeated probabilistic ratings and at least one genuinely independent blinded rater, and redefine M4 as a feasibility/reliability case series.

2. **Replace the heuristic updater with one coherent observation model:** known opportunity or count-based emissions, episode clustering, joint phase handling, robust contamination mixtures, and one likelihood contribution per underlying datum; remove KL/run caps and adaptive Beta claim weighting.

3. **Radically simplify v1 for \(N=5\)–10:** coarse static pillar states, uncertain expert Bayes factors, no archetype effects or stage dynamics, and an identified or explicitly derived overall score; add hierarchical dynamics only after locked held-out prediction demonstrates incremental value.