V2 — is each proposal's likelihood estimate unbiased, and how efficient is it?
| Status | registered 2026-10-31; amended 2026-10-31; read 2026-10-31 on 7,436 of 7,440 trees |
| Series | the validation series (replaces E3, E6b, E7) |
| Package build | emphasis 182055b (1.0.0.9000), R 4.5.3 |
| Driver | sim/V2.R |
| Data | results/V2/jobs/<cell>-r<rho>-k<kk>.csv, results/V2/build.txt |
Question
MCEM weighs every augmented tree by \(f(y, z\mid\theta)/q(z)\). The weights average to the likelihood only if \(q\) is the density the sampler draws from; a wrong density biases every log-likelihood, every AIC and every M-step. Earlier proposal studies measured the effective sample only, at the generating values, on builds since changed. V2 measures both: whether each proposal's estimate is unbiased where an exact likelihood exists, whether the two proposals agree where it does not, and how many effective draws each keeps, by model, size, turnover and sampling fraction.
Design
| cells | every reachable E43 cell (50–400 tips, turnover 0.10 and 0.40), at \(\rho\) = 1, 0.8, 0.5 |
| trees | 20 per (cell, \(\rho\)) |
| proposals | surrogate (all models but k, nk); thinning (all; ED-aware for ED models) |
| draws | four independent sets of 1000 per tree and proposal, at the generating values |
| exact | cr at every \(\rho\) (Bernoulli closed form); dd at \(\rho = 1\) (ll_dd, repaired in c641dca) |
| jobs | 7440 |
Pre-set rules
- Unbiased: per cell with an exact reference, the mean of \(\hat L/L\) has a 95% interval containing 1.
- Agreement: where no exact reference exists, the median difference between the two proposals' pooled \(\log\hat L\) lies within its 95% interval of 0, on trees where both keep at least 10 effective draws.
- Efficiency (descriptive): median effective sample and share of trees below 10, per proposal, cell, \(\rho\).
Results
Read on 7,436 of 7,440 trees (sim/V2-analysis.py; results/V2/rule1.csv, rule2.csv, rule3.csv, ess-diagnostic.csv).
Rule 1, unbiasedness against the exact likelihood (31 cells with a reference: cr at every \(\rho\), dd at \(\rho = 1\)).
- Surrogate: holds in 30 of 31 cells. For
crits estimate equals the exact likelihood on every set of draws (mean \(\hat L/L\) = 1.000, interval of zero width): it samples the posterior exactly. Forddthe mean lies within [0.95, 1.003] with the interval containing 1 in six of seven cells; the seventh, 50 tips at turnover 0.40, is 0.990 [0.984, 0.995]. - Thinning: holds in 15 of 31 cells, always failing below 1. The shortfall follows the effective sample: the median \(\log(\hat L/L)\) is -5 to -51 where the median effective sample of a 1000-draw set is 1–2, -0.3 to -2.2 at 3–9, -0.02 to -0.18 at 12–54 and -0.005 to -0.025 at 114–269. An average of importance weights is unbiased for \(L\), but with a few effective draws it lands below \(L\) on most runs; the surrogate, scored against the same references on the same trees, shows that the references are the likelihood the proposals estimate.
Rule 2, agreement between the proposals where no reference exists (dd at \(\rho < 1\) and the lineage-level models with both proposals; trees where both pooled effective samples reach 10).
- The proposals agree in 66 cells and differ in 150; 62 cells have fewer than five such trees.
- The proposal with the smaller effective sample gives the lower estimate: the surrogate is higher for
d,dd,ndandedc(median of the differing cells 0.4–1.4 per model), thinning is higher fored,nedandnedc(2–7), the models on which the surrogate keeps 2–7 effective draws. - The difference vanishes as both are sampled well. By the smaller pooled effective sample of the two (4000 draws each):
| smaller effective sample | trees | median difference of \(\log\hat L\) | share where the less-sampled proposal is lower |
|---|---|---|---|
| under 10 | 1,972 | 10.6 | 0.77 |
| 10–40 | 2,187 | 2.06 | 0.73 |
| 40–100 | 643 | 0.37 | 0.68 |
| 100–400 | 415 | 0.11 | 0.61 |
| over 400 | 174 | 0.03 | 0.55 |
Rule 3, efficiency (median effective sample of a 1000-draw set at the generating values; share of sets under 10 in brackets):
| model | surrogate, \(\rho\) 1 / 0.8 / 0.5 | thinning, \(\rho\) 1 / 0.8 / 0.5 |
|---|---|---|
cr |
1000 / 1000 / 1000 (0) | 7.7 / 3.8 / 3.1 (0.53–0.72) |
dd |
788 / 862 / 744 (0) | 5.2 / 3.4 / 3.8 (0.68–0.77) |
nd |
242 / 217 / 182 (0.02–0.05) | 4.6 / 3.1 / 2.9 (0.72–0.84) |
d |
118 / 56 / 42 (0.14–0.26) | 6.3 / 2.7 / 2.2 (0.59–0.87) |
edc |
64 / 18 / 8.4 (0.26–0.54) | 11.3 / 3.7 / 3.0 (0.48–0.73) |
ed |
6.4 / 3.1 / 2.0 (0.58–0.92) | 14.3 / 4.4 / 3.5 (0.45–0.73) |
ned |
7.3 / 4.1 / 2.4 (0.57–0.90) | 5.3 / 3.1 / 3.0 (0.61–0.76) |
nedc |
4.2 / 3.0 / 2.2 (0.77–0.95) | 5.1 / 3.4 / 3.6 (0.67–0.78) |
k |
— | 50.4 / 11.2 / 12.7 (0.35–0.48) |
nk |
— | 3.8 / 3.2 / 3.4 (0.71–0.79) |
Conclusion
Both proposal densities are the ones their samplers draw from: the surrogate reproduces the exact likelihood, and the two proposals converge on the same value where both are well sampled. What separates them is efficiency. The surrogate keeps hundreds of effective draws for cr, dd, nd and d and is the proposal to use there; for ed, ned and nedc neither proposal keeps more than a handful at the generating values, and an estimate of \(\log L\) from so few draws lies below the truth by an amount that grows as the effective sample falls. Sampling fractions below 1 lower the effective sample of every proposal.