Status complete (E8 and E8b, 120 rankings each; E8b's coefficient arm, 20 fits); the bias magnitudes hold; the guard's silence on variance describes the build of the run: from emphasis f30a791 the package warns when the chosen fit has an ESS below 10
Package build library e8 at 4f42b27; the audit found it identical to a clean build except for ned on the thinning proposal, which E8 does not use
Driver / analysis sim/E8-unequal-precision.R; coefficient arm sim/E8b-edcoef.R; figure fig-study-guard in sim/figures-study.R
Data results/E8/unequal-precision.csv, results/E8b/unequal-precision.csv (120 rows each), results/E8coef/edcoef.csv (20 rows)

Question

AIC is built from a log-likelihood that is the log of a Monte Carlo mean, so each model's value is biased downward (Jensen's inequality) by an amount that grows as its own effective sample falls. The amounts do not cancel between models: the model that is harder to sample is penalised for being harder to sample. compare_models() carries each fit's effective sample and Jensen gap and flags a ranking when the gaps could reorder it, but that guard had been exercised only on constructed fits and two real ones. E2 measured selection without making this asymmetry a factor. E8 asks whether the bias decides rankings in practice and whether the guard fires when it should.

Design

parameter value meaning
generators cr, dd, ned 20 trees each, the same trees in E8 and E8b
candidates cr, dd, ed, ned every tree fitted with all four, in both arms
target tips 100 (accepted 60-160) observed medians 106.5 (cr), 106.5 (dd), 136.5 (ned)
\(\lambda_0\), turnover, crown 0.5, 0.25, 15 turnover from the driver constant TURN <- 0.25; not a CSV column
ned effect \(-0.15\lambda\) \(\beta_{ED} = -0.15\,\lambda_0/\overline{\mathrm{ED}}\), \(K\) calibrated to 100 tips
equal arm 300 draws for every candidate the design said 400
unequal arm generator 100 draws, every rival 800 the design said 100 vs 1600; the files record 800
iterations E8: max_iter 30; E8b: max_iter 100 from the stop records and the max_iter column
clock 1800 s per fit
proposal, link surrogate (bdi), linear
box and start from the observed tree alone (sim/box.R) \(\lambda_0 = \log n/T\) sets the intercept range; covariate slots scaled by \(\lambda_0/n\) and \(\lambda_0/\overline{\mathrm{ED}}\)

The handicap in the unequal arm is aimed at the true model on purpose: that is the setting that can produce a flip, and a user cannot avoid it because they do not know which model is true. The box comes from the tree rather than from the generating values so that rival candidates, which have no generating values, are not boxed arbitrarily. Chosen model = minimum AIC of the four.

The draw counts disagree between sources: 00-design.md and the driver defaults give equal 400 and unequal 100 vs 1600; the files record 300/300 and 100/800 (draws_gen, draws_rival); report/report.tex says 100 vs 800; the supplement and methods-study.tex say 1600. The files are used here.

Pre-set rule

Descriptive (00-design.md §E8). Readout: a \(2\times2\) of (model chosen correctly / incorrectly) \(\times\) (guard flagged / silent), per generating model and arm; and for every wrong choice, whether twice the gap difference covers the AIC margin, i.e. whether the gap alone could account for it. Decision: whether the guard's threshold is where it should be.

Results

Share of trees on which the generator is chosen, the chosen counts (cr/dd/ed/ned), and the median ESS of the generator's fit:

run generator arm correct chosen cr/dd/ed/ned ESS of generator fit flagged
E8 cr equal 0.85 17/0/3/0 300 0
E8 cr unequal 0.95 19/0/1/0 100 0
E8 dd equal 0.85 2/17/0/1 164 0
E8 dd unequal 0.85 2/17/0/1 51 0
E8 ned equal 0.05 2/16/1/1 35 (2 < 10) 0
E8 ned unequal 0.05 1/17/1/1 27 (3 < 10) 0
E8b cr equal 0.90 18/0/2/0 300 0
E8b cr unequal 0.90 18/0/1/1 100 0
E8b dd equal 0.70 5/14/0/1 195 0
E8b dd unequal 0.80 4/16/0/0 68 0
E8b ned equal 0.05 6/12/1/1 53 (2 < 10) 0
E8b ned unequal 0.00 5/14/1/0 30 (1 < 10) 0
E8 guard
Figure 1. Each point is one of the 101 wrong choices: the AIC deficit of the generating model (x) against how far correcting the Jensen gaps would move the ranking toward it (y), log scales. Only points above the diagonal would be reversed; one is, with a deficit of 0.003 AIC. The bottom row holds the 14 corrections that are zero or move away from the generator.

E8b — is the ED term unfound, or found and too small to buy its penalty?

00-design.md has the heading for E8b and no body; its content is in the driver header of sim/E8b-edcoef.R and in the files. E8b has two parts.

The rerun at 100 iterations (results/E8b/unequal-precision.csv): the E8 design on the same trees (identical tip counts tree by tree) with max_iter 100 instead of 30. Its rows are in the table above. The correct-choice rates move by at most 0.15 (dd equal arm 0.85 → 0.70) and the ned rows stay at 0.00-0.05.

The coefficient arm (results/E8coef/edcoef.csv, called E8coef in ledger B). E8 lost ned to dd on 1 of 20 and then 0 of 20 trees, and records AIC, gap and ESS but not coefficients, so it could not tell an estimator that does not find the effect from one that finds it while AIC refuses the two extra parameters. E8b fits ned alone to E8's own ned trees (same seeds, same design, same box), 300 draws, max_iter 100, 1800 s clock, and records \(\hat\beta_{ED}\).

quantity value
median \(\hat\beta_{ED}\) vs generating -0.0169 vs -0.0224 → ratio 0.75 (quartiles 0.12 to 1.43)
right sign 15 of 20
median ESS 58.8 (1 fit below 10)
stops 5 on the iteration cap, 15 on the clock, 0 converged
estimates at a box bound 0

No decision rule was registered for E8b.

E8b coefficient
Figure 2. One ned fit per tree on E8's 20 ned trees: the fitted \(\beta_{ED}\) against the generating -0.0224 (dark line). Shaded: within a factor of two of the generating value; 10 of 20 points fall in it and 5 are on the wrong side of zero.

E8b's reading: the coefficient is found, at 0.75 of its value with the right sign on 15 of 20 trees, on fits that mostly ended on their clock; ned loses to dd because a -0.15\(\lambda\) effect at turnover 0.25 does not pay for two parameters at 100 tips, not because the estimator misses it. That is the question E12 and E16 take up.

Conclusion

Descriptive rule; the readout is complete. The Monte Carlo bias is measurable and does not decide these rankings: corrections are of order 0.01-0.25 AIC against deficits of order 1, and only a tie at 0.003 AIC would flip. The guard's silence on all 240 rankings is consistent with that. The unequal handicap did not lower the rate at which cr and dd were recovered. The ned rows measure something else: at turnover 0.25 an effect of -0.15\(\lambda\) is below the power floor (E12, E16), so dd wins on 12 to 17 of 20 ned trees in every arm and those rows are not a test of the guard. In this run the guard did not cover variance: leaders on fewer than 10 effective draws occur among wrong choices and passed without a warning. From emphasis f30a791 the package warns when the chosen fit has an ESS below 10, so that statement describes the build E8 ran on.