V7 — does model choice name the generating model, and how often a richer one?
| Status | complete on 12 November 2026 (3,580 fits under emphasis b21083f and projected; arrays v7 on Bre): 22 of 32 cells read, 10 budget-limited; false choice holds where decided, power fails in 18 of 19 cells; first pass (182055b) archived under results/V7-182055b |
| Series | the validation series (replaces E8, E23, E27, E31 (E47's main arm)) |
| Package build | rerun: emphasis b21083f (1.1.0.9000), R 4.5.3, rule projected; first pass: 182055b (1.0.0.9000), rule mc_error |
| Driver | sim/V7.R |
| Data / analysis | results/V7/jobs/ on Bre; merged rows and tables in results/V7/; sim/V7-analysis.R |
Question
Model choice is what a user reads first. Every earlier selection study ran on capped fits, the unconditioned likelihood or the old \(D\) likelihood (audit).
Design
| cells | the 100-tip E43 cells: 32; the 200-tip cells as a second wave |
| trees | 20 a cell |
| fits | the ladder's models (cr, dd, nd, ned, nk) and the generating model, one per job, conditioned, from the box's default start |
| readings | the ladder's choice (fewest parameters within 2 AIC) and the pairwise choice (generator vs the model it nests) |
| jobs | 3580 |
Pre-set rules
- False choice: on
crandddtrees a lineage-level model is chosen on at most 10%. - Power: on each lineage-level generator's trees the ladder names the generator or a model containing its covariate on at least 80%.
- A cell with more than a tenth of fits on the cap or the clock is budget-limited, not read.
Results
Complete, 12 November 2026 (emphasis b21083f, projected). Of the 3,580 fits, 1,121 met the stopping rule, 2,151 reached the 1000-iteration cap (read as the estimate), 248 ended on the 22-hour clock, 58 were stopped by the cluster's 24-hour limit before writing a row (counted on the clock, amendment of 2026-11-12) and 2 ended on an E-step failure. A cell with at most a tenth of its fits on the clock is read on the trees whose five ladder fits each finished with a log-likelihood, and the verdict is given under both extreme readings of the unread trees (amendment of 2026-11-11). 22 of the 32 cells are read; 399 of the 640 trees have a complete ladder.
Rule 1, false choice (a lineage-level model on at most 10% of cr and dd trees).
| cell | trees read (unread) | lineage-level model chosen | verdict | the ladder's choices |
|---|---|---|---|---|
dd, turnover 0.10 |
19 (1) | 0 | holds | dd 16, cr 3 |
cr, turnover 0.10 |
14 (6) | 0 | not decided | cr 14 |
dd, turnover 0.40 |
12 (8) | 0 | not decided | cr 9, dd 3 |
cr, turnover 0.40 |
9 (11) | 0 | budget-limited, not read | cr 7, dd 2 |
No lineage-level model is chosen on any of the 54 cr and dd trees with a complete ladder. The two cells not decided have too many unread trees to exclude a share above 10% (Wilson upper bounds 0.22 and 0.24).
Rule 2, power (the generating covariate's ladder model on at least 80% of trees). It fails in 18 of the 19 lineage-level cells read and is not decided in one.
| generator, effect | turnover | trees read (unread) | hits | share (Wilson 95%) | verdict | the ladder's choices |
|---|---|---|---|---|---|---|
nk +0.35 |
0.10 | 20 (0) | 12 | 0.60 (0.39–0.78) | fails | nk 12, dd 7, nd 1 |
nd −0.35 |
0.10 | 19 (1) | 10 | 0.53 (0.32–0.73) | fails | nd 10, dd 9 |
d −0.35 |
0.10 | 8 (12) | 4 | 0.50 (0.22–0.78) | not decided | nd 4, cr 3, nk 1 |
ed −0.35 |
0.10 | 12 (8) | 3 | 0.25 (0.09–0.53) | fails | nd 5, ned 3, nk 3, dd 1 |
k −0.35 |
0.10 | 16 (4) | 4 | 0.25 (0.10–0.49) | fails | dd 7, cr 5, nk 4 |
nedc +0.35 |
0.10 | 19 (1) | 3 | 0.16 (0.06–0.38) | fails | dd 14, ned 3, cr 2 |
d +0.35 |
0.10 | 17 (3) | 2 | 0.12 (0.03–0.34) | fails | cr 14, nd 2, dd 1 |
nd +0.35 |
0.10 | 18 (2) | 2 | 0.11 (0.03–0.33) | fails | dd 14, cr 2, nd 2 |
nedc −0.35 |
0.10 | 19 (1) | 2 | 0.11 (0.03–0.31) | fails | dd 9, nd 6, ned 2, cr 1, nk 1 |
ed +0.35 |
0.10 | 11 (9) | 1 | 0.09 (0.02–0.38) | fails | cr 10, ned 1 |
edc +0.35 |
0.10 | 13 (7) | 1 | 0.08 (0.01–0.33) | fails | cr 12, ned 1 |
ned −0.35 |
0.10 | 19 (1) | 1 | 0.05 (0.01–0.25) | fails | dd 14, cr 4, ned 1 |
ned +0.35 |
0.10 | 19 (1) | 1 | 0.05 (0.01–0.25) | fails | dd 18, ned 1 |
nk +0.35 |
0.40 | 13 (7) | 3 | 0.23 (0.08–0.50) | fails | cr 6, dd 3, nk 3, ned 1 |
d +0.35 |
0.40 | 12 (8) | 1 | 0.08 (0.01–0.35) | fails | cr 11, nd 1 |
nd −0.35 |
0.40 | 10 (10) | 0 | 0.00 (0.00–0.28) | fails | cr 9, dd 1 |
nd +0.35 |
0.40 | 10 (10) | 0 | 0.00 (0.00–0.28) | fails | cr 10 |
ned +0.35 |
0.40 | 18 (2) | 0 | 0.00 (0.00–0.18) | fails | cr 12, dd 5, nd 1 |
nedc +0.35 |
0.40 | 15 (5) | 0 | 0.00 (0.00–0.20) | fails | cr 8, dd 6, nk 1 |
Rule 3, budget. Ten cells have more than a tenth of their fits on the clock and are not read: cr at turnover 0.40; edc −0.35 and k +0.35 at 0.10; d −0.35, ed −0.35 and +0.35, edc −0.35 and +0.35, nedc −0.35 and k +0.35 at 0.40. The clock falls on the ned rung (145 fits on the clock and 35 stopped by the cluster, of 640) and the nk rung (65 and 19); cr has none and dd 3.
What the ladder chooses instead. At turnover 0.10 the 256 complete ladders go to dd on 110 trees and cr on 78: the generators that contain diversity dependence are read as dd (ned 32 of 38, nd 23 of 37, nedc 23 of 38) and those with a single covariate as cr (d 17 of 25, edc 14 of 20, k 11 of 22, ed 10 of 23). At turnover 0.40, cr is chosen on 103 of 143 trees, among them 9 of 12 dd trees.
Pairwise, descriptive (the generator against the model it nests, on trees where both fits finished; no rule). The generator is chosen, at turnover 0.10: ed −0.35 on 18 of 20, edc −0.35 18 of 20, k −0.35 15 of 20, d −0.35 14 of 20, nk +0.35 13 of 20, ed +0.35 11 of 20, nedc −0.35 11 of 19, nd −0.35 10 of 20, d +0.35 8 of 19, edc +0.35 8 of 20, k +0.35 4 of 17, nd +0.35 3 of 19, nedc +0.35 2 of 20, ned −0.35 and +0.35 1 of 20 each. At turnover 0.40: edc −0.35 15 of 20, d −0.35 11 of 20, ed +0.35 11 of 20, ed −0.35 10 of 17, nedc −0.35 9 of 17, nk +0.35 9 of 20, edc +0.35 5 of 20, k +0.35 4 of 18, d +0.35 2 of 20, ned +0.35 1 of 18, nd −0.35 and +0.35 0 of 18, nedc +0.35 0 of 16.
Fits without a log-likelihood. Twenty-one fits ended on the cap without a log-likelihood; their trees are unread. The cause is in the package, not the trees: the M-step's optimiser returned an estimate one rounding error (\(5.6 \times 10^{-17}\)) past a bound of the box, and NLopt refused every later step that started there, the one that evaluates the log-likelihood included. emphasis aeb2fe8 puts the start inside the box; on one of the 21 (nd, 23 tips) the fit then has no failed step and returns a log-likelihood.
First pass (182055b, mc_error, results/V7-182055b), on the 992 fits of 5 November; not read against the rerun. "The cap" is the 1000-iteration limit (max_iter), "the clock" the 22-hour limit (time_budget); the tenth in rule 3 is of the planned fits (100 or 120 a cell).
Outcomes: 584 converged, 160 on the cap, 246 on the clock (123 of them with no conditioned iteration: the clock went to the unconditioned start, see Open issues), 2 E-step failures. Of the written fits, nd, ned and nk end on the cap or the clock on 91%, 80% and 90%.
Rule 3. 16 of 32 cells are budget-limited whatever the remaining fits do: both cr and both dd cells, all four d cells, nd t10 +0.35 and t40 +0.35, ed t40 −0.35, edc t40 +0.35, ned t10 −0.35, nedc t10 −0.35, k t10 +0.35 and t40 +0.35. Rule 1 (false choice) therefore cannot be read in any cell. The other 16 cells are 1–6 cap or clock fits away from the same state.
Ladders. 73 trees have all five ladder fits written; none has all five converged (converged share on those trees: cr 0.95, dd 0.84, nd 0.07, ned 0.15, nk 0.07).
Descriptive, not a rule reading. Pairwise choice on the 38 trees with both fits converged: the generator is chosen on 19 (d 4/8, ed 8/14, edc 4/5, k 2/7, nedc 1/2, ned 0/1, nk 0/1). On two trees a converged richer fit lies more than 1 log-likelihood unit below the model it nests (k t10 +0.35: k08 ned against dd, −1.45; k07 nd against cr, −1.02); fits start from the box default, not the nested estimate.
Conclusion
On trees of about 100 tips with an effect of ±0.35 per scale unit, the ladder does not choose a lineage-level model where there is none (0 of 54 cr and dd trees; the rule holds in the one cell that can be decided) and names the generating covariate on at most 60% of trees where there is one: the power rule fails in 18 of 19 cells. A lineage-level choice on a tree of this size is evidence of an effect; the absence of one is not evidence against it. The ladder is also the wrong instrument for a single covariate without diversity dependence: ed, edc, d and k with a negative effect are chosen over cr on 14 to 18 of 20 trees pairwise and reach the ladder's choice on at most 4.
Related
- V3: how a fit on the cap is read. V5: recovery of the coefficients on the same generators.
- V11: the ladder on the real trees. V9: false choice on trees from misspecified models.
- Estimability.