Experiments
These are the pre-registered experiments behind the emphasis package: each tests the Monte Carlo EM estimator against an exact answer or a generating value, or asks what a dated tree can estimate. The validation series V1–V12 validates every method the two papers use, under one protocol: calibrated generators with no size window, the likelihood conditioned on survival, the exponential link, the package's default stopping rule, and a fit on the 1000-iteration cap read as the estimate (build 1.1.0.9000, b21083f, for the runs since 7 November). The experiments E1–E48 before it are the earlier record; each row below says what of it holds now. The conclusions that hold now are collected in What we know, what a tree of a given size estimates in Estimability, and the terms in the Glossary.
Identifiable, estimable, informative
These pages keep three properties apart, because each belongs to a different thing.
- Identifiable is a property of the model together with what is observed: a reconstructed tree of the sampled extant lineages, the sampling fraction \(\rho\), conditioning on survival. Distinct parameter values give distinct distributions of trees. It holds or fails for every tree at once, and no tree size repairs a failure: rates that vary freely in time are not identifiable from a reconstructed tree (Louca and Pennell 2020).
- Estimable at \(n\) tips is a property of the model and the design (size, turnover, effect size, \(\rho\)): an identifiable parameter is recovered within an experiment's rule on trees of that design. This is what the simulation experiments measure. Extinction under constant rates is identifiable, and at turnover 0.10 it is not estimable up to 1600 tips (E38).
- The information a tree carries is a property of one tree under a model: how sharply its likelihood falls away from the maximum, read from a profile or from a fit that ends on the edge of the parameter box. A tree with little spread in pendant ages says little about \(\beta_D\) even at a size where \(\beta_D\) is estimable, and one such tree is not evidence that the parameter is not identifiable.
The simulation experiments report the second, the real-tree analyses (E26, E28, E40, E48) the third. Experiment pages say "identified at \(n\) tips" in the second sense.
Reading the status
| status | meaning |
|---|---|
| Holds | the conclusion is a current result |
| Holds within a scope | it is a current result within the scope its page states |
| Partly holds | part of it is a current result; the page says which part |
| Superseded by E<n> | the result will be replaced when E<n> reports; until then it is the measurement on record |
| Historical | a method-development step, not cited as a result |
| Merged into E<n> | its record is carried by another page's table |
| Running | started; results so far are on its page |
| Planned | registered or designed, not started |
Conflicts between the record and the data are listed in Open issues.
At a glance
Validation series: complete 4 · complete for two rules 1 · running 4 · registered 3.
E1–E48: Holds 9 · Holds within a scope 5 · Partly holds 15 · Superseded 13 · Historical 6 · Merged 1 · Running 1.
Reading a page
Every experiment page has the same headings: Question, Design, Pre-set rule (as registered before the data existed), Results, Conclusion, Related. Variants (E8b, E10b, E12b, E19b, E24b, E34r, E36c and others) are sections of the parent page. Three markings recur:
- sampler-limited: the cell's median effective sample is below 10; its rates are reported, not read against the rule.
- clock-limited: the wall clock ended more than 10% of the cell's fits; its rates are floors.
- unfinished: the fits ran to their iteration cap without meeting the stopping rule.
The validation series V1–V12
Registered 2026-10-31. Each experiment validates one method of the papers; the series page gives the common protocol. Findings are as of 12 November 2026.
| Experiment | Question | Status | Finding so far |
|---|---|---|---|
| V1 | On complete trees, is each model's likelihood its simulator's? | Complete | All ten models, 2560 complete trees: pooled over each cell's 40 trees, 138 of 150 coefficients lie within 10% of their generating values; the per-tree rule holds in 29 of 64 cells, the rest being small-sample error except \(\beta_N\) of the \(N\) models (0.85–1.21). |
| V2 | Is each proposal's likelihood estimate unbiased, and how many effective draws does it keep? | Complete | Both densities are their samplers': the surrogate reproduces the exact likelihood (cr exactly; dd within 1% in six of seven cells), and the two proposals converge where both are well sampled (median difference 0.03 nats above 400 effective draws). The surrogate keeps hundreds of draws for cr, dd, nd, d; ed, ned, nedc keep a handful on either proposal, and there \(\log\hat L\) falls below the truth. |
| V3 | Does each stopping rule stop where the estimate has settled, inside the box? | Complete for two rules | The default rule stops every cr fit, 11 of 20 dd and 28 of 280 lineage-level fits; the capped fits' speciation coefficients are within 2% of the box span of the reference for 7 of 10 models at turnover 0.10 and 1 of 8 read at 0.40; extinction coefficients drift. A fit on the cap is read as the estimate. |
| V4 | On real and simulated trees, does MCEM reach the exact conditioned maximum? | Complete | On the 175 real trees the rule holds for cr and dd under both stopping rules (median shortfall -0.01 to -0.16); mc_error lands closer on 75% of trees; Picidae under dd stops 14.8 below. On simulated trees (E45, repaired dd reference, complete sampling) the rule holds in 11 of 14 dd cells. |
| V5 | From how many tips is each lineage-level parameter reliable? | Running | 423 of 1,710 fits (12 November), no cell read: over 414 finished fits the covariate slope is within a factor of 2 on 74%, the right sign on 92%; the extinction intercept within a factor of 2 on 11%. First pass: E46. |
| V6 | What do incomplete sampling and a misstated sampling fraction cost? | Running | Rerun under b21083f and projected submitted 8 November (3,200 jobs, behind V7); first pass archived. |
| V7 | Does model choice name the generating model, and how often a richer one? | Complete | 22 of 32 cells read at 100 tips and ±0.35. No lineage-level model on any of 54 null trees (false choice holds in the cell decided). Power fails in 18 of 19 cells: the ladder names the generating covariate on at most 12 of 20 trees (nk) and 10 of 19 (nd). |
| V8 | Does the covariate path name the generating covariates at a fitted estimate? | Registered | Runs on V7's trees once their dd fits are written. |
| V9 | On trees with no lineage-level effect, how often is one chosen? | Running | Calibrated after the amendment of 1 November: 15 of 16 cells within 70-140 tips (the latent rate at \(\sigma = 0.40\), turnover 0.40, reaches 35 and enters no rule); rerun under b21083f and projected submitted 8 November (2,400 jobs, behind V7); first pass archived. |
| V10 | Where does a tree identify extinction? | Registered | A reading of E45, E46 and E38. |
| V11 | The analysis of record on real phylogenies | Running | Submitted 11 November (875 fits: 175 trees × the ladder, b21083f, projected), ahead of the readings of V5–V10. |
| V12 | What a fit costs | Registered | A reading of the fits' recorded times. |
1. The estimator against the exact answer, and the proposals
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E1 | Does MCEM reach the exact MLE under cr and dd? |
Holds within a scope (unconditioned, linear link, 20–100 tips; replaced by V4) | MCEM at the exact maximum (median deficit ≥ -0.5 nats) on 27 of 30 cell × proposal rows; surrogate shortfall a median 0.00-0.32 nats. |
| S | The same comparison from 25 to 800 tips | Superseded by E45 and V4 | Under cr MCEM lies within a tenth of the exact MLE's spread, 1-6% high; cells from 100 tips clock-limited. |
| E3 | ED-aware or mean-field thinning for ned? |
Holds within a scope (where thinning is used) | ED-aware is ahead on 77% of trees, short of the 95% required; it completes on 23 trees where mean-field fails. |
| E4 | What does one E-step cost? | Partly holds | 0.06-4 s from 25 to 400 tips; the surrogate costs 4.7-7.0× thinning under cr, less per effective draw from 100 tips. |
| E6 | Which proposal keeps more draws as sampling falls? | Partly holds | Linear link, older build: the ranking holds in 17 of its 20 cells on E6b; where they differ, E6b's reading replaces it. |
| E6b | Does E6's proposal ranking hold on the current build and the exponential link? | Holds | Same winner in 17 of 20 cells; ed and ned cross between \(\rho\) 0.4 and 0.2; ED models below 10 effective draws on both proposals; the surrogate leads dd 11-68×, d 22-80×, nd 18-40×. |
| E7 | Where is the surrogate proposal usable? | Partly holds | Surrogate ESS below 10 in 7 of 67 cells, all with strong negative ED effects; thinning below 10 in 54. |
| E9 | Does the mc_error rule stop once the estimate has settled? |
Holds within a scope | It never stopped while the trace moved (0 of 17) and lands about 2% from a 400-iteration reference. |
| E10 | Which relaxation step converges the surrogate's mean field? | Holds | At 1000 sweeps a fixed ⅛ converges 107 of 139 diverging units, "auto" 77. |
| E10c | Does a relaxation step of ⅛ win at the shipped sweep budget? | Holds | Its rule is met, but at \(\rho\) = 1 ⅛ converges 0 of 192 units in 20 sweeps against 131 for "auto"; the default is "auto" at \(\rho\) = 1 and ⅛ below. |
| E13 | How noisy is one iterate, and when does a fit settle? | Partly holds (the noise floor) | Noise falls about as 1/√draws (1-7%); the settling iteration depends on the criterion and does not scale. |
| E15 | Where does the time of a ned iteration go? |
Historical | One iteration from the box takes 77 s, 78 min and 5.1 h at 100, 200, 400 tips; its parts under 2 min. |
| E42 | MCEM against the exact MLE on real trees | Partly holds (cr; replaced by V4) |
On 75 trees the median deficit is -0.079 nats under cr and -0.005 under dd on complete trees. |
2. Recovery and selection of the ED and single-covariate models
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E2 | ned recovery and selection over size, \(\rho\) and turnover |
Historical | \(\beta_{ED}\) recovered and detected in 1 of 16 effect cells; turnover 0.5 biases \(\beta_{ED}\) by 1-4 SD. |
| E14 | Do emphasis and evesim simulate the same trees? |
Holds | No KS rejection in 30 comparisons; completion rates match once max_tries = 0. |
| E11 | Does the estimator invent an ED effect on evesim null trees? |
Partly holds (selection; fits unfinished) | ned is selected on 0.05 and 0.00 of null trees. |
| E12 | Where does the ED term become selectable? | Superseded by V5 and V7 | The 0.80 selection contour lies between -0.35\(\lambda\) and -0.60\(\lambda\) at 100-400 tips; false positives 3.3%. |
| E16 | The same at turnover 0.25 | Superseded by V5 and V7 | Selection 0.67 at -0.35\(\lambda\) and 1.00 at -0.60\(\lambda\); no false positive. |
| E18 | Does centring ED lift selection? | Superseded by V5 and V7 | No: 0.10-0.20 at -0.15\(\lambda\) against 0.5 required; nedc behaves as ned. |
| E24 | Recovery and selection for ed, edc, d and k |
Partly holds | edc meets every half (ratio 0.96, 1.02; selection 0.95, 0.85); d has 0.15 false positives. |
3. The pendant-age model and the ancestral-splits covariate
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E19 | Is \(\beta_D\) recovered, and nd selected over dd? |
Superseded by V5 | Sign recovered, size not: ratio 0.30-0.47 on thinning and 0.43-0.63 on the surrogate at ±0.35. |
| E30 | E19b's design with 300 iterations and a 4-hour clock | Superseded by V5 | Ratio 0.67 at -0.35; 0 of 39 fits met the stopping rule. |
| E37 | Does the pendant-age fit finish within 1000 iterations? | Historical | Stopped when the D change altered the likelihood it maximised; 6 of 40 fits written. |
| E31 | Can the ladder tell nd from ned at 200 tips? |
Superseded by V5 and V7 | No: nd chosen on 5 of 20 of its own trees, the pair on 9; \(\beta_D\) at 0.68. |
| E21 | Is \(\beta_K\) recovered, by fitting and on the path? | Superseded by V5 and V7 | \(\beta_K\) at 0.95 and 0.92 at ∓0.35; selection 0.15 and 0.60; on the path ED carries K's signal. |
4. Model choice: the ladder and the covariate path
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E8 | Can Monte Carlo bias reorder an AIC ranking? | Partly holds | Correcting it moves rankings by 0.01-0.25 AIC and reverses 1 of 101 wrong choices, a 0.003-AIC tie. |
| E20 | Does one dgLARS path select the generating set? | Superseded by V7 and V8 | Met for cr only (0.90) at 100 tips; D and ED are not separable. |
| E22 | The path at 50, 200 and 400 tips | Superseded by V7 and V8 | Null chosen on 0.85-1.00 of trees; dd's N alone on 0.40-0.70; ned's N + ED never on 0.80. |
| E23 | What does the one-call ladder choose? | Superseded by V7 and V8 | The generator is chosen on 0.80 of cr and 0.90 of dd trees; nk 0.70, ned 0.20, nd 0.00. |
| E27 | The ladder across eight conditions | Superseded by V5 and V7 | nk chosen on 11-15 of 20 of its trees, ned on at most 2; turnover 0.25-0.40 moves null trees to cr. |
| E33 | The path at a fitted estimate; the extinction path | Superseded by V7 and V8 | At a fitted estimate dd's exact set falls to 0.35 and 0.25; an extinction driver is not detected. |
5. Extinction, conditioning and what a tree carries
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E29 | On finished fits at 200 tips, is extinction recovered? | Partly holds | No: the fitted rate is a median 26× below 0.05; \(\beta_{ED}\) returns at 1.02 on the 7 finished fits. |
| E32 | Does a wider box or an exact estimator recover extinction? | Holds | No: estimates follow the box for a 0.01 log-likelihood gain; bd_ML puts \(\mu\) at zero on 37 of 40 trees. |
| E34 | Is zero extinction an artefact of not conditioning? | Holds | No: conditioned extinction is within 2× of 0.05 on 1 of 20 trees (E34r: 0 of 20). |
| E36 | How accurate is the survival emulator? | Partly holds | It labelled cap-passing clades as extinct; with that fixed (d3dee92) it meets its rule at 2000 training clades. |
| E35 | A K driver at turnover 0.25; a misstated \(\rho\) | Holds within a scope | The negative K slope returns at 0.75 of its value; \(\rho\) off by 0.1 leaves the choice unchanged on ≥ 18 of 20. |
| E38 | How much does a tree say about extinction, by turnover and size? | Holds within a scope | At turnover 0.10 the exact cr likelihood excludes zero on fewer than 80% of trees up to 1600 tips; dd from 100. |
6. Real trees
The results on the 170 bird and mammal trees of E40, with figures, are on Real phylogenies.
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E5 | Does the workflow run on 47 family trees? | Historical | cr within 0.53 nats of bd_ML on all 46 fitted trees (linear link). |
| E17 | Are the families' margins what the fitted model predicts? | Historical | All three real ned-over-dd margins lie inside the central 90% of their bootstrap. |
| E25 | Three bird families under the ladder and the path | Historical | nk on Corvidae (12.0 AIC over ned), ned on Icteridae, dd on Cotingidae. |
| E26 | The family trees under the whole workflow | Merged into E28 | 36 of 46 families usable; by minimum AIC 21 choose a lineage-level model (nk 12, ned 9). |
| E28 | The 29 families in the calibrated range, refitted | Partly holds | 25 usable; parsimonious choice cr 4, dd 5, nd 0, ned 8, nk 8. |
| E39 | What do the covariates share within a tree? | Holds | ED-K rank correlation median -0.44 on family trees against -0.18 on simulated ones, stronger with imbalance. |
| E40 | Birds and mammals cut into clade-sized trees | Partly holds | Bird clades choose ned or nk on 72 of 77; clade and family tree agree on 10 of 29 pairs (rule: 70%). |
| E41 | Do E40's fits change when continued, with edges moved out? | Partly holds | Choice unchanged on 16 of 16 trees; the edge binds on 4 of 63 fits. The nd and nk fits kept their starting draws (the stopping rule could not grow them on that build), so their convergence is measured by V3. |
| E48 | Does refitting nd under the corrected D likelihood change the real-tree readings? |
Holds | Choice changes on 5 of 155 readable E40 trees, 1 of 25 E28 and 1 of 9 E26 (only that set crosses 10%); the pairs rule still fails; \(\beta_D < 0\) on 28 of 30 read family fits. |
7. Estimability under the conditioned likelihood
How well each model's parameters are estimated, and from how many tips, with the likelihood conditioned on the survival of the clade: what these experiments and the validation series have measured so far is read together on Estimability.
| Experiment | Question | Status | Finding |
|---|---|---|---|
| E44 | On complete trees, is the likelihood the simulator's? | Partly holds (the \(D\) fix; replaced by V1) | Yes, with D constant between events: d's \(\beta_D\) returns at 0.95 and 1.16 of its value, against 0.70 and 1.46 before. |
| E43 | The survival factor of each model, exact or emulated | Holds | No emulator meets the rule for any lineage-level model (error 0.5-2.6 against 0.05), so they are conditioned by a direct Monte Carlo estimate of \(P(S)\); the reference matches the exact cr/dd values (median error 0.007, 0.008). |
| E45 | Conditioned cr and dd against the exact estimate |
Partly holds (cr; dd on the repaired reference) |
1720 of 1720 fits: MCEM at the exact maximum in all 24 cr cells, \(\lambda\) reliable from 25 tips at turnover 0.10; on the repaired dd reference at complete sampling the rule holds in 11 of 14 dd cells (all nine of 25–100 tips; misses at 400 tips and turnover 0.10, and 200 and 400 tips at 0.25, where most fits ended on the clock); 747 of 756 rescored trees have a valid maximum. Reliable sizes for dd: pending. |
| E46 | The eight lineage-level models, conditioned, by size, turnover and effect | Running (the series' V5) | 57 cells at 50 and 100 tips, larger only where a parameter is not reliable; 19 of 57 cell–proposal pairs are sampler-limited; the lineage-level fits rerun under b21083f after V3. |
Chronology
- Exact comparisons and proposals: E1, S, E2, E3, E4, E6.
- Feasibility and convergence: E7 (from S's failed
nedarm), E9, E10, E13. - ED power and specificity: E8, E12, E5, E16, E17, E18, E14, E11.
- Pendant age, K and the path: E19, E21, E20, E22, E15 (which gave the ladder its nested starts).
- The ladder and the families: E23, E24, E25, E26, E27, E28 (October 2026).
- Extinction and the information in a tree: E29, E31, E30, E32, E34, E36, E35, E37, E38, E33, E39.
- Class-tree clades: E40 (closed 2026-10-18), E42, E41 (closed at the pilot 2026-10-19).
- Estimability under the conditioned likelihood (from 2026-10-15): E44, E43, E45, then E10c, E6b, E48, E46.
- The validation series (from 2026-10-31): V1–V12.