E46 — the eight lineage-level models under the conditioned likelihood: accuracy by size, turnover and effect
| Status | ran on Bre (USI cluster) 2026-10-31 to 2026-11-04 under the amendment of 2026-10-31 (build 182055b: the thinning path's mc_error rule could not fire in 2143bde, so every thinning fit was rerun); nothing queued since 4 November; 1,072 fits written, 1 of 38 open pairs read (see Results). Rerun as V5 from 10 November on two local hosts (32 workers): the lineage-level models at 50 and 100 tips, 30 trees a cell, under emphasis b21083f and projected, a fit on the cap read as the estimate (amendment of 2026-11-10). Ran on CSCS Daint 27–28 October before that (stopped: the allocation belongs to a course) |
| Package build | emphasis 1.0.0, the main branch of 2026-10-21 (2143bde) |
| Driver / analysis | sim/E46-lib.R, sim/E46-trees.R, sim/E46-run-claimed.R, sim/E46-launch.sh |
| Data | results/E46/ (trees, one CSV and one record per job, the decisions the queue takes) |
Question
Phase 2 of the estimability plan. For d, nd, ed, edc, ned, nedc, k and nk there is no exact likelihood, so the reference is the generating value. Per model, coefficient, turnover and effect: from how many tips is each parameter of the conditioned MCEM estimate reliable, and from how many is the lineage-level coefficient identified. E46 replaces the recovery results of E12, E16, E18, E19 (with E19b), E21 (E21a), E24's d arm, E27's ned and nk ratios, E29's \(\beta_{ED}\) recovery, E30, E31 at 200 tips, and E7's ed and ned cells.
E43 found no survival emulator accurate enough for any of these models, so each conditioned fit needs a direct Monte Carlo estimate of \(P(S)\), the probability that both crown lineages survive. Supplying one at every parameter value the M-step evaluates (about 1300-2900 evaluations an iteration under ned) would cost 0.7-1.6 hours of simulation an iteration. A pilot on 2026-10-20/21, not a registered experiment, compared ways of supplying it and chose the one described below.
Design
| generators | E43's calibrated generators: exponential link, crown age 15, kept if both crown lineages survive, no size window, calibrated to the median surviving size, E43's lineage cap |
| models | d, ed, edc, k (one covariate); nd, ned, nedc, nk (\(N\) plus one) |
| turnover at the crown | 0.10, 0.40 |
| lineage-level speciation coefficient | \(-0.35\) and \(+0.35\) per scale unit of its covariate |
| extinction slope | generated at 0, estimated |
| \(\rho\) | 1 |
| tips | 50 and 100 for every cell the generator reaches (57 cells); 200 only where a parameter is not reliable at 100; 400 only where one is still not reliable at 200 |
| not run (the generator cannot reach the size) | ned and k at \(-0.35\), turnover 0.40; nk at \(-0.35\) from 100 tips at turnover 0.10 and at every size at 0.40; at 400 tips nd at 0.40, ned at \(-0.35\) and 0.10, nedc at \(+0.35\) and 0.40 |
| trees per cell | 30 |
| effective sample before the fit | one E-step of 2400 draws at the generating values on every tree, on the fit's proposal; a cell whose median is below 10 is sampler-limited and not fitted |
| proposal | the package default (surrogate for the ED models, thinning for d, nd, k, nk); the 100-tip d and nd cells on both, and E19b's adoption rule picks the proposal of the other d and nd cells |
| fit | the generating model only; mc_error (300 draws grown by 1.5 up to 2400); at most 1000 iterations a stage; box from auto_bounds(); started from the exact conditioned cr estimate (one-covariate models) or dd estimate (\(N\) models) with zero lineage-level slopes |
| clock | 24 hours for the first three trees of each cell; four times the longest of those three for the rest |
| box edges | a parameter on an edge has that edge moved out (an intercept by 4 log units, a slope to three times its distance from zero) and the conditioned stage is rerun; a parameter on the moved edge is reported as not bounded by the tree |
Conditioning. Four steps per tree.
- The unconditioned fit, to the stopping rule.
- \(\log P(S)\) by direct simulation at \(2p + 1\) points around that estimate (\(p\) the number of coefficients of a quadratic in the parameters: 31 points for four parameters, 57 for six), in a box of half-width 0.03 of the fit's box per coordinate, the same clade seeds at every point, enough clades for a standard error of 0.02 at the centre; a quadratic fitted by weighted least squares, with the tangent plane outside the box.
- The fit continued from the unconditioned estimate with that surface as the conditioning factor, in segments of 25 iterations; whenever the iterate leaves the inner half of the surface's box, the surface is rebuilt around it. It ends when the stopping rule is met inside the inner half.
- When the unconditioned extinction intercept is on its box edge, step 3 also runs from the nested start, and the higher conditioned log-likelihood is kept.
The conditioned log-likelihood reported uses a direct \(\log P(S)\) at the estimate (standard error 0.01). In the pilot, on 100-tip cr and dd trees where the exact conditioned maximum is known, steps 1-3 inside MCEM ended a median 0.0011 nats below it (worst 0.13), against 0.0074 for MCEM with the exact factor and 0.26 for the unconditioned fit; without step 4, a second, lower maximum on the extinction edge was kept on 4 of 10 cr trees at turnover 0.10. The surfaces cost under 0.3% of a cr or dd fit's time, 0.4-1.8% under ned and 3-9% under nk.
Pre-set rule
- A cell is read only if at least 80% of its fits finished (both stages) and its median final effective sample is at least 10; otherwise it is reported as clock-limited (more than 10% of fits on the clock), sampler-limited or unfinished, and that is the answer at that size. For
ndandnk, a fit on the iteration cap is also counted as finished if its speciation coefficients alone meet the stopping rule over its last 50 iterations (all at 2400 draws, and the means of the first and last 25 iterates of each speciation coefficient differing by less than one standard error of that difference); its extinction parameters are then reported, not read. Both readings are reported. - A parameter is reliable at a size when its median ratio to the generating value lies in [0.8, 1.25] and the central 80% of ratios within [0.5, 2]. The extinction slope, generated at 0, is read on its difference from 0 in units of the speciation slope's generating size, with the same band widths. There is no coverage condition: the package returns no interval for an MCEM fit.
- A lineage-level coefficient is identified at a size when it is reliable and its sign is right on at least 95% of trees. The answer is the smallest size from which rules 2 and 3 hold at every size run above it.
- The conditioning's own noise: on 3 trees a cell, step 3 is repeated with other surface seeds; a coefficient whose spread across seeds exceeds a tenth of its spread across the cell's trees makes the cell limited by the conditioning, reported and not read.
- A tree whose conditioned maximum lies where the direct estimate of \(P(S)\) found no surviving clade is reported, not read.
A cell not read at a size is not run at a larger one. The d and nd proposal is chosen by E19b's rule: the surrogate if its median effective sample is above 10 in every 100-tip cell where thinning's is below 10, and no cell's median ratio of \(\beta_D\) is further from 1 on the surrogate than on thinning.
Budget
From the pilot, per tree on one core at 100 tips with both stages and the nested start: nk about 7 hours, ned at least 10; taken for the others: nd 7, nedc 10, ed and edc 6, d and k 4. At 50 tips 0.6 of that, at 200 tips 3.1 times (from E27's size ladders). The 50- and 100-tip cells with the second d/nd proposal: about 10 500 CPU-hours, about 14 days on 24 workers. Each 200-tip cell costs 3.1 times its 100-tip cell (at most a further 17 500 CPU-hours if no parameter is reliable at 100), a 400-tip cell about 10 times. These are lower bounds where the pilot's fits ended on their clock.
Results
Interim, on the rows of 4 November (sim/E46-analysis.R, tables in results/E46/). Read under the amendment: 958 fits on 182055b (Bre) and 114 surrogate fits on 2143bde (Daint and a local host); 1,710 effective-sample checks; 103 conditioning repeats.
| pairs | |
|---|---|
| cell–proposal pairs | 65 (57 cells) |
| sampler-limited before the fit | 19 |
not queued (d/nd at 50 tips: E19b's choice waits on two 100-tip nd fits) |
8 |
| clock-limited | 34 |
unfinished (2 of 30 written: k t40 +0.35 100 tips, nk t40 +0.35 50 tips; edc t10 +0.35 50 tips) |
3 |
| read | 1 |
Written: 1,072 of the 1,140 fits and 103 of the 114 repeats of the 38 open pairs. The read pair, ed turnover 0.10, +0.35, 50 tips (surrogate; 25 of 30 finished; rule 4 met): \(\beta_0\) reliable (median ratio 1.17, central 80% 0.88–1.52); \(\beta_{ED}\) not reliable (1.36, 0.65–1.89; sign right on 25 of 25); \(\gamma_0\) not bounded by the tree on 20 of 25. Its 100-tip pair is clock-limited, so the answer for that triple is the limit at 100 tips. No model has a smallest reliable size.
In partly finished pairs the finished trees are the smaller ones (median 51 against 111 tips in d t40 −0.35, 91 against 147 in ed t10 +0.35 at 100 tips). The design's clock (four times the longest of a cell's first three trees) exceeds Bre's 24-hour allocation on 547 of the 551 fits that ended on it; 303 fits ended together when the allocations ended at 19:15–19:36 on 4 November. Conditioned stages also end on the 1000-iteration cap on the surrogate proposal (83 of 180 edc fits, 45 of 60 nedc).
Conclusion
Not evaluable until the run reports.
Related
- Estimability: the plan, Phase 2.
- E43: the generators, and why the conditioning is a direct Monte Carlo estimate.
- E45: Phase 1,
crandddagainst the exact conditioned estimate. - E41: the fit protocol and the box edges;
ndandnkunfinished at 1000 iterations. - E6 (E6b) and E7: the effective sample of the proposals.
- E19 (E19b): the adoption rule for the
dandndproposal. - E47: model choice on E46's trees, the effect grid, turnover 0.25, \(\rho\) below 1, and the extinction-driver arm (
nedandnkat turnover 0.40), deferred from E46 because no generator with an extinction driver has been calibrated.