V3 — does each stopping rule stop where the estimate has settled, and does the box hold it?
| Status | complete for the reference and two rules (ref, proj, mcfix; 320 trees each, emphasis b21083f); the rel and mc arms, cancelled on 8 November to give V7 the cluster, are to be rerun; the first pass on 182055b (852 of 960 runs, rule 2 unreadable) is archived under results/V3-182055b |
| Series | the validation series (replaces E9, E13, E41) |
| Package build | rerun: emphasis b21083f (1.1.0.9000), R 4.5.3; first pass: 182055b (1.0.0.9000) |
| Driver | sim/V3.R |
| Data / analysis | results/V3/jobs/ on Bre; merged rows and rule tables in results/V3/; sim/V3-analysis.R |
Question
A fit is only as good as the place it stops. The package's default rule (rel_change) and the Monte Carlo error rule (mc_error) are checked against a reference run that never stops by rule, on every model; E41's claim that nd and nk do not settle within 1000 iterations was made on fits whose draws never grew (fixed in 182055b).
Design
| cells | the 100-tip E43 cells of all ten models: 32 |
| trees | 10 a cell |
| arms | proj (the package's default rule since 2026-11-01, projected; added in the rerun), mcfix (mc_error at 200 draws fixed: the step within its own Monte Carlo error stops the fit, draws not grown; added 8 November after the interim reading of proj), rel (rel_change, the previous default), mc (mc_error, draws to 2400, the rule the series registered), ref (1200 draws, 400 iterations, never stops by rule; estimate = mean of the last 200 iterates of the conditioned continuation) |
| fit | estimate_rates() defaults, conditioned on survival, the box of auto_bounds(), 22-hour clock; since b21083f the unconditioned start gets at most a quarter of the clock and every stage is recorded |
| jobs | 1,600 (first pass: 960, three arms) |
Pre-set rules
- A rule stops by itself on at least 90% of trees.
- Its estimate lies within 2 reference sds of the reference mean on at least 90% of trees, for every coefficient; read since 8 November also in span units: every speciation coefficient within 2% of the box span of the reference mean (
rule2_span_2pct), the registered band being narrower than a 200-draw fit's Monte Carlo error (see the interim reading). - The reference mean lies inside the box, more than \(10^{-3}\) of the span from each edge, on at least 90% of trees.
Results
Rerun (b21083f): submitted 7 November. What it decides: the stopping rule of the relaunch of V6, V7 and V9 is the rule that passes rules 1 and 2 on the most model–turnover cells.
Complete for ref, proj and mcfix (320 trees each; rel and mc wait). Rule 1: projected stops by itself on every cr tree (median 20 and 35 iterations at the two turnovers) and on 11 of 20 dd trees; on the eight lineage-level models it stops 28 of 280 fits, 23 of them ed, and the rest reach the 1000-iteration cap in a median 0.4 to 6.6 hours. mcfix adds dd (18 of 20) and a few d, edc and k fits (6, 6 and 3); on nd, ned, nedc and nk it stops 2 of 130. Rule 2 in span units (every speciation coefficient within 2% of the box span of the reference mean on at least 90% of trees): projected passes 8 of 20 model–turnover pairs — cr, dd, ed, edc, ned, nedc and nk at turnover 0.10, nk at 0.40 — fails 10 (d, k, nd at 0.10 with 11, 13 and 13 of 20 trees within; at 0.40 the median distance of the failing pairs is 1.0–2.1% of the span) and 2 are not read; mcfix passes 7. The registered band (two reference sds) fails every pair for both rules. Rule 3: the reference mean is inside the box on 15 of 20 pairs; it fails on cr and dd at turnover 0.10 (3 and 6 of 10 trees with a coefficient at an edge), nk at 0.10 and dd at 0.40 (2 of 10 each), and k at 0.40 is not read. The decision that followed is the amendment of 8 November (evening): the series runs under projected and a fit on the cap is read as the estimate.
Interim, 8 November, on 307 of 320 references and 293 of 320 projected fits (the rel and mc arms wait). Every reference has its 200 iterates from the full trace. Rule 1: projected stops by itself on every cr tree (20–35 iterations) and on 8 of 10 dd trees at turnover 0.10; on the eight lineage-level models it reaches the 1000-iteration cap on 83–100% of trees, in 0.4–5 hours a fit (median 1.3). Rule 2 fails for projected on every model–turnover pair, cr included (within on 3 of 10 and 1 of 10 trees): the cr estimate sits a median 0.036 (\(\beta_0\)) and 0.14 (\(\gamma_0\)) from the reference mean — 1–2% of the box span — against a reference sd of 0.002 and 0.011, so the registered yardstick, the sd of a 1200-draw reference's iterates, is narrower than the Monte Carlo error of a 200-draw fit. On the lineage-level models the speciation coefficients of the capped fits lie a median 0.03–0.8% of the span from the reference (ed \(\beta_{ED}\) 0.0002 of 0.8, nd \(\beta_D\) 0.006, nk \(\beta_K\) 0.001) and the extinction coefficients 4–12%. Rule 3 passes on 12 of 20 pairs, fails on cr 0.10, dd (both), nk 0.10 (\(\gamma_0\), \(\gamma_N\), \(\beta_N\) at an edge), 4 not read. What this leaves to the author: as registered, no rule passes rules 1 and 2 for any lineage-level model; either the cap is read as a stop when the estimate is at the reference within a stated share of the span, or rule 2's yardstick becomes the fit's own Monte Carlo standard error.
Interim, 8 November evening, mcfix on 241 of 320 fits (its N + lineage-level fits still running). Rule 1: mcfix fires where proj did not on dd (9 of 10 at both turnovers against 8 and 3) and a little more often on d (5 against 0 of 20), edc and k (3 against 0–1); on nd neither rule fires (0 and 1 of 20), nor so far on ned, nedc, nk: in the N + lineage-level models the EM step stays above its own Monte Carlo error for 1000 iterations — drift in the extinction coefficients, not noise. Rule 2 in span units: the two rules are equally accurate; at turnover 0.10 the speciation coefficients of cr, dd, ed, edc, ned, nedc and nk lie within 2% of the box span of the reference on at least 90% of trees under proj (nd on 13 and 17 of 20 under proj and mcfix); at turnover 0.40 most pairs fail narrowly (medians 1.0–2.1% of the span, cr among them with 7 of 10 within). The registered band fails every pair for both rules. What follows: mcfix does not earn a relaunch and V7 stays under proj; what changes is how a capped fit is read — as the estimate, gated by its distance to the reference in span units, rather than as budget-limited — and the gate's value is the author's decision (the data support 2% of the span for speciation at turnover 0.10, about 2% at 0.40, extinction not identified at either).
First pass (182055b, results/V3-182055b), on the 852 rows of 5 November; not read against the rerun. A job without a row counts against rule 1.
Missing rows (108). Killed at Bre's 24-hour limit (a 22-hour fit clock plus the tree's simulation): 96 mc fits (nedc all 39, ned 29, nk 19, nd 8, k 1), 11 rel fits on ned and nedc, one reference.
Rule 1 holds for both arms on cr at both turnovers and dd at 0.10 (3 of 20 model–turnover pairs) and fails on the other 17. rel ends on its 1000-iteration cap on 209 of its 309 written fits; mc ends on the 22-hour clock or has no row for most nd, ned, nedc and nk trees.
| model | rel: cap / clock / no row |
mc: cap / clock / no row |
|---|---|---|
| cr | 0 / 0 / 0 | 0 / 0 / 0 |
| dd | 5 / 0 / 0 | 2 / 0 / 0 |
| d | 37 / 0 / 0 | 19 / 8 / 0 |
| ed | 9 / 0 / 0 | 4 / 12 / 0 |
| edc | 35 / 0 / 0 | 8 / 22 / 0 |
| k | 20 / 2 / 0 | 11 / 6 / 1 |
| nd | 40 / 0 / 0 | 5 / 26 / 8 |
| ned | 13 / 1 / 9 | 0 / 0 / 29 |
| nedc | 33 / 0 / 2 | 0 / 0 / 39 |
| nk | 17 / 0 / 0 | 0 / 1 / 19 |
Rule 2 is not read. With cond = "surface", estimate_rates() runs the conditioned fit in segments of 25 iterations and returns the last segment's trace as details$mcem; the full trace is in conditioning. sim/V3.R took the last 200 rows of details$mcem, so the reference mean and sd recorded are over at most 26 iterates (26 on 294 of 319 references). Reading the rule needs the references rerun with the full trace kept.
Rule 3, on the recorded reference mean: passes on 7 pairs (d, edc, nd at 0.10; cr, d, edc, nk at 0.40), fails on 11 (cr, dd, ed, k, ned, nedc, nk at 0.10; dd, ed, nd, nedc at 0.40), not read on 2 (k 0.40: a reference missing; ned 0.40: four references cut under 100 iterations). The coefficients at the edge are mostly extinction ones (\(\gamma_0\), \(\gamma_N\), \(\gamma_{ED}\)).
These first-pass results are for rel_change and mc_error on 182055b; the rerun adds projected, the package default since 2026-11-01, and reads the reference on the whole trace.