Glossary
Definitions follow the methods paper (report/paper2/s2-models.tex, s3-inference.tex) and ledger E (mathematics against code). Time runs forward from the crown (\(t = 0\)) to the present (\(T\)); \(A(t)\) is the set of lineages alive at \(t\) on the complete tree.
The rates
\[ \lambda_s(t) = g\big(\beta_0 + \beta_N N(t) + \beta_D D_s(t) + \beta_{ED}\,\mathrm{ED}_s(t) + \beta_K K_s(t)\big),\qquad \mu_s(t) = g\big(\gamma_0 + \gamma_N N(t) + \gamma_D D_s(t) + \gamma_{ED}\,\mathrm{ED}_s(t) + \gamma_K K_s(t)\big) \]
A model leaves some coefficients free and fixes the rest at zero. Every covariate a model uses enters both rates, so each adds two free parameters to the two intercepts.
The models
| model | covariates (both rates) | free parameters | default proposal (alternative) | exact reference |
|---|---|---|---|---|
cr |
none | 2 | surrogate (thinning) | DDD::bd_ML |
dd |
\(N\) | 4 | surrogate (thinning) | DDD::dd_ML (linear link, constant \(\mu\)) |
d |
\(D\) | 4 | thinning (surrogate) | none |
nd |
\(N\), \(D\) | 6 | thinning (surrogate) | none |
ed |
ED | 4 | surrogate (thinning) | none |
edc |
centred ED | 4 | surrogate (thinning) | none |
ned |
\(N\), ED | 6 | surrogate (thinning) | none |
nedc |
\(N\), centred ED | 6 | surrogate (thinning) | none |
k |
\(K\) | 4 | thinning (the surrogate refuses \(K\)) | none |
nk |
\(N\), \(K\) | 6 | thinning (the surrogate refuses \(K\)) | none |
"Lineage-level model": any model with \(D\), ED or \(K\); "clade-level": cr and dd.
The covariates
All are read on the complete tree (hidden lineages included).
- \(N(t)\): the number of lineages alive, \(|A(t)|\).
- Pendant age \(E_s(t)\): the time since lineage \(s\) was born or last split, \(E_s(t) = t - \tau_s(t)\); the pendant start \(\tau_s\) restarts at every split, observed or hidden.
- \(D\), centred pendant age (age imbalance): the pendant age minus its clade mean, \(D_s(t) = E_s(t) - M(t)\) with \(M(t) = \frac{1}{N(t)}\sum_{s \in A(t)} E_s(t)\); the \(D_s\) sum to zero.
- ED, fair-proportion evolutionary distinctiveness: each branch's length shared among the lineages alive below it, \(\mathrm{ED}_s(t) = \sum_{b \succeq s} \ell_b(t)/n_b(t)\); the \(\mathrm{ED}_s\) sum to Faith's phylogenetic diversity PD, so their clade mean is \(\mathrm{PD}/N\), a clade-level quantity.
- Centred ED (\(\mathrm{ED}^c\),
edc,nedc): \(\mathrm{ED}^c_s = \mathrm{ED}_s - \mathrm{PD}/N\), which sums to zero. - \(K\), ancestral splits: the number of speciation events on the path from the crown to \(s\), hidden splits included; at a split both daughters carry one more than the parent.
- NND (used in E39 and, as
evesim's metric, in E11): distance to the nearest living relative; not a package covariate.
Effect scales. Simulation effects are stated per scale unit of the covariate: under the linear link as a fraction of \(\beta_0\) at one clade-mean (ED) or one standard deviation (\(D\)) unit (e.g. "-0.15\(\lambda\)"); under the exponential link as \(\log(1+e)\) per standard deviation (e.g. \(\log 0.65\), so the rate one unit above the mean is 0.65 times the rate at the mean). Real-tree coefficients are reported as the factor \(\exp(\hat\beta_X\,\mathrm{sd}_X)\) per standard deviation.
Links
- Linear: \(g(\eta) = \max(0, \eta)\); a coefficient is a change in rate per unit of its covariate. An observed event at a rate truncated to zero gives a draw of zero weight.
- Exponential: \(g(\eta) = e^\eta\); a coefficient is a change in log rate. Used by every ladder and real-data experiment from E20 on.
- Gaussian: rate \(\beta_0\exp\{-(\eta'-1)^2/2\}\), peaking at \(\beta_0\); appears only in E10 and is not a link of the package.
The tree and its parameters
- \(\rho\), sampling fraction: each lineage alive at \(T\) is in the tree independently with known probability \(\rho\) (emphasis);
DDDinstead fixes the number missing (E42). Below \(\rho\) = 1 some covariates of a fit are partly drawn. - Turnover \(\varepsilon\): \(\mu/\lambda\) of the generator (at \(N = 0\) for
dd-type generators), or of a fit read at the crown. - Identifiable: distinct parameter values give distinct distributions of reconstructed trees, given \(\rho\) and the conditioning; a property of the model and the observation, not of a tree.
- Estimable at \(n\) tips: recovered within an experiment's rule on trees of that design (size, turnover, effect size); a property of the model and the design. Experiment pages say "identified at \(n\) tips" in this sense.
- Information a tree carries: the curvature of one tree's likelihood about its maximum; a property of that tree under the model.
- Crown age: the age of the first split, \(T\). Simulation generators use 15 throughout, except E17 and the real trees.
- Complete / reconstructed tree: the complete tree holds every lineage that arose; the reconstructed tree \(y\) the sampled extant lineages and their ancestry. The hidden part \(z\) is the difference; the likelihood is \(L(\theta) = \int f(y, z \mid \theta)\,dz\).
The two proposals
- Thinning (Richter et al. 2021): draws hidden births forward in time as a Poisson process of intensity \(N(t)\lambda_{seg}(1 - \rho e^{-\mu_{seg}(T - t)})\) by thinning, with rates held per segment; parents drawn uniformly over attachments. For ED and \(K\) models the default is the ED-aware variant, which reads per-lineage rates; the mean-field variant is offered.
- Surrogate (
bdi, formerly "BDI"): the exact conditional law of the hidden part under time-dependent rates, a birth-death process with immigration given \(p(t)\), the probability a lineage leaves a sampled descendant. Undercrit is the model's own conditional, so weights are constant and ESS equals the number of draws. Under other models its rates come from a mean field (expected hidden lineages and pendant length), solved as a fixed point by Picard sweeps. - Relaxation \(\omega\): the sweep update \(x \leftarrow (1-\omega)x + \omega F(x)\). The shipped
"auto"starts at 1 and halves on a growing residual, down to 1/64 (E10).
Importance sampling and its diagnostics
- ESS, effective sample size: \((\sum_k w_k)^2/\sum_k w_k^2\) over the importance weights \(w_k = f/q\); equals the number of draws when the weights are equal and falls toward 1 as one draw dominates.
- Jensen gap: \(\log \hat L\) is biased below \(\log L\); to second order by \(\tfrac12(1/\mathrm{ESS} - 1/K)\). A draw set with ESS < K/10 is heavy and its gap not read as a size. The swing \(2(g_m - g_{best})\) is how far correcting the gaps could move model \(m\) toward the leader;
compare_models()flags a ranking as at risk when the swing reaches the AIC difference (E8).
Fitting
- MCEM: each iteration draws hidden parts at the current iterate (E-step), weights them and maximises the weighted complete-data log-likelihood over the box (M-step, Subplex).
rel_change(default stopping rule): stop once the largest relative parameter step \(\delta_k < 10^{-2}\) on three consecutive iterations (floor \(10^{-2}\), or \(10^{-2}/T\) on the linear link).mc_error(offered): measures the step in units of its Monte Carlo standard error from five batches; a step under 1 SE counts as noise and multiplies the draws by 1.5 up to 8 times the initial number; three noise-sized steps at the cap end the run with the mean of the last three iterates (E9).- Caps: 200 iterations and 3600 s by default; the clock is read between iterations, so an iteration in progress is not interrupted (E30). Experiments set their own
max_iterand clock. - The box: the parameter region the M-step searches.
- Study box (
sim/box.R): computed from the observed tree alone, without simulation; with \(\lambda_0 = \log n/T\), intercepts in \([\lambda_0/5, 5\lambda_0]\) (speciation) and \([0, 3\lambda_0]\) (extinction), slopes within \(\pm 3\lambda_0/X\) of zero at the covariate's observed scale \(X\) (on the exponential link the logarithms of these, with extinction from \(\lambda_0/100\), and slopes within \(\pm 3/X\)); start at the midpoint. Used by the pairwise, specificity, guard, noise and stopping experiments (E8, E11, E12, E16, E18, E19-E22, E24, E30, and E25c on real trees). auto_bounds(): the package's box, the range of feasible points (forward simulations giving clades of \(0.1n\) to \(10n\) tips) widened by half, clamped to slope caps \(3/\text{scale}\) and an extinction-intercept floor \(\log(0.1\hat r) - 3\), \(\hat r = \log(n/2)/T\). At 200 tips and crown age 15 this allows ±0.40 per unit of ED and extinction rates down to 0.0014-0.0017 (E29). Used by the ladder and family experiments (E23, E25d, E26-E29, E31-E35, E40).
- Study box (
Model choice
- AIC \(= 2k - 2\hat\ell\), with \(\hat\ell\) from a final E-step at the returned estimate.
- Minimum AIC: the model with the lowest AIC.
- Parsimony reading: the model with the fewest parameters among those within 2 AIC of the minimum; a nested model two parameters richer is then chosen only on a likelihood-ratio statistic above 6.
compare_models()reports it from emphasisf30a791(E23, E27). - The ladder:
estimate_rates(model = "all")fitscr→dd→ {nd,ned,nk}, each rung started from the estimate of the model it nests, new coefficients at zero, each with its ownauto_bounds()box. - The dgLARS path:
covariate_path(), differential-geometric least angle regression (Augugliaro, Mineo & Wit 2013) on the Poisson form of each rate's log-linear point process over the per-lineage segment table of one draw set; covariates enter in order of their Rao score; BIC picks the set. The study draws 3000 hidden parts and enters the 150 of largest weight (package default 40). On a reconstructed tree with no hidden lineages the extinction path is empty.
Conditioning
- Unconditional likelihood: what every fit of the study maximised until E34, and every real-tree fit through E40, on trees kept only if both crown lineages survive; the conditioned fits are E34, E36 (b) and E45.
- Conditioned on survival: \(\ell_S(\theta) = \log L(\theta) - \log P(S \mid \theta)\), with \(S\) the event that both crown lineages leave sampled descendants (E34).
- Survival emulator: \(\hat P(\theta)\), an additive logistic GAM (
mgcv) trained byauto_bounds()on Latin-hypercube points of the box with one simulated clade each. Fromd3dee92: 2000 clades, and a clade that passes the lineage cap counts as survived (before: 500, counted as extinct) (E36).
Markings
- Sampler-limited: median ESS below 10 in a cell (some later designs: below 10 on more than half the fits). Reported, not read against the rule.
- Clock-limited: the wall clock ended more than 10% of the fits (the convention fixed on 2026-10-07 after E31); the budget of a fit is its iteration cap, the clock a safety. Rates are floors.
- Unfinished: fits that ran to their iteration cap without meeting the stopping rule.
Exact references
bd_ML(DDD): the Nee et al. (1994) likelihood of a reconstructed tree under constant rates;cond = 0unconditioned,cond = 1conditioned on crown survival; missing species asmissnumspec.dd_ML(DDD, Etienne et al. 2012): linear diversity dependence of speciation, \(\max\{0, \lambda_0 - (\lambda_0 - \mu_0)N/K\}\), with constant extinction; it is emphasis's linear-linkddon the subspace \(\beta_0 = \lambda_0\), \(\beta_N = -(\lambda_0-\mu_0)/K\), \(\gamma_0 = \mu_0\), \(\gamma_N = 0\). It returns the sentinel -1 outside its parameter space (E38). E42 uses its master-equation likelihood with E40's exponential-link rates.evesim(Qin, Valente & Etienne 2025): an independent simulator, used as a cross-check (E14, E11).