The documented functions of emphasis version 1.1.0.9000: the 12 exported functions first, then the methods and the internal functions that the package documents.

Exported functions

auto_bounds Automatically determine parameter bounds via forward simulation

Description

Starts from a safe centre point (low rates, zero covariates) and expands outward along each parameter axis using bisection to find the feasibility boundary — where trees transition from surviving with sensible size to going extinct or exploding. Much more efficient than blanket LHS sampling over a wide box. Optionally trains a survival GAM over the detected region; the bounds do not depend on it.

Usage

auto_bounds(
  tree,
  model = "cr",
  link = "exponential",
  n_test = 5L,
  bisect_steps = 8L,
  margin = 0.5,
  tip_range = c(0.1, 10),
  train_surv_gam = TRUE,
  num_threads = 1L,
  verbose = TRUE,
  rho = 1,
  surv_tries = 20L
)

Arguments

tree

A phylo object or anything accepted by estimate_rates.

model

Model specification: "cr", "dd", etc.

link

Link function: "exponential" (default), rate = exp(eta), under which the intercepts beta_0 and gamma_0 are log rates; "linear", rate = max(0, eta), under which the parameters are rates and a covariate can drive a rate to zero, where it is clipped.

n_test

Number of replicate simulations per candidate point to assess feasibility (default 5).

bisect_steps

Number of bisection steps per axis direction (default 8; gives ~1/256 resolution).

margin

Fractional expansion of detected bounds (default 0.5).

tip_range

Multiplier range for acceptable tip counts relative to the observed tree. Default c(0.1, 10).

train_surv_gam

If TRUE (default), train a survival GAM on the detected region, from 2000 single forward draws. The GAM is not accurate enough to condition a fit on (experiment E43; see train_GAM).

num_threads

Parallel threads for batch simulation (default 1).

verbose

Print progress (default TRUE).

rho

Sampling fraction (0, 1]. Default 1 (complete sampling).

surv_tries

Draws allowed per simulation, the first included, when judging whether a parameter vector produces surviving clades of a usable size: an extinct clade is redrawn up to this many times in all (passed to simulate_tree as max_tries). Default 20.

Value

A list with components:

lower_bound

Numeric vector of lower bounds (compact layout).

upper_bound

Numeric vector of upper bounds (compact layout).

survival_gam

A gam object if trained, else NULL.

center

The safe centre point used as starting location.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
# A coarse search without the survival GAM
ab <- auto_bounds(phy, model = "cr", n_test = 3L, bisect_steps = 3L,
                  train_surv_gam = FALSE, verbose = FALSE)
ab$lower_bound
ab$upper_bound

# The default search, which also trains the survival GAM (see train_GAM()
# for why a fit should not be conditioned on it)
ab2 <- auto_bounds(phy, model = "cr", verbose = FALSE)
predict_survival(ab2$survival_gam, as.data.frame(t(ab2$center)))

compare_models Compare fitted models by AIC, carrying the Monte Carlo handicap

Description

Ranks two or more fits from estimate_rates by AIC. Each fit's log-likelihood is a log of a Monte Carlo mean, so it sits below the log-likelihood it estimates; how far below grows as that fit's own effective sample falls, and the amounts do not cancel between models. A model that is harder to sample is therefore penalised for being harder to sample, and the AIC difference between two fits can be smaller than the difference in their biases.

Usage

compare_models(..., margin = 2)

Arguments

...

Two or more emphasis_fit objects. Names become the row labels; unnamed arguments are labelled from the fitted model.

margin

AIC units within which two models are not told apart. Default 2.

Details

The returned table carries ESS and loglik_gap per fit and AIC_swing, twice the gap relative to the leader's, which is how much of the AIC difference the bias alone could account for.

ranking_at_risk names the models that could overtake the leader once both biases are corrected. A model qualifies only if its own gap is larger than the leader's — so that correcting both closes the distance — and then either the swing reaches its delta_AIC or its draw is heavy enough that the swing understates the gap. A warning is issued with it.

Two things follow that a caller should not have to infer. The attribute is about the ORDER, not about precision: when the leader is itself the heavily-handicapped fit, correcting the biases only widens its lead, so nothing is named and nothing is wrong with that. And it covers bias, not variance. A fit can lead on a log-likelihood estimated from a handful of effective draws — measured: a winner with an effective sample of 6.5 of 300 — and be named by nothing here, because its expected position is safe while its realised one is noisy. loglik_se and ESS are the columns for that, and they are in the table to be read.

The row with the lowest AIC is not the model the table chooses. Among nested models the richer one has the lower AIC whenever its two extra parameters raise the log-likelihood by more than 2, which they do on a share of trees generated without them: with two extra parameters the likelihood-ratio statistic is chi-squared on two degrees of freedom under the null, so the richer model takes the minimum on about 13% of null trees, and with three richer models offered beside the null, on more. Measured, on 20 trees a cell from a diversity-dependent generator at 50 to 200 tips: a lineage-level model had the minimum AIC on 2 to 7 of 20, and led by more than 2 AIC on 0 to 2 of 20 (E27). The chosen attribute is therefore the model with the fewest parameters among those within margin AIC units of the minimum, the one with the lower AIC when several share that number of parameters, and within_margin marks those rows; the full ranking is kept so that the margin can be read.

A warning names the chosen model when its own effective sample is below 10 draws: its log-likelihood is then an estimate from a handful of histories and the margin it rests on is noise as much as evidence. The ranking_at_risk attribute covers the bias of unequal precision; this covers its variance.

Value

A data frame, one row per fit, with columns model, n_pars, loglik, AIC, loglik_se, ESS, loglik_gap, gap_heavy, delta_AIC, AICw, AIC_swing and within_margin, ordered by AIC, with attributes chosen (the most parsimonious model within margin of the minimum), chosen_starved (whether that model's effective sample is below 10) and ranking_at_risk.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
ctrl <- list(max_iter = 5, num_trees = 30)
fit_cr <- estimate_rates(phy, method = "mcem", model = "cr",
  control = c(ctrl, list(lower_bound = c(-3, -5), upper_bound = c(1, 0))))
fit_dd <- estimate_rates(phy, method = "mcem", model = "dd",
  control = c(ctrl, list(lower_bound = c(-3, -0.1, -5, -0.1),
                         upper_bound = c(1, 0.1, 0, 0.1))))
compare_models(cr = fit_cr, dd = fit_dd)

covariate_path Selection along a path over the covariates

Description

Instead of comparing nested models one pair at a time, this traces the differential-geometric LARS path of Augugliaro, Mineo and Wit (2013) over all the covariates a rate can read — N, D and ED — on the same augmented trees an M-step maximises over. Under the exponential link the complete-data log-likelihood is a log-linear point process: one row per lineage and segment between consecutive events, with N, D and K at their values on the segment (each is constant between events), ED at the segment's midpoint, the segment's length as exposure, and a count of one when that lineage's own event ends the segment. The speciation and extinction halves separate, so each rate gets its own path.

Usage

covariate_path(
  trees,
  log_weights = NULL,
  n_trees = 40L,
  covariates = c("N", "D", "ED"),
  rate = c("speciation", "extinction"),
  criterion = c("BIC", "AIC"),
  n_gamma = 60L
)

Arguments

trees

A list of augmented-tree data frames with the topology, as augment_trees returns them with parent ids, or a single phylo taken as fully observed.

log_weights

Optional log importance weights, one per tree; without them every tree weighs the same.

n_trees

How many trees to use: the n_trees of largest weight (the first n_trees without weights), renormalised.

covariates

Which covariates to offer the path; any of "N", "D", "ED", "EDc" (ED centred on the alive lineages), "K" (the number of speciation events on the path from the crown).

rate

"speciation", "extinction" or both.

criterion

"BIC" or "AIC" along the path.

n_gamma

Number of points on the path.

Details

The path is the curve on which the Rao score statistics of the active covariates are equal in size and the inactive ones smaller, \(|r_j(\beta)| = \gamma\) for \(j\) active, traced from the \(\gamma\) at which the first covariate enters down to zero, with the intercept always free. It is solved here on a grid of \(\gamma\) by Newton steps rather than by dglars's predictor–corrector, because the point process needs a true exposure offset, which that package does not take; on a constant-exposure problem the two agree (tests/testthat/test-lineage-table.R). The order of entry says which covariates the data ask for first; AIC and BIC along the path pick a point, with BIC's sample size the number of events.

Importance weights enter the likelihood as row weights, one per tree and normalised to sum to one, so the objective is the expected complete-data log-likelihood of one tree — the M-step's Q function — and the sample size behind BIC is the weighted mean number of events per tree.

Value

A list with one element per rate: the path (a data frame with one row per \(\gamma\): the coefficients, the log-likelihood, the active set size, AIC and BIC), the entry order of the covariates (the \(\gamma\) at which each enters), the coefficients and active set at the criterion's optimum, and the frame's size. A rate with no event in the trees (the extinction rate of a fully observed tree) gets an empty path: path is NULL, active and entry_order are empty and n_events is 0.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
cp <- covariate_path(phy, rate = "speciation")
cp$speciation$entry_order

diagnose_cem Diagnostics for a CEM fit

Description

Computes convergence, importance-sampling quality, and population-diversity diagnostics from the output of emphasis_cem or estimate_rates (method "cem").

Usage

diagnose_cem(
  x,
  plot = TRUE,
  lower_bound = NULL,
  upper_bound = NULL,
  true_pars = NULL
)

Arguments

x

An emphasis_fit object (from estimate_rates with method = "cem") or the raw list from emphasis_cem.

plot

Logical; if TRUE (default), produce diagnostic plots.

lower_bound, upper_bound

Optional numeric vectors of the same length as the parameter vector. When provided, dashed horizontal lines are drawn at the lower (orange) and upper (purple) bounds in each parameter-trace panel, making it easy to see whether the estimate has converged away from the boundary.

true_pars

Optional named or unnamed numeric vector of true parameter values. Useful in simulation studies where the generative parameters are known. When provided, a red dashed line is added to each parameter-trace panel at the true value. In real-data analyses this argument should be omitted (the true parameters are unknown).

Details

The function works on the details component of an emphasis_fit object or directly on the list returned by emphasis_cem. It requires that the fit was produced with the current version of the package (which stores history, final_pop, and best_IS in the output).

Convergence ($convergence)

A data frame with one row per iteration containing:

iteration

Iteration index.

best_loglik

IS log-likelihood of the best particle.

n_valid

Number of particles with a finite fhat.

rej_lambda

Trees rejected due to max_lambda exceedance.

rej_overruns

Trees rejected due to max_missing exceedance.

Population spread ($population_spread)

Per-iteration summary of the fhat distribution across all valid particles: median, interquartile range, and total range. A narrowing range indicates population convergence.

IS quality ($IS_quality)

Importance-sampling diagnostics computed from the final evaluation of the globally best parameter vector (with max(sample_size, n_boot, 1) augmented trees):

ESS

Effective sample size: \((\sum w_i)^2 / \sum w_i^2\).

ESS_fraction

ESS as a fraction of n_trees.

mean_lw, sd_lw

Mean and SD of IS log-weights.

fhat

IS log-likelihood estimate.

n_trees, n_rejected

Trees simulated and rejected.

Final population ($final_pop)

The full particle population at the last completed iteration (before resampling), with columns for parameters and fhat per particle. Useful for inspecting the distribution of surviving particles.

Raw IS data ($best_IS)

The raw IS output for the best particle: logf, log_q, lw (log-weights), and trees (augmented tree data frames in C++ format).

Value

A list of class "cem_diagnostics" (invisibly) with components convergence, population_spread, IS_quality, final_pop, and best_IS.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
# Exponential link (the default): the intercepts are log rates
lb <- c(-3, -5)
ub <- c(1, 0)
fit <- estimate_rates(phy, method = "cem", model = "cr",
  control = list(lower_bound = lb, upper_bound = ub,
                 max_iter = 5, num_particles = 20))
diag <- diagnose_cem(fit, lower_bound = lb, upper_bound = ub,
                     true_pars = log(c(0.5, 0.1)))
diag$convergence

diagnose_gam Diagnostics for a GAM-based MLE fit

Description

Produces diagnostic plots and returns summary statistics from a estimate_rates fit obtained with method = "gam".

Usage

diagnose_gam(x, plot = TRUE, true_pars = NULL)

Arguments

x

An emphasis_fit object from estimate_rates with method = "gam".

plot

Logical; if TRUE (default), produce diagnostic plots.

true_pars

Optional numeric vector of true parameter values (for simulation studies).

Surface coverage ($surface_summary)

Summary of the IS log-likelihood grid: total grid points, number with finite fhat, fhat range, and mean trees per grid point.

GAM quality ($gam_summary)

Key statistics from the fitted GAM: deviance explained, GCV score, and effective degrees of freedom.

MLE ($mle)

The MLE parameter estimates and predicted log-likelihood at the optimum.

Value

A list of class "gam_diagnostics" (invisibly) with components surface_summary, gam_summary, and mle.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
fit <- estimate_rates(phy, method = "gam", model = "cr",
  control = list(lower_bound = c(-3, -5), upper_bound = c(1, 0),
                 grid_points = 8, sample_size = 20))
diagnose_gam(fit)

diagnose_mcem Diagnostics for an MCEM fit

Description

Computes convergence and importance-sampling quality diagnostics from the output of estimate_rates with method = "mcem".

Usage

diagnose_mcem(
  x,
  plot = TRUE,
  lower_bound = NULL,
  upper_bound = NULL,
  true_pars = NULL
)

Arguments

x

An emphasis_fit object (from estimate_rates with method = "mcem") or the raw list from .mcem_dynamic_fresh.

plot

Logical; if TRUE (default), produce diagnostic plots.

lower_bound, upper_bound

Optional numeric vectors for bound lines.

true_pars

Optional numeric vector of true parameter values (for simulation studies). Red dashed lines are drawn at these values.

Convergence ($convergence)

A data frame with one row per EM iteration. The trace's last row, the final E-step at the returned parameters, is not an iteration and is not included; the table therefore has fit$iterations rows and its fhat of the final E-step is reported as $final_fhat. Columns:

iteration

Iteration index.

fhat

IS log-likelihood at the previous iterate (the E-step value the M-step of this iteration maximised).

delta_max

Maximum relative parameter change from the previous iteration, each coordinate measured against its own magnitude.

rejected

Draws rejected in the iteration's E-step: n_rejected (the sum over rejection channels) where the trace carries it, else the trace's own rejected column.

num_trees

Number of augmented trees used.

time

Wall-clock time for the iteration (ms).

IS quality ($IS_quality)

Importance-sampling diagnostics from the final EM iteration:

ESS

Effective sample size.

ESS_fraction

ESS as a fraction of num_trees.

mean_lw, sd_lw

Mean and SD of IS log-weights.

fhat

IS log-likelihood at the final parameters.

loglik_var

Bootstrap variance of fhat.

Value

A list of class "mcem_diagnostics" (invisibly) with components convergence, stop_reason, final_fhat, IS_quality, and final_IS.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
fit <- estimate_rates(phy, method = "mcem", model = "cr",
  control = list(lower_bound = c(-3, -5), upper_bound = c(1, 0),
                 max_iter = 10, num_trees = 50))
diagnose_mcem(fit, lower_bound = c(-3, -5), upper_bound = c(1, 0))

emphasis_pipeline Full estimation pipeline: auto_bounds -> GAM -> CEM -> MCEM

Description

Runs a multi-stage estimation workflow with automatic bounds detection, exploratory fits (GAM and CEM), and optional MCEM refinement. Each stage is timed and logged. If MCEM fails, the best exploratory result is returned.

Usage

emphasis_pipeline(
  tree,
  model = "cr",
  link = "exponential",
  stages = c("bounds", "gam", "cem", "mcem"),
  control = list(),
  verbose = TRUE
)

Arguments

tree

A phylo object, branching-time vector, or simulate_tree result.

model

Model specification: "cr", "dd", "d", "nd", "ed", "ned", "k", "nk", a formula (e.g. ~ N + D, ~ N + ED, ~ N + K), or a binary vector.

link

Link function: "exponential" (default), rate = exp(eta), under which the intercepts beta_0 and gamma_0 are log rates; "linear", rate = max(0, eta), under which the parameters are rates and a covariate can drive a rate to zero, where it is clipped.

stages

Character vector of stages to run. Default c("bounds", "gam", "cem", "mcem"). Remove stages to skip them, e.g. c("bounds", "gam") for a quick exploratory fit.

control

Named list with optional sub-lists bounds, gam, cem, mcem to override defaults for each stage. Common parameters (num_threads, default 1, max_missing, max_time, rho) can be set at the top level and are inherited by all stages. rho, the sampling fraction in (0, 1], is inherited by the bounds, gam, cem and mcem stages alike, and a value nested under control$gam, control$cem or control$mcem overrides the inherited one for that stage.

verbose

Logical. If TRUE (default), print progress and a summary log to the console.

Stages

  1. Bounds: auto_bounds detects parameter bounds and trains a survival GAM, which the later stages condition on. The GAM is not accurate enough for that (experiment E43; see train_GAM). To condition on another survival probability, or not at all, leave this stage out and give control$lower_bound, control$upper_bound and, to condition, control$survival_gam in any form estimate_rates(cond = ) accepts.

  2. GAM: Fits a smooth likelihood surface over a grid and optimises it. Fast, one-shot.

  3. CEM: Cross-entropy global search. Stochastic, no init needed. Explores the space differently than GAM.

  4. MCEM: Monte Carlo EM starting from the best init found in stages 2–3. Local refinement for precise estimates.

Value

A list of class "emphasis_pipeline" with:

pars

Named parameter vector from the best stage.

loglik

Log-likelihood from the best stage.

AIC

AIC from the best stage.

best_stage

Which stage produced the final result.

rho

The sampling fraction the best stage ran at.

log

Data frame with one row per stage: stage name, pars, loglik, AIC, elapsed seconds, status, rho.

mcem_trace

If MCEM ran, a data frame of per-iteration fhat and parameter values. NULL otherwise.

bounds

The auto_bounds result (bounds + survival GAM).

fits

Named list of individual emphasis_fit objects.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
# CEM stage only, with the search box given (log rates: the
# exponential link is the default)
res <- emphasis_pipeline(phy, model = "cr", stages = "cem",
  control = list(lower_bound = c(-3, -5), upper_bound = c(1, 0),
                 cem = list(max_iter = 3, num_particles = 10)),
  verbose = FALSE)
res

# Bounds found automatically, then CEM
res2 <- emphasis_pipeline(phy, model = "cr", stages = c("bounds", "cem"),
  control = list(cem = list(max_iter = 3, num_particles = 10)),
  verbose = FALSE)
res2$log

estimate_rates Estimate diversification rates from a phylogenetic tree

Description

Unified entry point for parameter inference under the general diversification model. Accepts a simulated or empirical tree, extracts branching times (pruning to extant tips if needed), and dispatches to the chosen optimisation back-end.

Usage

estimate_rates(
  tree,
  method = c("mcem", "cem", "gam"),
  model = "cr",
  init_pars = NULL,
  control = list(),
  link = "exponential",
  cond = NULL
)

Arguments

tree

One of:

  • a phylo object (extant tips only recommended),

  • the list returned by simulate_tree ($tes is used automatically, falling back to a pruned $tas), or

  • a numeric vector of branching times (crown-age ordering).

The covariate D is a lineage's pendant age – measured from the last time that lineage became a tip – centred on the mean over the alive lineages, so it is a function of the topology and not of the branching times alone. A phylo (or a simulate_tree result) carries it; a bare numeric vector does not, and every observed lineage is then recorded as dating from the crown, which leaves D = 0 at every observed branching event. Fit "d" or "nd" from a tree, not from branching times.

method

Optimisation method:

"mcem"

Monte Carlo Expectation-Maximisation (default). Iterates E- and M-steps until parameters stabilise (max relative change < tol for patience consecutive iterations) or max_iter is reached. Requires init_pars.

"cem"

Monte Carlo Cross-Entropy Method. Initialises a particle population randomly within the bounds and evolves it until convergence (annealing exhausted, plateau, or control$max_iter reached). Does not need init_pars.

"gam"

GAM-based MLE. Evaluates the IS log-likelihood on a grid spanning the bounds, fits a smooth GAM surface, and optimises it via L-BFGS-B. One-shot (non-iterative); no init_pars needed. Requires the mgcv package.

model

Model specification. Accepts:

  • a formula such as ~ N, ~ N + D, ~ N + ED or ~ N + K,

  • a string shortcut ("cr", "dd", "d", "nd", "ed", "ned", "edc", "nedc", "k", "nk"), or

  • an integer vector c(use_N, use_M, use_D, use_ED, use_K) (use_M is internal only and should be 0; a shorter vector is an earlier layout and is padded; use_ED = 3 selects the centred covariate).

model = "all" fits the nested ladder cr, dd, nd, ned, nk (the last two need a phylo), each started from the estimate of the model it nests, with each model's box from auto_bounds unless control$bounds names one per model, and returns the fits with their compare_models table and its chosen model, the fewest parameters within 2 AIC of the minimum (an emphasis_fits object); control$models restricts the ladder. Default "cr" (constant rate). "nd" (~ N + D) is diversity and age-imbalance; "ned" (~ N + ED) diversity and evolutionary distinctiveness, which needs a phylo. "nedc" (~ N + EDc) reads ED minus its mean over the lineages alive at the same time, a covariate with clade mean zero, so that beta_ED carries only the within-clade contrast and none of the clade-level level that N tracks. "nk" (~ N + K) is diversity and the number of speciation events on the path from the crown to the lineage, hidden splits counted; it needs a phylo and runs on the thinning sampler.

init_pars

Starting parameter vector. Required for "mcem"; ignored for "cem". If NULL with "mcem", the midpoint of the bounds is used, with covariate slopes started at 0 (linear link) so that lambda stays positive during the early E-steps.

control

Named list of tuning parameters. Must include lower_bound and upper_bound. Use estimate_rates_control to inspect other defaults. Key entries:

lower_bound

Numeric vector of lower parameter bounds. Length must be 2 + 2*sum(model). Order: c(beta_0, [beta_N], [beta_D], gamma_0, [gamma_N], [gamma_D]) (D = age-imbalance covariate; M is internal only).

upper_bound

Numeric vector of upper parameter bounds (same order).

verbose

Print progress messages. Default FALSE.

link

Link function: "exponential" (default), rate = exp(eta), under which the intercepts beta_0 and gamma_0 are log rates; "linear", rate = max(0, eta), under which the parameters are rates and a covariate can drive a rate to zero, where it is clipped.

cond

The survival probability to condition on, or NULL (default) for no conditioning. When provided, the log-likelihood is conditioned: \(\ell_{\rm cond} = \ell - \log P_\theta(S)\), where \(P_\theta(S)\) is the probability that both crown lineages leave a sampled descendant at the present. Three forms are accepted: a function of the parameter data frame that returns \(P_\theta(S)\) (the probability, not its log), a GAM from train_GAM, or a nnet object (see predict_survival). The GAM is not accurate enough to condition on; the section ‘Conditioning on survival’ says what to pass instead.

Conditioning on survival

The survival emulator of train_GAM is not adequate for any lineage-level model. In experiment E43 (https://franciscorichter.github.io/emphasis/experiments/E43.html), GAMs trained on 2000, 5000 and 20 000 simulated clades for each of d, nd, ed, edc, ned, nedc, k and nk had a 90th-percentile error of \(\log P(S)\) of 0.44 to 2.56, against the 0.05 its adequacy rule required, and the error did not fall with the training size.

Under cr and dd, condition on the exact survival probability. The package does not export a function for it: pass one to cond. The function receives a data frame with one row per parameter point and columns named as the parameters (beta_0, gamma_0 under cr), and returns \(P_\theta(S)\), one value per row. Under constant rates on the exponential link, with \(\lambda = e^{\beta_0}\), \(\mu = e^{\gamma_0}\), \(d = \lambda - \mu\), crown age \(T\) and sampling fraction \(\rho\), one crown lineage leaves a sampled descendant at the present with probability

\[p = \frac{\rho d}{\rho\lambda + (\lambda(1-\rho) - \mu)e^{-dT}}\]

(\(p = \rho / (1 + \rho\lambda T)\) when \(\lambda = \mu\)); the two crown lineages are independent, so \(P_\theta(S) = p^2\). The last example below codes it. Under dd the exact value solves the master equation of the clade size, which the package does not provide. E43's Monte Carlo reference agreed with the exact values to a median error of 0.007 (cr) and 0.008 (dd) in \(\log P\).

Under a lineage-level model, either fit without conditioning or pass a function that returns a direct Monte Carlo estimate of \(P_\theta(S)\) at each row: the share of single forward draws from simulate_tree (max_tries = 1) in which both crown lineages leave a sampled descendant, a clade that reaches the lineage cap counted as surviving (at \(\rho = 1\), the share with status "done" or "too_large"). At a standard error of 0.01 in \(\log P\), one such estimate took E43 a median of 2 to 29 seconds of one core, depending on the model, and the M-step evaluates cond at every parameter value it tries.

Value

A list with class "emphasis_fit" containing:

pars

Named numeric vector of parameter estimates.

loglik

Final log-likelihood estimate. For MCEM this is the importance-sampling estimate at the returned parameters, and is NA when the E-step evaluating them failed.

loglik_var

Bootstrap variance of loglik due to Monte Carlo noise. Populated automatically when control$num_trees >= 2 (B = 200 replicates); NA when num_trees = 1.

n_pars

Number of free parameters (used for AIC): parameters with lower_bound == upper_bound are fixed and not counted.

AIC

Akaike Information Criterion: -2 * loglik + 2 * n_pars.

method

Method used ("mcem" or "cem").

model

Resolved binary model vector.

rho

The sampling fraction the fit was computed at, after the control merge.

stop_reason

Why the MCEM run stopped ("converged", "max_iter", "time_budget", "e_step_failure", "m_step_failure"); NA for other methods.

iterations

Number of completed MCEM iterations (0 when every E-step failed); NA for other methods.

n_failed

Number of E- or M-step failures over the MCEM run; NA for other methods.

notes

Named character vector of what the measurements and the run say about this fit: size, starved and turnover (emitted as a warning or messages when the fit is made), mean_field (the surrogate proposal's mean-field iteration was relaxed below the full step, ran under a damping other than "auto", or did not converge within its sweep budget: the setting, the sweeps taken, the relaxation and residual of the last E-step, and how many E-steps of the run relaxed or failed to converge) and clock (max_time ran out inside an E-step or an M-step). print() shows the last two.

details

Full output from the back-end function (.mcem_dynamic_fresh or emphasis_cem).

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
# Exponential link (the default): the intercepts are log rates
fit <- estimate_rates(phy, method = "mcem", model = "cr",
  control = list(lower_bound = c(-3, -5), upper_bound = c(1, 0),
                 max_iter = 10, num_trees = 50))
fit

# Diversity-dependent model on the same tree
fit_dd <- estimate_rates(phy, method = "mcem", model = "dd",
  control = list(lower_bound = c(-3, -0.1, -5, -0.1),
                 upper_bound = c(1, 0.1, 0, 0.1),
                 max_iter = 10, num_trees = 50))
fit_dd$pars

# Conditioning a cr fit on its exact survival probability, P(S) = p^2
cond_cr <- function(T, rho = 1) function(pars) {
  lambda <- exp(pars$beta_0)
  mu     <- exp(pars$gamma_0)
  d      <- lambda - mu
  p <- ifelse(abs(d) < 1e-12, rho / (1 + rho * lambda * T),
              rho * d / (rho * lambda + (lambda * (1 - rho) - mu) * exp(-d * T)))
  p^2
}
crown_age <- max(ape::branching.times(phy))
fit_cond <- estimate_rates(phy, method = "mcem", model = "cr",
  cond = cond_cr(T = crown_age),
  control = list(lower_bound = c(-3, -5), upper_bound = c(1, 0),
                 max_iter = 10, num_trees = 50))
fit_cond

lineage_table The per-lineage design of an augmented tree

Description

One row per lineage alive on each segment between consecutive events of an augmented tree: the segment's bounds, the covariates the rates read there (N, the clade mean pendant age M at the segment's end t1, the lineage's pendant start ts, so that D = (t1 - ts) - M, constant on the segment, and its ED at the segment start, growing at slope one within it), and whether the lineage's own event ends the segment (event: 0 none, 1 speciation, 2 extinction). These are the numbers the likelihood is built on; covariate_path runs a selection path over them.

Usage

lineage_table(tree)

Arguments

tree

An augmented-tree data frame with the topology (the frames augment_trees returns when given parent ids), or a phylo object, which is taken as fully observed.

Value

A data frame with columns seg, lineage, id, event, t0, t1, N, M, ts, ed0.

Examples

set.seed(1)
tr  <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
phy <- tr$tes
head(lineage_table(phy))

predict_survival Predict survival probability from a trained emulator

Description

Evaluates the survival probability estimate_rates(cond = ) conditions through, in any of its three forms: a function that takes the parameter data frame and returns probabilities, a gam from train_GAM, or a nnet object.

Usage

predict_survival(gam_fit, newpars)

Arguments

gam_fit

A gam object returned by train_GAM, a nnet object trained on the same columns, or a function of a parameter data frame.

newpars

Numeric matrix or data frame of parameter values to predict at. Column names must match those used in train_GAM.

Details

The GAM is not accurate enough to condition on: in experiment E43 (https://franciscorichter.github.io/emphasis/experiments/E43.html) its 90th-percentile error of \(\log P(S)\) was 0.44 to 2.56 for every lineage-level model, against 0.05 required. Under cr and dd pass the exact survival probability as a function; the section ‘Conditioning on survival’ of estimate_rates gives the cr formula and the alternatives.

Value

Numeric vector of predicted survival probabilities.

Examples

set.seed(1)
# Exponential link (the default): the intercepts are log rates
pars_mat <- cbind(beta_0  = log(runif(100, 0.3, 1.2)),
                  gamma_0 = log(runif(100, 0.05, 0.5)))
sims <- simulate_tree(pars = pars_mat, max_t = 4, model = "cr",
                      max_tries = 1)
gam_fit <- train_GAM(sims$simulations, pars_mat, model = "cr")
predict_survival(gam_fit, data.frame(beta_0  = log(c(0.5, 1.0)),
                                     gamma_0 = log(c(0.1, 0.3))))

simulate_tree Simulate a phylogenetic tree under the general diversification model

Description

Unified entry point for forward and conditional tree simulation. When tree = NULL (default) a complete tree is simulated forward from the crown; when tree is supplied the observed extant tree is augmented with stochastically drawn extinct lineages.

Usage

simulate_tree(
  tree = NULL,
  pars,
  max_t = 1,
  model = "cr",
  max_lin = 1e+06,
  max_tries = 1,
  useDDD = TRUE,
  n_trees = 1L,
  link = "exponential",
  max_missing = 10000,
  max_lambda = 500,
  maxN = NULL,
  num_threads = 1L,
  rho = 1,
  method = "bdi"
)

Arguments

tree

Optional observed extant tree for augmentation. Accepts NULL (forward sim), a simulate_tree result, a phylo object, or a numeric branching-time vector. A phylo also supplies the topology the pendant-age covariates M and D are computed from; with a bare branching-time vector every observed lineage is recorded as dating from the crown and D = 0 at every observed branching event.

pars

Numeric parameter vector or matrix. When a matrix each row produces one forward simulation.

max_t

Crown age (forward simulation only). Default 1.

model

Model specification (string, formula, or binary vector). Default "cr".

max_lin

Maximum lineages alive before declaring the tree too large; a clade that has had ten times as many lineages in all, alive and dead, is too large as well, which bounds the cost of a simulation that churns (high speciation and extinction over a long crown) under the cap. Default 1e6.

max_tries

Maximum number of forward simulations in all, the first included (forward simulation only): a clade that goes extinct is drawn again until one survives or max_tries draws have been made, so the default 1 is a single draw, unconditioned on survival. A positive integer; 0, which asked for a single draw when the argument counted the draws after the first, is read as 1 with a warning. A clade that overflows max_lin is not retried: the overflow is a property of the parameters, not of the draw, and every retry would run to the cap again. The draws are counted at the R level (attempts) to compute survival_prob. Default 1.

useDDD

Convert L-table to phylo via DDD. Default TRUE.

n_trees

Number of augmented trees to draw (augmentation only). Default 1L.

link

Link function: "exponential" (default), rate = exp(eta), under which the intercepts beta_0 and gamma_0 are log rates; "linear", rate = max(0, eta), under which the parameters are rates and a covariate can drive a rate to zero, where it is clipped.

max_missing

Maximum extinct lineages per augmented tree. Default 1e4.

max_lambda

Maximum speciation rate (thinning bound) for augmentation. Default 500.

maxN

Maximum total augmentation attempts. NULL (default) sets this to max(2000, 200 * n_trees).

num_threads

Threads for parallel augmentation. Default 1L.

rho

Sampling fraction in (0, 1]. Default 1 (complete sampling). With rho < 1 a random fraction of extant tips is dropped (forward simulation) or unsampled extant lineages are inserted (augmentation), matching incomplete taxon sampling.

method

Augmentation method: "bdi" (default, exact BDI sampler) or "thinning" (C++ thinning proposal). The BDI sampler draws from the exact conditional distribution under constant rates (ESS = S, zero variance) and uses a self-consistent backward-forward iteration under diversity dependence. It covers N-only models ("cr", "dd") on the linear and exponential links; any other model or link falls back to thinning.

method = "bdi" returns NULL for part of its draws. The BDI sampler has no notion of the two crown lineages, so a lineage born before the first observed branching is recorded as a daughter of observed node 0, which is not yet born then. Such a draw cannot be turned into a tas: it is refused, and that element of trees (or tas, with n_trees = 1) is NULL. On seven trees measured for this release, 0 to 168 of 200 draws built; a tree whose first observed branching is late loses most of them. log_q is returned for every draw either way. It was previously silent and wrong – every such draw produced a phylo with an edge of negative length. The "thinning" method draws the parent from every lineage alive, the two crown lineages included, and is not affected.

Model specification

The model's covariates are diversity N, age-imbalance D = E - M (a lineage's isolation relative to the clade mean M, which is used only internally to centre D) and evolutionary distinctiveness ED (the fair-proportion score on the complete tree: the branches from the crown to the lineage, each divided by the alive lineages below it; its last term is the pendant age and its sum over lineages is Faith's phylogenetic diversity). Selected by named shortcuts or a formula:

"cr" constant rate
"dd" diversity dependence (N)
"d" age-imbalance (D)
"nd" N + D
"ed" evolutionary distinctiveness (ED)
"ned" N + ED
"edc" centred ED (ED minus its mean over the lineages alive)
"nedc" N + centred ED
"k" ancestral splits (K: the speciation events on the path from the crown)
"nk" N + K

Formulas ~ N, ~ N + D, ~ N + ED, ~ N + EDc and ~ N + K are also accepted (ep is a legacy alias for d). An ED or K model needs the tree's topology (a phylo, not a branching-time vector).

Parameter vector

Compact layout; beta_D is the age-imbalance coefficient and beta_ED the distinctiveness coefficient (M is internal only, never part of the user vector):

c(beta_0,  [beta_N],  [beta_D],  [beta_ED],
  gamma_0, [gamma_N], [gamma_D], [gamma_ED])

Only active covariates are included; length = 2 + 2 * sum(model).

Reproducibility

Every call draws its seed for the C++ sampler from R's generator, so set.seed() fixes both the forward simulator and the thinning augmenter: two runs under one seed return the same tree, two runs under different seeds do not. The retries a forward simulation makes draw a seed each, so they are distinct trees rather than one tree redrawn. Under parallel::mclapply each child has its own R stream and therefore its own seeds. Augmentation with num_threads > 1 is not reproducible: each worker draws from its own substream of the seed, but the sample keeps the first n_trees augmentations to finish.

Output – forward simulation (single)

tes

Extant-only phylogeny (phylo), or NULL.

tas

Full phylogeny with extinct lineages (phylo), or NULL.

L

L-table (DDD format), or NULL if simulation failed.

status

"done", "extinct", or "too_large".

survival_prob

Empirical survival probability: 1 / n_attempts when the simulation succeeded, 0 otherwise.

attempts

Number of forward simulations run, counting the retries after extinction; 1 when the first draw completed or overflowed.

Output – forward simulation (batch, matrix pars)

simulations

List of individual results (one per row of pars), each with the fields above.

survival_prob

Fraction of simulations with status == "done".

Output – conditional simulation / augmentation

When n_trees = 1:

tas

Augmented phylogeny (phylo) including extinct lineages, or NULL on failure or when useDDD = FALSE.

log_q

Log sampling probability \(\log q(z \mid \text{obs}, \theta)\) of the augmentation. NA on failure.

When n_trees > 1:

trees

List of tas phylogenies (length up to n_trees; failed draws are NULL).

log_q

Numeric vector of log sampling probabilities, one per successful draw.

Value

A list whose components depend on the mode: for a single forward simulation tes, tas, L, status, survival_prob and attempts; for a matrix pars a list with simulations and survival_prob; for an augmentation the augmented tree(s) and their proposal log-densities log_q. The sections below list every component.

Examples

# Forward simulation.  Under the exponential link (the default) the
# intercepts are log rates: speciation 0.5, extinction 0.1
set.seed(1)
tr <- simulate_tree(pars = log(c(0.5, 0.1)), max_t = 4, model = "cr")
tr$status
tr$survival_prob

# Batch: 20 constant-rate simulations
pm <- cbind(beta_0  = log(runif(20, 0.3, 0.8)),
            gamma_0 = log(runif(20, 0.05, 0.3)))
sims <- simulate_tree(pars = pm, max_t = 4, model = "cr")
sims$survival_prob

# Augmentation of the extant tree
aug <- simulate_tree(tree = tr, pars = log(c(0.5, 0.1)), model = "cr")
aug$log_q

# Many augmented trees
augN <- simulate_tree(tree = tr, pars = log(c(0.5, 0.1)), model = "cr",
                      n_trees = 20L)
length(augN$log_q)

# The linear link: the parameters are the rates themselves
tr_lin <- simulate_tree(pars = c(0.5, 0.1), max_t = 4, model = "cr",
                        link = "linear")

train_GAM Train a survival-probability GAM on forward simulation results

Description

Fits a binomial GAM to estimate P(survival | parameters) from forward simulations run with max_tries = 1 (one draw each, unconditioned on survival). Each simulation row has a binary outcome: 1 if both crown lineages reached the present (status "done") or the clade passed the lineage cap ("too_large"), 0 if it went extinct. A clade stopped at the cap has survived so far, and the conditional likelihood divides by the probability of survival; counting it as extinct teaches the GAM a survival near zero wherever clades grow fast, which then rewards the fit for moving there (E36c of the paper).

Usage

train_GAM(
  simulations,
  pars_mat,
  model = "cr",
  spline_type = c("univariate", "bivariate"),
  verbose = TRUE
)

Arguments

simulations

List of simulate_tree() outputs (one per row of pars_mat), produced with max_tries = 1.

pars_mat

Numeric matrix whose rows are the parameter vectors used to generate simulations. Column names are used as predictor names; if absent, names are auto-generated from model.

model

String or binary integer vector (length 3 or 4) matching the model used in simulation. Used only when pars_mat has no column names. Default "cr".

spline_type

"univariate" (default) fits independent smooth terms s(x) for each varying predictor. "bivariate" fits tensor-product smooths te(x, y) for every pair of varying predictors, which can capture interaction effects.

verbose

Report the number of simulations and the survival rate (default TRUE).

Details

The GAM is not accurate enough to condition a fit on. In experiment E43 (https://franciscorichter.github.io/emphasis/experiments/E43.html), GAMs trained on 2000, 5000 and 20 000 clades for each of the eight lineage-level models had a 90th-percentile error of \(\log P(S)\) of 0.44 to 2.56, against the 0.05 its adequacy rule required, and the error did not fall with the training size. Condition cr and dd fits on the exact survival probability, and lineage-level fits on a direct Monte Carlo estimate or not at all; the section ‘Conditioning on survival’ of estimate_rates shows how.

Value

A gam object (binomial family) representing \(P(\text{survival} \mid \theta)\).

Examples

set.seed(1)
# Exponential link (the default): the intercepts are log rates
pars_mat <- cbind(beta_0  = log(runif(100, 0.3, 1.2)),
                  gamma_0 = log(runif(100, 0.05, 0.5)))
sims <- simulate_tree(pars = pars_mat, max_t = 4, model = "cr",
                      max_tries = 1)
gam_uni <- train_GAM(sims$simulations, pars_mat, model = "cr")
predict_survival(gam_uni, data.frame(beta_0 = log(0.8), gamma_0 = log(0.2)))

Methods

print.cem_diagnostics Print method for CEM diagnostics

Description

Print method for CEM diagnostics

Usage

## S3 method for class 'cem_diagnostics'
print(x, ...)

Arguments

x

Output of diagnose_cem.

...

Additional arguments (ignored).

Value

x, invisibly.

print.emphasis_fit Print method for emphasis_fit objects

Description

Print method for emphasis_fit objects

Usage

## S3 method for class 'emphasis_fit'
print(x, ...)

Arguments

x

An emphasis_fit object returned by estimate_rates.

...

Additional arguments (ignored).

Value

x, invisibly.

print.emphasis_fits Print method for the fits of estimate_rates(model = "all")

Description

Print method for the fits of estimate_rates(model = "all")

Usage

## S3 method for class 'emphasis_fits'
print(x, ...)

Arguments

x

An emphasis_fits object.

...

Ignored.

Value

x, invisibly.

print.emphasis_pipeline Print method for emphasis_pipeline objects

Description

Print method for emphasis_pipeline objects

Usage

## S3 method for class 'emphasis_pipeline'
print(x, ...)

Arguments

x

An emphasis_pipeline result.

...

Additional arguments (ignored).

Value

x, invisibly.

print.gam_diagnostics Print method for GAM diagnostics

Description

Print method for GAM diagnostics

Usage

## S3 method for class 'gam_diagnostics'
print(x, ...)

Arguments

x

Output of diagnose_gam.

...

Additional arguments (ignored).

Value

x, invisibly.

print.mcem_diagnostics Print method for MCEM diagnostics

Description

Print method for MCEM diagnostics

Usage

## S3 method for class 'mcem_diagnostics'
print(x, ...)

Arguments

x

Output of diagnose_mcem.

...

Additional arguments (ignored).

Value

x, invisibly.

Internal functions

augment_trees Draw augmented trees via importance sampling

Description

Augments an observed extant tree (given by its branching times) with stochastically drawn extinct lineages, and returns the simulated trees together with their log proposal probabilities. Scoring (logf) and IS aggregation (fhat) are intentionally left to the R layer via eval_logf and .is_fhat.

Usage

augment_trees(
  brts,
  pars,
  sample_size,
  maxN,
  max_missing,
  max_lambda,
  num_threads,
  model = as.integer(c(0, 0, 0)),
  link = 0L,
  rho = 1,
  parent_tip_start = as.numeric(c()),
  seed = 0L,
  parent_id = as.integer(c())
)

Arguments

brts

Numeric vector of branching times (crown age first, sorted decreasing).

pars

Numeric vector of 8 model parameters c(beta_0, beta_N, beta_M, beta_D, gamma_0, gamma_N, gamma_M, gamma_D).

sample_size

Number of valid augmented trees to collect.

maxN

Maximum total augmentation attempts (including failures).

max_missing

Maximum extinct lineages per augmented tree.

max_lambda

Upper bound on the speciation rate (thinning bound).

num_threads

Threads for parallel augmentation.

model

Integer vector c(use_N, use_P, use_E).

link

Link function: 0 = linear, 1 = exponential.

rho

Sampling fraction in (0, 1].

parent_tip_start

Tip start of the lineage that splits at each observed branching event, in the same forward-time order as brts (that is, rev of the decreasing brts); one entry per event, or one per node with the last ignored. Empty (the default) means the topology is not available: every observed lineage is then recorded as dating from the crown and every observed event gets D = 0, which is what a bare branching-time vector has always produced.

seed

Positive integer seeding the C++ sampler. 0 (the default) draws one from R's generator, so set.seed() reaches the sampler whether or not a seed is passed. Two calls carrying the same seed, the same arguments and num_threads = 1 draw the same trees.

Value

A named list:

trees

List of augmented-tree data frames.

logf

Per-tree log p(obs, z | theta).

logg

Per-tree log q(z | obs, theta).

rejected

Unhandled rejections.

rejected_overruns

Rejected: too many extinct lineages.

rejected_lambda

Rejected: lambda bound exceeded.

rejected_zero_weights

Rejected: zero IS weight (log weight -Inf).

rejected_nonfinite

Rejected: log weight +Inf or NaN.

num_trees

Number of trees returned (length(trees)).

envelope_violations

Thinning candidates drawn during this call whose acceptance probability exceeded 1, i.e. the envelope did not dominate the rate. Non-zero means the draws are not from the density logg charges them; a warning is issued.

time

Elapsed time (ms).

.attach_w Attachment weights: each lineage's own rate, around the aggregate

Description

The proposal's aggregate rate already carries the clade mean of the covariate, so a lineage's own rate is that aggregate plus its own departure from the mean, lam + beta * (c_s - mean(c)) – not lam + beta * c_s, which shifts every lineage instead of spreading them and leaves the mean in twice.

Usage

.attach_w(cov, lam, beta)

Arguments

cov

Per-lineage covariate values (here the pendant age).

lam

The aggregate rate the mean field supplies.

beta

The covariate's coefficient.

Details

Weights are floored away from zero rather than at it. A lineage the model gives a positive rate must stay reachable, or the proposal no longer covers the target and the estimate is biased rather than merely noisy. The floor is a thousandth of the aggregate, which costs a bounded amount of weight variance and keeps the support.

Value

A positive weight per lineage, averaging lam.

.augment_tree_bdi BDI augmentation: draw sample_size augmented trees.

Description

Returns the same structure as the augment_trees() C++ function, list(trees, logf, logg, weights, fhat), plus the draw counts.

Usage

.augment_tree_bdi(
  tree,
  pars,
  model_bin = c(0L, 0L, 0L),
  sample_size = 1L,
  max_missing = 10000,
  link = 0L,
  rho = 1,
  use_gaussian_closure = TRUE,
  topology = NULL,
  mesh = NULL,
  attach = c("rate", "uniform"),
  damping = "auto",
  deadline = Inf
)

Details

Draws are attempted until sample_size have completed or the budget of 5 * sample_size attempts is spent. A draw is rejected when its missing-lineage count exceeds max_missing (counted in n_rejected_max_missing) or, under DD, when a missing lineage is still alive at the present (counted in n_rejected; such a tree has f = 0 at rho = 1). Completed draws with a non-finite log-weight are dropped from the returned tree set (the M-step set) but stay in the fhat denominator with weight zero.

fhat = log(sum_w / n_valid) + max_lw + log(acc), where n_valid counts every completed draw and acc = n_valid / (n_valid + n_rejected) is the acceptance rate of the survivor channel. max_missing overflows are excluded from the denominator, as in the thinning E-step (S_completed in src/E_step.cpp).

The log(acc) correction is derived at rho = 1, where a surviving missing lineage has f = 0 and the rejection is a property of the target. At rho < 1 an unsampled survivor is a legitimate configuration and is emitted rather than rejected, so n_rejected is 0, acc is 1 and the factor is inert. acc is the stop-at-n_valid plug-in estimate of the acceptance rate, which is biased at small sample_size: at the function's own default sample_size = 1L a draw with three survivor rejections moves fhat by log(1/4) = -1.39. At the MCEM default of a few hundred draws the bias is of order 1e-3.

Every pre-first-split lineage is attached to observed node 0

.bdi_to_tree_df records, as the parent of an augmented lineage born at t, the last observed branching at or before t (max(which(bt_sorted <= birth) - 1L)). For a lineage born before the first observed branching there is none, and the expression falls back to observed node 0 – a lineage that is not yet born at t. The sampler has no notion of the two crown lineages, which is where such a birth really attaches, and it never records the reserved crown ids the thinning sampler uses.

The consequence used to be silent and wrong: tas carried an edge of negative length wherever it happened. On seven trees measured here, the parent build turned all 200 draws into a phylo and 5 to 200 of them had a negative edge, the shortest -7.15.

.aug_to_Ltable now refuses an attachment older than its recorded parent and returns NULL, so those draws produce no tree at all. The failure is loud where it used to be silent, and the throughput is the price: on the same seven trees, simulate_tree(method = "bdi") built 0, 106, 124, 129, 153, 156 and 168 trees out of 200, with the refused draws in every case exactly those carrying a lineage born before the first observed branching. A tree whose first observed branching is late loses most of its draws; one whose crown splits early loses few.

The parent assignment itself is unchanged and still wrong. Fixing it means giving the BDI sampler the two crown lineages and the same uniform draw over labelled attachments the thinning sampler makes (audit finding H45).

Value

List: trees, logf, logg, weights (finite-weight draws only), fhat, n_valid (completed draws), n_nonfinite (completed draws dropped for a non-finite log-weight), n_attempts, n_rejected (survivors at tp), n_rejected_max_missing, acc, the mean-field diagnostics mf_converged, mf_delta (final residual), mf_sweeps, mf_omega (relaxation of the last sweep), mf_budget and mf_damping (all NA under cr), and timed_out (the deadline, an absolute proc.time() elapsed value, passed before sample_size draws completed: the draws made are returned and no shortfall warning is issued). trees is empty when no draw completed or every completed draw has a non-finite log-weight; a warning is issued when fewer than sample_size draws completed.

.bdi_augment_one Draw one augmented tree via the BDI process.

Description

Under CR: uses exact time-change method with analytical cumulative hazard. The event times are drawn from the correct inhomogeneous BDI process, giving zero-variance IS weights (logf - logg = constant). Under DD: uses approximate Gillespie with piecewise-constant rates. IS weights have nonzero variance (importance sampling, not exact).

Usage

.bdi_augment_one(
  bt,
  pars8,
  model_bin,
  link,
  tp,
  p_fun = NULL,
  Nhat_fun = NULL,
  Phat_fun = NULL,
  Ehat_fun = NULL,
  max_missing = 10000L,
  rho = 1,
  track_parents = FALSE,
  first_aug_id = 0L,
  step_max = Inf,
  obs_pid = NULL,
  beta_ed = 0,
  attach_fn = NULL,
  deadline = Inf
)

Arguments

bt

Branching times, measured forward from the crown.

pars8

The rate vector in the packed slot layout.

model_bin

Which covariate slots the model uses.

link

Link index: 0 linear, 1 exponential.

tp

The present, as a time from the crown.

p_fun, Nhat_fun, Phat_fun, Ehat_fun

The converged mean field, as functions of time; NULL under CR, where the rates are exact.

max_missing

Draw budget for hidden lineages before the draw is abandoned.

rho

Sampling fraction.

track_parents

Whether to record, for every inserted lineage, which lineage it was attached to. The parent is decoded from the same uniform that chose the event, so recording it costs no extra draw and leaves the stream bit-identical to a run without it.

first_aug_id

The id the inserted lineages are numbered from, so that ids stay distinct from the observed ones.

step_max

Cap on the Gillespie step, which is how often the frozen rates are re-evaluated. Inf leaves the step uncapped.

obs_pid

Parent ids of the observed nodes, when the topology is known.

beta_ed

The ED coefficient, which tilts the attachment choice toward the lineages the model gives a higher rate. Zero attaches uniformly.

attach_fn

Exact attachment weights for a D model, or NULL.

deadline

The fit's clock as an absolute proc.time() elapsed value; the draw is abandoned with reason "deadline" once it has passed. Inf (the default) is no deadline.

Details

At rho < 1 a missing lineage still alive at tp is not a rejection but an unsampled extant lineage: the conditioning event is "leaves no sampled descendant", and at the present that is satisfied by being alive and unsampled, an event of probability 1 - rho. Those lineages are returned in $unsampled (birth times) and written with the 5e10 sentinel by .bdi_to_tree_df, so N(t) counts them.

Value

List with $reason: "accepted" (then also $species, $unsampled, $n_alive_at_tp, $logg, and the parent records when track_parents), "max_missing" (more than max_missing missing lineages drawn), "deadline" (the clock ran out during the draw) or "survivor" (a missing lineage still alive at tp; only reachable at rho = 1, where such a tree has f = 0).

.bdi_find_event_time_cr Find next event time using exact time-change for CR.

Description

Given cumulative hazard H(s) = .bdi_integral_cr(t_cur, s, ...), find t* such that H(t*) = U where U ~ Exp(1).

Usage

.bdi_find_event_time_cr(t_cur, U, n, k, lam0, mu0, tp, t_max, rho = 1)

Value

Event time, or value > t_max if no event before boundary.

.bdi_integral_cr Exact integral of total BDI rate over [t1, t2] for CR with constant n, k.

Description

total_rate(t) = (n+2k)*lam0*(1-p(t)) + n*mu0/(1-p(t))

Usage

.bdi_integral_cr(t1, t2, n, k, lam0, mu0, tp, rho = 1)

Details

Uses the identities: int lam0*(1-p) dt = mu0*(t2-t1) + ln[p(t1)/p(t2)] int mu0/(1-p) dt = lam0*(t2-t1) + ln[(1-p(t1))/(1-p(t2))]

Valid on both sides of lam0 = mu0. Both log-ratios are evaluated with s = |lam0 - mu0| in the exponent, so the argument of exp is never positive and the mu0 > lam0 side does not overflow; writing the factors as s - r*expm1(-s*tau) also removes the cancellation of lam0 - mu0*E near lam0 = mu0.

Under incomplete sampling the same two identities hold with p replaced by its rho-sampled counterpart – they follow from dp/dt = lam*p^2 - (lam-mu)*p alone, which carries no terminal condition – so only the closed forms of the two log-ratios change. With

A(t) = d*E + rho*lam0*(1 - E) (so p = rho*d/A) B(t) = A(t) - rho*d (so 1 - p = B/A)

and E = exp(-d*(tp - t)), the integrals are

int lam0*(1-p) dt = min(lam0,mu0)*(t2-t1) + ln[A(t2)/A(t1)] int mu0/(1-p) dt = max(lam0,mu0)*(t2-t1) + ln[B(t1)/B(t2)]

At rho = 1, B(tp) = 0 and the second integral diverges as t2 -> tp: a doomed lineage must die before the present. At rho < 1, B(tp) = d*(1-rho) is non-zero and the integral is finite – a doomed lineage may survive to the present as an unsampled extant tip, which is the configuration .bdi_augment_one then has to emit.

Value

The integral value (Inf when rho = 1, n > 0 and t2 == tp).

.bdi_integral_cr_rho The rho < 1 branch of .bdi_integral_cr.

Description

Kept in its own function so that the complete-sampling path is the code the validation study measured, byte for byte.

Usage

.bdi_integral_cr_rho(t1, t2, n, k, lam0, mu0, tp, rho)

Details

A and B are written so that every exponential argument is non-positive and every term of every sum is non-negative, on both sides of lam0 = mu0:

d > 0: A = d*(1 + em) - rho*lam0*em, em = expm1(-d*(tp - t)) B = d*(1 - rho) + em*(d - rho*lam0) d < 0: both multiplied by exp(-s*(tp - t)), s = |d|, and negated A = s - rho*lam0*em, em = expm1(-s*(tp - t)) B = s*(1 - rho)*exp(-s*(tp-t)) + em*(s + rho*lam0) the scale factor exp(-s*(tp - t)) contributes -s*(t2 - t1) to ln[A2/A1] and +s*(t2 - t1) to ln[B1/B2], which is exactly what turns the min/max prefactors around.

B is 1 - p times A and p is in (0, 1], so B > 0 throughout for either sign of d - rho*lam0.

.bdi_iterate Backward-forward iteration for self-consistent BDI rates.

Description

Mean-field state: Nhat(t) = k(t) + mhat(t), Phat(t) = P_obs(t) + Phat_miss(t), Ehat(t) = Phat(t) / Nhat(t). All three enter the rate function; inactive covariates (pars8 slopes pinned to 0) make the extension transparent for CR/DD and activate PD/EP/mixed models. The sweep budget is 60 because the iteration slows as rho falls: the unsampled extant lineages raise Nhat, which feeds back into the rates that produced them. On the 20-tip DD tree at K = 20 the fixed point is reached in 6 sweeps at rho = 1, 7 at 0.9, 14 at 0.5 and 27 at 0.2. The old budget of 20 therefore returned a non-converged mean field at small rho – delta = 2.9e-3 against tol = 1e-4 in that last case – and the loop still breaks on tol, so raising the cap cannot change a run that already converged.

Usage

.bdi_iterate(
  pars8,
  model_bin,
  link,
  bt,
  tp,
  max_iter = NULL,
  tol = 1e-04,
  n_grid = 500,
  use_gaussian_closure = TRUE,
  rho = 1,
  pd_mode = c("pendant", "faith"),
  damping = "auto"
)

Value

List with p_fun, Nhat_fun, Phat_fun, Ehat_fun, and the fixed-point diagnostics converged / delta / iterations, with omega (the relaxation of the last sweep), damping (the setting) and max_iter (the budget).

.bdi_solve_mean_forward Solve the forward mean ODE jointly for (m, P_miss).

Description

State: s = (m, P_miss) with dm/dt = (lam_bdi - mu_bdi) * m + nu (self-consistent BDI mean) dP_miss/dt = m (mean-field: each alive missing lineage extends its pendant edge at rate 1)

Usage

.bdi_solve_mean_forward(
  pars8,
  model_bin,
  link,
  bt,
  tp,
  p_fun,
  Nhat_fun,
  Phat_fun,
  Ehat_fun,
  t_grid
)

Details

Rates lam, mu are evaluated at (Nhat, Phat, Ehat) to support DD/PD/EP + mixed.

Value

list(m_vals, pm_vals) on t_grid.

.bdi_solve_meanvar_forward Solve the forward mean + second-moment ODE (Gaussian closure).

Description

Augments '.bdi_solve_mean_forward' with variance state needed to cancel the process-level Jensen residual E[P/N] - Phat/Nhat ~ -Cov(P,N)/Nhat^2 + Phat Var(N)/Nhat^3. State: s = (m, P_miss, Var(N), Cov(N,P)) with dm/dt = (lam_bdi - mu_bdi) m + nu dP_miss/dt = m dVar(N)/dt = 2(lam_bdi - mu_bdi) Var(N) + (lam_bdi + mu_bdi) m + nu dCov(N,P)/dt = (lam_bdi - mu_bdi) Cov(N,P) + Var(N)

Usage

.bdi_solve_meanvar_forward(
  pars8,
  model_bin,
  link,
  bt,
  tp,
  p_fun,
  Nhat_fun,
  Phat_fun,
  Ehat_fun,
  t_grid
)

Details

The Var(N) ODE is the standard jump-diffusion variance infusion for a BDI process; the Cov(N,P) ODE comes from the deterministic coupling dP = N dt between events. Cross-jumps at deaths (DeltaN = -1, DeltaP = -age) are dropped at this order – they are second-order corrections to cNP and would require tracking the age distribution.

Value

list(m_vals, pm_vals, vN_vals, cNP_vals) on t_grid.

.bdi_solve_p_backward Solve survival probability p(t) backward from tp to 0. ODE: dp/dt = lam(t)*p^2 - (lam(t)-mu(t))*p, p(tp) = rho. Rates use the mean-field covariates (Nhat, Phat, Ehat) -- supports DD/PD/EP and mixed models. For CR (all model_bin = 0) the covariate arguments are ignored inside .bdi_lam/.bdi_mu.

Description

The ODE itself carries no sampling fraction; rho enters only as the terminal condition p(tp) = rho, which is the probability that a lineage alive at the present leaves a sampled descendant (itself). That is the whole of the incomplete-sampling change on the backward side, and it reproduces the closed form of .bdi_p_cr under constant rates.

Usage

.bdi_solve_p_backward(
  pars8,
  model_bin,
  link,
  bt,
  tp,
  Nhat_fun,
  Phat_fun,
  Ehat_fun,
  t_grid,
  rho = 1
)

.bdi_supported Can the BDI sampler be used for this model/link/rho?

Description

The sampler conditions on a rate that is a function of the lineage count alone: .bdi_p_cr and .bdi_solve_p_backward solve one survival probability p(t) shared by every lineage alive at t, and the aggregate Gillespie of .bdi_augment_one moves n_alive lineages with one pair of rates. That is what an N-only model gives it.

Usage

.bdi_supported(model_bin, link, rho = 1)

Arguments

model_bin

Length-3 binary model vector c(use_N, use_M, use_D).

link

Integer link code: 0 linear, 1 exponential.

rho

Sampling fraction in (0, 1]. Default 1.

Details

A lineage-level model (D, model_bin[3] != 0, or ED, model_bin[4] != 0) gives each lineage its own rate, so no single p(t) is the model's. The sampler then conditions on the rate of the clade's average lineage instead: an ED of P-hat/N-hat, which the mean-field iteration solves for, and a D of zero, since D is centred. The proposal is exact for that surrogate model and the importance weights carry each lineage's departure from the average; the attachment of a birth is tilted by the parent's pendant age, where D and ED read the tree. This is available on the linear and exponential links. The M covariate (model_bin[2]) is not in scope: the sampler's own mean-field state would have to be closed on P as well as N, which is written but not gated.

The linear and exponential dd rates are monotone in N. On a 20-tip tree over a 4 x 8 x 4 grid in (beta_0, beta_N, rho) with a 200-sweep budget, the Picard iteration of .bdi_iterate reached the fixed point for both in every cell, in at most 27 iterations at rho = 1. It is slower below it: an exponential dd cell needs 63 sweeps at rho = 0.2, and converging linear dd cells have needed 126 and 134, which is what sets the rho < 1 budget.

Incomplete sampling is in scope for every model and link the gate otherwise accepts. p(t) becomes the probability of leaving a sampled descendant (.bdi_p_cr, .bdi_solve_p_backward, both of which take rho), and .bdi_to_tree_df emits the 5e10 sentinel for a missing lineage still alive at the present, so the draws carry unsampled extant lineages and N(t) counts them.

.bdi_to_tree_df Convert BDI species list to the tree data frame format used by emphasis. Columns: brts, n, t_ext, pd, tip_start, id, parent_id

Description

The parent recorded for a lineage born at t is the last observed branching at or before t. A lineage born before the first observed branching has none, and max(0L, ...) makes it observed node 0, which is not yet born then; .aug_to_Ltable refuses that attachment rather than building a tas with a negative edge. See the .augment_tree_bdi documentation for what that costs and what fixing it would take.

Usage

.bdi_to_tree_df(
  species,
  bt,
  tp,
  unsampled = numeric(0),
  obs_parent_id = NULL,
  obs_focal_ts = NULL,
  aug_id = NULL,
  aug_par = NULL,
  uns_id = NULL,
  uns_par = NULL
)

.bdi_unsupported_reason Why .bdi_supported refused, as a phrase for a message.

Description

Returns NULL when the sampler is available. The reasons are ordered so that the first thing wrong with the call is the one reported.

Usage

.bdi_unsupported_reason(model_bin, link, rho = 1)

Arguments

model_bin

Length-3 binary model vector c(use_N, use_M, use_D).

link

Integer link code: 0 linear, 1 exponential.

rho

Sampling fraction in (0, 1]. Default 1.

.bootstrap_fhat_var Bootstrap variance of the bias-corrected IS log-likelihood estimate

Description

Draws B bootstrap replicates (sampling with replacement from the IS log-weights lw = logf - log_q) and computes \(\hat\ell_{\rm BC,K}\) for each replicate. Returns the sample variance of the B estimates, which approximates \(\mathrm{Var}(\hat\ell_{\rm BC,K}(L))\) due to Monte Carlo noise.

Usage

.bootstrap_fhat_var(logf, log_q, K = 2L, B = 200L)

Arguments

logf

Numeric vector of log p(obs, z | theta) for valid trees.

log_q

Numeric vector of log q(z | obs, theta) for valid trees.

K

Moment truncation order (default 2).

B

Number of bootstrap replicates (default 200).

Details

This variance is used downstream for AIC uncertainty quantification: \(\mathrm{Var}(\Delta\mathrm{AIC}) = 4[\mathrm{Var}(\hat\ell_1) + \mathrm{Var}(\hat\ell_2)]\), enabling a Gaussian test of equal AIC.

Value

Numeric scalar: estimated variance of the log-likelihood estimator. NA if fewer than 2 valid log-weights are available.

.build_cond_fun Build a conditioning function from a survival GAM

Description

Returns a function that takes an 8-element parameter vector (C++ layout) and returns log(P_tree(pars)), the log-probability that both basal lineages survive to the present.

Usage

.build_cond_fun(gam_fit, model_bin)

.check_rho Validate a sampling fraction

Description

rho is the fraction of extant species present in the tree, so it lives in (0, 1]. The C++ layer (model.hpp) replaces any value outside that range by 1 and reports nothing, so a call made with rho = 80 or rho = -1 returns a complete-sampling fit under the label the user asked for (audit finding H79). Every R entry point that accepts rho checks it here first.

Usage

.check_rho(rho, arg = "control$rho")

Arguments

rho

The value supplied by the caller.

arg

Name to quote in the error message.

Value

rho as a length-1 double.

.ess_from_lw Effective Sample Size from IS log-weights

Description

\(\text{ESS} = (\sum w_i)^2 / \sum w_i^2\), computed in log-space for numerical stability. Measures IS efficiency: ESS close to \(n\) means all trees contribute roughly equally; ESS close to 1 means one tree dominates.

Usage

.ess_from_lw(lw)

.eval_independent Evaluate particles independently (Mode 1)

Description

Only evaluates particles with NA fhat. For each such particle \(\theta_i\), simulates sample_size trees from \(q(z \mid \text{obs}, \theta_i)\) and computes:

\[\hat{f}(\theta_i) = \log\text{mean}\exp(\text{logf}(z, \theta_i) - \log q(z, \theta_i))\]

Elite particles with cached fhat are skipped entirely. Their cached trees and log_q values are also stored for potential use by the shared evaluator if the mode is later switched.

Usage

.eval_independent(pop, input, num_threads)

Value

Named list: updated pop, rej_lambda and rej_overruns vectors (length = number of particles evaluated).

.eval_particles Dispatch particle evaluation to the appropriate mode

Description

Dispatch particle evaluation to the appropriate mode

Usage

.eval_particles(pop, input, num_threads)

.eval_shared Evaluate particles using the shared (pooled) tree estimator (Mode 2)

Description

Simulates fresh trees for every particle, pools them into a single set, and cross-evaluates: for each particle \(\theta_j\), computes

\[\hat{f}(\theta_j) = \log\text{mean}\exp\bigl( \text{logf}(z_i, \theta_j) - \log q(z_i, \theta_j)\bigr)\]

where both numerator and denominator are evaluated at the target particle \(\theta_j\). The trees \(z_i\) may have been simulated at any particle; what matters for IS correctness is that the proposal density \(q\) is evaluated at the same \(\theta_j\) used for \(p\).

Usage

.eval_shared(pop, input, num_threads)

Details

Tree caches are cleared between iterations (see .resample_particles with clear_cache = TRUE), so the pool contains only the current iteration's trees.

Value

Named list: updated pop, rej_lambda, rej_overruns, n_simulated.

.extract_brts Branching times, and the topology the covariates need

Description

Returns the branching times in the crown-first (decreasing) order the C++ layer expects, carrying the per-node "parent_tip_start" attribute when the input was a tree rather than a bare numeric vector. Read it with .pts() and hand it to augment_trees() / em_cpp().

Usage

.extract_brts(tree)

Details

The attribute has one entry per node: the tip start of the lineage that splits at that observed branching event, then -1 for the terminal marker at the present, which is not an event. It is numeric(0) for a numeric input, where no topology exists.

.fit_ess The effective sample size behind a fit's log-likelihood

Description

Wherever the fit recorded it: the MCEM drivers leave it in details$final_IS, the CEM driver in details$best_IS.

Usage

.fit_ess(f)

.fit_gap The amount by which a fit's log-likelihood sits below the truth

Description

Recomputed from the recorded effective sample size when the fit predates the field, so that an old fit still compares safely.

Usage

.fit_gap(f)

.fit_gap_heavy Whether a fit's draw is heavy enough that its gap is unquantified

Description

Whether a fit's draw is heavy enough that its gap is unquantified

Usage

.fit_gap_heavy(f)

.init_population Initialise the particle population

Description

Returns a list with four parallel components of length num_points:

pars

Data frame of parameter vectors.

fhat

Numeric; NA = needs evaluation.

log_q

List; per-particle vector of log q(z | obs, theta) for cached trees. NULL = no trees cached.

trees

List; per-particle list of raw augmented-tree data frames (C++ format for loglikelihood()). NULL = not cached.

Usage

.init_population(num_points, lower_bound, upper_bound)

.is_fhat IS log-likelihood estimate (with optional bias correction)

Description

Two estimators are available:

Usage

.is_fhat(logf, log_q, bias_correct = FALSE, K = 2L, n_zero_weight = 0L)

Arguments

logf

Numeric vector of log p(obs, z | theta) for valid trees.

log_q

Numeric vector of log q(z | obs, theta) for valid trees.

bias_correct

Logical; use moment-based bias correction (default FALSE).

K

Integer truncation order for the moment series (default 2).

n_zero_weight

Integer; number of completed augmentations that had zero IS weight. These contribute w = 0 to the IS sum and inflate the denominator accordingly (default 0).

Details

Default (bias_correct = FALSE): the standard log-sum-exp estimator \(\log\bar{L} = \log\!\left(\frac{1}{n}\sum_i L_i\right)\), computed in numerically stable form. \(\bar{L}\) is an unbiased estimator of \(p(\mathrm{obs}\mid\theta)\), but \(\log\bar{L}\) slightly underestimates \(\log p\) (Jensen's inequality).

Bias-corrected (bias_correct = TRUE): uses the exact identity

\[\log\bar{L} = \hat\ell + \log\!\left(1 + \sum_{k=2}^{\infty} \frac{m_k}{k!}\right), \qquad m_k = \frac{1}{n}\sum_i(\log L_i - \hat\ell)^k\]

truncated at order K (default 2). The K=2 approximation is \(\hat\ell + \log(1 + m_2/2)\), equivalent to the log-normal correction \(\mu + \sigma^2/2\). With the full series (\(K\to\infty\)) the two estimators are identical; the truncation is the approximation.

Trees that hit computational limits (max_missing, max_lambda) are discarded entirely – they are simulator failures, not IS samples. Trees that completed augmentation but have zero IS weight (w = 0 because lambda = 0 under the current parameters) are legitimate IS samples and are included in the denominator via n_zero_weight.

.is_summary The importance-sampling summary of one draw: the estimate and the ESS

Description

One denominator for both proposals. Three kinds of draw carry weight zero and all three belong in it: a draw whose log-weight is -Inf because the model gives its tree probability zero; a draw the proposal rejected for a reason that depends on the parameters – a missing lineage still alive at the present under complete sampling, whose density under \(f\) is zero; and nothing else. A draw rejected for a reason that does not depend on the parameters, an overrun of the missing-lineage budget, is not a sample and stays out.

Usage

.is_summary(lw, n_zero_weight = 0L, n_rejected = 0L)

Arguments

lw

Log-weights of the completed draws.

n_zero_weight

Completed draws whose weight is zero.

n_rejected

Draws rejected for a parameter-dependent reason.

Details

The two proposals reached this denominator by different routes – the thinning path through n_zero_weight, the conditional path through a separate acceptance factor – and the difference between the two routes is the size of the effect they are used to measure.

Value

A list: fhat, ess, n_finite, n_den.

.jensen_gap The downward bias of log(mean(w)), from the effective sample size

Description

\(\widehat\ell = \log(\frac1N \sum_i w_i)\) estimates \(\log E[w]\), and a log of a mean is not the mean of a log: by Jensen's inequality \(E[\widehat\ell] \le \log E[w]\), always downward, never up. The EM objective is unaffected – it is an expectation of a log, and the M-step averages \(\log f\) directly (src/M_step.cpp) – but the log-likelihood a fit reports is exactly this estimator, and so is the AIC built from it.

Usage

.jensen_gap(ess, n)

Arguments

ess

Effective sample size of the draw.

n

Number of draws the estimator averaged over.

Details

The delta method gives the bias as \(-\mathrm{Var}(w)/(2N E[w]^2)\), which in terms of the effective sample size \(n_{\mathrm{eff}} = N/(1+\mathrm{CV}^2)\) is \(-\frac12(1/n_{\mathrm{eff}} - 1/N)\): a quantity already on record for every fit.

It is an indicator of size, not a bound in either direction, and its error runs the wrong way where the quantity matters. Measured on log-normal weights with \(N = 200\), as the ratio of what this returns to the true bias: 4.9 at \(n_{\mathrm{eff}} = 188\), 2.0 at 157, 0.90 at 82, 0.75 at 36, 0.46 at 17, 0.27 at 9, 0.16 at 6. It overstates only where the bias is negligible anyway (below \(2\times10^{-4}\)), and understates once the weights are heavy – by a factor of six at \(n_{\mathrm{eff}} = 6\), where the true bias is half a nat. heavy marks that regime, \(n_{\mathrm{eff}} < N/10\), which is where the ratio first falls below one half. A caller deciding anything on the size of the gap must treat a heavy draw as unquantified rather than as the number returned.

Value

A list: gap (a non-negative magnitude, the amount by which the reported log-likelihood sits below the truth) and heavy (TRUE when gap understates it).

.mcem_bdi MCEM with BDI augmentation.

Description

Drop-in replacement for .mcem_dynamic_fresh(). Uses BDI exact sampling for the E-step and m_cpp() for the M-step.

Usage

.mcem_bdi(
  brts,
  pars,
  sample_size,
  max_missing,
  lower_bound,
  upper_bound,
  max_iter,
  xtol,
  tol = 0.01,
  patience,
  num_threads,
  verbose = FALSE,
  conditional = NULL,
  model = c(0L, 0L, 0L),
  link = 0L,
  max_time = NULL,
  rho = 1,
  stop_rule = c("rel_change", "mc_error"),
  mc_batches = 5L,
  mc_z = 1,
  mc_grow = 1.5,
  max_draws = NULL,
  mesh = NULL,
  damping = "auto"
)

Details

Under CR: zero-variance IS, so every E-step gives the same fhat regardless of the augmented trees. Convergence depends only on the M-step.

Stopping rule. With theta_k the M-step output of iteration k,

\[\delta_k = \max_j |\theta_k[j] - \theta_{k-1}[j]| / \max(|\theta_{k-1}[j]|, \epsilon)\]

(.rel_change) with \(\epsilon\) from .rel_floor, and the run stops as "converged" once \(\delta_k < tol\) for patience consecutive iterations. The scale is the parameter itself, not the search box, so the same fit stops at the same point whatever bounds the user passes; the floor carries the units of the parameters, so it also stops at the same point when the tree is measured in another unit of time. An iteration whose M-step returned its starting point unchanged carries no information about stability and resets the streak. The trace also records the absolute step abs_step and drift, the same relative displacement measured over the last patience iterations: under pure Monte Carlo noise drift is of the order of sqrt(patience) steps, under a deterministic EM drift it is patience steps.

The reported likelihood is a separate E-step at the final iterate, so fhat, ESS, final_IS and loglik_var all describe pars; that E-step is the last row of mcem (m_step = FALSE, no step columns) and is not counted in iterations. When that E-step fails and the final iterate is not the last one whose E-step succeeded, pars falls back to that iterate and the E-step is run there: a theta the sampler cannot evaluate is not returned as an estimate.

Draw counts come from the E-step, which drops non-finite draws before returning: n_valid is the number of completed draws and the fhat denominator, num_trees the finite subset handed to the M-step, and n_nonfinite the difference. rejected and rejected_max_missing are the two rejection channels.

Eight E-step failures end the run with stop_reason = "e_step_failure" whether or not they are consecutive, so an M-step that keeps proposing an iterate the sampler cannot evaluate does not alternate to max_iter.

Value

A list: mcem (trace, one row per iteration plus the final E-step), pars, iterations (number of completed E+M iterations), stop_reason, loglik (fhat at pars, NA when the final E-step failed), loglik_var, final_IS, n_failed (E- or M-step failures), final_truncated, time_note and damping (the setting the mean field ran under). The trace carries the mean field of every E-step: mf_converged, mf_delta (final residual), mf_sweeps, mf_omega (the relaxation of the last sweep) and mf_budget (the sweep budget).

Clock. max_time binds inside the E-step and the M-step: the sampler is given the deadline and stops drawing when it passes, between draws and inside one, and m_cpp is given the seconds left and reads them before every evaluation of its objective. An iteration whose E-step ran out is discarded and the run stops with "time_budget" and a time_note. An iteration whose M-step ran out keeps the best point the M-step had evaluated when that point improves on the current iterate, and is discarded otherwise; either way the run stops with "time_budget" and a time_note naming the M-step. The final E-step is given a clock of its own, max_time again, and when it runs out the log-likelihood rests on the draws it made (final_truncated), so a fit returns within about twice max_time.

.mcem_dynamic_fresh MCEM with thinning augmentation.

Description

E-step and M-step of one iteration are a single em_cpp() call: the augmented trees are drawn by the thinning sampler, up to maxN attempts per E-step.

Usage

.mcem_dynamic_fresh(
  brts,
  pars,
  sample_size,
  maxN,
  max_missing,
  lower_bound,
  upper_bound,
  max_iter,
  xtol,
  tol = 0.01,
  patience = 3L,
  num_threads,
  verbose = FALSE,
  conditional = NULL,
  model = c(0L, 0L, 0L),
  link = 0L,
  max_time = NULL,
  rho = 1,
  rel_floor = 0.01,
  stop_rule = c("rel_change", "mc_error"),
  mc_batches = 5L,
  mc_z = 1,
  mc_grow = 1.5,
  max_draws = NULL
)

Details

Stopping rule, the same statistic .mcem_bdi uses. With theta_k the M-step output of iteration k,

\[\delta_k = \max_j |\theta_k[j] - \theta_{k-1}[j]| / \max(|\theta_{k-1}[j]|, \epsilon)\]

and the run stops as "converged" once \(\delta_k < tol\) for patience consecutive iterations. The search box does not enter, so the same fit stops at the same point whatever bounds the user passes. The floor \(\epsilon\) comes from .rel_floor(brts, link, rel_floor) and carries the units of the parameters (rel_floor / crown_age for the rate-valued links, rel_floor for the log-rates of the exponential link), so the statistic is unchanged when the tree and the rates are expressed in another unit of time. An iteration whose M-step returned its starting point unchanged carries no information about stability and resets the streak (trace column m_moved). The trace also records the absolute step abs_step and drift, the same relative displacement measured over the last patience iterations.

Trace. Row k of mcem pairs theta_k with the E-step value fhat(theta_{k-1}) it was computed from, and with the rejection counters of that E-step. The likelihood of the returned point is a separate E-step at the final iterate, appended as the last row (m_step = FALSE, step columns NA); only that row is evaluated at pars, and it is not counted in iterations. It is attempted whenever at least one iteration succeeded, including after a "e_step_failure" or "time_budget" stop, and may itself fail, in which case there is no such row and loglik, loglik_var and final_IS report nothing rather than the previous iterate's values.

Rejection columns. rejected_errors (unhandled exception), rejected_overruns (max_missing exceeded), rejected_lambda (max_lambda exceeded) and rejected_nonfinite (log weight +Inf or NaN) are the four disjoint ways a draw is discarded; n_rejected is their sum, the quantity final_IS$n_rejected reports, and rejected is that same total under the name the BDI trace uses. rejected_zero_weights is counted apart: those draws completed, have weight zero and stay in the fhat denominator.

Value

A list: mcem (trace, one row per completed iteration plus the final E-step), pars, iterations (number of completed E+M iterations, not loop passes and not nrow(mcem)), n_failed (E-step failures over the whole run), stop_reason, final_estep (did the final E-step at pars succeed), maxN (the ratcheted attempt cap: doubled on every E-step failure up to 50000, never reset after a success), loglik (fhat at pars, NA when the final E-step failed), loglik_var, final_IS, final_truncated (the final E-step ran out of its clock and loglik rests on the draws it made) and time_note (what the clock cut short, or NULL).

Clock. max_time binds inside the E-step and the M-step: em_cpp is given the seconds left, the E-step stops drawing when they run out, between draws and inside one, and the M-step reads the same deadline before every evaluation of its objective. An iteration whose E-step ran out is discarded and the run stops with "time_budget". An iteration whose M-step ran out keeps the best point the M-step had evaluated when that point improves on the current iterate, and is discarded otherwise; either way the run stops with "time_budget" and a time_note naming the M-step. The clock is also read between iterations. The final E-step is given a clock of its own, max_time again, so a fit returns within about twice max_time.

.mcem_iter_rows Trace rows that are MCEM iterations

Description

Both drivers append the final E-step at the parameters they return as a trace row. That row is not an iteration: it carries no M-step and no step metrics. .mcem_bdi flags it m_step = FALSE and .mcem_dynamic_fresh flags it final_estep = TRUE; both flags are read here, so that anything counting or plotting the trace reports K rows for a K-iteration fit.

Usage

.mcem_iter_rows(mcem_df)

Arguments

mcem_df

An MCEM trace data frame, or NULL.

Value

A logical vector over the rows of mcem_df.

.observed_parent_id The lineage that splits at each observed branching event

Description

The ED covariate needs to know which lineage is which across events, which the tip start of .observed_parent_tip_start does not say. Lineages are named as the C++ layer names them: the two crown lineages are -2 (the one that splits first) and -3; the daughter born at the k-th observed branching event (forward-time order, 0-based) is lineage k. Which child of a node continues the parent's lineage is fixed by a convention that any consistent rule would do for: the child whose subtree holds the lowest tip label continues, the other is the daughter.

Usage

.observed_parent_id(phy)

Arguments

phy

A phylo object.

Value

Integer vector of length Nnode(phy) - 1, in forward-time order: the id of the lineage splitting at each event.

.observed_parent_tip_start Tip start of the lineage that splits at each observed branching event

Description

The covariates \(M\) and \(D\) are functions of the pendant ages of the alive lineages, and a lineage's pendant age is measured from the last time it became a tip – its tip start, which is reset every time it speciates. For an observed branching event that is the forward time of the parent of the splitting node, and 0 for the two children of the crown.

Usage

.observed_parent_tip_start(phy)

Arguments

phy

A phylo object.

Details

Events are returned in forward-time order, the order the C++ layer builds the observed tree in, so the k-th entry belongs to the k-th observed node. A 2-tip tree has no non-crown internal node and returns numeric(0).

Value

Numeric vector of length Nnode(phy) - 1.

.pid The lineage ids carried by a branching-time vector, or integer(0).

Description

The lineage ids carried by a branching-time vector, or integer(0).

Usage

.pid(brts)

.pts The parent tip starts carried by a branching-time vector

Description

numeric(0) when there are none, which is how the C++ layer is told that no topology is available.

Usage

.pts(brts)

.rel_change Relative parameter change, the statistic both MCEM stopping rules use

Description

The step in each coordinate is measured against that coordinate's own magnitude, with a floor so that a coordinate at or near zero is not measured against itself. The floor carries the units of the parameter (see .rel_floor), so the statistic is unchanged when the tree and the rates are expressed in another unit of time.

Usage

.rel_change(new, old, floor_val)

Arguments

new, old

Parameter vectors of equal length.

floor_val

Floor of the denominator, from .rel_floor.

Value

The largest relative coordinate change.

.rel_floor Floor of the relative-change denominator

Description

Under the linear link the parameters are rates, of dimension 1/time, and the only rate scale in the problem is the inverse crown age: the floor is rel_floor / crown_age, which rescales with the rates themselves. Under the exponential link the parameters are log-rates, where an absolute step already is a relative change of the rate, so the floor is rel_floor.

Usage

.rel_floor(brts, link = 0L, rel_floor = 0.01)

Arguments

brts

Branching times (crown age first, or any order).

link

Integer link code: 0 linear, 1 exponential.

rel_floor

Dimensionless floor. Default 1e-2.

Value

A positive scalar.

.resample_particles Resample particle population

Description

Selects the top disc_prop fraction of valid particles (elites) and fills the remaining slots with perturbed copies of elites. All particles (including elites) have their fhat reset to NA so they are re-evaluated at the next iteration, preventing "lucky particle" lock-in.

Usage

.resample_particles(
  pop,
  num_points,
  disc_prop,
  lower_bound,
  upper_bound,
  sd_vec,
  clear_cache = FALSE
)

Arguments

clear_cache

Logical; if TRUE, discard all tree/log_q caches.

Details

When clear_cache = FALSE (Mode 1), elite tree/log_q caches are carried forward. When clear_cache = TRUE (Mode 2), all caches are discarded so every particle simulates fresh trees at the next iteration. This prevents stale trees from unconverged early iterations from polluting the shared pool.

Value

Updated population list.

.resolve_control_aliases Resolve control-parameter aliases

Description

Supports both old names (num_points, sample_size) and new names (num_particles, num_trees). User-supplied values take precedence; if both old and new are present, the new name wins. Internally the code always reads the OLD names for backwards compatibility with emphasis_cem and the C++ layer.

Usage

.resolve_control_aliases(ctrl, method, user_ctrl = list())

.simulate_particle Draw augmented trees at a single particle (internal inference use only)

Description

Calls augment_trees to sample sample_size augmented trees from the proposal \(q(z \mid \text{obs}, \theta)\).

Usage

.simulate_particle(
  brts,
  pars8,
  model_bin,
  link,
  sample_size,
  maxN,
  max_missing,
  max_lambda,
  num_threads,
  rho = 1
)

Details

Returns the raw augment_trees output: logf, logg, trees, and rejection counts for adaptive limit escalation.

.validate_linear_init_pars Validate init_pars for linear link models

Description

For linear link, lambda = max(0, beta_0 + beta_X * X). If the slope is too negative, lambda becomes zero at observed covariate values, causing IS collapse. This clamps slopes so lambda stays positive at the maximum covariate value seen in the tree.

Usage

.validate_linear_init_pars(
  init_pars,
  model_bin,
  brts,
  lower_bound,
  upper_bound,
  verbose = FALSE
)

ed_fair_proportion Evolutionary distinctiveness of a lineage forest at a time

Description

The fair-proportion routine the likelihood and the simulator share (inst/include/ed_covariate.hpp), exposed so that it can be checked against an independent computation on an ape tree.

Usage

ed_fair_proportion(parent, birth, alive, t)

Arguments

parent

Integer vector: for each lineage, the (1-based) index of the lineage it split from, or 0 for a crown lineage.

birth

Numeric vector: each lineage's birth time (forward).

alive

Logical vector: whether each lineage is alive on the segment being evaluated.

t

The evaluation time.

Value

A data frame with one row per lineage: c and ts such that ED(u) = c + (u - ts) for u in the segment (NA for a lineage not alive), ed the value at t, and n_desc the alive lineages in its subtree.

em_cpp function to perform one step of the E-M algorithm

Description

function to perform one step of the E-M algorithm

Usage

em_cpp(
  brts,
  init_pars,
  sample_size,
  maxN,
  max_missing,
  max_lambda,
  lower_bound,
  upper_bound,
  xtol_rel,
  num_threads,
  copy_trees,
  model = as.integer(c(0, 0, 0)),
  link = 0L,
  rho = 1,
  rconditional = NULL,
  parent_tip_start = as.numeric(c()),
  seed = 0L,
  parent_id = as.integer(c()),
  time_left = -1
)

Arguments

brts

vector of branching times

init_pars

vector of initial parameter files

sample_size

number of samples

maxN

maximum number of failed trees

max_missing

maximum number of species missing

max_lambda

maximum speciation rate

lower_bound

vector of lower bound values for optimization, should be equal in length to the vector of init_pars

upper_bound

vector of upper bound values for optimization, should be equal in length to the vector of init_pars

xtol_rel

relative tolerance for optimization

num_threads

number of threads used.

copy_trees

if set to true, the trees generated are returned as well

model

integer vector of length 3: c(use_N, use_P, use_E)

link

link function: 0 = linear (max(0,...)), 1 = exponential

rho

sampling fraction in (0, 1]

rconditional

R function that evaluates the GAM function.

parent_tip_start

tip start of the lineage that splits at each observed branching event, in the same forward-time order as brts; empty (the default) when only branching times are available, in which case every observed lineage is recorded as dating from the crown and D = 0 at every observed event.

seed

positive integer seeding the C++ sampler; 0 (the default) draws one from R's generator, so set.seed() reaches the sampler whether or not a seed is passed.

time_left

seconds the fit has left on its clock; the E-step stops drawing when they run out, inside a draw as well as between draws, and returns the trees it has with timed_out = TRUE and no M-step. The same deadline binds the M-step, which reads it before every evaluation of its objective and stops once it has passed (m_timed_out). Negative (the default) means no deadline.

Value

a list with the following components:

trees

list of trees

rejected

number of rejected trees

rejected_overruns

number of trees rejected due to too large size

rejected_lambda

number of trees rejected due to lambda errors

rejected_zero_weights

number of trees rejected due to zero weight (log weight -Inf)

rejected_nonfinite

number of trees rejected due to a +Inf or NaN log weight

num_trees

number of trees the E-step returned

estimates

vector of estimates

nlopt

nlopt status

fhat

vector of fhat values

time

time elapsed

weights

vector of weights

logf

vector of log p(obs, z_i | theta) for each valid tree

logg

vector of log q(z_i | obs, theta) for each valid tree

timed_out

whether the deadline passed before sample_size trees were drawn; when TRUE no M-step was run and estimates is init_pars

m_timed_out

whether the deadline passed during the M-step

m_improved

when m_timed_out: whether the best point the M-step had evaluated improves on init_pars; estimates is that point when TRUE and init_pars when FALSE

emphasis_cem Monte Carlo Cross-Entropy Method for diversification models

Description

Implements the Cross-Entropy Method (CEM) with an IS log-likelihood objective. A population of parameter particles is evaluated via IS (fhat), the elite fraction is retained, and the remainder is refilled by perturbing elite copies.

Usage

emphasis_cem(
  brts,
  max_iter,
  num_points,
  max_missing,
  sd_vec,
  lower_bound,
  upper_bound,
  maxN = 10L,
  sample_size = 1L,
  shared_trees = FALSE,
  bias_correct = FALSE,
  disc_prop = 0.5,
  tol = 1e-04,
  patience = 5L,
  sd_decay = 0.85,
  sd_min_frac = 0.01,
  verbose = FALSE,
  num_threads = 1L,
  model = c(0L, 0L, 0L),
  link = 0L,
  cond_fun = NULL,
  max_time = NULL,
  rho = 1
)

Arguments

brts

Numeric vector of branching times.

max_iter

Maximum number of iterations (hard cap).

num_points

Population size.

max_missing

Max extinct lineages per augmentation.

sd_vec

Initial perturbation SD per parameter (8-element).

lower_bound, upper_bound

Search bounds (8-element).

maxN

Max augmentation attempts per particle. Default 10.

sample_size

Trees simulated per particle per evaluation. Default 1.

shared_trees

If TRUE, use pooled-tree estimator (Mode 2). Default FALSE (independent, Mode 1).

bias_correct

If TRUE, apply moment-based bias correction to the IS log-likelihood. Default FALSE.

disc_prop

Elite fraction retained per iteration. Default 0.5.

tol

Minimum improvement in best fhat required to avoid incrementing the plateau counter. Default 1e-4.

patience

Consecutive plateau iterations before early stopping. Default 5.

sd_decay

Multiplicative factor applied to sd_vec when the best log-likelihood has not improved. Default 0.85. Set to NULL to use the legacy linear annealing schedule.

sd_min_frac

Minimum SD as a fraction of the initial sd_vec. Default 0.01 (1%). SD will not decay below this floor.

verbose

Print iteration summaries. Default FALSE.

num_threads

Parallel threads. Default 1.

model

Length-3 binary integer vector c(use_N, use_M, use_D).

link

Integer link code: 0 = linear, 1 = exponential.

rho

Sampling fraction (0, 1]. Default 1 (complete sampling).

Details

Adaptive SD annealing: The perturbation SD is decayed multiplicatively (factor sd_decay, default 0.85) only when the best log-likelihood has not improved by more than tol in the current iteration. When the likelihood is still improving, the SD is held constant to maintain exploration. A minimum SD floor (sd_min_frac * initial_sd, default 1% of initial) prevents complete collapse. This replaces the former linear annealing schedule which coupled the SD decay to max_iter and could cause premature collapse on easy problems or insufficient exploration on hard ones.

Stopping rules (first triggered wins):

  1. Plateau: best fhat has not improved by more than tol for patience consecutive iterations and the SD has reached its floor – the population has both converged in likelihood and exhausted its perturbation budget.

  2. Hard cap: max_iter iterations reached (safety limit).

Value

A list:

best_loglik

fhat of best particle per completed iteration (length = actual iterations run, \(\leq\) max_iter).

best_pars

Parameter matrix, one row per iteration.

obtained_estim

Globally best parameter vector.

loglik_var

Bootstrap variance of fhat at best particle (NA if sample_size = 1).

converged

Character string naming the stopping rule that fired: "annealing", "plateau", or "max_iter".

time

Elapsed proc.time().

estimate_likelihood_surface Estimate log-likelihood at a grid of parameter values via importance sampling

Description

For each row of pars_mat, augments sample_size trees via IS and computes the IS log-likelihood estimate (fhat). Returns a data frame ready for train_likelihood_GAM.

Usage

estimate_likelihood_surface(
  tree,
  pars_mat,
  model = "cr",
  sample_size = 200L,
  link = "exponential",
  max_missing = 10000,
  max_lambda = 500,
  maxN = NULL,
  num_threads = 1L,
  bias_correct = FALSE,
  verbose = TRUE,
  max_time = NULL,
  rho = 1
)

Arguments

tree

A phylo object, branching-time vector, or anything accepted by estimate_rates.

pars_mat

Numeric matrix whose rows are compact parameter vectors (same layout as pars in estimate_rates).

model

Model specification: "cr", "dd", "d", "nd", "ed", "ned", "k", "nk", or a binary integer vector (length 3 to 5).

sample_size

Number of augmented trees per grid point (default 200).

link

"exponential" (default) or "linear"; see estimate_rates.

max_missing

Maximum missing lineages per augmentation (default 1e4).

max_lambda

Maximum speciation rate (default 500).

maxN

Total augmentation attempts per grid point. If NULL, set automatically.

num_threads

Number of threads for the C++ E-step.

bias_correct

Logical; if TRUE, use the moment-based bias-corrected IS estimator (truncated at order K). Default FALSE uses the standard log-mean-exp estimator.

verbose

Print progress (default TRUE).

rho

Sampling fraction (0, 1]. Default 1 (complete sampling).

Value

A data frame with columns for each parameter plus fhat (IS log-likelihood) and n_trees (number of valid trees obtained). Rows where IS failed have fhat = NA.

estimate_rates_control Default control parameters for estimate_rates

Description

Returns a list of default tuning parameters for estimate_rates. Override individual elements and pass the modified list via the control argument.

Usage

estimate_rates_control(method = c("mcem", "cem", "gam"), n_pars = 4)

Arguments

method

Which method's defaults to return: "mcem", "cem", or "gam".

n_pars

Number of parameters (used to set the default sd_vec for CEM when NULL).

Value

A named list of control parameters.

Shared parameters (both methods):

lower_bound

Numeric vector of lower parameter bounds. Length must be 2 + 2*sum(model). Required (no default).

upper_bound

Numeric vector of upper parameter bounds (same order). Required (no default).

verbose

Print progress messages. Default FALSE.

max_missing

Max extinct lineages per augmented tree. Trees that require more than this many extinct lineages are discarded. Default 1e4.

num_threads

Threads for parallel computation. Default 1.

rho

Sampling fraction: the proportion of extant species present in the tree, in (0, 1]. Default 1 (complete sampling). Values outside (0, 1] are an error. Under rho < 1 the augmentation inserts unsampled extant lineages as well as extinct ones; both samplers do so, and the choice of sampler does not depend on rho.

max_time

Wall-clock budget of the fit in seconds; NULL disables it. Default 3600. Under "mcem" the clock binds inside the E-step and the M-step, and the fit ends with stop_reason = "time_budget" and a clock note naming the step it cut. Both samplers stop drawing when it runs out, inside a draw as well as between draws, and the iteration in progress is discarded. The M-step's optimiser reads it before every evaluation of its objective; cut short, the iteration keeps the best point the optimiser had evaluated when that point improves on the current iterate, and is discarded otherwise. The E-step that evaluates the returned parameters then gets max_time again, and when it runs out its log-likelihood rests on the draws it made, so an MCEM fit returns within about twice max_time. Under "cem" and "gam" the clock is read between evaluations.

MCEM-specific parameters:

sampling

Sampling scheme: "bdi" (default), the birth-death-with-immigration sampler, which covers the cr and dd models on the linear and exponential links, at any rho in (0, 1]; or "dynamic_fresh", the thinning sampler, which covers every model, link and rho. Under constant rates the BDI proposal is the exact conditional distribution, so its weights are constant and its effective sample size is the number of draws; under diversity dependence and under ED it is a mean-field approximation and the weights vary. "bdi" falls back to "dynamic_fresh", with a message naming the reason, for a D- or M-dependent model.

damping

BDI only: under-relaxation of the mean-field sweep, which is a Picard iteration \(x \leftarrow F(x)\). Where its map expands the iteration diverges and a larger budget makes it worse: on a 416-tip tree at rates whose equilibrium is critical the residual is 305 after 20 sweeps and 329 after 200. A partial step, \(x \leftarrow (1-\omega)x + \omega F(x)\), contracts it. "auto" starts at \(\omega = 1\) and halves it whenever the residual grows; on that tree it converges in 40 sweeps, and where the map already contracts it is the plain iteration to the bit. A number in \((0, 1]\) fixes \(\omega\) instead; 1 is the plain iteration. "default" (the default) is "auto" at rho = 1 and \(\omega = 1/8\) at rho < 1: in experiment E10c a fixed 1/8 converged 0 of 192 units within the 20 sweeps at rho = 1 against 131 for "auto", at 1.4–3.3 times the E-step time and no gain in effective sample, and below rho = 1 it converged more (142 against 131 at rho = 0.2); an explicit "auto" or number is used as given. Relaxation below about \(1/20\) converges too slowly to be useful. It fixes the iteration and nothing else: at 416 tips the effective sample is 2.1 of 97 even with a converged mean field, because the proposal must invent some 600 missing lineages there (audit H108). Every E-step's mean field is in the trace (mf_converged, mf_delta, the final residual, mf_sweeps, mf_omega, the relaxation of the last sweep, and mf_budget); when some E-step relaxed the iteration below the full step or did not converge within its sweep budget, or the setting is not "auto", the fit's notes carry a mean_field entry with the resolved setting (and whether it is the default for the fit's rho), the sweeps, relaxation and residual of the last E-step and whether it converged, and print() shows it.

mesh

BDI only, and only where its Gillespie branch runs (any model but "cr"): the number of steps per crown age the frozen rates are re-evaluated on. NULL (default) leaves the step uncapped, which is a rate held over a whole waiting time. A mesh tightens the weights — on a 60-tip dd tree the standard deviation of the log-weight falls from 0.465 to 0.061 at 2000 steps per crown age, and the effective sample from 600 draws rises from 527 to 598 — and does not move the estimate (dev/mfc_mesh.R).

proposal

The thinning sampler's proposal for an ED model ("ed", "ned", ~ ED): "ed" (default) draws the missing births at the model's total speciation rate over the alive lineages, each at its own ED, and attaches a birth to a lineage in proportion to that lineage's rate; "meanfield" draws them at N * lambda(N, M) with the ED term at zero and attaches uniformly, as for every other model. The two are the same proposal at beta_ED = gamma_ED = 0. Ignored for a model without ED.

num_trees

Augmented trees per EM iteration. Default 200. Alias: sample_size.

xtol

Relative tolerance for the M-step optimiser. Default 1e-3.

max_iter

Maximum EM iterations. Default 200.

maxN

Total augmentation attempts per E-step (accepted plus rejected). Read by the thinning sampler only; the BDI sampler has no such cap and ignores it. Under "dynamic_fresh" it must be at least num_trees, and estimate_rates stops with an error otherwise. Increase for models with high rejection rates (e.g. D/DD with strongly negative covariate slope). Default NULL, and a non-finite value is treated as NULL, which sets max(2000, 10 * num_trees).

tol

Parameter-stability convergence threshold. MCEM is considered converged when the largest relative parameter change, \(\max_j |\theta_k[j] - \theta_{k-1}[j]| / \max(|\theta_{k-1}[j]|, c)\), is below tol for patience consecutive iterations. The bound box does not enter. The floor \(c\) carries the units of the parameters (see .rel_floor): tol / crown_age where they are rates (linear link) and tol where they are log-rates (exponential link), so the statistic does not change when the tree is expressed in another unit of time. Default 1e-2: every coordinate must move by less than 1% of its own magnitude, patience times in a row. The step the sampler itself produces sets whether that is reachable – on a 12-tip CR tree at num_trees = 200 the statistic has median 0.02 and falls below 1e-2 in about a fifth of iterations – so runs at these defaults commonly end at max_iter. Raising num_trees shrinks the Monte Carlo part of the step.

patience

Number of consecutive iterations with parameter change below tol required to declare convergence. Default 3.

CEM-specific parameters:

max_iter

Maximum CEM iterations (hard cap). Default 20.

num_particles

Particle population size (parameter vectors per iteration). Default 50. Alias: num_points.

sd_vec

Initial perturbation SDs. NULL (default) sets each to (upper - lower) / 4 at call time.

maxN

Max augmentation attempts per particle evaluation. Default 10.

num_trees

Augmented trees simulated per particle per iteration. Default 1. Alias: sample_size.

shared_trees

If FALSE (default), each particle is evaluated on its own simulated trees. If TRUE, trees are pooled across all particles and every particle's fhat is re-evaluated over the full pool (lower variance; bias correction not yet implemented).

disc_prop

Elite fraction retained per iteration (Cross-Entropy Method). Default 0.5.

bias_correct

Apply moment-based bias correction to the IS log-likelihood. Default FALSE.

num_trees

When > 1, bootstrap variance of the log-likelihood is computed automatically (B = 200 replicates) from the IS weights already drawn. Minimum recommendation: 5. loglik_var in the fit is NA when num_trees = 1.

eval_attachments What the proposal could attach an augmented lineage to

Description

Replays an augmented tree through the forward sweep the thinning sampler carries, and reports at every augmented birth the lineages the proposal had to choose from, the labelled attachments they carry, and the parent that was recorded. candidates must be the alive count n and attachments the 2 * tips + Ne that Model::sampling_prob charges -log of; the two crown lineages carry no node, so they appear only here and in their reserved ids.

Usage

eval_attachments(tree)

Arguments

tree

One augmented-tree data frame from augment_trees.

Details

The convention the replay runs under is the clade column, which augment_trees carries out with the tree; a data frame built without it is read under the legacy convention, as a bare branching-time tree has always been. Nothing reported here separates the two: the alive set, the attachments and whether the recorded parent is among them are carried by the lineage ids and the event order, which both conventions share.

Value

A data frame with one row per augmented birth: brts, n, candidates, attachments, parent_id and parent_alive.

eval_logf Evaluate log p(obs, z | theta) and log q(z | obs, theta) for augmented trees

Description

Given a list of augmented trees (produced by augment_trees) and a parameter vector, computes for each tree:

  • logf[i] = log p(obs, z_i | theta) (model log-likelihood)

  • logg[i] = log q(z_i | obs, theta) (proposal log-probability)

Both are evaluated at the supplied pars, regardless of which parameters were used to simulate the trees. This is essential for shared-tree (Mode 2) evaluation where trees simulated at one particle are scored at another.

Usage

eval_logf(pars, trees, model = as.integer(c(0, 0, 0)), link = 0L, rho = 1)

Arguments

pars

Numeric vector of 8 model parameters.

trees

List of augmented-tree data frames (output of augment_trees).

model

Integer vector c(use_N, use_P, use_E).

link

Link function: 0 = linear, 1 = exponential.

Value

A named list:

logf

Numeric vector of log p(obs, z_i | theta).

logg

Numeric vector of log q(z_i | obs, theta).

eval_nh_rate The pendant PD and thinning rate the sampler sees at arbitrary times

Description

Model::pendant_pd_at reads P off the node that governs the segment a candidate time falls in; Model::nh_rate is the intensity of the sampler's birth process there. This exposes both, so a test can hold P against an independent recomputation from the augmented tree and against pd + n * (t - brts) of the governing node, and can scan the intensity over a segment to check that its value at the segment start dominates it.

Usage

eval_nh_rate(
  pars,
  tree,
  times,
  model = as.integer(c(0, 0, 0)),
  link = 0L,
  rho = 1
)

Arguments

pars

Numeric vector of 8 model parameters.

tree

One augmented-tree data frame.

times

Numeric vector of times at which to evaluate.

model

Integer vector c(use_N, use_M, use_D).

link

Link function: 0 = linear, 1 = exponential.

rho

Sampling fraction.

Value

A named list with pd (the pendant PD used) and nh (the non-homogeneous thinning rate) at each time.

eval_pendant_sweep Replay the forward pendant sweep over an augmented tree

Description

One application of the sweep the thinning sampler carries forward: it rewrites tip_start, focal_tip_start and pd on every node from the event list alone. With parent_tip_start supplied the sweep is driven by the observed topology, exactly as the closing pass of augment_trees is: the k-th observed node is given the k-th entry as the tip start of the lineage that splits there, and the tree is read under the topology convention. With it empty the tree is replayed as it stands, under the convention its clade column records and with the focal_tip_start it already carries as the match key.

Usage

eval_pendant_sweep(
  tree,
  parent_tip_start = as.numeric(c()),
  parent_id = as.integer(c())
)

Arguments

tree

One augmented-tree data frame from augment_trees.

parent_tip_start

The tip starts the augmentation was driven by, or numeric(0) to replay the tree as it stands.

Details

An augmented tree returned by augment_trees is a fixed point of the first form: the state the sampler carried is the tree that comes back.

Value

The same data frame with tip_start, focal_tip_start and pd rewritten by the sweep.

find_MLE Find MLE by optimizing a fitted likelihood GAM

Description

Uses optim (L-BFGS-B) to maximize the GAM-predicted log-likelihood surface.

Usage

find_MLE(gam_fit, lower_bound, upper_bound, start = NULL, par_names = NULL)

Arguments

gam_fit

A gam object returned by train_likelihood_GAM.

lower_bound

Numeric vector of lower bounds for each parameter.

upper_bound

Numeric vector of upper bounds for each parameter.

start

Optional numeric vector of starting values. If NULL, the grid point with the highest predicted value is used.

par_names

Character vector of parameter names matching the GAM predictors. If NULL, inferred from the GAM formula.

Value

A list with components:

pars

Named numeric vector of MLE estimates.

loglik

Predicted log-likelihood at the MLE.

convergence

Convergence code from optim (0 = success).

optim

Full optim output.

lineage_table_cpp The per-lineage design an augmented tree's likelihood is built on

Description

One row per lineage alive on each segment between consecutive events: the segment, the lineage (index and id), whether that lineage's event ends the segment (0 none, 1 speciation, 2 extinction), the segment's bounds, N, M, the lineage's pendant start (so D = (t - ts) - M) and its ED at the segment start (ED grows at slope one within the segment).

Usage

lineage_table_cpp(tree)

Arguments

tree

One augmented-tree data frame with the topology (ids).

m_cpp function to perform one step of the E-M algorithm

Description

function to perform one step of the E-M algorithm

Usage

m_cpp(
  e_step,
  init_pars,
  plugin,
  lower_bound,
  upper_bound,
  xtol_rel,
  num_threads,
  model = as.integer(c(0, 0, 0)),
  link = 0L,
  rho = 1,
  rconditional = NULL,
  time_left = -1
)

Arguments

e_step

result of e_step function, as a list

init_pars

vector of initial parameter values

plugin

string indicating plugin used, currently available: 'rpd1' and 'rpd5c'

lower_bound

vector of lower bound values for optimization, should be equal in length to the vector of init_pars

upper_bound

vector of upper bound values for optimization, should be equal in length to the vector of init_pars

xtol_rel

relative tolerance for optimization

num_threads

number of threads used.

model

integer vector of length 3: c(use_N, use_P, use_E)

link

link function: 0 = linear (max(0,...)), 1 = exponential

rconditional

R function that evaluates the GAM function.

time_left

seconds the fit has left on its clock; the optimiser reads it before every evaluation of the objective and stops once it has run out. Negative (the default) means no deadline.

Value

list with the following entries:

estimates

vector of estimates

nlopt

nlopt status

time

used computation time

timed_out

whether the deadline stopped the optimiser

improved

when timed_out: whether the best point evaluated improves on init_pars; estimates is that point when TRUE and init_pars when FALSE

prune_to_extant Prune extinct tips from a phylogenetic tree

Description

Drops tips that did not survive to the present (i.e. tips whose root-to-tip distance is shorter than the maximum, within a small numerical tolerance).

Usage

prune_to_extant(phy, tol = 1e-08)

Arguments

phy

A phylo object, possibly containing extinct lineages.

tol

Tolerance for deciding whether a tip is at the present. Default 1e-8.

Value

A phylo object with only extant tips retained.

simulate_div_tree_cpp Simulate a single phylogenetic tree under the general diversification model

Description

Simulate a single phylogenetic tree under the general diversification model

Usage

simulate_div_tree_cpp(
  pars,
  model,
  max_t,
  max_N,
  max_tries,
  link = 0L,
  seed = 0L
)

Arguments

pars

Numeric vector of length 8: c(beta_0, beta_N, beta_M, beta_D, gamma_0, gamma_N, gamma_M, gamma_D). Inactive parameters (those not selected by model) should be set to 0.

model

Integer vector of length 3: c(use_N, use_P, use_E) (each 0 or 1).

max_t

Crown age (forward simulation end time).

max_N

Maximum number of lineages before simulation is declared too large.

max_tries

Maximum retries after extinction; an overflow is not retried.

link

Link function: 0 = linear (max(0,...)), 1 = exponential.

seed

Positive integer seeding the simulator. 0 (the default) draws one from R's generator, so set.seed() reaches the simulator whether or not a seed is passed. Note that max_tries retries continue the same stream: a retry is a fresh tree, not the same one again.

Value

A named list with Ltable (4-column numeric matrix in DDD format) and status (one of "done", "extinct", "too_large").

thinning_envelope_violations Count thinning candidates with acceptance probability above 1

Description

The thinning sampler accepts a candidate speciation time with probability nh(t) / lambda_max. A value above 1 means the envelope lambda_max did not dominate the rate on that segment. The counter accumulates over every augmentation call in the session.

Usage

thinning_envelope_violations(reset = FALSE)

Arguments

reset

Logical; zero the counter after reading it.

Value

Number of candidates with acceptance probability above 1 since the last reset.

train_likelihood_GAM Fit a GAM to the IS log-likelihood surface

Description

Takes the output of estimate_likelihood_surface and fits a smooth GAM approximation to the log-likelihood as a function of the model parameters.

Usage

train_likelihood_GAM(
  surface,
  par_names = NULL,
  spline_type = c("univariate", "bivariate"),
  verbose = TRUE
)

Arguments

surface

Data frame returned by estimate_likelihood_surface. Rows with fhat = NA are dropped automatically.

par_names

Character vector of column names in surface to use as predictors. If NULL (default), all columns except fhat and n_trees are used.

spline_type

"univariate" (default) or "bivariate". See train_GAM for details.

verbose

Report the number of grid points (default TRUE).

Value

A gam object (Gaussian family) representing \(\hat\ell(\theta) \approx \log L(\theta \mid \text{data})\).