E42 — the exact maximum-likelihood estimate on E40's trees
| Status | complete on the 75 trees E40 had finished on 2026-10-15; amended before any deficit was read (boxed, odeint route) |
| Package build | f30a791 |
| Driver / analysis | sim/E42-exact.R (the run); sim/E42-allmodels.R (the pre-registration probe); probes not in the repository: sim/E42-reported.R, E42-offset.R, E42-convention.R, E42-crosscheck.R, E42-timing.R |
| Data | results/E42/exact.csv (150 rows: per tree and model the exact and the E40 estimate, the deficit dl, valid, on_box); outside the repository, results/E42/jobs/, results/E42-unboxed/ (the first run, not read) and the log results/E42.log |
Question
Two of E40's five models, cr and dd, have an exact likelihood, so on every E40 tree the Monte Carlo EM fit can be set against the exact maximum. This is the study's claim that MCEM reaches the exact MLE, tested on simulated trees in E1, carried to real trees, at E40's budget, with E40's link.
Design
| model | exact likelihood | maximisation |
|---|---|---|
cr |
DDD::bd_ML, constant rates, unconditioned (cond = 0, as E40's fits are), soc = 2, btorph = 1, the unsampled species as missnumspec \(= \mathrm{round}(n/\rho - n)\) |
bd_ML |
dd |
E40's exponential-link dd, \(\lambda = e^{\beta_0 + \beta_N N}\), \(\mu = e^{\gamma_0 + \gamma_N N}\), all four free. DDD has no log-linear form, so its master-equation likelihood (dd_loglik, analytical) is used with its rate function DDD:::lambdamu replaced, for a new model code 9, by these rates |
Subplex (nloptr::sbplx, xtol_rel \(10^{-6}\), 800 evaluations) from E40's dd estimate and from the exact cr estimate with zero slopes |
The hidden-lineage count is truncated at \(n + m + 60\) and the maximum checked at \(n + m + 100\). Checks recorded per tree: with both slopes at zero the replaced likelihood equals bd_loglik; the truncation check. Per tree and model: the exact estimate; the deficit \(\Delta\ell = \ell_{exact}(\hat\theta_{E40}) - \ell_{exact}(\hat\theta_{MLE}) \le 0\); each parameter's difference; and \(g\), the reported-minus-exact log-likelihood at E40's estimate. Where E41 continued a fit, its estimate is set against the same maximum.
Pre-registration check (complete). Before the design was registered, on the 31 E40 trees then finished (sim/E42-allmodels.R, on each tree exactly as the fit read it, tip edges extended to the deepest tip), the exact log-likelihood at each E40 estimate was compared with the value E40 reported:
| model | trees | exact against reported, at E40's own estimate |
|---|---|---|
cr |
complete trees (\(\rho = 1\)) | 0.000 |
dd |
complete trees (\(\rho = 1\)) | within 0.01 |
cr, dd |
\(\rho < 1\) | the reported value 0.01 to 0.17 above the exact one (reported minus exact positive) |
Where \(\rho < 1\) the two codes model sampling differently: emphasis samples each species independently with probability \(\rho\), DDD fixes the number missing, so E40's reported log-likelihood sits 0.01 to 0.17 nats above DDD's exact value on those trees. On complete trees the two sampling models coincide, and there the reported value is the exact likelihood. The probe printed its comparison to the console; no file of it is kept.
Pre-set rule
The rule of E1, per model and per set: median \(\Delta\ell \ge -0.5\) nats and 90th percentile of \(|\Delta\ell| \le 2\) nats; read separately for the complete trees (\(\rho = 1\), where the two sampling models coincide) and the rest. A pilot runs on one worker on the trees E40 has finished; the rest once E40 ends.
Results
The first pilot maximised the replaced likelihood without bounds through DDD's matrix-exponential route, which is unstable past about 110 hidden lineages: on Muscicapidae it returned a "maximum" of -22.6 (at \(\gamma_0 = -48.9\), \(\beta_N > 0\)) against -131.7 at E40's estimate. It is set aside (results/E42-unboxed/, not read), and the design was amended before any deficit was read: DDD's odeint Runge-Kutta route (equal to bd_loglik with zero slopes at truncations 110, 150 and 250), a boxed maximisation, and a validity test on every maximum (stable when the truncation grows by 40, not below the value at E40's estimate).
On the 75 trees E40 had finished (results/E42/exact.csv):
| model | trees | deficit, median | 90th percentile of \(|\Delta\ell|\) | worst | rule | |---|---|---|---|---|---| | cr, \(\rho = 1\) | 50 | -0.079 | 0.19 | -0.38 | met | | cr, \(\rho < 1\) | 25 | -0.056 | 0.17 | -0.27 | met | | dd, \(\rho = 1\) | 48 (2 invalid not read) | -0.0045 | 0.011 | -0.093 | met | | dd, \(\rho < 1\) | 21 (4 invalid not read) | -0.0069 | 0.14 | -0.99 | met |
Reported minus exact log-likelihood at E40's own estimates: a median 0.000 (cr) and 0.004 (dd) on complete trees. At the exact dd maximum the extinction slope on \(N\) sits on its \(-3/n\) cap on 61 of 69 valid trees (E40's own fits: 50), extinction at the present is a median \(2 \cdot 10^{-6}\) against speciation 0.059, and the extinction intercept a median -10.0 against E40's -7.0; the exact bd_ML likewise puts extinction at zero.
Conclusion
Met. On real family and clade trees of 50-130 tips, E40's Monte Carlo EM fits of cr and dd sit at the exact maximum: a median 0.005-0.08 nats below it, 90th percentile at most 0.19. Where E40's extinction parameters ended on a box edge, the exact maximum moves them further toward zero extinction for a median gain of 0.005 nats: the likelihood of these trees is flat toward zero extinction, and widening the box moves the number without improving the fit (see E41). Both are statements about the unconditioned likelihood E40 maximises; the conditioned comparison is Phase 1 of the estimability plan.