nested: choose dynesty's sampling mode by dimension, and let callers sweep it - #253
Merged
Conversation
…sweep it `sample="rwalk"` was hardcoded. dynesty recommends rwalk only for 10 <= ndim <= 20 and slice sampling above that, so DC2018-128's d=27 observable arm was being sampled outside the mode's documented regime. It showed. That run TERMINATED in 1.6 h -- against ultranest's 5.4 days without finishing on the same config -- and landed up to 40 SIGMA from truth on the light-curve observables with absurdly small errors: parameter truth median sd pull source.u_0 0.141832 0.138032 0.00024 -15.55 planet.log_q -2.91657 -3.01526 0.00246 -40.15 Large offsets with tiny errors is a collapsed live-point set, not efficient exploration. nlive=500 is also thin at d=27 (guidance ~50*ndim = 1350). `sample=None` now means "choose by dimension" instead of "rwalk regardless", `bound` is exposed alongside it, and both are overridable so a caller can sweep them -- job array 15430323 is doing exactly that over 4 modes x 3 nlive, scored against TRUTH rather than wall clock. The chosen mode is logged so a run's own log says which it used. NOTE the dimension-aware default is a hypothesis until that grid reports. It follows dynesty's own guidance, and it cannot be worse than hardcoding a mode outside its regime, but "rslice fixes it" is not yet measured. Also adds the two truth-check harnesses, which exist because the previous round read speed as success: they score the LIGHT-CURVE observables only, and deliberately exclude theta_E/pi_rel, which need an SED and inherit 8.6.7's mu_rel error whatever the sampler does.
The multimodal results.csv header is # parname, mode, weight, weight_err, value, up_err, low_err seven fields, and the reader hardcoded six -- omitting `weight_err`. Every column after `weight` therefore shifted by one: `value` picked up the usually-EMPTY weight_err cell and parsed to None, `up_err` picked up the value. So the per-event comparison table reported an EMPTY exozippy column while run_event.py exited non-zero with no explanation. MULTIMODAL IS THE NORM for microlensing -- the +/-u_0 degeneracy is always there -- so this silently voided the comparison for most events. Measured on DC2018 events 152, 194 and 223: all three fitted, all three wrote results.csv, all three compared to nothing. Event 128 escaped it only by being parsed from a unimodal file, which is also why the RESULTS_CSV_MAP rewrite in #246 was validated without catching this. The field list is now read FROM THE HEADER rather than hardcoded, with the old lists kept as the fallback, so it survives the writer gaining another column. With the fix the same three events produce real pulls -- e.g. 194: t_0 +2.84, u_0 +0.30, t_E -0.21, s -0.36, q -0.08.
Measured on DC2018-128's d=27 observable arm, max|pull| is MONOTONIC in n_eff: dynesty rwalk 500 4,803 eff 25.5 dynesty rwalk 1350 33,752 eff 27.6 dynesty rwalk 2700 80,138 eff 43.2 dynesty rslice 500 16,770 eff 51.1 PTDE (control) converged 39.4 The best-sampled run scores WORST and converges toward PTDE's, because resolving a posterior better makes it narrower and narrow-in-the-wrong-mode is what a pull punishes. PTDE's log_s is -0.01051 +/- 0.00019, cleanly inside one mode with truth outside it (39 sigma); dynesty rwalk/500's is -0.03230 +/- 0.03261, 170x broader, straddling both modes so truth falls inside and the pull is 0.9. Neither found truth's mode; one was less willing to commit. This is 7.13.1's lesson in a new costume -- there the t_E marginal was skewed and the PIT quantile was the right statistic; here the posterior is multimodal and the pull is wrong again. The scorer asks instead: is truth's mode PRESENT (nearest joint approach), what WEIGHT does it carry, and within it is the fit unbiased. On PTDE's observable arm the answer is that truth's mode is never visited, and the blocker is log_q -- 12% off with 0.4% claimed precision (27 sigma), nearest draw 1.35x tolerance. So event 128's geometry is slightly wrong in a way no sampler choice fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sample="rwalk"was hardcoded in the dynesty branch. dynesty recommendsrwalkonly for 10 ≤ ndim ≤ 20 and slice sampling above that — so DC2018-128's d=27 observable arm was sampled outside the mode's documented regime.It showed. That run terminated in 1.6 h (against ultranest's 5.4 days without finishing on the same config) and landed up to 40 sigma from truth with absurdly small errors:
source.u_0planet.log_qLarge offsets with tiny errors is a collapsed live-point set, not efficient exploration.
nlive=500is also thin at d=27 (guidance ~50×ndim = 1350).What changes
sample=Nonenow means choose by dimension rather than rwalk regardless;boundis exposed alongside it; both are overridable so callers can sweep them. The chosen mode is logged, so a run's own log says which it used.Caveat
The dimension-aware default is a hypothesis until the grid reports. It follows dynesty's own guidance and can't be worse than hardcoding a mode outside its regime, but "rslice fixes it" is not yet measured — job array 15430323 is testing 4 modes × 3 nlive, scored against truth rather than wall clock, with
rwalkas the control andunifas a deliberate negative control.Also adds the two truth-check harnesses. They exist because the previous round read speed as success: they score the light-curve observables only and deliberately exclude
theta_E/pi_rel, which need an SED and inherit 8.6.7's mu_rel error regardless of sampler.