Adding pred_measure feature - #363
Draft
florence-bockting wants to merge 67 commits into
Draft
florence-bockting wants to merge 67 commits into
florence-bockting wants to merge 67 commits into
Conversation
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if 0e7d193 is merged into loo-v3.0.0:
|
This was referenced Jun 8, 2026
Contributor
Author
|
Linking here to Issue #179 as it seems to be about loo scores. We should see whether it is still relevant and can be implemented within loo refactoring.
|
Contributor
Author
|
There is an open issue requesting a subset() method for psis objects #343 and an (unrelated) open PR #110. Might be worth considering whether we can address this as well in this refactoring PR. If yes, we should
Moved this to PR #379 |
Closed
6 tasks
florence-bockting
changed the title
pred_measure feature
added 19 commits
July 2, 2026 09:16
The test read res_roaches$predperf_loo$log_weights. The fixture field is predperf, not predperf_loo, so the test passed log_weights = NULL and checked the unweighted path instead. No stored weight set in the fixture is normalized, so the test now normalizes res_roaches$log_weights itself. It also asserts the real invariant: measure_elpd() normalizes internally, so a normalized input gives the same result as a raw one.
test_data_generation.R wrote ylp_train to test_data_sleep_cv.Rds and mupred_loo to test_data_sleep.Rds. No test and no vignette reads either. The loo::E_loo() call that produced mupred_loo is gone with it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #281
Summary
Adds a unified predictive performance measures API on top of
loo-v3.0.0. Density scores, scoring rules, and point-prediction metrics can be computed under in-sample, PSIS-LOO, k-fold, and holdout evaluation via*_pred_measure().Design log:
developer-notes.md(edit:developer-notes.Rmd)Issues addressed
This PR consolidates work that was tracked across several related issues:
ypreddraw matrix (measure_rps(),measure_srps())loo_pred_measure()accepts a pre-computedpsis_object(or extracts one from alooresult withsave_psis = TRUE)measure_r2()as a built-in*_pred_measure()—loo_compareintegration still outstanding (see Known limitations)How to navigate this PR
This is a large PR. These documents explain design, scope, and current status:
developer-notes.md:central reference: status, open tasks, design decisions, numerical comparisons
overview-measures.Rmd: supported scores and metrics, with mathematical definitionspred-measure-workflow.Rmd: conceptual introduction and end-to-end workflow examplesmigration-guide.Rmd: mapping from deprecated functions to the new APIWhat's new
insample_pred_measure(),loo_pred_measure(),kfold_pred_measure(),test_pred_measure(),pred_measure()measure_*()+supported_measures_list()(incl.r2, classification metrics, Brier, MAE/MSE/RMSE, ELPD/IC)measure_rps()/measure_srps()— singleypredmatrix, PWM/ECDF estimator; covers continuous and ordered categorical outcomes (CRPS/RPS and scaled variants in the new API)measurepsis_objector reuse weights from a priorloo_pred_measure()result viapredperfpred_measureobjects (incl. Pareto-k-hat for LOO)migration-guidevignette + website-onlyoverview-measuresandpred-measure-workflowarticles.Rdsfixtures, pkgdown vignette-data generation stepDeprecations (implementations retained)
Existing functions are deprecated but still work with their current APIs:
elpd()measure_elpd()/*_pred_measure()crps(),scrps()(x,x2)measure_rps(),measure_srps()(ypred)loo_crps(),loo_scrps()loo_pred_measure(..., measure = "rps" | "srps")loo_predictive_metric()loo_pred_measure()In
*_pred_measure(),elpdis computed automatically whenylpis supplied; requestmeasure = "ic"separately for the information criterion.The new RPS functions use a different estimator (one draw matrix, PWM) than deprecated CRPS (two draw matrices, permutation). Results are highly correlated but not identical — see the comparison section and figures in
developer-notes.Rmd. Comparison tests are intests/testthat/test_crps.R.Dependencies
Vignettes, test-data generation, and pkgdown CI install brms from GitHub
master(paul-buerkner/brms), because k-fold support for categorical / multinomial models (brms#1890, fixing brms#1889) is not yet on CRAN. Once a CRAN release includes that fix, the GitHub pin can be dropped.Known limitations
group_idsgrouping not yet implementedloo_compareintegration still outstandingSee
developer-notes.Rmdfor open design questions andremaining tasks.
Acknowledgements
This work builds on an initial prototype of the
pred_measurefeature implemented by @VisruthSK (see VisruthSK/loo-sandbox) under guidance from @jgabry.Test plan
devtools::test()anddevtools::check()passdevtools::test(): 1 failure — unrelatedtest_psislw.Rsnapshot drift (Warning in psislw()→Warning:)R CMD check: 1 error —kfold_pred_measure()roxygen example usesmeasure = "rmse"withoutmupredtest_deprecated_measures.R)test_crps.R; not re-diffed againstloo-v3.0.0branch)test_crps.R, 32 tests)*_pred_measure()examples run (migration guide + workflow vignette; rendered locally withNOT_CRAN=true)brms(workflow + migration-guide articles render; fullpkgdown::build_site()not verified locally — existingdocs/blocks build)