Skip to content

Add local Modal leaderboard mode - #78

Merged
msaroufim merged 1 commit into
mainfrom
feat/local-modal-mode
Aug 20, 2026
Merged

Add local Modal leaderboard mode#78
msaroufim merged 1 commit into
mainfrom
feat/local-modal-mode

Conversation

@msaroufim

Copy link
Copy Markdown
Member

Summary

  • add popcorn submit --local for running public evaluations in the user Modal workspace
  • reuse KernelBot task construction and evaluation for test, benchmark, and leaderboard modes
  • format per-case results and compute the configured leaderboard score locally
  • bypass Popcorn registration and the hosted API for local runs
  • document Modal token setup, supported GPUs, and unofficial-result limitations

Main-branch freshness

  • resolve gpu-mode/reference-kernels and gpu-mode/kernelbot main to immutable commit SHAs on every invocation
  • include those SHAs in the Modal image definition so a new upstream commit invalidates the cached image
  • clone and evaluate those exact commits, then print both SHAs in the result
  • retry GitHub resolution and fail closed instead of falling back to a potentially stale symbolic main
  • allow explicit ref pinning with POPCORN_REFERENCE_KERNELS_REF and POPCORN_KERNELBOT_REF

Limitations

Local leaderboard mode reproduces the public test, benchmark, and ranked stages. It cannot reproduce server-only private seeds or create an official gpumode.com submission.

Test plan

  • cargo fmt --all -- --check
  • cargo test (53 passed)
  • cargo clippy --all-targets --all-features -- -D warnings
  • embedded Python helper syntax check
  • git diff --check

A paid Modal GPU end-to-end run was not performed because this machine does not have Modal credentials configured.

@msaroufim
msaroufim marked this pull request as ready for review August 20, 2026 17:24
@msaroufim
msaroufim merged commit 32461c1 into main Aug 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant