Cache and automatically run type checker benchmarks

[Bill Schnurr (bschnurr)]

Summary

• Automatically benchmark pull requests that change packages/pyright-internal/src/analyzer/.
• Keep /benchmark as a maintainer-only rerun trigger for other pull requests.
• Compare each PR synthetic merge commit against the exact main commit used as its first parent.
• Cache validated main benchmark results by commit and hosted measurement profile.
• Proactively populate the cache on relevant pushes to main and retain each raw result as a 90-day artifact.
• Publish comparisons through artifacts, the job summary, and one updated bot comment without modifying PR branches.

What is compared

The candidate is GitHub's synthetic merge commit, which combines the PR's current head with its current main base. The baseline is that same exact main base commit:

main commit B ----------------------------> baseline benchmark
           \
            + PR head H -> synthetic M ---> candidate benchmark

comparison: M (B + PR changes) versus B

The trusted report workflow verifies that B is the merge commit's first parent. If the PR head or base changes before reporting validates the revisions, the report stops and a new run is required.

Cache and security

• Relevant pushes to main benchmark GITHUB_SHA, save an exact-SHA/profile cache entry, and upload the result.
• PR reports restore that same cache entry or benchmark the validated base on a cache miss.
• The candidate workflow has read-only permissions and cannot access the shared cache.
• The trusted report independently verifies analyzer path changes or an authorized rerun attempt.
• No shared request label is used, so overlapping requests cannot revoke each other's authorization.
• Candidate output crosses the trust boundary only as bounded, provenance-validated JSON.
• Benchmark workflows never commit generated baseline files into contributor branches.

Benchmark history

The checked-in bootstrap results under build/benchmark/baselines/ include source commit metadata. build/benchmark/benchmark_history.ipynb can graph those results and export the static dashboard under docs/benchmark-results/. Automatic main results are retained through the Actions cache and 90-day artifacts rather than source commits.

Validation

• python -m unittest discover -s build/benchmark -p "test_*.py" (52 tests)
• Workflow YAML parsing
• Prettier checks for changed workflows and documentation

[Rich Chiodo (rchiodo)]

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

[Rich Chiodo (rchiodo)]

Result: ⚠️ partially-verified

Verification details

Verification: Isolated verification observed failures that were not classified as caused by this PR: Module-style benchmark test invocation, Benchmark unittest suite.

Summary: Five new comparator/profile tests passed. The 52-test benchmark suite ran 50 tests successfully, but two workflow tests errored because the container lacks Node.js. Verification is therefore partial; no PR-caused assertion failure was observed.

Test runs: 1 passed, 2 failed

• ❌ Failed | unrelated to this PR | Module-style benchmark test invocation | python -m unittest build.benchmark.test_compare_benchmarks
• ❌ Failed | unrelated to this PR | Benchmark unittest suite | python -m unittest discover -s build/benchmark -p "test_*.py"
• ✅ Passed | New comparator and profile tests | python -m unittest test_compare_benchmarks.CompareBenchmarksTest.test_rejects_unexpected_source_revision test_compare_benchmarks.CompareBenchmarksTest.test_report_identifies_compared_revisions test_compare_benchmarks.CompareBenchmarksTest.test_allows_successful_benchmark_profile_change test_compare_benchmarks.CompareBenchmarksTest.test_incompatible_results_still_require_successful_measurements test_compare_benchmarks.CompareBenchmarksTest.test_pr_workflow_uses_matching_base_and_candidate_profiles

❌ Module-style benchmark test invocation diagnostic output

ModuleNotFoundError: No module named 'build.benchmark'
Ran 1 test in 0.000s
FAILED (errors=1)

❌ Benchmark unittest suite diagnostic output

Two workflow tests errored in _load_yaml with FileNotFoundError: [Errno 2] No such file or directory: 'node'.
Ran 52 tests in 0.057s
FAILED (errors=2)