ci: reduce test shards and redundant jobs - #750
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Validated head The first hosted Node CI run completed all 25 jobs:
Against the latest main run, the observed tradeoff was 2.7 fewer runner minutes (about 4.9%) and 21 seconds more waiting. This single comparison is not a controlled benchmark; it is consistent with modest resource savings and does not establish a latency improvement. The uploaded JUnit inventories and outcomes match the latest main run exactly: 2,137 cases on Linux, 2,137 on macOS, and 2,134 on Windows, with 111 shared test files per OS and no duplicate cases across shards. Typechecking and formatting ran successfully exactly once on Linux shard 1. The separate Windows policy check passed on shard 3, and the real Windows Node 24 package check passed. The full local Bun 1.3.14 run with seed 67890 passed: 2,095 passed, 43 skipped, 0 failed. Seed 12345 finished with 2,094 passed, 43 skipped, and one process cleanup assertion failure. The three related process tests passed in isolated reruns on both unchanged main and this head. The affected implementation and test are unchanged in this PR. This is an alternative to #745: it reuses a Linux test job for static checks, with later static-only failure feedback and fewer jobs. That PR and the branch protection rules were left unchanged. Human approval is still required; this PR has not been merged. |
1 participant
Summary
node-ciexpands to 29 jobs, with repeated setup across 13 SDK test shards. This change reduces it to 25 jobs while retaining every shared test file on Linux, macOS, and Windows, the supported Node runtime checks, and all three required status names.Changes
windows-latest / node-24result aggregator. The actual Windows Node 24 installed-package check remains, as do all three required result aggregators and the dedicated Windows machine-policy test.CI tradeoff analysis
The baseline is the mean of four successful, first-attempt main runs with the same workflow layout: 33292937659, 33292854751, 33266575958, and 33257840080. Their observed waiting times range from 5m26s to 5m57s. The intervening failed run is excluded; this is not a retry-cost or flake-rate analysis.
Runner minutes sum each job's start-to-completion time. Waiting runs from workflow creation to the final job's completion, including dependency and queue waits. These are resource minutes, not rounded billing minutes.
The following table changes only sharding. The baseline is measured; every alternative is an estimate using 52 JUnit reports and the checked-in file scheduler and duration weights.
Each removed shard saves approximately 42 seconds of repeated overhead on Linux, 49 seconds on macOS, or 63 seconds on Windows. Test work remains: fewer shards primarily trade repeated setup for a longer slowest shard. The 1/1/1 case would exceed the existing Windows test timeout and would require moving the separate policy check; it is not proposed here.
The replay assigns each run's observed per-file test durations to the scheduler's new groups, holds average overhead constant within each OS, and applies the change to that run's observed completion path. Other jobs and observed queue/gate delays remain. It assumes serial Bun execution within each shard. Grouping effects, cache variation, future source changes, and reduced queue contention are not modeled. Test case counts vary slightly across the four commits; each run uses its own durations and the same 111-file inventory.
This PR combines 2/2/6 sharding with moving static checks and removing one aggregator, yielding 25 jobs. Static checks occupied 17–25 seconds of the shared package job in the samples. The combined model estimates 52.8 runner minutes and 5m36s waiting, or roughly 5% less runner time with broadly unchanged waiting. The projected waiting-time difference is smaller than observed run-to-run variation; this is not a measured speedup or a controlled benchmark.
Duplicate amd64 container validation, standalone title validation, and any migration to one required status are outside this PR. Supported runtime coverage and branch rules are unchanged.
Testing
git diff --checkpassed.Risk and rollout
Fewer shards increase work repeated when a shard must be rerun. Static-only failures are reported later than with a separate early static-check job, although the package build still compiles production TypeScript. The first Linux shard must pass static checks for the existing required result checks to succeed.
No production code, dependency versions, test selection, required check names, branch rules, or release authorization boundaries change. Compare the first hosted run with the historical baseline before treating the estimates as observed savings. The shard counts and static-check placement can be reverted without a product migration.
Public disclosure review