Skip to content

Benchmark aggregation reports skipped workloads as successful timing samples #2495

Description

@chubes4

Observed failure

The MDI decision matrix completed with PASS for both backends, including a boot-timing workload that returned skipped: missing_bench_boot_phase. Aggregation reported approximately 0.004 ms as its duration, presenting the skip path as a benchmark result rather than a skipped scenario.

Reproduction workload: Automattic/markdown-database-integration, tests/bench/boot-timing.php, matrix source 21267e18d41c88396e4883e227422158ba5c16b2. Invoke the decision benchmark profile without BENCH_BOOT_PHASE. Operator evidence: Homeboy job 7018d7ce-fdf0-442d-a95c-ad96eedac2c5, SQLite run b28d11dd-e33d-40cf-8944-683aacb5fbc3, native run 2cc3b041-9291-4e8b-823f-e05b92376607. The source and normal workload invocation are the portable reproduction; IDs require operator access.

Expected contract

The workload result boundary preserves typed skip status and reason. Skipped workloads contribute no performance sample and are explicitly represented in matrix completeness. A required skipped scenario leaves acceptance incomplete, even when other executed scenarios pass. Test aggregation with executed, skipped, and failed callables through the real dispatcher.

Inspect owning layers before implementation: if WP Codebox already preserves skip status and the loss occurs downstream, route the repair to that aggregator rather than duplicating logic.

AI Assistance

GPT-6 Astra through OpenCode inspected the benchmark output and workload source, searched existing trackers, and drafted this report under Chris Huber’s direction. Direct OpenCode repair runs use openai/gpt-5.6-terra.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions