wordpress.phpunit: database-type=mdi-native excludes 100% of candidates (excluded=1335, routed=0) on multisite + managed bootstrap

[chubes4]

Summary

With database-type=mdi-native, wordpress.phpunit discovers the full candidate set and then excludes 100% of it, routing zero tests. The same component on database-type=mysql runs 1,177 tests normally.

FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0

Discovery clearly works — it finds 1,335 candidates. Routing/selection is where everything is dropped.

Reproduction

• wp-codebox: d4e0f713 (current main), so this includes wordpress.phpunit: add mdi-native backend #2490 (mdi-native backend), fix: preinstall native multisite PHPUnit state #2494 (native multisite PHPUnit state), Allow explicit native MDI sources #2497 (explicit native MDI sources), and Retain complete PHPUnit failure artifacts #2498 (retain PHPUnit failure artifacts).
• Consumer: Homeboy wordpress extension, managed install, provenance-valid (homeboy extension setup wordpress run clean; version reported 0.26.8 at the managed source path).
• Component under test: a WordPress plugin with
  • wp_codebox_multisite: true
  • wp_codebox_phpunit_bootstrap_mode: "managed"
  • 101 test files / 1,335 PHPUnit candidates

Control vs variable, same component, same commit, only database_type changed:

database_type	Result
mysql (external service)	1,177 passed, 7 skipped, 1,177 total, 577s
mdi-native	0 passed, 0 failed, 0 skipped, 0 total — excluded=1335

Behaviour detail

• The run does not fail fast at the point of exclusion. After reporting excluded=1335 it continues until the harness budget is consumed. In one run it reported test suite consumed 1336s of its 1500s budget (89%) having executed nothing.
• With Retain complete PHPUnit failure artifacts #2498 present the run surfaces failed test command did not expose an observed failed test ID rather than silently producing nothing, which is an improvement, but there is still no diagnostic explaining why 1,335 candidates were excluded.

Why this matters

mdi-native is the difference between "tests need an external MySQL/MariaDB with provisioned credentials" and "tests need nothing." For consumers, that is the difference between a suite any contributor or agent can run locally and one gated behind secrets. Right now the backend validates, plans, and reports success at the config layer while running zero tests — so it cannot be trusted as a MySQL-compatible harness.

Credit where due: the consumer side handles this safely. Homeboy classifies zero executed tests as invalid_evidence → passed: false, so a release preflight on this backend is correctly blocked rather than shipping untested code. The danger is not a silent false pass in that toolchain, but the failure is silent enough that another consumer could plausibly treat "exit 0, no failures" as success.

Asks

1. Emit a reason for exclusion. excluded=1335 with no per-candidate or aggregate cause is not diagnosable. Even a single aggregate reason code would have made this self-service.
2. Fix the routing/selection path for mdi-native so candidates route the way they do under mysql. The identical candidate count (1,335) under both backends suggests discovery is shared and only selection/routing diverges.
3. Fail fast on selected=0 when candidates>0. Consuming the full time budget after excluding everything wastes ~22 minutes per run and obscures the real error.

Possibly relevant

The combination here is multisite + managed bootstrap. #2494 specifically addressed native multisite PHPUnit state, so this may be an unhandled remainder of that work rather than a general mdi-native failure — I have not tested single-site to isolate that, and would be happy to if it helps.

[chubes4]

Correction to my own report — the excluded=1335 framing is wrong, please disregard it

I ran the isolation tests I offered and they invalidate the headline diagnosis. Posting before anyone spends time on it.

excluded=1335 is a red herring

I anchored the report on:

FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0

That line is identical on a backend where tests DO run. Control run with database-type=sqlite, same component, same commit:

FULL_SUITE_STANDALONE_PHP_SUMMARY:candidates=1335 selected=0 routed=0 excluded=1335 passed=0 failed=0
...
total: 1232   passed: 1112   failed: 111

So that summary belongs to a separate standalone-PHP harness and says nothing about PHPUnit routing. My "selection/routing drops every candidate" conclusion was drawn from it and is not supported. Ask #1 in the original issue (emit a reason for exclusion) is therefore not the real problem either.

Corrected comparison

Same component, same commit, only database_type changed:

Backend	Executed	Result
mysql (external)	1,177	1,177 passed, 7 skipped
sqlite	1,232	1,112 passed, 111 failed
mdi-native	0	nothing executed, run consumes full budget

sqlite executes the suite. mdi-native executes nothing. That is the actual, narrower bug.

Multisite ruled out

I also tested wp_codebox_multisite: false with mdi-native — still zero executed. So this is not a remainder of #2494 (native multisite PHPUnit state) as I speculated. It reproduces on single-site too.

Note on the sqlite failures

The 111 sqlite failures are almost entirely mysqli_sql_exception: Access denied for user root@localhost — tests in this component that open their own mysqli connection rather than going through $wpdb. That is a property of the component, not a wp-codebox defect, and it is orthogonal to this issue. I mention it only so the 111 is not mistaken for backend breakage.

Revised ask

1. Emit a reason for exclusion — withdrawn, based on a misread.
2. Why does mdi-native execute zero tests when sqlite executes 1,232 on the identical configuration? That is the whole issue.
3. Fail fast when nothing executes. Still stands — the run consumed 1336s of a 1500s budget having executed nothing.

Apologies for the noise in the original report. Environment for all three runs: wp-codebox d4e0f713 (current main, includes #2490/#2494/#2497/#2498), managed install with valid provenance, PHP 8.4, Node 25.

[chubes4]

Second correction — mdi-native DOES execute tests. Retitling the real problem.

I ran a control on a different component and it overturns the headline again. I would rather post this than leave a wrong report standing.

mdi-native runs fine on another component

Extra-Chill/extrachill-api — same wp_codebox_multisite: true, same wp_codebox_phpunit_bootstrap_mode: "managed", no external database service:

Backend	Executed	Result
default (no database_type set)	231	231 passed, 0 failed
mdi-native	231	216 passed, 15 failed

So mdi-native is not broken as a backend. It discovers, routes, and executes. My original report — "executes zero tests" — is only true of one component and should not have been generalised.

What is actually broken, part 1: a MySQL type-fidelity gap

All 15 mdi-native failures on that component are the same assertion shape:

AssertionFailedError: Failed asserting that null is identical to 0.
  Route_AffinityTest::test_authenticated_caller_identity_survives_loopback
  Route_AffinityTest::test_booking_availability_affinity_preserves_body_and_identity
  ...

These tests pass on the default backend and fail on mdi-native, same commit, same config, only the backend changed. mdi-native is returning null where MySQL returns the string 0.

This is the thing that matters for treating mdi-native as a MySQL-compatible harness: a backend that returns null instead of 0 will silently pass or fail tests differently from production, which is worse than not running at all. Strict-comparison assertions against $wpdb results are extremely common in WordPress suites.

What is actually broken, part 2: zero execution on one component

The original zero-execution symptom is real but component-specific, not general. The affected component differs from the working one in ways worth checking:

• validation_dependencies: ["data-machine"] (a second plugin must be installed into the sandbox)
• wp_codebox_prepare_steps running npm ci + npm run build across three block directories
• a much larger suite (1,335 candidates vs 231)

I have not yet isolated which of those triggers it. If it would help I can bisect, but the dependency-installation path is my first suspect given the other component has no validation_dependencies.

Revised asks

1. Emit a reason for exclusion — withdrawn (the excluded=1335 line is a separate standalone-PHP harness; it appears identically on runs that execute 1,232 tests).
2. mdi-native executes zero tests — withdrawn as a general claim; true only for one component, see above.
3. NEW, and the important one: mdi-native returns null where MySQL returns 0. Reproducible, 15 tests, clean control.
4. Investigate zero execution when validation_dependencies and/or prepare_steps are present.
5. Fail fast when nothing executes — still stands; that run consumed 1336s of a 1500s budget having executed nothing.

Environment for all runs: wp-codebox d4e0f713 (current main, includes #2490/#2494/#2497/#2498), managed install with valid provenance, PHP 8.4, Node 25.

Sorry for the churn on this issue. Each correction came from adding a control I should have run before filing.

[chubes4]

Bisect complete — zero-execution is not caused by any config difference

Followed up on the component-specific zero-execution. Removed each candidate cause in turn, full run each time.

Config removed	Backend	Executed
baseline (everything present)	mdi-native	0
validation_dependencies	mdi-native	0
...also wp_codebox_prepare_steps	mdi-native	0
...also committed phpunit.xml + phpunit.contracts.xml	mdi-native	0
...also all but one test file (95 → 1)	mdi-native	0
wp_codebox_multisite: false (separate run)	mdi-native	0

After the third row the component config is byte-identical in the relevant fields to extrachill-api, which executes 231 tests on mdi-native — only wp_codebox_multisite: true and wp_codebox_phpunit_bootstrap_mode: "managed".

What this rules out

• validation_dependencies (my earlier prime suspect) — not the cause
• prepare_steps / npm build steps — not the cause
• Committed PHPUnit configs — not the cause
• Suite scale (1,335 candidates vs 231) — not the cause; one single test file still executes zero
• Multisite — not the cause

What it points to

Because a single test file still yields zero executed, the failure happens during environment or plugin bootstrap, before PHPUnit reaches any test. It is a property of the plugin under test, not of the recipe configuration.

One concrete difference worth checking: this plugin creates a custom database table on activation via dbDelta (inc/Core/EventDatesTable.php), which extrachill-api does not do. If mdi-native cannot service dbDelta / CREATE TABLE during plugin activation, activation would abort and the harness would come up with nothing to run — which matches the observed "candidates discovered, zero executed, budget consumed" signature exactly.

I have not confirmed that hypothesis; there are no retained artifacts on the failing path to check against. Which leads to:

The diagnosability gap is the real blocker here

Across ~6 full runs (~20 min each) the failing path produced no error, no stack trace, and no retained artifact — only zero executed tests and a consumed budget. #2498 improved this for one path (a run now surfaces failed test command did not expose an observed failed test ID instead of hanging silently) but the bootstrap-failure path still yields nothing actionable.

If activation or bootstrap fails under mdi-native, that error needs to surface. Right now the only way to narrow it is to bisect the consuming repo by hand, which is what I just spent several hours doing.

Standing asks, in priority order

1. mdi-native returns null where MySQL returns 0 (15 reproducible failures on extrachill-api, clean control). This is the blocker for trusting it as a MySQL-compatible harness.
2. Surface bootstrap/activation errors on the mdi-native path. Silence here is what made everything above expensive.
3. Investigate dbDelta / CREATE TABLE during plugin activation under mdi-native as the likely cause of zero-execution.
4. Fail fast when candidates > 0 and nothing executes rather than consuming the full budget.

Environment: wp-codebox d4e0f713 (current main), managed install, valid provenance, PHP 8.4, Node 25.