Test infra (optional): extract test-runner auditors to a plugin (pytest/go/cargo as peers)

CodeyBox.slnx

@@ -41,6 +41,7 @@
   </Folder>
   <Folder Name="/plugins/">
     <Project Path="plugins/CodeyBox.FileSizeLimitsAuditorPlugin/CodeyBox.FileSizeLimitsAuditorPlugin.csproj" />
+    <Project Path="plugins/CodeyBox.DotnetTestRunnerPlugin/CodeyBox.DotnetTestRunnerPlugin.csproj" />
     <Project Path="plugins/CodeyBox.OpencodeGoQuotaPlugin/CodeyBox.OpencodeGoQuotaPlugin.csproj" />
     <Project Path="plugins/CodeyBox.QuotaResetNotifier/CodeyBox.QuotaResetNotifier.csproj" />
     <Project Path="plugins/CodeyBox.StatisticsPlugin/CodeyBox.StatisticsPlugin.csproj" />

docs/extending/auditor-plugins.md

@@ -233,6 +233,15 @@ GET /plugins
 → [{ "pluginId": "myorg.no-var-keyword", "displayName": "MyOrg: Ban var keyword" }]
 ```
 
+## Test-runner auditors
+
+Test execution is a specialised plugin shape: `ITestRunnerAuditor`
+(`CodeyBox.Core`) owns invocation building, test-universe enumeration, and
+result classification for one framework. The bundled `dotnet test` runner and
+a pytest reference stub ship in `plugins/CodeyBox.DotnetTestRunnerPlugin/`.
+See [`docs/extending/test-runner-plugins.md`](test-runner-plugins.md) for the
+packaging, the peer-runner recipe, and the bundling caveat.
+
 ## Sample plugin
 
 A fully working sample is provided at `samples/CodeyBox.SampleAuditorPlugin/`.

docs/extending/plugins.md

@@ -210,7 +210,7 @@ the operator explicitly opts in. This is intentional.
 
 Every plugin declares the minimum host API version it requires via
 `minHostApiVersion` on `[CodeyBoxPlugin]`. The host rejects plugins that
-require a version newer than `CodeyBoxApiVersion.Current` (currently `"1.1"`).
+require a version newer than `CodeyBoxApiVersion.Current` (currently `"1.3"`).
 
 ### Version bump rules
 

docs/extending/test-runner-plugins.md

@@ -0,0 +1,84 @@
+# Test-runner plugins
+
+The merge gate's test step is an `ITestRunnerAuditor` (`CodeyBox.Core`), not a
+bespoke host service. The bundled `dotnet test` runner ships as a plugin
+package — `plugins/CodeyBox.DotnetTestRunnerPlugin/` — so `pytest`, `go test`,
+and `cargo test` can be peer `ITestRunnerAuditor` plugins without touching
+`CodeyBox.Core`. `TestFramework` already declares all four members.
+
+This is a packaging seam, not a behavioural one: the default host still
+registers the dotnet runner explicitly (same byte-identical command, same
+soundness tests), with or without any allowlist entry.
+
+## Contents
+
+- [`DotnetTestAuditor`](#dotnettestauditor) — the bundled implementation
+- [`DotnetTestRunner`](#dotnettestrunner) — the attributed plugin entry
+- [`PytestTestRunner`](#pytesttestrunner) — the reference stub
+- [Writing a peer runner](#writing-a-peer-runner)
+- [Bundling caveat](#bundling-caveat)
+
+## `DotnetTestAuditor`
+
+Lives in the plugin package (`CodeyBox.DotnetTestRunnerPlugin` namespace).
+Owns the full `dotnet test` invocation — base command, test-selection
+`--filter`, `--blame-hang` args — carries its own result classifier, and
+delegates each run to `ShellCommandAuditor` so tool-presence handling stays
+identical to the generic shell path. With an all-tests selection and default
+options the command is byte-identical to `["dotnet", "test", "--no-build"]`.
+
+The host default-registers one instance (`csharp:test-pass`, build-test-gate
+role) in `Program.cs`, and the `csharp` language preset builds per-project
+instances with the same hot-reloadable run options. The VSTest escaping guard
+(`VstestFilterEscaping`) and the output parser (`DotnetTestOutputParser`) stay
+in `CodeyBox.Audit.Shell` as the single shared copy both the plugin and the
+shell attribution path reuse.
+
+## `DotnetTestRunner`
+
+The `[CodeyBoxPlugin]` entry (`id: "codeybox.dotnet-test-runner"`). A thin,
+parameterless-constructible wrapper over the canonical `csharp:test-pass`
+configuration — parameterless because the plugin loader registers entry types
+without constructor arguments.
+
+Run options are static per load: `BlameHangTimeout` / `AuditorIdleTimeout`
+under `CodeyBox:Plugins:codeybox.dotnet-test-runner:` are read once in
+`InitializeAsync`. Invalid values log a warning and keep
+`TestRunOptions.Default`. The host's default registration (live
+`Func<TestRunOptions>`) remains the hot-reloadable path.
+
+## `PytestTestRunner`
+
+A reference-only stub (`id: "codeybox.pytest-test-runner"`) proving a second
+framework needs no Core change: a real `TestSuiteDescriptor`
+(`pytest --collect-only -q`) and a real `BuildInvocation` shape (bare `pytest`
+for the whole suite, `pytest -k <or-joined>` when narrowed). `RunAsync`
+deliberately throws `NotSupportedException` instead of returning a fabricated
+pass, so the stub can never green a gate it did not run. It is not referenced
+by any catalog or DI registration. A production pytest runner would replace the
+throw with a sandboxed `pytest` invocation plus a pytest output classifier.
+
+## Writing a peer runner
+
+1. Reference `CodeyBox.Core` + `CodeyBox.PluginSdk` only (never the
+   orchestrator or API). The pytest stub is the template: it needs nothing
+   else.
+2. Implement `ITestRunnerAuditor`: a stable `Name`, the `TestSuiteDescriptor`
+   (framework + enumeration argv the selector reasons about), `BuildInvocation`
+   (apply the narrowed `TestSelection` through runner-native filter syntax —
+   never raw passthrough of untrusted baseline names), and a `ResultClassifier`
+   that distinguishes genuine test failures from an unrunnable environment.
+3. Decorate with `[CodeyBoxPlugin]` and a new `TestFramework` member only if
+   the framework is not already declared. `Pytest`, `GoTest`, and `CargoTest`
+   are pre-declared.
+4. Keep the runner out of the default panel until the host gains multi-runner
+   selection: the single-`ITestRunnerAuditor` default registration assumes one
+   canonical runner.
+
+## Bundling caveat
+
+The dotnet entry is bundled AND default-registered. Do not add
+`codeybox.dotnet-test-runner` to the allowlist of a host that already
+default-registers it — the discovered copy would join the audit panel as a
+second test gate alongside the default one. The plugin id exists so downstream
+hosts can load this package as an external plugin instead of referencing it.

docs/quality/audit.md

@@ -230,7 +230,7 @@ non-zero on findings.
 
 Capability: `None`.
 
-### `DotnetTestAuditor` (`CodeyBox.Audit.Shell`)
+### `DotnetTestAuditor` (`CodeyBox.DotnetTestRunnerPlugin`)
 
 Backs the built-in `csharp:test-pass` gate. It is a first-class
 `ITestRunnerAuditor` rather than a generic `ShellCommandAuditor`, so

plugins/CodeyBox.DotnetTestRunnerPlugin/CodeyBox.DotnetTestRunnerPlugin.csproj

@@ -0,0 +1,22 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net10.0</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="..\..\src\CodeyBox.Core\CodeyBox.Core.csproj" />
+    <ProjectReference Include="..\..\src\CodeyBox.PluginSdk\CodeyBox.PluginSdk.csproj" />
+    <!--
+      Execution reuse: the runner delegates each run to ShellCommandAuditor so
+      the tool-presence probe, missing-tool handling and result classification
+      stay identical to the generic shell path. Only Core and PluginSdk cross
+      the plugin load-context boundary; the shell copy is an implementation
+      detail of this first-party package (see DotnetTestRunnerPlugin remarks).
+    -->
+    <ProjectReference Include="..\..\src\CodeyBox.Audit.Shell\CodeyBox.Audit.Shell.csproj" />
+  </ItemGroup>
+
+</Project>

src/CodeyBox.Audit.Shell/DotnetTestAuditor.cs → plugins/CodeyBox.DotnetTestRunnerPlugin/DotnetTestAuditor.cs

@@ -1,13 +1,15 @@
 using System.Globalization;
 using System.Text.Json;
-using CodeyBox.Audit;
+using CodeyBox.Audit.Shell;
 using CodeyBox.Core;
 
-namespace CodeyBox.Audit.Shell;
+namespace CodeyBox.DotnetTestRunnerPlugin;
 
 /// <summary>
-/// First-class <c>dotnet test</c> auditor. Replaces the previous arrangement
-/// where <c>csharp:test-pass</c> was a generic <see cref="ShellCommandAuditor"/>
+/// First-class <c>dotnet test</c> auditor, shipped as a CodeyBox plugin (see
+/// <see cref="DotnetTestRunner"/>, the attributed entry point). It replaces
+/// the previous arrangement where <c>csharp:test-pass</c> was a generic
+/// <see cref="ShellCommandAuditor"/>
 /// that three separate call sites had to sniff as "really a dotnet test"
 /// (result-classifier selection by <c>argv[1]=="test"</c>, per-test hang
 /// handling, and a future <c>--filter</c> injection).

plugins/CodeyBox.DotnetTestRunnerPlugin/DotnetTestRunner.cs

@@ -0,0 +1,134 @@
+using System.Globalization;
+using CodeyBox.Audit.Shell;
+using CodeyBox.Core;
+using CodeyBox.PluginSdk;
+using Microsoft.Extensions.Logging;
+
+namespace CodeyBox.DotnetTestRunnerPlugin;
+
+/// <summary>
+/// Plugin entry point for the bundled <c>dotnet test</c> runner. A thin,
+/// parameterless-constructible <see cref="ITestRunnerAuditor"/> over the shared
+/// <see cref="DotnetTestAuditor"/> implementation, so the plugin loader (which
+/// registers entry types without constructor arguments) can load this package
+/// while the host keeps default-registering the implementation directly with
+/// live hot-reloadable options.
+///
+/// <para>Do NOT add this plugin id to the allowlist of a host that already
+/// default-registers the dotnet runner (this repository's host does): the
+/// discovered copy would join the audit panel as a second test gate alongside
+/// the default one. The id exists so downstream hosts can load this package as
+/// an external plugin instead of referencing it.</para>
+///
+/// <para>Run options are static per load (not hot-reloadable): the optional
+/// <c>BlameHangTimeout</c> / <c>AuditorIdleTimeout</c> scoped-config values are
+/// read once in <see cref="InitializeAsync"/>. An invalid value logs a warning
+/// and keeps <see cref="TestRunOptions.Default"/> — a bad config value must
+/// never break plugin load, mirroring the auditor's fail-safe mode fallback.
+/// The host's default registration (live <c>Func{TestRunOptions}</c>) is the
+/// hot-reloadable path. Test selection, failure attribution, and shadow
+/// telemetry are not wired on the standalone entry, so its runs always execute
+/// the full suite (the fail-safe default).</para>
+/// </summary>
+[CodeyBoxPlugin(
+    id: PluginId,
+    displayName: "CodeyBox: dotnet test runner",
+    minHostApiVersion: "1.3")]
+public sealed class DotnetTestRunner : ITestRunnerAuditor, IShellAuditorArgvProvider, IPluginInitializer
+{
+    /// <summary>Allowlist / config-scope id for this plugin.</summary>
+    public const string PluginId = "codeybox.dotnet-test-runner";
+
+    /// <summary>Runner name, identical to the host's default registration.</summary>
+    public const string RunnerName = "csharp:test-pass";
+
+    /// <summary>Canonical base command the selection seam enumerates against.</summary>
+    public static readonly IReadOnlyList<string> CanonicalBaseArgv = ["dotnet", "test", "--no-build"];
+
+    private TestRunOptions _runOptions = TestRunOptions.Default;
+    private ILogger _logger = Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance;
+
+    /// <summary>
+    /// Parameterless for the plugin loader. Builds the canonical
+    /// <c>csharp:test-pass</c> build-test-gate runner over
+    /// <see cref="TestRunOptions.Default"/> until <see cref="InitializeAsync"/>
+    /// optionally narrows it from scoped config.
+    /// </summary>
+    public DotnetTestRunner()
+    {
+    }
+
+    public string Name => RunnerName;
+
+    public string Kind => "shell";
+
+    public AuditCapabilities Required => AuditCapabilities.None;
+
+    public bool CanShortCircuitOnBlockingFinding => true;
+
+    public AuditorRole Role => AuditorRole.BuildTestGate;
+
+    public BuildTestGateEvidence BuildTestGateEvidence => BuildTestGateEvidence.Test;
+
+    public TestSuiteDescriptor TestSuite =>
+        new(TestFramework.DotnetTest, [.. CanonicalBaseArgv, "--list-tests"]);
+
+    public IAuditResultClassifier ResultClassifier { get; } = new DotnetTestCommandResultClassifier();
+
+    public TestRunOptions CurrentRunOptions => _runOptions;
+
+    public IReadOnlyList<string> Argv => BuildInner().Argv;
+
+    public IReadOnlyList<string> BuildInvocation(TestSelection selection, TestRunOptions options)
+        => BuildInner().BuildInvocation(selection, options);
+
+    public Task<AuditResult> RunAsync(
+        ISandbox sandbox,
+        string workingDirectory,
+        AuditContext context,
+        CancellationToken ct = default)
+        => BuildInner().RunAsync(sandbox, workingDirectory, context, ct);
+
+    public Task InitializeAsync(PluginContext context, CancellationToken ct = default)
+    {
+        ArgumentNullException.ThrowIfNull(context);
+        _logger = context.Logger;
+        var blameHang = ParseOptionalTimeout(context.ScopedConfig["BlameHangTimeout"], "BlameHangTimeout");
+        var idle = ParseOptionalTimeout(context.ScopedConfig["AuditorIdleTimeout"], "AuditorIdleTimeout");
+        if (blameHang is null && idle is null)
+            return Task.CompletedTask;
+        _runOptions = new TestRunOptions
+        {
+            BlameHangTimeout = blameHang ?? _runOptions.BlameHangTimeout,
+            IdleTimeout = idle ?? _runOptions.IdleTimeout,
+        };
+        _logger.LogInformation(
+            "DotnetTestRunner initialized: blameHang={BlameHang} idleTimeout={IdleTimeout}",
+            _runOptions.BlameHangTimeout, _runOptions.IdleTimeout);
+        return Task.CompletedTask;
+    }
+
+    private TimeSpan? ParseOptionalTimeout(string? value, string key)
+    {
+        if (string.IsNullOrWhiteSpace(value))
+            return null;
+        if (TimeSpan.TryParse(value, CultureInfo.InvariantCulture, out var parsed) && parsed > TimeSpan.Zero)
+            return parsed;
+        _logger.LogWarning(
+            "DotnetTestRunner: ignoring invalid {Key}={Value}; keeping the current run options",
+            key, value);
+        return null;
+    }
+
+    private DotnetTestAuditor BuildInner()
+        => new(new DotnetTestAuditorOptions
+        {
+            Name = RunnerName,
+            BaseArgv = CanonicalBaseArgv,
+            CanShortCircuitOnBlockingFinding = true,
+            Role = AuditorRole.BuildTestGate,
+            BuildTestGateEvidence = BuildTestGateEvidence.Test,
+            RunOptionsAccessor = () => _runOptions,
+            SelfHealNuGetHome = true,
+        });
+}

plugins/CodeyBox.DotnetTestRunnerPlugin/PytestTestRunner.cs

@@ -0,0 +1,84 @@
+using CodeyBox.Core;
+using CodeyBox.PluginSdk;
+
+namespace CodeyBox.DotnetTestRunnerPlugin;
+
+/// <summary>
+/// Reference-only stub demonstrating the <see cref="ITestRunnerAuditor"/> seam
+/// for a second framework. It carries a real <see cref="TestSuiteDescriptor"/>
+/// and a real <see cref="BuildInvocation"/> shape (<c>pytest</c>, narrowing via
+/// <c>-k</c>), but <see cref="RunAsync"/> is intentionally unimplemented: this
+/// host has no Python projects and the stub is NOT registered in any catalog or
+/// DI container. A production pytest runner would replace the throw with a
+/// sandboxed <c>pytest</c> invocation (mirroring how
+/// <see cref="DotnetTestAuditor"/> delegates to <c>ShellCommandAuditor</c>)
+/// plus a pytest output classifier — without touching <c>CodeyBox.Core</c>.
+/// </summary>
+[CodeyBoxPlugin(
+    id: PluginId,
+    displayName: "CodeyBox: pytest test runner (reference stub)",
+    minHostApiVersion: "1.3")]
+public sealed class PytestTestRunner : ITestRunnerAuditor
+{
+    /// <summary>Allowlist / config-scope id for this stub.</summary>
+    public const string PluginId = "codeybox.pytest-test-runner";
+
+    /// <inheritdoc/>
+    public string Name => "pytest:test-pass";
+
+    /// <inheritdoc/>
+    public string Kind => "shell";
+
+    /// <inheritdoc/>
+    public AuditCapabilities Required => AuditCapabilities.None;
+
+    /// <inheritdoc/>
+    public TestSuiteDescriptor TestSuite =>
+        new(TestFramework.Pytest, ["pytest", "--collect-only", "-q"]);
+
+    /// <inheritdoc/>
+    public IAuditResultClassifier ResultClassifier { get; } = new PytestPassthroughClassifier();
+
+    /// <inheritdoc/>
+    public TestRunOptions CurrentRunOptions => TestRunOptions.Default;
+
+    /// <summary>
+    /// Builds the <c>pytest</c> argv: bare <c>pytest</c> for the whole suite,
+    /// otherwise <c>pytest -k &lt;expr&gt;</c> with the selected filters
+    /// <c>or</c>-joined. pytest <c>-k</c> treats each entry as a substring
+    /// match, so entries are passed through verbatim (no VSTest escaping —
+    /// that guard belongs to the dotnet runner only).
+    /// </summary>
+    public IReadOnlyList<string> BuildInvocation(TestSelection selection, TestRunOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(selection);
+        ArgumentNullException.ThrowIfNull(options);
+        if (selection.IsAll)
+            return ["pytest"];
+        return ["pytest", "-k", string.Join(" or ", selection.Filters)];
+    }
+
+    /// <summary>
+    /// Not implemented by design: this is a reference stub, never registered.
+    /// Always throws <see cref="NotSupportedException"/> rather than returning
+    /// a fabricated pass so the stub can never green a gate it did not run.
+    /// </summary>
+    public Task<AuditResult> RunAsync(
+        ISandbox sandbox,
+        string workingDirectory,
+        AuditContext context,
+        CancellationToken ct = default)
+        => throw new NotSupportedException(
+            "PytestTestRunner is a reference stub demonstrating the ITestRunnerAuditor seam; " +
+            "it is not registered and cannot execute a test run.");
+
+    /// <summary>
+    /// No refinement: without a real pytest output grammar there is nothing
+    /// sound to classify, so failed commands fall back to the generic
+    /// command-failure result.
+    /// </summary>
+    private sealed class PytestPassthroughClassifier : IAuditResultClassifier
+    {
+        public AuditResult? ClassifyFailedCommand(AuditResultClassificationContext context) => null;
+    }
+}

src/CodeyBox.Api/CodeyBox.Api.csproj

@@ -21,6 +21,7 @@
     <ProjectReference Include="..\CodeyBox.Audit.Presets\CodeyBox.Audit.Presets.csproj" />
     <ProjectReference Include="..\CodeyBox.Audit.Llm\CodeyBox.Audit.Llm.csproj" />
     <ProjectReference Include="..\CodeyBox.Audit.Shell\CodeyBox.Audit.Shell.csproj" />
+    <ProjectReference Include="..\..\plugins\CodeyBox.DotnetTestRunnerPlugin\CodeyBox.DotnetTestRunnerPlugin.csproj" />
     <ProjectReference Include="..\CodeyBox.Sandbox.Bubblewrap\CodeyBox.Sandbox.Bubblewrap.csproj" />
     <ProjectReference Include="..\CodeyBox.Sandbox.Incus\CodeyBox.Sandbox.Incus.csproj" />
     <ProjectReference Include="..\CodeyBox.Sandbox.Multipass\CodeyBox.Sandbox.Multipass.csproj" />