Core .NET Framework unit tests: the MSBuild Test target reports success while running zero tests

[gfraiteur]

Summary

The MSBuild Test target of the three .NET Framework unit test projects under Tests/Core passes an invalid path to nunit3-console.exe. The runner terminates before loading the test assembly, and because the target ignores the exit code, the build reports success while executing zero tests.

The affected projects are:

• Tests/Core/PostSharp.Core.Test.NetFramework/PostSharp.Core.Test.NetFramework.csproj
• Tests/Core/PostSharp.Compiler.Client.Test/PostSharp.Compiler.Client.Test.csproj
• Tests/Core/PostSharp.Compiler.SymbolLocationDb.Test/PostSharp.Compiler.SymbolLocationDb.Test.csproj

Reproduction

Build/Scripts/msbuild.ps1 Tests/Core/PostSharp.Core.Test.NetFramework/PostSharp.Core.Test.NetFramework.proj /t:Test

Exits with code 0 and prints no test output. The binary log shows that nunit3-console.exe was started and terminated immediately:

System.NotSupportedException : The given path's format is not supported.
   at System.Security.Permissions.FileIOPermission.EmulateFileIOPermissionChecks(String fullPath)
   at NUnit.Engine.TestPackage..ctor(String filePath)
exit code -100

Root cause

Each of the three projects contained a copy-pasted Test target of this shape:

<Exec Command="&quot;...nunit3-console.exe&quot; &quot;$(MSBuildProjectDirectory)\$(PublishDir)\$(AssemblyName).dll&quot; --labels=On --teamcity --x86"
      IgnoreExitCode="True" IgnoreStandardErrorWarningFormat="True" />

$(PublishDir) is defined in Tests/Core/PostSharp.Core.Test/Publish.targets. Commit cb16095378 (2025-07-16, "Replace relative paths to $(PostSharpBinDir)") changed it from the relative ..\..\..\Build\bin\bin.$(Configuration)\tests\$(AssemblyName)\ to the absolute $(PostSharpBinDir)bin.$(Configuration)\tests\$(AssemblyName)\. The three Test targets were not updated, so they now compose a project directory with an absolute path and produce a path with a drive letter in the middle:

C:\...\Tests\Core\PostSharp.Core.Test.NetFramework\C:\...\Build\bin\bin.Debug\tests\...\PostSharp.Core.Test.NetFramework.dll

Two independent defects combine here:

1. The malformed path, which makes the runner fail before it can execute anything.
2. IgnoreExitCode="True" without any inspection of the exit code. Not failing the build on failed tests is by design in this repository — the failures are reported to TeamCity by the runner's own service messages and the run continues so that the remaining suites still execute. The defect is that this makes a run in which the runner itself never started indistinguishable from a clean run: nothing is reported to TeamCity either, so the suite silently counts as executed.

Impact on CI

The TeamCity job TestNetFxOther ("Windows X64 + .NET Framework 4.8.1 + .NET SDK 8.0 + VS 17.0: Other") runs the make target NetFxTestOtherFromArtifacts, which runs make -C Tests\Core ClientTests CoreTests SerializationTests OtherTests. ClientTests and CoreTests invoke the .proj wrappers of the three projects, which in turn invoke the broken Test target. The CI path is therefore the same path, and these unit tests have not been executed anywhere since 2025-07-16.

The job stayed green because the reporting relies on the --teamcity service messages emitted by the runner: a runner that dies before starting emits no message at all, so TeamCity observed no test and no failure.

Test counts that were silently skipped (per architecture, each suite being run twice, 32-bit and 64-bit):

Project	Tests
PostSharp.Core.Test.NetFramework	357
PostSharp.Compiler.Client.Test	146
PostSharp.Compiler.SymbolLocationDb.Test	29

Secondary defects found in the same targets

• Neither run passed a result path, so nunit3-console.exe wrote TestResult.xml into the working directory (the project's source directory — hence the TestResult.xml entry in .gitignore), and the second run overwrote the result of the first one. The <Delete Files="$(OutDir)\$(AssemblyName).nunit.xml" /> task at the top of each target referred to a file that nothing ever produced.
• Tests/Core/PostSharp.Compiler.SymbolLocationDb.Test/PostSharp.Compiler.SymbolLocationDb.Test.proj declared <TestName>Compiler.Client.Test</TestName>, so its results would have been reported to TeamCity under the name of another suite.

Fix

The three copies of the target are replaced by a single shared implementation in Tests/Core/PostSharp.Core.Test/NUnitTest.targets, imported by the three projects, which:

• passes $(PublishDir)$(AssemblyName).dll and fails with a clear message if the test assembly or the runner is missing;
• keeps the existing reporting model unchanged: both architectures are always run, failed tests do not fail the build, and they continue to be reported to TeamCity by the runner's --teamcity service messages;
• captures the exit code of each run and turns a negative one into an MSBuild error. A negative code means the runner itself could not run, so no test was started and nothing was reported at all — the case this issue is about. It routes into the Test initialization failed handling that UnitTests.Common.targets already implements;
• writes a distinct result file per architecture next to the test assembly instead of into the source tree.

State after the fix

The suites run and report. msbuild PostSharp.Core.Test.NetFramework.proj /t:Test now exits 0 and emits 38 ##teamcity[testFailed] messages, so TeamCity — not MSBuild — decides what to do with them. The .NET Framework Core suite is not green; the failures are pre-existing and were simply never visible:

• 64-bit run: 357 total, 351 passed, 6 failed — one CodeModel.TestCustomAttributes fixture that cannot resolve mscorlib, and five User.* VCS licensing tests that depend on the state of the local Git/TFVC working copy.
• 32-bit run on a developer machine: 357 total, 325 passed, 32 failed. The 26 extra failures are all Strong name validation failed on the delay-signed PostSharp.Core.Test.Assets, because the skip-verification entry is registered only under HKLM\SOFTWARE\Microsoft\StrongName\Verification and not under the Wow6432Node view that the 32-bit CLR reads. The build image registers both views (eng/dockerfiles/postsharp-2024.0-win-x64-build.Dockerfile), so this is expected to be a developer-machine artifact, but it has never been observed on CI and needs to be confirmed by a real run.

The PostSharp.Compiler.Client.Test (146) and PostSharp.Compiler.SymbolLocationDb.Test (29) suites pass fully in both architectures.

Related

On the release/2026.0 line, commit 344e7ee821 (2025-11-11, "Fixing Core tests.") corrected the path in PostSharp.Core.Test.NetFramework.csproj only. The two sibling projects still carry the defect there, and none of the copies inspects the exit code. The forward merge of this fix resolves both.

-- Claude for Gael

[gfraiteur]

Fixed by postsharp/PostSharp#88 (targets release/2024.0). Cross-repository closing keywords do not auto-close, so this issue needs closing by hand once that PR is merged.

-- Claude for Gael