fix(xcresult): attribute a failure to the test's own frame, not a dependency's - #1159
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## claude/xcresult-attribution-types #1159 +/- ##
=====================================================================
+ Coverage 83.28% 83.34% +0.06%
=====================================================================
Files 72 72
Lines 16110 16180 +70
=====================================================================
+ Hits 13417 13486 +69
- Misses 2693 2694 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…endency's The file we report for a test is whatever the failure summary points at, and for a failure raised inside a helper that is the helper's file: a snapshot trait, a mocking framework, a page object, a launch helper. The path lands under the package checkout (`Tuist/.build/checkouts/...`, `DerivedData/SourcePackages/checkouts/...`), and since codeowners are resolved from that path the test is then owned by whoever owns the vendored directory. Adds `FileSource::TestFrame`, the frame whose symbol names the test. Frames run innermost first, so the test's own frame sits in the middle of the stack — helpers it called below it, the framework that invoked it above — which is why taking the last frame lands on a dependency. `TestIdentity` owns the spellings a frame can use: Swift (`Suite.testCase()`), Objective-C (`-[Suite testCase]`), closures declared inside the test, and top-level swift-testing functions, which have no suite. That frame is the only source that identifies the test rather than the failure, which `FileSource::is_positive_identification` states. Everything else is vetted against `ReportedPath::is_vendored_dependency` in a single `find`, so a source added later cannot quietly skip the check — previously the same filter was repeated at each link of the chain. The stack fallback now yields its frames outermost-first instead of collapsing to a single "last frame", so rejecting one lands on the next frame out rather than giving up. When a test crashes or fails to launch it never reaches its own frame and every source points into a dependency; we then report no file at all rather than one that would re-own the test. Consumers already treat a missing file as "unchanged" rather than "cleared", so the test keeps the path and owners it last had. Note this also changes attribution for a failure raised inside an in-repo helper: the test's own file now wins over the helper's. That is the intended reading of "the file of the test case". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
e15e605 to
52b4923
Compare
TylerJang27
left a comment
There was a problem hiding this comment.
Admittedly a bit over my head. Maybe we need a xcresult brown bag at some point 🙃
| pub fn is_named_by(&self, symbol: &str) -> bool { | ||
| let expected = match self.suite { | ||
| Some(suite) => vec![ | ||
| format!("{}.{}", suite, self.case), |
There was a problem hiding this comment.
Is this overly generic for the frame text patterns?
There was a problem hiding this comment.
The two arms mirror the Swift demangler's own grammar. From printEntity, the function that formats every demangled Swift symbol
(NodePrinter.cpp:3593):
// Either we print the context in prefix form "<context>.<name>" or in // suffix form "<name> in <context>".
Those are the only two shapes a symbol's context can take, and they map 1:1 to the matcher. The
ObjC spelling -[Suite testCase] is also exact-match. No contains anywhere, and all three shapes are confirmed against real symbol names in the #1160 bundles.
|
This pull request was merged into |
Stack 2/3 — depends on #1158.
The file we report for a test is whatever the failure summary points at, and for a failure raised inside a helper that is the helper's file: a snapshot trait, a mocking framework, a page object, a launch helper. The path lands under the package checkout (
Tuist/.build/checkouts/...,DerivedData/SourcePackages/checkouts/...), and since codeowners are resolved from that path the test is then owned by whoever owns the vendored directory.The test's own frame
Adds
FileSource::TestFrame, the frame whose symbol names the test. Frames run innermost first, so the test's own frame sits in the middle of the stack — helpers it called below it, the framework that invoked it above — which is why taking the last frame lands on a dependency.TestIdentityowns the spellings a frame can use: Swift (Suite.testCase()), Objective-C (-[Suite testCase]), closures declared inside the test, and top-level swift-testing functions, which have no suite.imageNamelooks like the natural discriminator here and is not: SPM dependencies are statically linked into the test bundle, so every frame reports the test bundle's name.Vetting in one place
That frame is the only source that identifies the test rather than the failure, which
FileSource::is_positive_identificationstates. Everything else is vetted againstReportedPath::is_vendored_dependencyin a singlefind:so a source added later cannot quietly skip the check — previously the same filter was repeated at each link of the chain.
The stack fallback now yields its frames outermost-first instead of collapsing to a single "last frame", so rejecting one lands on the next frame out rather than giving up.
When nothing survives
When a test crashes or fails to launch it never reaches its own frame and every source points into a dependency; we then report no file at all rather than one that would re-own the test. Consumers already treat a missing file as "unchanged" rather than "cleared", so the test keeps the path and owners it last had.
Note this also changes attribution for a failure raised inside an in-repo helper: the test's own file now wins over the helper's. That is the intended reading of "the file of the test case".
Verified against a customer bundle
Confirmed on a customer UI test bundle from this thread, whose single failure
comes from a page-object helper in a vendored dependency:
.../Tuist/.build/checkouts/<dependency>/...That is the exact shape this PR targets, on real data: a
Tuist/.build/checkouts/paththat would resolve codeowners to whoever owns the vendored checkout.
It reports no file rather than the test's own file because of a separate schema bug that
stops the call stack being read at all; #1162 fixes that and turns this into the test's
own source file.
Real bundles reproducing each of these shapes are in the follow-up PR.
🤖 Generated with Claude Code