Skip to content

fix(xcresult): attribute a failure to the test's own frame, not a dependency's - #1159

Merged
trunk-io[bot] merged 1 commit into
claude/xcresult-attribution-typesfrom
claude/xcresult-test-frame-attribution
Aug 19, 2026
Merged

fix(xcresult): attribute a failure to the test's own frame, not a dependency's#1159
trunk-io[bot] merged 1 commit into
claude/xcresult-attribution-typesfrom
claude/xcresult-test-frame-attribution

Conversation

@dfrankland

@dfrankland dfrankland commented Aug 18, 2026

Copy link
Copy Markdown
Member

Stack 2/3 — depends on #1158.

The file we report for a test is whatever the failure summary points at, and for a failure raised inside a helper that is the helper's file: a snapshot trait, a mocking framework, a page object, a launch helper. The path lands under the package checkout (Tuist/.build/checkouts/..., DerivedData/SourcePackages/checkouts/...), and since codeowners are resolved from that path the test is then owned by whoever owns the vendored directory.

The test's own frame

Adds FileSource::TestFrame, the frame whose symbol names the test. Frames run innermost first, so the test's own frame sits in the middle of the stack — helpers it called below it, the framework that invoked it above — which is why taking the last frame lands on a dependency.

TestIdentity owns the spellings a frame can use: Swift (Suite.testCase()), Objective-C (-[Suite testCase]), closures declared inside the test, and top-level swift-testing functions, which have no suite.

imageName looks like the natural discriminator here and is not: SPM dependencies are statically linked into the test bundle, so every frame reports the test bundle's name.

Vetting in one place

That frame is the only source that identifies the test rather than the failure, which FileSource::is_positive_identification states. Everything else is vetted against ReportedPath::is_vendored_dependency in a single find:

FileCandidate::from_failure_summary(failure_summary, identity)
    .into_iter()
    .find(Self::is_reportable)

so a source added later cannot quietly skip the check — previously the same filter was repeated at each link of the chain.

The stack fallback now yields its frames outermost-first instead of collapsing to a single "last frame", so rejecting one lands on the next frame out rather than giving up.

When nothing survives

When a test crashes or fails to launch it never reaches its own frame and every source points into a dependency; we then report no file at all rather than one that would re-own the test. Consumers already treat a missing file as "unchanged" rather than "cleared", so the test keeps the path and owners it last had.

Note this also changes attribution for a failure raised inside an in-repo helper: the test's own file now wins over the helper's. That is the intended reading of "the file of the test case".

Verified against a customer bundle

Confirmed on a customer UI test bundle from this thread, whose single failure
comes from a page-object helper in a vendored dependency:

reported file
before a path under .../Tuist/.build/checkouts/<dependency>/...
after (none — the vendored path is rejected)

That is the exact shape this PR targets, on real data: a Tuist/.build/checkouts/ path
that would resolve codeowners to whoever owns the vendored checkout.

It reports no file rather than the test's own file because of a separate schema bug that
stops the call stack being read at all; #1162 fixes that and turns this into the test's
own source file.

Real bundles reproducing each of these shapes are in the follow-up PR.

🤖 Generated with Claude Code

@trunk-staging-io

trunk-staging-io Bot commented Aug 18, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

Failed Test Failure Summary Logs
pending_quarantine_test should be quarantined when run with variant A test marked as pending was expected to fail but unexpectedly passed. Logs ↗︎
variant_quarantine_test should be quarantined when run with variant A test expected the sum of 2 + 2 to be 5, but it was actually 4, indicating a failing assertion. Logs ↗︎

View Full Report ↗︎Docs

@codecov-commenter

codecov-commenter commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.02913% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.34%. Comparing base (8579cd1) to head (52b4923).

Files with missing lines Patch % Lines
xcresult/src/file_attribution.rs 98.70% 1 Missing ⚠️
Additional details and impacted files
@@                          Coverage Diff                          @@
##           claude/xcresult-attribution-types    #1159      +/-   ##
=====================================================================
+ Coverage                              83.28%   83.34%   +0.06%     
=====================================================================
  Files                                     72       72              
  Lines                                  16110    16180      +70     
=====================================================================
+ Hits                                   13417    13486      +69     
- Misses                                  2693     2694       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@trunk-io

trunk-io Bot commented Aug 18, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

…endency's

The file we report for a test is whatever the failure summary points at, and for
a failure raised inside a helper that is the helper's file: a snapshot trait, a
mocking framework, a page object, a launch helper. The path lands under the
package checkout (`Tuist/.build/checkouts/...`,
`DerivedData/SourcePackages/checkouts/...`), and since codeowners are resolved
from that path the test is then owned by whoever owns the vendored directory.

Adds `FileSource::TestFrame`, the frame whose symbol names the test. Frames run
innermost first, so the test's own frame sits in the middle of the stack —
helpers it called below it, the framework that invoked it above — which is why
taking the last frame lands on a dependency. `TestIdentity` owns the spellings a
frame can use: Swift (`Suite.testCase()`), Objective-C (`-[Suite testCase]`),
closures declared inside the test, and top-level swift-testing functions, which
have no suite.

That frame is the only source that identifies the test rather than the failure,
which `FileSource::is_positive_identification` states. Everything else is vetted
against `ReportedPath::is_vendored_dependency` in a single `find`, so a source
added later cannot quietly skip the check — previously the same filter was
repeated at each link of the chain.

The stack fallback now yields its frames outermost-first instead of collapsing
to a single "last frame", so rejecting one lands on the next frame out rather
than giving up. When a test crashes or fails to launch it never reaches its own
frame and every source points into a dependency; we then report no file at all
rather than one that would re-own the test. Consumers already treat a missing
file as "unchanged" rather than "cleared", so the test keeps the path and owners
it last had.

Note this also changes attribution for a failure raised inside an in-repo
helper: the test's own file now wins over the helper's. That is the intended
reading of "the file of the test case".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dfrankland
dfrankland force-pushed the claude/xcresult-test-frame-attribution branch from e15e605 to 52b4923 Compare August 18, 2026 23:32

@TylerJang27 TylerJang27 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Admittedly a bit over my head. Maybe we need a xcresult brown bag at some point 🙃

pub fn is_named_by(&self, symbol: &str) -> bool {
let expected = match self.suite {
Some(suite) => vec![
format!("{}.{}", suite, self.case),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this overly generic for the frame text patterns?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two arms mirror the Swift demangler's own grammar. From printEntity, the function that formats every demangled Swift symbol
(NodePrinter.cpp:3593):

// Either we print the context in prefix form "<context>.<name>" or in
// suffix form "<name> in <context>".

Those are the only two shapes a symbol's context can take, and they map 1:1 to the matcher. The
ObjC spelling -[Suite testCase] is also exact-match. No contains anywhere, and all three shapes are confirmed against real symbol names in the #1160 bundles.

@trunk-io
trunk-io Bot merged commit 0e48e67 into main Aug 19, 2026
27 checks passed
@trunk-io

trunk-io Bot commented Aug 19, 2026

Copy link
Copy Markdown

This pull request was merged into main as part of stacked PR 1162.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants