Flaky Tests not getting reported

[irfankhan14]

Hi,
I am not able to get the Flaky Test Report, I uploaded test reports with both Passed and Failed for same test manually.
But it is not reported as Flaky, please help me to resolve this issue.

I have attached the xml and the workflow too

Single Test Failure:

<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
    </failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.38" >
</testcase>

Double Test Failure:

<testcase name="testButtonClickAdd" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="1.438" />
<testcase name="testButtonClickSubtract" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"' 
    </failure>
</testcase>
<testcase name="testButtonClickMultiply" classname="net.yasik.sampleuitest.ExampleInstrumentedTest" time="2.06" >
    <failure>androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'with text: is "21"'
    </failure>
</testcase>

Yaml File (Wokflow)

    - name: Publish Test Report with Insights
      uses: ctrf-io/github-test-reporter@v1
      with:
        report-path: 'sample.json'
        summary-delta-report: true
        insights-report: true
        flaky-report: true
        flaky-rate-report: true
        fail-rate-report: true
        slowest-report: true
        previous-results-report: true
        fetch-previous-results: true
        upload-artifact: true
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      if: always()

Can u please provide the Workflow to report Flaky Test and also please provide the sample xml

[davidgammon-mettle]

It looks like you're running into an issue I've seen before. To get the JUnit to CTRF conversion to work, you need to make sure that flaky test failures are correctly marked in the JUnit reports.

Here’s how you can do that:

1. Add the org.gradle.test-retry plugin to your Gradle build. This will automatically retry any tests that fail.
2. In your build.gradle file, configure the JUnit XML reports to merge the results of reruns by setting mergeReruns = true.

This will ensure that tests that pass after a retry are properly marked as flaky, which should resolve the conversion issue.