Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[20843] Add action to generate flaky test reports #107

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

EduPonz
Copy link

@EduPonz EduPonz commented Jun 12, 2024

Description

This PR adds a flakiness_report action to create flaky tests reports in nightly workflows.
This stems from the original work done by @elianalf in:

TODO:

  • Add Windows support
  • Add macOS support

Contributor Checklist

  • Commit messages follow the company guidelines.
  • Changes are API compatible.
  • New features have been added to the versions.md and README.md files (if applicable).

Reviewer Checklist

  • The title and description correctly express the PR's purpose.
  • The Contributor checklist is correctly filled.

@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch 6 times, most recently from 3369cbb to 976d703 Compare June 13, 2024 10:16
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch from d07a07f to 1685099 Compare June 14, 2024 08:54
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch from 1685099 to 35aaca1 Compare June 17, 2024 12:46
@@ -15,7 +16,8 @@
failures="{$numberOfFailures}"
errors="{$numberOfErrors}"
skipped="{$numberOfSkipped}"
time="{$buildTime}">
time="{$buildTime}"
timestamp="{$input_timestamp}">
<xsl:for-each select="//Site/Testing/Test">
<xsl:variable name="testName" select="translate(Name, '-', '_')"/>
<xsl:variable name="duration" select="Results/NamedMeasurement[@name='Execution Time']/Value"/>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is just a reminder for us: if we will have problems related to storage space of the artifacts, we may need to to create a reduced version of this .xsl file saving only the necessary information related to the test results. A possible solution is the following:

Suggested change
<xsl:variable name="duration" select="Results/NamedMeasurement[@name='Execution Time']/Value"/>
<xsl:variable name="duration" select="Results/NamedMeasurement[@name='Execution Time']/Value"/>
<xsl:variable name="status" select="@Status"/>
<xsl:variable name="output" select="Results/Measurement/Value"/>
<xsl:variable name="className" select="translate(Path, '/.', '.')"/>
<testcase classname="projectroot{$className}"
name="{$testName}"
status="{$status}"
time="{$duration}">
</testcase>
</xsl:for-each>
</testsuite>
</testsuites>
</xsl:template>
</xsl:stylesheet>

resources/junit/junit_utils.py Show resolved Hide resolved
resources/junit/junit_utils.py Outdated Show resolved Hide resolved
resources/flaky/publisher.py Outdated Show resolved Hide resolved
external/upload-artifact/action.yml Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants