-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update plugin: display list of unit tests #886
Comments
Hi @Evangelink sure thanks for info. I'll try with suggested workaround while waiting fro your feedback. |
@Evangelink we also have nunit reports that we pass using /d:sonar.cs.nunit.reportsPaths="Nunit\Result.xml" -> this is xml why the results are same - just static number ? In c++ there is https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Get-test-execution-metrics |
Hi @tijana88, We have decided that we should provide this feature to align with the other plugins. Regarding your last comment I am not 100% sure what you meant. The list will look exactly like for other languages (for example for Java: https://sonarcloud.io/component_measures?id=com.softserve.academy%3ASpaced.Repetition&metric=tests). |
I created a workaround for this. Instead of passing SonarQube a trx directly, I converted to SonarQube's Generic Test Data format [1]. You can then provide the argument and path to the Scanner to the transformed XML, and you'll get the standard level of results. A link to the .XSLT transform is at [2]. [1] https://docs.sonarqube.org/display/SONAR/Generic+Test+Data |
@awesley can you provide a tool name or example how to do the transformation? |
@maulwuff I used Saxonica. Here's a gist of a Groovy script I wrote to call it: https://gist.github.com/awesley/8993eb0a2646203261df4a35f251328d In there you can see the line with Consuming that script from a Jenkins pipeline step:
|
Can you guys tell if this issue is fixed and is available in the latest SonarQube version 7.1 ? |
Hi @uprasenjit, As you can see the ticket is still open which means it wasn't yet fixed. |
Please add this in as a feature! It feels as though you've done all the hard work (i.e. mechanism to import results), but the easy bit of making it clearly visible isn't there! |
+1 here interested in this feature |
Any news? |
An update on this topic - this year we are planning to improve the user experience regarding test code, including showing the unit test results in SQ. At this moment, I don't have a more specific ETA to share. |
The workaround from this community thread
|
Currently, we seem to generate (see UnitTestResultsTest )
However, we don't push them to SQ. More info in #2764 |
I have an update on this issue, after a discussion with Product Management. We are closing this issue as
Our products should be used on a working build. Unit tests / integration tests should be used as functional regression tests, and regressions should not be allowed in the code base. If unit / integration tests are failing, we believe that the build should fail. |
@andrei-epure-sonarsource I don't see how your reason for closing this is related to the original report. The original report of this issue was that if test results were reported in a specific format (NUnit, XUnit, ...) it will show the number of test cases, but not the individual tests. If the same data is reported through generic XML format it will also show the individual tests. I don't think this has anything to do with failing tests. |
thanks for your reply, @pascalberger. To understand better, what is the use case of having the list of executed unit tests in SonarQube? |
@andrei-epure-sonarsource First of all it would not be a new feature, as it is already there, just not working. So having a number of tests shown, but while clicking on it showing an empty list is confusing and inconsistent at best, but I would classify it as a bug, especially since it works for other formats. For the use case of the feature, independent of the format in which the tests were reported: It gives the ability to have SonarQube as central place for everything quality related of a build. Of course I can report test cases somewhere else (e.g. to build server), but for developers this would mean they need to check just another tool: One which shows them test results and another one which shows code coverage. So the reverse questions would be: What is the use case that SonarQube can show code coverage, numbers of tests, and has an UI for listing tests which is empty if tests are reported in a certain format? |
Completely agree. Would b great to have a central location to look up all quality metrics (Static Analysis, Test Results and Code Coverage) rather than having to go lookup in various different tools depending on scenarios and have to aggregate manually. |
@andrei-epure-sonarsource - very disappointed that SonarQube have decided they know better than the community. Not really clear why #2764 is a duplicate of this, I guess you are saying it has the same root cause and is therefore the same? (Still not really a duplicate though, as it is addressing two separate issues). |
Perhaps some of the drivers to pull in reporting on unit test could be SonarQube as a centre for build quality and security. We wanted to use a single system as our current CI (GitHub Actions) doesn't have good functionality to report on Unit Test failures (Azure Pipelines, TeamCity, etc. have much richer functionality in this area). SonarQube (we actually evaluate SonarCloud) would be a prefect solution giving us flexibility to use different CI systems (for different projects) while keeping one system that guards the quality of development. We would love for SonarQube to re-evaluate the position - for the time being we are forced to look at alternate solutions. Thanks, |
This sounds suspiciously a lot like
It's a very poor excuse when there is clear demand, to the point where the community at large is engineering their own workarounds to make it happen. I was also looking to centralize code quality checks, including test execution results, in SonarQube. Because it's just that more convenient than having to keep eyes on multiple tools.
That's Sonar's philosophy. And frankly, it's heavily opinionated and to the point: quite flawed. Operating from the flawed point of perception of a build being an atomic thing that runs through to live code. In reality, even within CI environments a 'build' consists of several steps that each fall into build; test; or deploy phases. It's the deploy phase which shouldn't go ahead when a test fails. So, you gate your deployments -- or rather: your automatic deployments. Even if a test is failing, you may still want to push a deployment ahead manually if it fixes important production issues and only breaks a largely non-consequential unit-test. It is however nonsense to gate your analysis tools. Infact; using the output of your analysis tools to drive the greenlighting of an automatic deployment is a good thing. That's why you have quality gates -- I mean, it's pretty much in the damn name, isn't it? A level of quality you want to uphold and gate actions such as publishing a release behind. For anyone stuck wanting to get decent source-code associations linked to trx reports: |
Hi @rjgotten, https://community.sonarsource.com/ is the correct place to suggest new features, raise traction from community and to get attention of our product managers. Please feel free to describe intended scenarios and use cases there. |
Hi guys. I've created a small tool to convert trx files to Generic Test Data expected by SonaCloud - https://github.com/gmarokov/dotnet-trx2sonar |
@gmarokov Is that solution still valid? I'm trying to use your tool like:
But it does not seem to have any effect, No test results are shown. |
We got green light from Product to fix this problem. |
Currently once I pass path to unit test report to MSBuils sonar scanner (I'm using latest c# plugin) it gets imported in SonarQube but only as static number. If I click on it does not list any file.
Is there a way to display all affected files? Especially important for failed tests.
The text was updated successfully, but these errors were encountered: