From: Peeter Piegaze <61758048+peeter-piegaze-sonarsource@users.noreply.github.com> Date: Wed, 11 May 2022 13:05:50 +0000 (+0200) Subject: MMF-2659 Provide guidance around setting up code coverage for our flagship languages... X-Git-Tag: 9.5.0.56709~145 X-Git-Url: https://source.dussan.org/?a=commitdiff_plain;h=18fd77805e2cca4e948bc08f711042270b6339ad;p=sonarqube.git MMF-2659 Provide guidance around setting up code coverage for our flagship languages in SQ docs --- diff --git a/server/sonar-docs/src/pages/analysis/analysis-parameters.md b/server/sonar-docs/src/pages/analysis/analysis-parameters.md index aeb7710e378..a88ba5da6dd 100644 --- a/server/sonar-docs/src/pages/analysis/analysis-parameters.md +++ b/server/sonar-docs/src/pages/analysis/analysis-parameters.md @@ -15,7 +15,7 @@ For example, if you override the `sonar.exclusions` parameter via command line f Most of the property keys shown in the interface at both global and project levels can also be set as analysis parameters, but the parameters listed below can _only_ be set at analysis time. -For language-specific parameters related to test coverage and execution, see [Test Coverage & Execution](/analysis/coverage/). +For language-specific parameters related to test coverage and execution, see [Test Coverage](/analysis/test-coverage/overview/). For language-specific parameters related to external issue reports, see [External Issues](/analysis/external-issues/). [[info]] diff --git a/server/sonar-docs/src/pages/analysis/coverage.md b/server/sonar-docs/src/pages/analysis/coverage.md deleted file mode 100644 index 46fa787aed7..00000000000 --- a/server/sonar-docs/src/pages/analysis/coverage.md +++ /dev/null @@ -1,236 +0,0 @@ ---- -title: Test Coverage & Execution -url: /analysis/coverage/ ---- - -Test coverage reports and test execution reports are important code quality metrics that you can import into SonarQube. Test coverage reports tell you the percentage of your code that is covered by your test cases. Test execution reports tell you which tests have been run and their results. - -[[info]] -| SonarQube doesn't run your tests or generate reports. To include coverage results in your analysis, you need to set up a third-party coverage tool to generate reports and configure SonarQube to import those reports. - -Below you'll find guidelines and resources as well as well as language- and tool-specific analysis parameters for importing coverage and execution reports. - -## General Guidlines -Before importing test coverage and execution reports, you need to have the appropriate [SonarScanner](/analysis/overview/) configured as part of your build pipeline. Then, complete the following steps: - -1. Set up your coverage tool to run as part of your build pipeline. Your coverage tool should be set up to run *before* the SonarScanner analysis. -1. Configure the coverage tool so that location and format of the output report files matches what the SonarScanner expects. -1. Configure the analysis parameters of the SonarScanner so that it can import the report files. - -Now, on each build of your project, your coverage tool performs its analysis and outputs its results to one or more files (usually one for test coverage and one for test execution). Then, the SonarScanner imports those files and sends the results to SonarQube as part of its analysis process. - -## Supported report formats -SonarQube supports a variety of formats from third-party tools. We also provide a [generic test data format](/analysis/generic-test/) to easily gather and import test coverage and execution information into SonarQube. - -## Language-specific guides -On the [SonarSource Community forum](https://community.sonarsource.com/), you can find guides with instructions on generating reports for the following languages: -* [Apex, C/C++, Objective-C, Go, JS/TS, and Python](https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-apex-c-c-objective-c-go-js-ts-and-python/) -* [Swift](https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-swift/) -* [C#, VB.net](https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-c-vb-net/) -* [JaCoCo](https://community.sonarsource.com/t/coverage-test-data-importing-jacoco-coverage-report-in-xml-format/) - -## Wildcards -Some analysis parameters support the following path wildcards. Properties that support wildcards are noted in the **Description** columns of the analysis parameter tables below: - -Symbol|Meaning ----|--- -`?`|a single character -`*`|any number of characters -`**`|any number of directories - -## Test coverage analysis parameters -Unless otherwise specified, these properties require values that are relative to project root. - -### All Languages - -Analysis parameter|Description -----|---- -`sonar.coverageReportPaths`|Path to coverage report in the [Generic Test Data](/analysis/generic-test/) format. - -### Apex - -Analysis parameter|Description -----|---- -`sonar.apex.coverage.reportPath`|Path to the `test-result-codecoverage.json` report file generated by the [`apex:test:run`](https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_force_apex.htm?search_text=apex%20test#cli_reference_test_run) command of the [Salesforce CLI](https://developer.salesforce.com/tools/sfdxcli). Note, you must have a [Salesforce DX project](https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_workspace_setup.htm) set up and linked to your Org - -### C/C++/Objective-C - -Analysis parameter|Description -----|---- -`sonar.cfamily.gcov.reportsPath`|Path to the directory containing native `*.gcov` reports (not the XML reports generated by gcovr) -`sonar.cfamily.llvm-cov.reportPath`| Path to a llvm-cov report -`sonar.cfamily.vscoveragexml.reportsPath`|Path may be absolute or relative to the solution directory. Path wildcards (see above) are supported. Note that the native `.coverage` report format offered by Visual Studio is not supported. This file format can be converted to the supported XML format with Microsoft's CodeCoverage.exe tool. -`sonar.cfamily.bullseye.reportPath`| Path to the report from Bullseye, version >= 8.9.63 (use [covxml](http://www.bullseye.com/help/ref-covxml.html) tool) - -### C# - -Analysis parameter|Description -----|---- -`sonar.cs.vscoveragexml.reportsPaths`|Path to Visual Studio Code Coverage report. Multiple paths may be comma-delimited, or included via wildcards. See _Notes on importing .NET reports_ below. -`sonar.cs.dotcover.reportsPaths`|Path to dotCover coverage report. See _Notes on importing .NET reports_ below. -`sonar.cs.opencover.reportsPaths`|Path to OpenCover coverage report. See _Notes on importing .NET reports_ below. -`sonar.cs.ncover3.reportsPaths`![](/images/cross.svg)|**Deprecated.** Path to NCover3 coverage report. See _Notes on importing .NET reports_ below. - -### Flex - -Analysis parameter|Description -----|---- -`sonar.flex.cobertura.reportPaths`|Path to the Cobertura XML reports. Multiple paths may be comma-delimited. May be absolute or relative to the project base directory. - -### Go - -Analysis parameter|Description -----|---- -`sonar.go.coverage.reportPaths`|Comma-delimited list of paths to coverage report files. Path wildcards are supported (see above) since SonarGo 1.1. - -### Java, Kotlin, Scala, and JVM - -Analysis parameter|Description -----|---- -`sonar.coverage.jacoco.xmlReportPaths`|Path to JaCoCo XML coverage reports. Path wildcards are supported (see above). - -### Javascript and Typescript - -Analysis parameter|Description -----|---- -`sonar.javascript.lcov.reportPaths`|Comma-delimited list of paths to LCOV coverage report files. Paths may be absolute or relative to project root. - -### PHP - -Analysis parameter|Description -----|---- -`sonar.php.coverage.reportPaths`|Comma-delimited list of paths to Clover XML-format coverage report files. Paths may be absolute or relative to project root. - -### Python - -Analysis parameter|Description -----|---- -`sonar.python.coverage.reportPaths`|Comma-delimited list of paths to coverage reports in the Cobertura XML format. Path wildcards are supported (see above). Leave unset to use the default (`coverage-reports/*coverage-*.xml`). - -### Ruby - -Analysis parameter|Description -----|---- -`sonar.ruby.coverage.reportPaths`|Comma-delimited list of paths to SimpleCov report files generated with the [JSON formatter](https://github.com/simplecov-ruby/simplecov#json-formatter) (availaible from SimpleCov 0.20). For SimpleCov versions < 0.18, you can provide `.resultset.json` report files (not recommended). Paths may be absolute or relative to project-root. - -### Scala - -Analysis parameter|Description -----|---- -`sonar.scala.coverage.reportPaths`|Comma-separated list of paths to `scoverage.xml` report files generaged by Scoverage. - -### Swift, Xcode 9.3+ - -Analysis parameter|Description -----|---- - |You can use the [xccov-to-sonarqube-generic.sh](https://github.com/SonarSource/sonar-scanning-examples/blob/master/swift-coverage/swift-coverage-example/xccov-to-sonarqube-generic.sh) script from the [sonar-scanning-examples/swift-coverage](https://github.com/SonarSource/sonar-scanning-examples/tree/master/swift-coverage) project convert output from Xcode 9.3's xccov tool to the [Generic Test Data](/analysis/generic-test/) format. - -### Swift, Xcode 7–9.2 - -Analysis parameter|Description -----|---- -`sonar.swift.coverage.reportPath`|Path to the report generated by `llvm-cov show`. Path may be absolute or relative to project root. - -### VB.NET - -Analysis parameter|Description -----|---- -`sonar.vbnet.vscoveragexml.reportsPaths`|Path to Visual Studio Code Coverage report. Multiple paths may be comma-delimited, or included via wildcards. See _Notes on importing .NET reports_ below. -`sonar.vbnet.dotcover.reportsPaths`|Path to dotCover coverage report. See _Notes on importing .NET reports_ below. -`sonar.vbnet.opencover.reportsPaths`|Path to OpenCover coverage report. See _Notes on importing .NET reports_ below. -`sonar.vbnet.ncover3.reportsPaths`![](/images/cross.svg)|**Deprecated.** Path to NCover3 coverage report. See _Notes on importing .NET reports_ below. - - -## Test execution analysis parameters -Unless otherwise specified, these properties require values that are relative to project root. - -### All Languages - -Analysis parameter|Description -----|---- -`sonar.testExecutionReportPaths`|Comma-delimited list of paths to execution reports in the [Generic Execution Data](/analysis/generic-test/) format. The paths must be relative to the project's base directory (root module). - -### C/C++/Objective-C - -Analysis parameter|Description -----|---- -`sonar.cfamily.cppunit.reportsPath`|Path to the directory holding the [CPPUnit](http://sourceforge.net/projects/cppunit/) reports. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. - -### C# - -Analysis parameter|Description -----|---- -`sonar.cs.vstest.reportsPaths`|Paths to VSTest reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. -`sonar.cs.nunit.reportsPaths`|Paths to NUnit execution reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. -`sonar.cs.xunit.reportsPaths`|Paths to xUnit execution reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. - -### Go - -Analysis parameter|Description -----|---- -`sonar.go.tests.reportPaths`|Comma-delimited list of paths to unit test report files. Paths may be absolute or relative to project root. - -### Java, Kotlin - -Analysis parameter|Description -----|---- -`sonar.junit.reportPaths`|Comma-delimited list of paths to Surefire XML-format reports. - -### Javascript and Typescript - -Analysis parameter|Description -----|---- - |You can use [jest-sonar-reporter](https://www.npmjs.com/package/jest-sonar-reporter) or [karma-sonarqube-unit-reporter](https://github.com/tornaia/karma-sonarqube-unit-reporter) to create reports in the [Generic Execution Data](/analysis/generic-test/) format. Both packages are available on npm. - -### PHP - -Analysis parameter|Description -----|---- -`sonar.php.tests.reportPath`|Path to the PHPUnit unit test execution report file. Path may be absolute or relative to project root. - -### Python - -Analysis parameter|Description -----|---- -`sonar.python.xunit.reportPath`|Path to unit test execution report. Leave unset to use the default (`xunit-reports/xunit-result-*.xml`). Path wildcards (see above) are supported. If any paths in the report are invalid, set `sonar.python.xunit.skipDetails=true` to collect only project-level details. - -### VB.NET - -Analysis parameter|Description -----|---- -`sonar.vbnet.vstest.reportsPaths`|Paths to VSTest execution reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. -`sonar.vbnet.nunit.reportsPaths`|Paths to NUnit execution reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. -`sonar.vbnet.xunit.reportsPaths`|Paths to xUnit execution reports. Multiple paths may be comma-delimited, or included via wildcards. Note that while measures such as the number of tests are displayed at project level, no drilldown is available. - -## Importing .NET reports -To import .NET reports, the report generation process must be executed after the begin step and before the end MSBuild command. The following steps detail importing .NET reports: - -1. Run the SonarScanner.MSBuild.exe `begin` command, specifying the absolute path where the reports _will be_ available using the `/d:propertyKey="path"` syntax ("propertyKey" depends on the tool) -1. Build your project using MSBuild -1. Run your test tool, instructing it to produce a report at the same location specified earlier to the MSBuild SonarQube Runner ([How to generate reports with different tools](https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-c-vb-net/9871)) -1. Run the SonarScanner.MSBuild.exe `end` command - -For more information, see the [Generate Reports for C#, VB.net Community Post](https://community.sonarsource.com/t/coverage-test-data-generate-reports-for-c-vb-net/9871). - -## Troubleshooting - -### Why doesn't coverage in SonarQube match my coverage tool? -Code coverage metrics may look different between SonarQube and the tool you've used to gather them. Often, this is because the two aren't using the same metrics. - -External tools often use line coverage, calculated by dividing covered lines (lines to cover - uncovered lines) by the total number of executable lines (lines to cover). - -SonarQube uses the covered lines from the imported coverage report and the executable lines (or lines to cover) to calculate its coverage metrics. SonarQube computes coverage as follows: - -``` -Coverage = (CT + CF + LC)/(2*B + EL) -where -CT = conditions that have been evaluated to ‘true’ at least once -CF = conditions that have been evaluated to ‘false’ at least once -LC = covered lines = lines_to_cover - uncovered_lines -B = total number of conditions -EL = total number of executable lines (lines_to_cover) -``` - -SonarQube might also compute line coverage differently from your external tool. This is because the lines to cover may not be the same according to SonarQube and to the tool. - -For more information on what SonarQube considers a line of code, see the **Tests** section on the [metric-definitions page](/user-guide/metric-definitions/). \ No newline at end of file diff --git a/server/sonar-docs/src/pages/analysis/generic-test.md b/server/sonar-docs/src/pages/analysis/generic-test.md deleted file mode 100644 index 525c2cc5c3a..00000000000 --- a/server/sonar-docs/src/pages/analysis/generic-test.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: Generic Test Data -url: /analysis/generic-test/ ---- - -Out of the box, SonarQube supports generic formats for test coverage and test execution import. If your coverage engines' native output formats aren't supported by SonarQube, simply covert them to these formats: - -## Generic Coverage -Report paths should be passed in a comma-delimited list to: - - * `sonar.coverageReportPaths` - -The supported format is described by the `sonar-generic-coverage.xsd`: - - - - - - - - - - - - - - - - - - - - - - - - - - -and looks like this: - - - - - - - - - - - -The root node should be named `coverage`. Its version attribute should be set to `1`. - -Insert a `file` element for each file which can be covered by tests. Its `path` attribute can be either absolute or relative to the root of the module. -Inside a `file` element, insert a `lineToCover` for each line which can be covered by unit tests. It can have the following attributes: -* `lineNumber` (mandatory) -* `covered` (mandatory): boolean value indicating whether tests actually hit that line -* `branchesToCover` (optional): number of branches which can be covered -* `coveredBranches` (optional): number of branches which are actually covered by tests - -## Generic Execution -Report paths should be passed in a comma-delimited list to: - -* `sonar.testExecutionReportPaths` - -The supported format looks like this: - - - - - - other - - - stacktrace - - - stacktrace - - - - -The root node should be named `testExecutions`. Its version attribute should be set to `1`. - -Insert a `file` element for each test file. Its `path` attribute can be either absolute or relative to the project's base directory (root module). - -**Note** unlike for coverage reports, the files present in the report must be test file names, not source code files covered by tests. - -Inside a `file` element, insert a `testCase` for each test run by unit tests. It can have the following attributes/children: - -* `testCase` (mandatory) - * `name` (mandatory): name of the test case - * `duration` (mandatory): long value in milliseconds - - * `failure|error|skipped` (optional): if the test is not OK, report the cause with a message and a long description - * `message` (mandatory): short message describing the cause - * `stacktrace` (optional): long message containing details about `failure|error|skipped` status diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/c-family-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/c-family-test-coverage.md new file mode 100644 index 00000000000..a7f9ca70836 --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/c-family-test-coverage.md @@ -0,0 +1,76 @@ +--- +title: C/C++/Objective-C Test Coverage +url: /analysis/test-coverage/c-family-test-coverage/ +--- + +SonarQube supports the reporting of test coverage information as part of the analysis of your C/C++/Objective-C project. + +However, SonarQube does not generate the coverage report itself. +Instead, you must use a third-party tool to produce the coverage report(s) as part of your build process (before the analysis step). +Later, during the analysis steps, the scanner picks up this report and sends it to SonarQube. +The coverage is then displayed on your SonarQube project dashboard along with the other analysis metrics. + + +## Adjust your setup + +To enable coverage you need to: + +* Adjust your build process so that the coverage tool generates the report(s). + This is done just after your unit tests as part of the clean build required to run analysis. +* Make sure that the coverage tool writes its report file to a defined path in the build environment. +* Configure the scanning step of your build so that the scanner picks up the report file from that defined path. + + +## Add coverage to your build process + +For C/C++/Objective-C projects SonarCloud supports a number of coverage tools. Each has an associated analysis parameter that must be set to the location of the coverage report that is produced by the tool. The parameters are: + +* `sonar.cfamily.llvm-cov.reportPath` +* `sonar.cfamily.vscoveragexml.reportsPath` +* `sonar.cfamily.gcov.reportsPath` +* `sonar.cfamily.bullseye.reportPath` +* `sonar.coverageReportPaths` + +Assuming that you have already set up your project, you will have seen the example projects (_without coverage_) referenced in the in-product tutorials: [sonarsource-cfamily-examples](https://github.com/orgs/sonarsource-cfamily-examples/). + +To help you add coverage to your project, we also provide, in the same GitHub organization, a few example repositories _with coverage_. + +Unlike the examples without coverage, not every possible combination of tooling and platform are represented and the examples given are only for SonarCloud. However, adapring them to your needs shuld not be difficult: + +* [windows-msbuild-vscoverage-gh-actions-sc](https://github.com/sonarsource-cfamily-examples/windows-msbuild-vscoverage-gh-actions-sc) + +* [windows-msbuild-vscoverage-azure-sc](https://github.com/sonarsource-cfamily-examples/windows-msbuild-vscoverage-azure-sc) + +* [macos-xcode-coverage-gh-actions-sc](https://github.com/sonarsource-cfamily-examples/macos-xcode-coverage-gh-actions-sc) + +* [linux-cmake-llvm-cov-gh-actions-sc](https://github.com/sonarsource-cfamily-examples/linux-cmake-llvm-cov-gh-actions-sc) + +* [linux-cmake-gcovr-gh-actions-sc](https://github.com/sonarsource-cfamily-examples/linux-cmake-gcovr-gh-actions-sc) + +* [linux-autotools-gcov-travis-sc](https://github.com/sonarsource-cfamily-examples/linux-autotools-gcov-travis-sc) + +These examples include the major free-to-use coverage tools for C/C++/Objective-C (VS Coverage, XCode Coverage, LLVM-COV, GCOVR, and GCOV). For information on the populat commercial Bullseye product, see https://www.bullseye.com/. + + +## Coverage parameters can be set in multiple places + +As with other analysis parameters, the coverage-related parameters for C/C++/Objective-C projects can be set in multiple places: + +* On the command line of the scanner invocation using the `-D` or `--define` switch. This is what is done in the examples above, inside the `build.yml` files of each example. + +* In the `sonar-project.properties` file. + +* In the SonarQube interface under + + **_Your Project_ > Project Settings > General Settings > Languages > C/C++/Objective-C > Coverage** + + for project-level settings, and + + **Administration > Configuration > General Settings > Languages > C/C++/Objective-C > Coverage** + + for global settings (applying to all projects). + + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/dotnet-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/dotnet-test-coverage.md new file mode 100644 index 00000000000..e5afe20b00f --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/dotnet-test-coverage.md @@ -0,0 +1,237 @@ +--- +title: .NET Test Coverage +url: /analysis/test-coverage/dotnet-test-coverage/ +--- + +SonarQube supports the reporting of test coverage information as part of the analysis of your .NET project. + +However, SonarQube does not compile the coverage report itself. +Instead, you must set up a third-party tool to produce the coverage report as part of your build process. +During the analysis, the scanner (in this case, the [SonarScanner for .NET](/analysis/scan/sonarscanner-for-msbuild/)) picks up this report and sends it to SonarQube. +The coverage is then displayed on your SonarQube project dashboard, along with the other analysis metrics. + +SonarQube supports the following .NET test coverage tools: + +* Visual Studio Code Coverage +* dotnet-coverage Code Coverage +* dotCover +* OpenCover +* Coverlet + +Additionally, a generic coverage format is also supported if you wish to use an unsupported tool +(though you will have to convert its output to the generic format yourself). +In this section, we discuss the directly supported tools. +For information on the generic format, see [Generic Test Data](/analysis/test-coverage/generic-test/). + + +## Adding coverage to your build process + +The .NET scanner comes in four variants depending on which version of .NET and which CI you are using +(_.NET Framework_, _.NET Core_, _.NET tool_ and _SonarScanner for Azure DevOps_). +The setup is slightly different for each variant +(see the [SonarScanner for .NET](/analysis/scan/sonarscanner-for-msbuild/) +and [SonarScanner for Azure DevOps](/analysis/scan/sonarscanner-for-azure-devops/) +sections for details), +but the essential steps are the same. + +The analysis is always split into two parts in your build process; +the begin step and the end step. +In between, you perform the actual build and your tests. +To enable coverage reporting, you need to make the following changes: + +* In the scanner begin step, add the appropriate parameter to specify the location of the coverage report file that will be produced. +* Just after the build step but before the scanner end step, ensure that your test step produces the coverage report file. + + +## Examples using the .NET tool scanner variant + +The SonarScanner for .NET comes in four major variants: .NET Framework 4.6, .NET Core, .NET tool, and Azure Pipelines extension. + + +### Visual Studio Code Coverage + +We only recommend the use of this tool when the build agent has Visual Studio Enterprise installed or when you are using an Azure DevOps Windows image for your build. +In these cases, the .NET Framework 4.6 scanner will automatically find the coverage output generated by the `--collect "Code Coverage"` parameter without the need for an explicit report path setting. +It will also automatically convert the generated report to XML. +No further configuration is required. +Here is an example: + +``` +SonarScanner.MSBuild.exe begin /k:"" /d:sonar.login="" +dotnet build --no-incremental +dotnet test --collect "Code Coverage" +SonarScanner.MSBuild.exe end /d:sonar.login="" +``` + + +### dotnet-coverage + +This is a modern alternative to the Visual Studio Code Coverage provided by Microsoft (see above) that outputs results in the same format, +is cross-platform and not dependent on having Visual Studio installed. +It requires .NET Core 3.1 or later. + +To use [dotnet-coverage](https://docs.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage), +you must install it as a global dotnet tool: + +``` +dotnet tool install --global dotnet-coverage +``` + +Using this tool, your build script would look like something like this: + +``` +dotnet sonarscanner begin /k:"" + /d:sonar.login="" + /d:sonar.cs.vscoveragexml.reportsPaths=coverage.xml +dotnet build --no-incremental +dotnet-coverage collect 'dotnet test' -f xml -o 'coverage.xml' +dotnet sonarscanner end /d:sonar.login="" +``` + +Note that we specify the path to the reports using `sonar.cs.vscoveragexml.reportsPaths` +because this tool’s output format is the same as the Visual Studio Code Coverage tool. +We use the `-f xml` parameter to specify that the output format is in XML. + + +### **dotCover** + +To use [dotCover](https://www.jetbrains.com/help/dotcover/dotCover__Coverage_Analysis_on_Third-Party_Server.html) +you must install it as a global dotnet tool: + +``` +dotnet tool install --global JetBrains.dotCover.GlobalTool +``` + +Using this tool, your build script would look like something like this: + +``` +dotnet sonarscanner begin /k:"" + /d:sonar.login="" + /d:sonar.cs.dotcover.reportsPaths=dotCover.Output.html +dotnet build –no-incremental +dotnet dotcover test --dcReportType=HTML +dotnet sonarscanner end /d:sonar.login="" +``` + +Note that we specify the path to the reports using `sonar.cs.dotcover.reportsPaths` because we are using dotCover. + + +### OpenCover + +To use [OpenCover](https://github.com/OpenCover/opencover/wiki/Usage) you must download it from [here](https://github.com/OpenCover/opencover/releases) and unzip it in an appropriate directory, for example: `C:\tools\opencover` + +Using this tool, your build script would look like something like this: + +``` +dotnet sonarscanner begin /k:"" + /d:sonar.login="" + /d:sonar.cs.opencover.reportsPaths=coverage.xml +dotnet build --no-incremental +& C:\tools\opencover\OpenCover.Console.exe -target:"dotnet.exe" + -targetargs:"test --no-build" + -returntargetcode + -output:coverage.xml + -register:user +dotnet sonarscanner end /d:sonar.login="" +``` + +Note that we specify the path to the reports using `sonar.cs.opencover.reportsPaths` because we are using OpenCover. + + +### **Coverlet** + +To use Coverlet, you must install it as a global dotnet tool: + +``` +dotnet tool install --global coverlet.console +``` + +You also have to install [the coverlet collector NuGet package](https://www.nuget.org/packages/coverlet.collector/) on your test project. + +Using this tool, your build script would look like something like this: + +``` +dotnet sonarscanner begin /k:"" + /d:sonar.login="" + /d:sonar.cs.opencover.reportsPaths=coverage.xml +dotnet build --no-incremental +coverlet .\CovExample.Tests\bin\Debug\net6.0\CovExample.Tests.dll + --target "dotnet" + --targetargs "test --no-build" + -f=opencover + -o="coverage.xml" +dotnet sonarscanner end /d:sonar.login="" +``` + +Note that we specify the path to the reports in `sonar.cs.opencover.reportsPaths` because Coverlet produces output in the same format as OpenCover. + + +## .NET Framework and .NET Core scanners + +In most of the examples above, we use the .NET tool scanner variant. +If you use the .NET Framework or .NET Core scanner, the commands will be a bit different but the pattern will be the same. +See [SonarScanner for .NET](/analysis/scan/sonarscanner-for-msbuild/) for details. + + +## SonarScanner for Azure DevOps + +Using the SonarScanner for Azure DevOps and Visual Studio Code Coverage with a C# project, your `azure-pipelines.yml` would look something like the example below. + +Note that with the SonarScanner for Azure DevOps extension, the scanner `begin` step is handled by the `SonarQubePrepare` task and the scanner `end` step is handled by the `SonarQubeAnalyze` task. + +Also note that because our build is running on Windows (we specify `vmImage: windows-latest`), we do not need to explicitly specify the path to the coverage report (there is no `sonar.cs.vscoveragexml.reportsPaths=coverage.xml`) nor do you need to run `codecoverage.exe` to convert the report to XML. + +``` +azure-pipelines.yml +trigger: +- master + +variables: +- name: system.debug + value: true + +pool: + vmImage: windows-latest + +steps: +- task: DotNetCoreCLI@2 + inputs: + command: 'restore' + projects: 'my-project.sln' + feedsToUse: 'select' + +- task: SonarQubePrepare@1 + inputs: + SonarQube: 'SonarQube' + scannerMode: 'MSBuild' + projectKey: 'my-project-key' + projectName: 'my-project' + +- task: DotNetCoreCLI@2 + inputs: + command: 'build' + projects: 'my-project.sln' + +- task: DotNetCoreCLI@2 + inputs: + command: 'test' + projects: 'tests/**/*.csproj' + arguments: '--collect "Code Coverage"' # This is all you need to add! + +- task: SonarQubeAnalyze@1 +``` + +## **VB.NET** + +The examples above are all for C# projects. For VB.NET projects the setup is identical except that you would use these parameters: + +* `sonar.vbnet.vscoveragexml.reportsPaths` for Visual Studio Code Coverage +* `sonar.vbnet.dotcover.reportsPaths` for dotCover +* `sonar.vbnet.opencover.reportsPaths` for OpenCover or Coverlet + +The parameter `sonar.cs.ncover3.reportsPaths` was formerly used for or NCover3. +This parameter has been deprecated. + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/generic-test.md b/server/sonar-docs/src/pages/analysis/test-coverage/generic-test.md new file mode 100644 index 00000000000..ae773690327 --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/generic-test.md @@ -0,0 +1,115 @@ +--- +title: Generic Test Data +url: /analysis/test-coverage/generic-test/ +--- + +SonarQube supports generic formats for test coverage and test execution import. +If your coverage engines' native output formats aren't supported by SonarQube, simply covert them to these formats: + + +## Generic Coverage + +Report paths should be passed in a comma-delimited list to: + +`sonar.coverageReportPaths` + +The supported format is described by the `sonar-generic-coverage.xsd`: + +``` + + + + + + + + + + + + + + + + + + + + + + + + +``` + +and looks like this: + +``` + + + + + + + + + +``` + +The root node should be named `coverage`. +Its version attribute should be set to `1`. + +Insert a `file` element for each file which can be covered by tests. +Its `path` attribute can be either absolute or relative to the root of the module. + +Inside a `file` element, insert a `lineToCover` for each line which can be covered by unit tests. +It can have the following attributes: + +* `lineNumber` (mandatory) +* `covered` (mandatory): boolean value indicating whether tests actually hit that line +* `branchesToCover` (optional): number of branches which can be covered +* `coveredBranches` (optional): number of branches which are actually covered by tests + + +## Generic Execution + +Report paths should be passed in a comma-delimited list to: + +`sonar.testExecutionReportPaths` + +The supported format looks like this: + +``` + + + + + other + + + stacktrace + + + stacktrace + + + +``` + +The root node should be named `testExecutions`. +Its version attribute should be set to `1`. + +Insert a `file` element for each test file. +Its `path` attribute can be either absolute or relative to the project's base directory (root module). + +Unlike for coverage reports, the files present in the report must be test file names, not source code files covered by tests. + +Inside a `file` element, insert a `testCase` for each test run by unit tests. +It can have the following attributes/children: + +* `testCase` (mandatory) + * `name` (mandatory): name of the test case + * `duration` (mandatory): long value in milliseconds + * `failure|error|skipped` (optional): if the test is not OK, report the cause with a message and a long description + * `message` (mandatory): short message describing the cause + * `stacktrace` (optional): long message containing details about `failure|error|skipped` status diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/java-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/java-test-coverage.md new file mode 100644 index 00000000000..c87d8c3d186 --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/java-test-coverage.md @@ -0,0 +1,207 @@ +--- +title: Java Test Coverage +url: /analysis/test-coverage/java-test-coverage/ +--- + +SonarQube supports the reporting of test coverage as part of the analysis of your Java project. + +However, SonarQube does not compile the coverage report itself. +Instead, you must set up a third-party tool to produce the coverage report as part of your build process. +The SonarScanner will pick up this report and send it to SonarQube where it will be displayed on your project dashboard +along with the other analysis metrics. + +For Java projects, SonarQube directly supports the JaCoCo coverage tool +(see [Generic Test Data](/analysis/test-coverage/generic-test/) for information on integrating other coverage tools). + + +## Adjust your setup + +To enable coverage you need to: + +* Adjust your build process so that JaCoCo report generation step runs _before_ the SonarScanner step. +* Make sure that JacCoCo writes its report file to a defined path in the build environment. +* Configure the scanning step of your build so that the SonarScanner picks up the report file from that defined path. + + +## Add coverage in a single-module Maven project + +To add coverage to your Maven project you need to use the [`jacoco-maven-plugin`](https://mvnrepository.com/artifact/org.jacoco/jacoco-maven-plugin) and its `report` goal to create a code coverage report. + +Typically, you would create a specific Maven profile for executing the unit tests with instrumentation and producing the coverage report only on demand. + +In the most basic case, we will need to execute two goals: `jacoco:prepare-agent`, which allows coverage info to be collected during unit tests execution, and `jacoco:report`, which uses data collected during unit test execution to generate a report. +By default, the tool generates XML, HTML, and CSV versions of the report. +Here, we explicitly specify XML, since that is the only one we need for SonarQube. +The `` section of your `pom.xml` should look something like this: + +``` + + coverage + + + + org.jacoco + jacoco-maven-plugin + 0.8.7 + + + prepare-agent + + prepare-agent + + + + report + + report + + + + XML + + + + + + ... + + + +``` + +By default the generated report will be saved under `target/site/jacoco/jacoco.xml`. +This location will be checked automatically by the scanner, so no further configuration is required. +Just launch: + +``` +mvn sonar:sonar -Pcoverage +``` + +as usual and the report will be picked up. + +If you need to change the directory where the report is generated, you can set the property either on the command line using Maven’s `-D` switch: + +``` +mvn -Dsonar.coverage.jacoco.xmlReportPaths= + ../app-it/target/site/jacoco-aggregate/jacoco.xml + sonar:sonar -Pcoverage +``` + +or in your `pom.xml`: + +``` + + + ../app-it/target/site/jacoco-aggregate/jacoco.xml + + +``` + +Wildcards and a comma-delimited list of paths are supported. +See [Coverage Analysis Parameters](/analysis/test-coverage/test-coverage-parameters/) for details. +The path can be either absolute or relative to the project root. + + +## **Add coverage in a multi-module Maven project** + +For multi-module Maven projects, you configure the `jacoco-maven-plugin` in a profile in the parent pom just as in the single module case, above. By default, a separate coverage report will be generated for each module. + +If you want to aggregate all the module-specific reports into one project-level report, the easiest solution is to create a special Maven module (alongside the ones you already have), that contains nothing except a `pom.xml` that uses the `report-aggregate` goal. Here is an example: + +``` + + my-project-report-aggregate + My Project + Aggregate Coverage Report + + + ${project.groupId} + my-module-1 + ${project.version} + + + ${project.groupId} + my-module-2 + ${project.version} + + + + + + org.jacoco + jacoco-maven-plugin + + + report-aggregate + verify + + report-aggregate + + + + + + + +``` + +When you invoke `maven clean verify` in the `report-aggregate-module` directory the aggregated report will be generated and placed inside that directory at the standard location `target/site/jacoco-aggregate/jacoco.xml`. +Then, in the top level `pom.xml` you set `sonar.coverage.jacoco.xmlReportPaths` to this location: + +``` +/ + + ${project.basedir}/report-aggregate/target/site/ + jacoco-aggregate/jacoco.xml + + +``` + +Wildcards and a comma-delimited list of paths are supported. +See [Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/) for details. + + +## Add coverage in a Gradle project + +To set up code coverage for your Gradle files, you just need to apply the JaCoCo plugin together with the SonarScanner for Gradle to the `build.gradle` file of your project as the JaCoCo is already integrated into the default gradle distribution: + +``` +plugins { + id "jacoco" + id "org.sonarqube" version "3.3" +} + +jacocoTestReport { + reports { + xml.enabled true + } +} +``` + +Your report will be automatically saved in the `build/reports/jacoco` directory. +The SonarQube plugin automatically detects this location so no further configuration is required. +To import coverage, launch: + +``` + gradle test jacocoTestReport sonarqube +``` + +For more details, see the [Gradle JaCoCo Plugin documentation](https://docs.gradle.org/current/userguide/jacoco_plugin.html) and + + +## Coverage parameter can also be set in the UI + +The `sonar.coverage.jacoco.xmlReportPaths` parameter can also be set in the SonarQube interface under + + **_Your Project_ > Project Settings > General Settings > JaCoCo** + + for project-level settings, and + + **Administration > Configuration > General Settings > JaCoCo** + + for global settings (applying to all projects). + + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/javascript-typescript-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/javascript-typescript-test-coverage.md new file mode 100644 index 00000000000..4024800383e --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/javascript-typescript-test-coverage.md @@ -0,0 +1,126 @@ +--- +title: Javascript/Typescript Test Coverage +url: /analysis/test-coverage/javascript-typescript-test-coverage/ +--- + +SonarQube supports the reporting of test coverage information as part of the analysis of your JS/TS project. + +However, SonarQube does not compile the coverage report itself. +Instead, you must set up a third-party tool to produce the coverage report as part of your build process (before the analysis step). +Later, during the analysis steps, the scanner picks up this report and sends it to SonarQube. +The coverage is then displayed on your SonarQube project dashboard, along with the other analysis metrics. + +For JS/TS projects, SonarQube directly supports all coverage tools that produce reports in the LCOV format. +Additionally, a generic coverage format is also supported if you wish to use an unsupported tool (though you will have to convert its output to the generic format yourself). + +In this section, we discuss the directly supported JS/TS LCOV coverage feature. +For information on the generic format, see Generic Test Data. + + +## Adjusting your setup + +To enable coverage you need to: + +* Adjust your build process so that the coverage tool runs before the scanner step. +* Make sure that the coverage tool writes its report file to a defined path in the build environment. +* Configure the scanning step of your build so that the scanner picks up the report file from that defined path. + + +## Adding coverage to your build process + +The details of setting up coverage within your build process depend on which tools you are using. + +The following illustrates how to do this for a JS/TS project that uses Yarn and Jest in the GitHub Actions CI. +Simply add the following to your build.yml file: + +``` +- name: Install dependencies + run: yarn +- name: Test and coverage + run: yarn jest --coverage +``` + +The resulting file should look something like this: + +``` +name: Build +on: + push: + branches: + - master + pull_request: + types: [opened, synchronize, reopened] +jobs: + sonarqube: + name: sonarqube + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + fetch-depth: 0 + - name: Install dependencies + run: yarn + - name: Test and coverage + run: yarn jest --coverage + - name: SonarQube Scan + uses: SonarSource/sonarqube-scan-action@master + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} +``` + +First, you install all your project dependencies and then invoke jest with the `--coverage` option to run your tests and write out the coverage data to a file. + +If, as here, you do not specify an output file, the default `./coverage/lcov.info` is used. + +If you are using a different package manager or a different testing tool, these details will be different. + +The essential requirements are that the tool produces its report in the LCOV format and writes it to a place from which the scanner can then pick it up. + + +## Adding the coverage analysis parameter + +The next step is to add `sonar.javascript.lcov.reportPaths` to your analysis parameters. +This parameter must be set to the path of the report file produced by your coverage tool. +The path can be either absolute or relative to the project root. +In this example, that path is set to the default produced by Jest: `./coverage/lcov.info`. +It is set in the `sonar-project.properties` file, located in the project root: + +``` +sonar.projectKey= +... +sonar.javascript.lcov.reportPaths=./coverage/lcov.info +``` + + +## Coverage parameters can be set in multiple places + +As with other analysis parameters, the coverage-related parameters for JavaScript/TypeScript projects (`sonar.javascript.lcov.reportPaths`) can be set in multiple places: + +* In the `sonar-project.properties` file, as mentioned above. + +* On the command line of the scanner invocation using the `-D` or `--define` switch, for example: + + `sonar-scanner -Dsonar.javascript.lcov.reportPaths=./coverage/lcov.info` + +* In the SonarQube interface under + + **_Your Project_ > Project Settings > General Settings > Languages > JavaScript/TypeScript > Tests and Coverage** + + for project-level settings, and + + **Administration > Configuration > General Settings > Languages > JavaScript/TypeScript > Tests and Coverage** + + for global settings (applying to all projects). + + +## Same parameter for JavaScript and TypeScript + +The parameter `sonar.typescript.lcov.reportPaths` was formerly used for typescript coverage. +This parameter has been deprecated. + +The parameter `sonar.javascript.lcov.reportPaths` is now used for both JavaScript and TypeScript. + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/overview.md b/server/sonar-docs/src/pages/analysis/test-coverage/overview.md new file mode 100644 index 00000000000..03968a06be1 --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/overview.md @@ -0,0 +1,64 @@ +--- +title: Overview +url: /analysis/test-coverage/overview/ +--- + +Test coverage reports and test execution reports are important metrics in assessing the quality of your code. +Test coverage reports tell you what percentage of your code is covered by your test cases. +Test execution reports tell you which tests have been run and their results. + +SonarQube itself does not calculate coverage. +To include coverage results in your analysis, you must set up a third-party coverage tool and configure SonarQube to import the results produced by that tool. + +Below, you'll find guidelines and resources, as well as language- and tool-specific analysis parameters. + + +## General guidelines + +Before importing test coverage, you need to have the appropriate SonarScanner configured to perform code analysis as part of your build pipeline. + +To enable coverage reporting, you must then do the following: + +1. Set up your coverage tool to run as part of your build pipeline. + Your coverage tool should be set up to run _before_ the SonarScanner analysis. +2. Configure the coverage tool so that the location and format of the output report files match what the SonarScanner expects. +3. Configure the analysis parameters of the SonarScanner so that it can import the report files. + +Now, on each build of your project, your coverage tool should perform its analysis and output its results to one or more files (usually one for test coverage and one for test execution). +Then, the SonarScanner, as part of its analysis process, will import those files and send the results to SonarQube. + + +## Coverage support + +SonarQube directly supports the import of coverage data in formats native to a variety of tools for a variety of languages. It also supports the import of a [generic format](/analysis/test-coverage/generic-test/) that can be used as a target for custom conversion of reports from tools that are not directly supported. + + +### Detailed guides + +Detailed guides for the following languages are provided in this section: + +* [Java Test Coverage](/analysis/test-coverage/java-test-coverage/) +* [JavaScript/TypeScript Test Coverage](/analysis/test-coverage/javascript-typescript-test-coverage/) +* [.NET Test Coverage](/analysis/test-coverage/dotnet-test-coverage/) +* [Python Test Coverage](/analysis/test-coverage/python-test-coverage/) +* [PHP Test Coverage](/analysis/test-coverage/php-test-coverage/) +* [C/C++/Objective-C Test Coverage](/analysis/test-coverage/c-family-test-coverage/) + + +### Generic format + +See [Generic Test Data](/analysis/test-coverage/generic-test/) for information on how the generic format works. + + +### Test coverage parameters + +See [Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/) for a reference on all coverage-related analysis parameters. + + +## Test execution reports + +This section is about _test coverage reports_, that is, reports that describe the percentage of your code that is tested by your test suite during a build. + +Test execution reports are a separate feature. +These describe which tests within your tests suite are executed during a build. +For details, see [Test Execution Parameters](/analysis/test-coverage/test-execution-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/php-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/php-test-coverage.md new file mode 100644 index 00000000000..5f370fce3ca --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/php-test-coverage.md @@ -0,0 +1,130 @@ +--- +title: PHP Test Coverage +url: /analysis/test-coverage/php-test-coverage/ +--- + +SonarQube supports the reporting of test coverage information as part of the analysis of your PHP project. + +However, SonarQube does not compile the coverage report itself. +Instead, you must use a third-party tool to produce the coverage report as part of your build process (before the analysis step). +Later, during the analysis step, the scanner picks up this report and sends it to SonarQube. +The coverage is then displayed on your SonarQube project dashboard along with the other analysis metrics. + +For PHP projects, we recommend PHPUnit for testing and coverage reporting. + + +## Adjust your setup + +To enable coverage you need to: + +* Adjust your build process so that the coverage tool runs _before_ the scanner report generation step runs. +* Make sure that the coverage tool writes its report file to a defined path in the build environment. +* Configure the scanning step of your build so that the scanner picks up the report file from that defined path. + + +## Add coverage to your build process + +The details of setting up coverage within your build process depend on which tools you are using. +In our example below we use: + +* Composer, as a package manager +* PHPUnit with Xdebug, to execute the tests +* Clover to do the coverage reporting, and +* GitHub Actions to perform the build. + +Simply add the following to your `.github/workflows/build.yml` file: + +``` +- name: Setup PHP with Xdebug + uses: shivammathur/setup-php@v2 + with: + php-version: '8.1' + coverage: xdebug + +- name: Install dependencies with composer + run: composer update --no-ansi --no-interaction --no-progress + +- name: Run tests with phpunit/phpunit + run: vendor/bin/phpunit --coverage-clover=coverage.xml + +- name: Fix code coverage paths + run: sed -i 's@'$GITHUB_WORKSPACE'@/github/workspace/@g' coverage.xml +``` + +The resulting file should look something like this: + +``` +name: build +on: + - pull_request + - push +jobs: + tests: + name: Tests + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v2 + with: + fetch-depth: 0 + - name: Setup PHP with Xdebug + uses: shivammathur/setup-php@v2 + with: + php-version: '8.1' + coverage: xdebug + - name: Install dependencies with composer + run: composer update --no-ansi --no-interaction --no-progress + - name: Run tests with phpunit/phpunit + run: vendor/bin/phpunit --coverage-clover=coverage.xml + - name: Fix code coverage paths + run: sed -i 's@'$GITHUB_WORKSPACE'@/github/workspace/@g' coverage.xml + - name: SonarQube Scan + uses: SonarSource/sonarqube-scan-action@master + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} +``` + +First you install all your project dependencies using Composer as a package manager and then invoke _PHPUnit with XDebug_ to run your tests and generate a coverage report file. + +The essential requirements are that the tool produces its report in the `clover.xml` format and writes it to a place from which the scanner can then pick it up. + + +## Add the coverage analysis parameter + +The next step is to add `sonar.php.coverage.reportPaths` to your analysis parameters. +This parameter must be set to the path of the report file produced by your coverage tool. +In this example, that path is set to the default. +It is set in the `sonar-project.properties` file, located in the project root: + +``` +sonar.projectKey= +... +sonar.php.coverage.reportPaths=coverage.xml +``` + + +## Coverage parameters can be set in multiple places + +As with other analysis parameters, `sonar.php.coverage.reportPaths` can be set in multiple places: + +* In the `sonar-project.properties` file, as mentioned above. + +* On the command line of the scanner invocation using the `-D` or `--define` + switch, for example: + + `sonar-scanner -Dsonar.php.coverage.reportPaths=coverage.xml` + +* In the SonarQube interface under + + **_Your Project_ > Project Settings > General Settings > Languages > PHP > PHPUnit** + + for project-level settings, and + + **Administration > Configuration > General Settings > Languages > PHP > PHPUnit** + + for global settings (applying to all projects). + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/python-test-coverage.md b/server/sonar-docs/src/pages/analysis/test-coverage/python-test-coverage.md new file mode 100644 index 00000000000..a5044e8267f --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/python-test-coverage.md @@ -0,0 +1,157 @@ +--- +title: Python Test Coverage +url: /analysis/test-coverage/python-test-coverage/ +--- + +SonarQube supports the reporting of test coverage information as part of the analysis of your Python project. + +However, SonarQube does not compile the coverage report itself. Instead, you must use a third-party tool to produce the coverage report as part of your build process (before the analysis step). Later, during the analysis step, the scanner picks up this report and sends it to SonarQube. The coverage is then displayed on your SonarQube project dashboard along with the other analysis metrics. + + +## Adjust your setup + +To enable coverage you need to: + +* Adjust your build process so that the coverage tool runs _before_ the scanner report generation step runs. +* Make sure that the coverage tool writes its report file to a defined path in the build environment. +* Configure the scanning step of your build so that the scanner picks up the report file from that defined path. + + +## Add coverage to your build process + +The details of setting up coverage within your build process depend on which tools you are using. +In our example we use: + +* Tox, to configure the tests +* Pytest, to execute the tests +* Coverage, (the Coverage.py tool,) to measure code coverage, and +* GitHub Actions, to perform the build. + +In this example, we invoke `pytest` and use the `pytest-cov` plugin which, in turn, uses Coverage.py. +Simply add the text below to the `tox.ini` file at the root of your project: + +``` +[tox] +envlist = py39 +skipsdist = True + +[testenv] +deps = + pytest + coverage +commands = pytest --cov=my_project --cov-report=xml --cov-config=tox.ini --cov-branch + +[coverage:run] +``` + +Alternatively, we could start the test by invoking the Coverage.py tool (the command `coverage`) with the `pytest` invocation as an argument, like this: + +``` +[tox] +envlist = py39 +skipsdist = True + +[testenv] +deps = + pytest + coverage +commands = + coverage run -m pytest + coverage xml + +[coverage:run] +relative_files = True +source = my_project/ +branch = True +``` + +Note that we specify `relative_files = True` in the `tox.ini` file to ensure that your coverage results are correctly parsed. + +The following shows how to configure the GitHub Actions build file for your Python project so that it works in conjunction with the `tox.ini` configuration file described above to generate code coverage. +Your `build.yml` file should look something like this: + + + +``` +name: Build +on: + push: + branches: + - main + pull_request: + types: [opened, synchronize, reopened] +jobs: + sonarqube: + name: SonarQube + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + fetch-depth: 0 + - name: Setup Python + uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python }} + - name: Install tox and any other packages + run: pip install tox + - name: Run tox + run: tox -e py + - name: SonarQube Scan + uses: SonarSource/sonarqube-scan-action@master + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} +``` + +First of all, install all of your project dependencies and then invoke `tox` to run your tests and generate a coverage report file. + +If, as here, you do not specify an output file, the scanner will look for report paths located under the default +`.coverage-reports/*coverage-*.xml.` + +If you are using a different package manager or a different testing tool these details will be different. + +The essential requirements are that the tool produces its report in the Cobertura XML format and writes it to a place from which the scanner can then pick it up. + + +## Add the coverage analysis parameter + +The next step is to add `sonar.python.coverage.reportPaths` to your analysis parameters. +This parameter must be set to the path of the report file produced by your coverage tool. +In this example, that path is set to the default produced by Coverage.py. +It is set in the `sonar-project.properties` file, located in the project root: + +``` +sonar.projectKey= +... +sonar.python.coverage.reportPaths=coverage.xml +``` + + +## Coverage parameters can be set in multiple places + +As with other analysis parameters, `sonar.pyhton.coverage.reportPaths` can be set in multiple places: + +* In the `sonar-project.properties` file, as mentioned above. + +* On the command line of the scanner invocation using the `-D` or `--define` + switch, for example: + + `sonar-scanner -Dsonar.python.coverage.reportPaths=coverage.xml` + +* In the SonarQube interface under + + **_Your Project_ > Project Settings > General Settings > Languages > Python > Tests and Coverage** + + for project-level settings, and + + **Administration > Configuration > General Settings > Languages > Python > Tests and Coverage** + + for global settings (applying to all projects). + + +## See Also + +[Test Coverage Parameters](/analysis/test-coverage/test-coverage-parameters/). diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/test-coverage-parameters.md b/server/sonar-docs/src/pages/analysis/test-coverage/test-coverage-parameters.md new file mode 100644 index 00000000000..90f00b07f61 --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/test-coverage-parameters.md @@ -0,0 +1,206 @@ +--- +title: Test Coverage Parameters +url: /analysis/test-coverage/test-coverage-parameters/ +--- + +_Test coverage reports_ describe the percentage of your code that has been tested by your test suite during a build. + +This differs from _test execution reports_, which describe which tests within your test suite have been run during a build. For details, see [Test Execution Parameters](/analysis/test-coverage/test-execution-parameters/). + +Test execution reports are not generated by SonarQube itself. +Much like tests coverage reports, test execution reports must be generated by an external tool and then imported into SonarQube by specifying a parameter telling the scanner where to look for the report. +The data is then displayed in your SonarQube analysis. + +Below, you will find language- and tool-specific analysis parameters for importing test coverage reports. + +For information on analysis parameters in general, see [Analysis Parameters](/analysis/analysis-parameters/). + +Unless otherwise specified, these properties require values that are relative to the project root. Some properties support the following wildcards in paths. The remarks for properties that support wildcards will mention this fact. If wildcards are not noted for a given property, then they are not supported for that property. + +Symbol | Meaning +-|- +`?` | a single character +`*` | any number of characters +`**` | any number of directories + + +## All languages + +### sonar.coverageReportPaths + +Path to coverage report in the [Generic Test Data](/analysis/test-coverage/generic-test/) format. + + +## Apex + +### sonar.apex.coverage.reportPath + +Path to the `test-result-codecoverage.json` report file generated by the [`apex:test:run`](https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_force_apex.htm?search_text=apex%20test#cli_reference_test_run) command of the [Salesforce CLI](https://developer.salesforce.com/tools/sfdxcli). +Note, you must have a [Salesforce DX project](https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_workspace_setup.htm) set up and linked to your organization. + + +## C/C++/Objective-C + +See [C/C++/Objective-C Test Coverage](/analysis/test-coverage/c-family-test-coverage/) for examples and details. + + +### sonar.cfamily.gcov.reportsPath + +Path to the directory containing native `*.gcov` reports (not the XML reports generated by `gcovr`). + + +### sonar.cfamily.llvm-cov.reportPath + +Path to the `llvm-cov` report. + + +### sonar.cfamily.vscoveragexml.reportsPath + +Path to the Visual Studio Code Coverage report. +The path may be absolute or relative to the solution directory. +Path wildcards (see above) are supported. +Note that the `.coverage` report format must be converted to `.coveragexml` format using the tool `CodeCoverage.exe` tool in order to be imported. + + +### sonar.cfamily.bullseye.reportPath + +Path to the report from Bullseye, version >= 8.9.63 (use the [`covxml`](http://www.bullseye.com/help/ref-covxml.html) tool) + + +## C# + +See [.NET Test Coverage](/analysis/test-coverage/dotnet-test-coverage/) for examples and details. + + +### sonar.cs.vscoveragexml.reportsPaths + +Comma-delimited list of paths to the coverage reports produced by Visual Studio Code Coverage or the `dotnet-coverage` tool. +Wildcards are supported. + + +### sonar.cs.dotcover.reportsPaths + +Path to the dotCover coverage report. + + +### sonar.cs.opencover.reportsPaths + +Path to the OpenCover or Coverlet coverage report. + + +## Flex + +### sonar.flex.cobertura.reportPaths + +Path to the Cobertura XML reports. +Multiple paths may be comma-delimited. +May be absolute or relative to the project base directory. + + +## Go + +### sonar.go.coverage.reportPaths + +Comma-delimited list of paths to coverage report files. +Wildcards are supported. + + +## Java/Kotlin/Scala/JVM + +See [Java Test Coverage](/analysis/test-coverage/java-test-coverage/) for examples and details. + + +### sonar.coverage.jacoco.xmlReportPaths + +Comma-delimited list of paths to JaCoCo XML coverage reports. +Wildcards are supported. + + +### sonar.jacoco.reportPaths + +Deprecated. +Use `sonar.coverage.jacoco.xmlReportPaths`. + + +## JavaScript/TypeScript + +See [JavaScript/TypeScript Test Coverage](/analysis/test-coverage/javascript-typescript-test-coverage/) for examples and details. + + +### sonar.javascript.lcov.reportPaths + +Comma-delimited list of paths to LCOV coverage report files. +Paths may be absolute or relative to the project root. + + +## PHP + +See [PHP Test Coverage](/analysis/test-coverage/php-test-coverage/) for examples and details. + + +### sonar.php.coverage.reportPaths + +Comma-delimited list of paths to Clover XML-format coverage report files. +Paths may be absolute or relative to the project root. + + +## Python + +See [Python Test Coverage](/analysis/test-coverage/python-test-coverage/) for examples and details. + + +### sonar.python.coverage.reportPaths + +Comma-delimited list of paths to coverage reports in the Cobertura XML format. +Wildcards are supported. +Leave unset to use the default, `coverage-reports/*coverage-*.xml`. + + +## Ruby + +### sonar.ruby.coverage.reportPaths + +Comma-delimited list of paths to SimpleCov report files generated with the [JSON formatter](https://github.com/simplecov-ruby/simplecov#json-formatter) (available from SimpleCov 0.20). +For SimpleCov versions before 0.18, you can provide `.resultset.json` report files (though we recommend updating to 0.20 and using the JSON formatter). +Paths may be absolute or relative to the project root. + + +## Scala + +### sonar.scala.coverage.reportPaths + +Comma-delimited list of paths to `scoverage.xml` report files generated by Scoverage. + + +## Swift with Xcode 9.3+ + +You can use the [xccov-to-sonarqube-generic.sh](https://github.com/SonarSource/sonar-scanning-examples/blob/master/swift-coverage/swift-coverage-example/xccov-to-sonarqube-generic.sh) script from the [sonar-scanning-examples/swift-coverage](https://github.com/SonarSource/sonar-scanning-examples/tree/master/swift-coverage) project to convert output from Xcode 9.3's `xccov` tool to the [Generic Test Data](https://docs.sonarcloud.io/enriching/test-coverage/generic-test-data/) format. + + +## Swift with Xcode 7-9.2 + +### sonar.swift.coverage.reportPaths + +Path to the report generated by `llvm-cov show`. +The path may be absolute or relative to the project root. + + +## VB.NET + +See [.NET Test Coverage](/analysis/test-coverage/dotnet-test-coverage/) for examples and details. + + +### sonar.vbnet.vscoveragexml.reportsPaths + +Comma-delimited list of paths to Visual Studio Code Coverage reports. +Wildcards are supported. + + +### sonar.vbnet.dotcover.reportsPaths + +Path to dotCover coverage report. + + +### sonar.vbnet.opencover.reportsPaths + +Path to OpenCover coverage report. diff --git a/server/sonar-docs/src/pages/analysis/test-coverage/test-execution-parameters.md b/server/sonar-docs/src/pages/analysis/test-coverage/test-execution-parameters.md new file mode 100644 index 00000000000..a9976ad08ab --- /dev/null +++ b/server/sonar-docs/src/pages/analysis/test-coverage/test-execution-parameters.md @@ -0,0 +1,141 @@ +--- +title: Test Execution Parameters +url: /analysis/test-coverage/test-execution-parameters/ +--- + +_Test execution reports_ describe which tests within your test suite have been run during a build. + +This differs from _test coverage reports_, which describe the percentage of your code that has been tested by your test suite during a build. +For details, see [Test Coverage](/analysis/test-coverage/overview/). + +Test execution reports are not generated by SonarQube itself. +Much like tests coverage reports, test execution reports must be generated by an external tool and then imported into SonarQube by specifying a parameter telling the scanner where to look for the report. +The data is then displayed in your SonarQube analysis. + +Below, you will find language- and tool-specific analysis parameters for importing test execution reports. + +For information on analysis parameters in general, see [Analysis Parameters](/analysis/analysis-parameters/). + +Unless otherwise specified, these properties require values that are relative to the project root. Some properties support the following wildcards in paths. The remarks for properties that support wildcards will mention this fact. If wildcards are not noted for a given property, then they are not supported for that property. + +Symbol | Meaning +-|- +`?` | a single character +`*` | any number of characters +`**` | any number of directories + + +## Branches only + +_SonarQube only supports test execution reports for the branches of a project (including the main branch), not for pull requests._ +This differs from the case with test coverage reports, where pull requests are supported. + + +## All languages + +### sonar.testExecutionReportPaths + +Comma-delimited list of paths to execution reports in the [Generic Test Data](/analysis/test-coverage/generic-test/) format. + + +## C/C++/Objective-C + +### sonar.cfamily.cppunit.reportsPath + +Path to the directory holding the [CPPUnit](https://sourceforge.net/projects/cppunit/) reports. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +## C# + +### sonar.cs.vstest.reportsPaths + +Paths to VSTest reports. Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.cs.nunit.reportsPaths + +Paths to NUnit execution reports. Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.cs.mstest.reportsPaths + +Paths to MSTest execution reports. Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.cs.xunit.reportsPaths + +Paths to xUnit execution reports. Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +## Go + +### sonar.go.tests.reportPaths + +Comma-delimited list of paths to unit test report files. +Paths may be absolute or relative to the project root. + + +## Java + +### sonar.junit.reportPaths + +Comma-delimited list of paths to Surefire XML-format reports. + + +## JavaScript/TypeScript + +You can use [jest-sonar-reporter](https://www.npmjs.com/package/jest-sonar-reporter) or [karma-sonarqube-unit-reporter](https://github.com/tornaia/karma-sonarqube-unit-reporter) to create reports in the [Generic Test Data](/analysis/test-coverage/generic-test/) format. +Both packages are available on `npm`. + + +## PHP + +### sonar.php.tests.reportPath + +Path to the PHPUnit unit test execution report file. +The path may be absolute or relative to the project root. + + +## Python + +### sonar.python.xunit.reportPath + +Path to unit test execution report. +Leave unset to use the default (`xunit-reports/xunit-result-*.xml`). +Path wildcards (see above) are supported. +If any paths in the report are invalid, set `sonar.python.xunit.skipDetails=true` to collect only project-level details. + + +## VB.NET + +### sonar.vbnet.vstest.reportsPaths + +Paths to VSTest execution reports. +Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.vbnet.nunit.reportsPaths + +Paths to NUnit execution reports. +Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.vbnet.mstest.reportsPaths + +Paths to MSTest execution reports. +Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. + + +### sonar.vbnet.xunit.reportsPaths + +Paths to xUnit execution reports. +Multiple paths may be comma-delimited or included via wildcards. +Note that while measures such as the number of tests are displayed at the project level, no drill-down is available. diff --git a/server/sonar-docs/static/SonarQubeNavigationTree.json b/server/sonar-docs/static/SonarQubeNavigationTree.json index 21bacc6ee5a..9912b15b66a 100644 --- a/server/sonar-docs/static/SonarQubeNavigationTree.json +++ b/server/sonar-docs/static/SonarQubeNavigationTree.json @@ -50,8 +50,21 @@ "/analysis/languages/xml/" ] }, - "/analysis/coverage/", - "/analysis/generic-test/", + { + "title": "Test Coverage", + "children": [ + "/analysis/test-coverage/overview/", + "/analysis/test-coverage/java-test-coverage/", + "/analysis/test-coverage/javascript-typescript-test-coverage/", + "/analysis/test-coverage/dotnet-test-coverage/", + "/analysis/test-coverage/python-test-coverage/", + "/analysis/test-coverage/php-test-coverage/", + "/analysis/test-coverage/c-family-test-coverage/", + "/analysis/test-coverage/generic-test/", + "/analysis/test-coverage/test-coverage-parameters/", + "/analysis/test-coverage/test-execution-parameters/" + ] + }, { "title": "Importing External Issues", "children": ["/analysis/external-issues/", "/analysis/generic-issue/"] diff --git a/server/sonar-docs/static/StaticNavigationTree.json b/server/sonar-docs/static/StaticNavigationTree.json index e2f09ea6c1c..17309d1405e 100644 --- a/server/sonar-docs/static/StaticNavigationTree.json +++ b/server/sonar-docs/static/StaticNavigationTree.json @@ -75,8 +75,21 @@ "/analysis/languages/xml/" ] }, - "/analysis/coverage/", - "/analysis/generic-test/", + { + "title": "Test Coverage", + "children": [ + "/analysis/test-coverage/overview/", + "/analysis/test-coverage/java-test-coverage/", + "/analysis/test-coverage/javascript-typescript-test-coverage/", + "/analysis/test-coverage/dotnet-test-coverage/", + "/analysis/test-coverage/python-test-coverage/", + "/analysis/test-coverage/php-test-coverage/", + "/analysis/test-coverage/c-family-test-coverage/", + "/analysis/test-coverage/generic-test/", + "/analysis/test-coverage/test-coverage-parameters/", + "/analysis/test-coverage/test-execution-parameters/" + ] + }, { "title": "Importing External Issues", "children": ["/analysis/external-issues/", "/analysis/generic-issue/"]