Switch code coverage from OpenCover to coverlet

[sharwell]

Summary

Transition StyleCop Analyzers from OpenCover-based code coverage to coverlet for both CI and local development:

• Replace OpenCover + xunit.console invocations in build scripts with coverlet.
• Keep using ReportGenerator and Codecov, but feed them coverlet-generated Cobertura reports.
• Update the local coverage script to use coverlet instead of OpenCover.
• Document new local workflows (including a “focus on changed code” option) in CONTRIBUTING.md.

The structure of this issue is intentionally detailed so it can be executed step-by-step and/or handed off to an automation agent later. :contentReference[oaicite:0]{index=0}

---

Background / current state

• Azure Pipelines uses the build/build-and-test.yml template. It runs tests for each language version via build/test.yml and then merges OpenCover reports with ReportGenerator into a Cobertura report that is uploaded to Codecov using the CodecovUploader tool. :contentReference[oaicite:1]{index=1}
• build/test.yml currently runs tests via xunit.console.x86.exe wrapped in OpenCover.Console.exe, writing OpenCover.StyleCopAnalyzers.CSharp{N}.xml files into build/OpenCover.Reports, then publishes those XMLs as artifacts for the coverage merge stage. :contentReference[oaicite:2]{index=2}
• The Publish_Code_Coverage_* stage in build/build-and-test.yml:
  • Downloads coverageResults-cs* artifacts.
  • Uses ReportGenerator.exe from the ReportGenerator NuGet package to merge OpenCover.*.xml files and emit Cobertura.xml.
  • Uses CodecovUploader to upload the merged Cobertura report to Codecov. :contentReference[oaicite:3]{index=3}
• There is a local helper script build/opencover-report.ps1 that:
  • Builds the solution (unless -NoBuild).
  • Locates OpenCover, ReportGenerator, and xunit.runner.console from .nuget\packages.config.
  • Runs each test assembly (C# 6–13) under OpenCover, merging results into a single OpenCover.StyleCopAnalyzers.xml.
  • Generates an HTML report with ReportGenerator and prints the path to OpenCover.Reports\index.htm. :contentReference[oaicite:4]{index=4}
• .codecov.yml configures Codecov to:
  • Disable project/patch status checks.
  • Apply a fixes mapping from build/StyleCop.Analyzers/ paths back to StyleCop.Analyzers/.
  • Group coverage by production and test flags.
  • Use a diff layout for comments. :contentReference[oaicite:5]{index=5}
• CONTRIBUTING.md is minimal and currently does not describe any local code coverage workflows. :contentReference[oaicite:6]{index=6}

OpenCover is effectively archived and unmaintained, while coverlet is the recommended cross‑platform tool for .NET code coverage. Moving to coverlet should simplify tooling and reduce future maintenance.

---

Goals

1. Replace OpenCover with coverlet for both CI and local coverage runs.
2. Retain Codecov integration in CI, using a merged Cobertura report as the primary artifact.
3. Provide a local coverage workflow that:
   • Builds the repository.
   • Runs tests under coverlet.
   • Produces human-readable HTML coverage (via ReportGenerator).
4. Add a “changed code” workflow for local use:
   • Allow contributors to focus coverage review on files/lines changed in the current branch relative to master (or main).
   • This can be approximate (file- and line-level filtering) rather than a full “two-coverage-runs delta” implementation.
5. Document the new workflows in CONTRIBUTING.md.

---

Non-goals

• Changing the test framework (still xUnit) or the structure of test projects.
• Changing which branches Codecov tracks or how Codecov comments on PRs, beyond what’s required to keep it working with coverlet.
• Enforcing new coverage thresholds in CI (can be a follow-up).

---

High-level approach

1. Introduce coverlet as the coverage engine (likely via coverlet.console or coverlet.msbuild, depending on what integrates best with the existing xUnit console / .NET Framework + .NET 6 test mix).
2. Update CI scripts:
   • Replace the OpenCover invocation in build/test.yml with coverlet, writing per-language Cobertura reports.
   • Adjust the merge/upload stage in build/build-and-test.yml to pick up coverlet’s Cobertura XML instead of OpenCover XML.
3. Update build/opencover-report.ps1:
   • Either migrate it in-place (keeping the name but using coverlet) or introduce a new build/coverage-report.ps1 and deprecate the old script.
4. Add an optional “diff-only” mode to the local script that:
   • Uses git diff vs. the main branch to find changed files.
   • Filters the ReportGenerator output to those files (and their uncovered lines) for focused reviews.
5. Update CONTRIBUTING.md with instructions for:
   • Running full coverage locally.
   • Running diff-focused coverage locally.
   • How these relate to codecov.io in CI.

---

Detailed steps

1. Introduce coverlet and update dependencies

1.1. Decide on coverlet integration mode

• Options:
  • coverlet.console wrapping xunit.console.x86.exe (most similar to current OpenCover.Console pattern).
  • coverlet.msbuild + dotnet test for the .NET 6 test targets (requires more restructuring for .NET Framework runs).
• Given the current CI pattern and the existing opencover-report.ps1 design, the simplest migration is likely:
  • Use coverlet.console with xUnit console for all frameworks.

1.2. Update .nuget/packages.config

• Add a new entry for the selected coverlet package (e.g., coverlet.console).
• Remove the OpenCover package entry.
• Keep ReportGenerator and CodecovUploader (unless we decide to change how we upload to Codecov).
• Keep xunit.runner.console while we’re still using it as the test runner.

Implementation note: the CI scripts already parse .nuget\packages.config to locate package versions, so the new coverlet entry should follow the same pattern used for OpenCover. :contentReference[oaicite:7]{index=7}

---

2. Replace OpenCover with coverlet in Azure Pipelines

2.1. Update build/test.yml “Run tests” script

Current behavior (simplified):

• Resolve OpenCover and xunit.runner.console from .nuget\packages.config.
• Compose the test assembly path for a given C# / framework version.
• Run:

&$opencover_console `
  -register:Path32 `
  -threshold:1 `
  -oldStyle `
  -returntargetcode `
  -hideskipped:All `
  -filter:"+[StyleCop*]*" `
  -excludebyattribute:*.ExcludeFromCodeCoverage* `
  -excludebyfile:*\*Designer.cs `
  -output:"$report_folder\OpenCover.StyleCopAnalyzers.CSharp${{ parameters.LangVersion }}.xml" `
  -target:"$xunit_runner_console_${{ parameters.FrameworkVersion }}" `
  -targetargs:"$target_dll_csharp${{ parameters.LangVersion }} -noshadow -xml StyleCopAnalyzers.CSharp${{ parameters.LangVersion }}.xunit.xml"

…then publish build/OpenCover.Reports as coverageResults-cs{LangVersion}. ([GitHub]1)

Target behavior:

• Replace the OpenCover.Console.exe call with coverlet, e.g.:

  • Locate coverlet.console.exe from .nuget\packages.config (similar to how OpenCover.Console.exe is located today).
  • Wrap xunit.console.x86.exe via coverlet’s --target and --targetargs options, generating Cobertura output for each C# version and configuration.
  • Write per-language Cobertura reports into a folder like build/coverlet/coverage.CSharp{LangVersion}.cobertura.xml (or similar), and publish that folder as the coverageResults-cs{LangVersion} artifact instead of OpenCover.Reports.

• Preserve:

  • Filters for which assemblies/files to instrument (i.e., +[StyleCop*]*, excluding *.ExcludeFromCodeCoverage* and *Designer.cs), adapted to coverlet’s syntax.
  • The test result XMLs (*.xunit.xml) for PublishTestResults@2 to consume, as done today.

Work items:

• Replace OpenCover-specific script block in build/test.yml with a coverlet invocation while keeping xUnit as the underlying runner.
• Ensure the run still produces xUnit XML files for PublishTestResults@2.
• Ensure coverage artifacts are published as coverageResults-cs{LangVersion} as before, but containing Cobertura coverage files instead of OpenCover XML.

2.2. Update the Publish_Code_Coverage_* stage in build/build-and-test.yml

Current behavior:

• Downloads coverageResults-cs* artifacts containing OpenCover.*.xml files.

• Runs ReportGenerator like:

  &$report_generator `
    -targetdir:$(Pipeline.Workspace)/coverageResults-final `
    -reporttypes:Cobertura `
    "-reports:$(Pipeline.Workspace)/coverageResults-*/OpenCover.*.xml"

• Uploads coverageResults-final/Cobertura.xml to Codecov via CodecovUploader. ([GitHub]2)

Target behavior:

• Keep ReportGenerator + CodecovUploader, but:

  • Point -reports: at coverlet’s Cobertura XML files (e.g., coverageResults-*/coverage.*.cobertura.xml).
  • Keep -reporttypes:Cobertura so we still produce a single Cobertura.xml for Codecov.
  • Optionally also generate HTML reports (e.g., HtmlInline_AzurePipelines) for Azure Pipelines’ browsing UX, if desired.

Work items:

• Update the ReportGenerator invocation in build/build-and-test.yml to consume coverlet Cobertura reports instead of OpenCover XMLs.
• Keep the output file name and location (Cobertura.xml under coverageResults-final) so the CodecovUploader invocation stays simple.
• Verify Codecov still correctly associates coverage with repo paths using the existing .codecov.yml fixes configuration. ([GitHub]3)

---

3. Migrate the local build/opencover-report.ps1 script

Current behavior (simplified): ([GitHub]4)

• Parameters: -Debug, -NoBuild, -NoReport, -AppVeyor, -Azure.
• Builds the solution (unless -NoBuild).
• Resolves OpenCover, ReportGenerator, xunit.runner.console from .nuget\packages.config.
• Runs each test assembly (C# 6–13) under OpenCover with merge-by-hash, producing a single OpenCover.StyleCopAnalyzers.xml.
• Uses ReportGenerator.exe to create HTML coverage in OpenCover.Reports and prints a friendly message.

Target behavior:

• Replace OpenCover.Console.exe usage with coverlet:

  • Resolve coverlet.console from .nuget\packages.config.

  • Either:

    • Run each test assembly under coverlet separately, then merge reports with ReportGenerator, or
    • Use coverlet’s built-in merge mechanisms if desired.

• Keep or refine the parameters:

  • -Debug / -Release mapping to configuration.
  • -NoBuild to skip the build.
  • -NoReport to skip HTML generation (for CI-like callers).

• Rename output directories and files to something like:

  • build\coverage\coverage.cobertura.xml (primary Cobertura report).
  • build\coverage\index.htm (HTML root from ReportGenerator).

• Consider renaming the script to coverage-report.ps1 and:

  • Keep opencover-report.ps1 as a thin wrapper that delegates to coverage-report.ps1 with a warning about deprecation, or
  • Update the existing script in-place but update references in docs to call it “coverage report” rather than “OpenCover report”.

Work items:

• Replace all references to OpenCover in build/opencover-report.ps1 with coverlet equivalents.

• Update the report folder name(s) to reflect coverlet (e.g., coverage instead of OpenCover.Reports).

• Keep ReportGenerator integration but point it at coverlet Cobertura reports.

• Validate that a local run:

  • Builds successfully.
  • Executes tests for C# 6–13.
  • Produces a usable HTML coverage report.
  • Returns a non-zero exit code if tests fail.

---

4. Local “changed code” coverage workflow

Goal: provide a workflow for contributors to focus on coverage of code they’ve changed in their branch relative to master (or main).

Proposed design (script-level):

Extend the (renamed) coverage-report.ps1 script with options:

• -DiffBase <branchOrCommit>:

  • Default to origin/master (or origin/main, whichever matches the repo).
  • Use git diff --name-only --diff-filter=AM <DiffBase>...HEAD to find changed source files (e.g., *.cs).

• -DiffOnly or -ChangedFilesOnly:

  • After generating the standard coverlet Cobertura report, call ReportGenerator with -filefilters restricted to the changed files (converted into appropriate patterns) and produce an additional output directory, e.g., build\coverage-diff.
  • Optionally use a text-oriented report type (e.g., TextSummary or MarkdownSummary) to dump a quick summary of coverage for only changed files to the console.

Example workflows (to be documented, see section 5):

• Full coverage:

  # Build + run tests + generate full HTML coverage report
  .\build\coverage-report.ps1

• Diff-only coverage against origin/master:

  # Only changed files / lines vs origin/master
  .\build\coverage-report.ps1 -DiffBase origin/master -DiffOnly

Work items:

• Add optional -DiffBase / -DiffOnly parameters to the coverage script.
• Use git diff to determine the set of changed files relative to the base.
• Use ReportGenerator’s filefilters (and/or classfilters) to generate a coverage report limited to the changed files.
• Print a concise console summary of coverage for changed files (for quick feedback), with a pointer to the HTML report for deeper inspection.

Note: Codecov already provides PR-level diff coverage in CI via its diff comment layout; this step is focused on giving contributors a similar view locally, before pushing. ([GitHub]3)

---

5. Update CONTRIBUTING.md with coverage workflows

Current CONTRIBUTING.md describes prerequisites and general guidance but not coverage workflows. ([GitHub]5)

Proposed additions:

• Add a new “Code coverage” (or “Running tests and coverage”) section, including:

  1. Prerequisites

     • Ensure the correct .NET SDK is installed via init.ps1.
     • Mention that coverage scripts rely on NuGet restore having completed at least once.

  2. Run all tests with coverage

     • Example commands, e.g.:

       # Build + test + HTML coverage (Release)
       .\build\coverage-report.ps1
       
       # Debug build
       .\build\coverage-report.ps1 -Debug

     • Expected artifacts:

       • Cobertura XML path (e.g., build\coverage\Cobertura.xml).
       • HTML report path (e.g., build\coverage\index.htm).

  3. Check coverage for only your changes

     • Example commands, e.g.:

       # Compare against origin/master
       .\build\coverage-report.ps1 -DiffBase origin/master -DiffOnly
       
       # Compare against a specific commit or branch
       .\build\coverage-report.ps1 -DiffBase my-team-branch -DiffOnly

     • Explain high-level behavior:

       • Which files are considered “changed”.
       • Where the diff-focused report is written.
       • How this relates to the Codecov comment that will appear on your PR.

  4. Relation to CI

     • Briefly document that Azure Pipelines runs coverlet-based coverage on CI builds and uploads the merged Cobertura report to Codecov.
     • Mention that contributors should expect Codecov to report on both overall and diff coverage for PRs.

Work items:

• Add a “Code coverage” section to CONTRIBUTING.md describing:

  • Full coverage runs.
  • Diff-focused coverage runs.
  • Where to find the generated reports.
  • How the local workflow lines up with CI and Codecov.

---

Acceptance criteria

• CI still passes on the primary branch and PRs.

• Azure Pipelines:

  • Runs tests successfully for all C# / framework combinations.
  • Produces a merged Cobertura report from coverlet output.
  • Successfully uploads coverage to Codecov (Codecov status comment shows coverage for the head commit and diff).

• Local script:

  • build/coverage-report.ps1 (or updated opencover-report.ps1) runs without requiring extra manual steps beyond init.ps1.
  • Produces readable HTML coverage.
  • Offers an option to focus coverage reports on changed files relative to a specified base ref.

• CONTRIBUTING.md:

  • Documents how to run the coverage script.
  • Documents how to run diff-focused coverage locally and explains its limitations.

---

Open questions

• Exact coverlet package & invocation

  • Do we want to standardize on coverlet.console for all test frameworks, or split between coverlet.console for .NET Framework and coverlet.msbuild for .NET 6?

• Script naming / compatibility

  • Should we preserve build/opencover-report.ps1 as the canonical entrypoint (despite the misleading name), or introduce build/coverage-report.ps1 and keep opencover-report.ps1 as a thin compatibility shim?

• Diff coverage depth

  • Is file-level filtering sufficient for the “changed code” view, or do we want to invest in a more advanced delta-based workflow (two coverage runs plus TextDeltaSummary), which is more complex but more accurate?

Please feel free to tweak any of the names (script names, parameter names, artifact folder names) to better align with existing conventions in the repo while implementing this.

::contentReference[oaicite:14]{index=14}