feat: JSON Summary Reports, Coverage Comparison & CI Enforcement

Project.toml

@@ -10,6 +10,7 @@ Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 DefaultApplication = "3f0dd361-4fe0-5fc6-8523-80b14ec94d85"
 DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
 EzXML = "8f5d6c58-4d21-5cfd-889c-e3ad7ee6a615"
+JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
 LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
 OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
@@ -22,6 +23,7 @@ Dates = "1.10"
 DefaultApplication = "1"
 DocStringExtensions = "0.8, 0.9"
 EzXML = "1"
+JSON = "0.21, 1"
 LibGit2 = "1.10"
 OrderedCollections = "1"
 Pkg = "1.10"

README.md

@@ -11,12 +11,15 @@ This is a collection of trivial functions to facilitate generating and exploring
 ```julia
 Pkg.add("LocalCoverage")
 ```
+
 or `]add LocalCoverage` from the Julia REPL.
 
 ### Optional Dependencies
+
 The package has several optional features which require additional dependencies.
 [`lcov`](https://github.com/linux-test-project/lcov) is required for generating HTML
 output.  You can install it via
+
 - Debian/Ubuntu: `sudo apt install lcov`
 - Arch/Manjaro: `yay -S lcov`
 
@@ -25,6 +28,7 @@ Note that the code in this package assumes a reasonably recent `lcov` version wh
 `LocalCoverage` also provides an option to generate a
 [Cobertura](https://cobertura.github.io/cobertura/) XML, which is used by JVM-related test
 suites such as Jenkins.  This can be done either with:
+
 - The native Julia implementation via the `write_lcov_to_xml` function,
 - The original implementation using the `generate_xml` function.
 
@@ -39,20 +43,129 @@ with packages added with the `Pkg.dev` installation option (which allows for eas
 manipulation of the package directory).
 
 To generate test coverage data do
+
 ```julia
 using LocalCoverage
 # pkg is the package name as a string, e.g. "LocalCoverage"
 generate_coverage(pkg = nothing; run_test = true) # defaults shown
 ```
+
+You'll see output during testing like:
+
+```text
+┌─────────────────────┬───────┬─────┬──────┬──────┬──────────┐
+│ Filename            │ Lines │ Hit │ Miss │    % │ Gaps     │
+├─────────────────────┼───────┼─────┼──────┼──────┼──────────┤
+│ src/DummyPackage.jl │     1 │   0 │    1 │   0% │ 6        │
+│ src/bar.jl          │     2 │   1 │    1 │  50% │ 3        │
+│ src/corge/corge.jl  │     1 │   1 │    0 │ 100% │          │
+│ src/corge/grault.jl │     1 │   0 │    1 │   0% │ 2        │
+│ src/qux.jl          │     2 │   0 │    2 │   0% │ 2, 5     │
+├─────────────────────┼───────┼─────┼──────┼──────┼──────────┤
+│ TOTAL               │     7 │   2 │    5 │  29% │          │
+└─────────────────────┴───────┴─────┴──────┴──────┴──────────┘
+```
+
 You can then navigate to the `coverage` subdirectory of the package directory (e.g.
 `~/.julia/dev/PackageName/coverage`) and see the generated coverage summaries. Note that the test execution step may be skipped if `*.cov` files were already generated (possibly by some external package).
 
 To generate, and optionally open, the coverage report HTML do
+
 ```julia
 html_coverage(coverage::PackageCoverage; open = false, dir = tempdir()) # defaults shown
 ```
 
+### Coverage summary JSON output
+
+To generate a Jest-compatible `coverage-summary.json` report, pass a `JSONSummaryOptions` instance to `generate_coverage()` via the `json_summary` keyword argument:
+
+```julia
+generate_coverage(pkg = nothing; json_summary = JSONSummaryOptions()) # defaults shown
+```
+
+If `test_args` are provided, the top level `"total"` key is replaced with the name of the test set.
+
+To generate a Jest-compatible `coverage-summary.json` report from existing `PackageCoverage` data do
+
+```julia
+generate_json_summary(coverage::PackageCoverage, filename = "coverage-summary.json"; test_args = [""]) # defaults shown
+```
+
+#### `JSONSummaryOptions` Fields
+
+`JSONSummaryOptions` has the following fields (with their default values when calling `JSONSummaryOptions()`):
+
+* `write::Bool = true`: Whether to write the coverage JSON summary to disk.
+* `filename::String = "coverage-summary.json"`: Filename of the generated JSON summary.
+* `comparison_filename::Union{Nothing,String} = nothing`: Path/string of previous coverage JSON summary to compare against.
+* `comparison_fail_on_decrease::Bool = false`: Determines whether to fail (throw an error) if the coverage has decreased.
+
+#### JSON Summary Schema
+
+The JSON summary mimics [Jest's standard `json-summary` reporter format](https://stackoverflow.com/a/60968723) ([relevant section of Jest docs](https://jestjs.io/docs/configuration#coveragereporters-arraystring--string-options)). It contains a top-level key for the package total (or the name of the test set if customized), followed by keys for each individual tracked file path.
+
+Each section contains sub-metrics for `lines`, `statements`, `functions`, and `branches` with the following structure:
+* `total`: Total number of lines tracked.
+* `covered`: Number of lines hit.
+* `skipped`: Always `0` (included for compatibility).
+* `pct`: Coverage percentage (as a float).
+
+```json
+{
+  "total": {
+    "lines": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
+    "statements": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
+    "functions": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
+    "branches": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 }
+  },
+  "src/bar.jl": {
+    "lines": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
+    "statements": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
+    "functions": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
+    "branches": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 }
+  }
+}
+```
+
+
+### Coverage Delta and Comparison (CI)
+
+You can compare two coverage JSON summaries (either from file paths or directly from JSON strings) to print a delta table and check if coverage has decreased:
+
+```julia
+# Compare two summaries. Returns true if coverage did not decrease, false if it did.
+compare_coverage_json_summaries("old-summary.json", "new-summary.json")
+```
+
+You can also integrate this check directly into `generate_coverage()`, which is very useful for continuous integration (CI) environments to fail a build if coverage drops. If the `comparison_filename` file is missing, it's an automatic pass:
+
+```julia
+# Runs tests, generates the current summary, prints a comparison table against
+# "previous-summary.json", and throws an error if overall coverage decreased.
+generate_coverage(pkg; 
+                  json_summary = JSONSummaryOptions(
+                      comparison_filename = "previous-summary.json",
+                      comparison_fail_on_decrease = true
+                  ))
+```
+
+The comparison output looks like:
+
+```text
+┌─────────────────────┬──────────────┬──────────────┬────────┐
+│ File/Section        │ Old Coverage │ New Coverage │  Delta │
+├─────────────────────┼──────────────┼──────────────┼────────┤
+│ total               │        20.0% │       28.57% │ +8.57% │
+│ src/DummyPackage.jl │            - │         0.0% │      - │
+│ src/bar.jl          │        20.0% │        50.0% │ +30.0% │
+│ src/corge/corge.jl  │            - │       100.0% │      - │
+│ src/corge/grault.jl │            - │         0.0% │      - │
+│ src/qux.jl          │            - │         0.0% │      - │
+└─────────────────────┴──────────────┴──────────────┴────────┘
+```
+
 A utility method is also provided to easily print coverage statistics and exit with a status reflecting if some given target coverage was met. It can be used from a shell by doing
+
 ```bash
 julia --project -e'using LocalCoverage; report_coverage_and_exit(target_coverage=90)'
 ```