feat: JSON Summary Reports, Coverage Comparison & CI Enforcement#75
Conversation
…move deprecated wrapper method
…overage thresholds
…issing summary files
|
Cool! The JSON schema should be documented so that users know what to expect and this package is semantically versioned to support that. I like documentating JSON similar to this: |
|
@jakobjpeters I don't entirely understand, but how's this (from the updated readme)? JSON Summary SchemaThe JSON summary mimics Jest's standard Each section contains sub-metrics for
{
"total": {
"lines": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
"statements": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
"functions": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 },
"branches": { "total": 10, "covered": 5, "skipped": 0, "pct": 50.0 }
},
"src/bar.jl": {
"lines": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
"statements": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
"functions": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 },
"branches": { "total": 5, "covered": 2, "skipped": 0, "pct": 40.0 }
}
} |
|
That's great, thank you! I think it needs to be part of the docstring too. Here's an example, if it's helpful, which provides more details for each field. I'm not set on the structure of it though so feel free to adapt if it suits you :) |
tpapp
left a comment
There was a problem hiding this comment.
Thanks for this very nice PR! I especially appreciate the thorough tests and documentation. I am very busy now so I just made a quick comment, I will play around with it in detail. If you do not hear from me in a week, feel free to ping, I don't want this to get forgotten.
tpapp
left a comment
There was a problem hiding this comment.
Further minor changes, mainly about code style.
If you address these and the previous suggestions, I am happy to merge this, it would be a great feature.
…ions refactor: use `parse_json_summary()` and `is_json_content()`
tpapp
left a comment
There was a problem hiding this comment.
Thanks, very nicely organized code! just asked for a few cosmetic changes, then this is good to go.
tpapp
left a comment
There was a problem hiding this comment.
LGTM, thanks for the persistence and the nice work!
This PR introduces a suite of robust coverage reporting, comparison, and continuous integration (CI) enforcement capabilities to
LocalCoverage.jl. These additions allow developers and automated CI pipelines to generate Jest-compatible JSON summaries, compare current coverage with baseline summaries, display coverage deltas in a clear table, and fail builds or scripts if overall coverage decreases or falls below a target percentage threshold.This PR addresses all requirements for coverage summary reporting.
Key Features & Improvements
1. Jest-Compatible JSON Coverage Summary
generate_json_summary(coverage::PackageCoverage, filename = "coverage-summary.json"; test_args = [""])which outputs a coverage report mimicking the structure of Jest'sjson-summaryreporter (tracking lines, statements, functions, and branches).generate_coveragepipeline via keywords:json_summary::Bool = false(automatically set totrueif comparison is requested).json_summary_filename::String = "coverage-summary.json"."total"key in the summary with customized test set names derived fromtest_args.2. Coverage Comparison & Delta Reporting
compare_coverage_json_summaries(old::AbstractString, new::AbstractString, io::IO = stdout)to parse and compare coverage summaries (from either JSON strings or files).PrettyTablesdetailing:+8.57%or-2.30%).generate_coverage:json_comparison_summary_filename::Union{Nothing,String} = nothingspecifies the baseline JSON path to compare against.json_summary_comparison_fail_on_decrease::Bool = falsedetermines whether to throw an error and halt execution if overall coverage has dropped.3. CI-Friendly CLI Enforcement (
report_coverage_and_exit)report_coverage_and_exitto facilitate CLI/CI automation.0if the target coverage threshold is met, or status code1otherwise.julia --project=@. -e 'using LocalCoverage; report_coverage_and_exit(; target_coverage=90)'4. PrettyTables 3.0 Compatibility
@static if pkgversion(PrettyTables) < v"3.0.0") to support both older PrettyTables releases and the modern v3.0 API. This includes adaptive usage ofHighlighter/TextHighlighter, tabular keywords, layout configurations, and text format styling.File Changes & Coverage Analysis
Project.toml: Added dependency onJSONwith appropriate[compat]bounds.README.md: Extensively documented the new functions, inputs, options, CLI usage, and CI integration, complete with layout previews.src/LocalCoverage.jl: Refactored the core logic to return thePackageCoverageobject on success, implemented XML generation utilities, addedgenerate_json_summary, comparison routines, andreport_coverage_and_exit.test/Project.toml: Added test-environment dependencies.test/runtests.jl: Added comprehensive test coverage (~200 lines of robust tests) to validate JSON parsing/saving, file comparisons, CLI subprocess exit codes, edge cases, missing file paths, and Aqua quality assurance checks.Usage Examples
Generating a JSON summary:
CI check failing on coverage decrease:
CI check enforcing a 90% threshold:
julia --project=@. -e 'using LocalCoverage; report_coverage_and_exit(target_coverage=90)'Verification & Testing
Tests pass locally with 100% success, confirming:
closes #47