Repository navigation
Add audited model experiments and measured evaluation reports - #3
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Review decisions are not bound to image hashes, and validation scores are not bound to their checkpoint or manifest, risking cross-split photo leakage and incorrect model selection.
6 open issues
Bind review decisions to the reviewed image hashes · New Verify validation metrics match the checkpoint and manifest · New Verify the download checksum before moving it to the archive path · New Include every confusion pair tied at the cutoff · New Name optional config keys in the validation error · New Show the Tuxedo note only when that class is present · New
What changed in this PR
Adds Cat Vision’s first measured Gano v3 study, extending MobileNetV2 with dataset auditing, validation-only recipe selection and offline evaluation reports. The published results report 75.4% accuracy on 836 held-out images.
Changes:
- Add checksum-pinned acquisition, reviewed similarity groups and group-disjoint dataset preparation.
- Add configurable augmentation, traced fine-tuning, bootstrap intervals and offline reporting.
- Publish experiment evidence and update documentation, regression tests, dependencies and CI.
This pull request requires changes. Critical issues remain in audit-decision freshness and validation-metric provenance, alongside the moderate issues and nits noted below.
| File | Reviewed changes and outstanding findings |
|---|---|
uv.lock |
Locks the expanded audit and reporting dependencies. |
tests/test_selection.py |
Tests candidate ranking, tiebreaks and artifact checks. |
tests/test_report.py |
Tests report generation, escaping and image-path safeguards. |
tests/test_pipeline.py |
Extends fine-tuning and split-specific evaluation coverage. |
tests/test_metrics.py |
Tests reproducible grouped bootstrap behavior. |
tests/test_fetch.py |
Tests archive checksums and extraction safeguards. |
tests/test_audit.py |
Tests similarity reviews, grouping and audited preparation. |
src/cat_vision/selection.py |
Adds validation-based selection. Critical: scores are not bound to the checkpoint/config/manifest that produced them. Moderate: the saved class mapping is not verified. |
src/cat_vision/report.py |
Generates figures and offline HTML. Moderate: diagnostic plots lack accessible textual equivalents. Nit: the Tuxedo note appears for unrelated datasets. |
src/cat_vision/pipeline.py |
Adds traced fine-tuning and expanded evaluation. Moderate: child initialization metadata can disagree with inherited weights. Nit: config errors omit the allowed optional keys. |
src/cat_vision/model.py |
Makes horizontal-flip augmentation configurable. |
src/cat_vision/metrics.py |
Implements class-stratified group-bootstrap intervals. |
src/cat_vision/data.py |
Adds grouped preparation and manifest validation. Moderate: forced initial assignments can substantially oversize validation/test partitions. |
src/cat_vision/cli.py |
Exposes audit, selection, reporting and new training/evaluation options. Nits: stale evaluation help and missing tests for the new CLI wiring. |
src/cat_vision/audit.py |
Adds similarity screening and reviewed groups. Critical: filename-only decisions can silently reuse stale judgments after image contents change. |
scripts/fetch_gano.py |
Downloads and extracts the pinned archive. Moderate: incorrect downloads reach the final archive path before verification, preventing automatic retry. |
README.md |
Presents measured results and updated usage commands. |
pyproject.toml |
Adds audit dependencies and optional reporting dependencies. |
Makefile |
Installs reporting dependencies for development. |
docs/verification.md |
Records software checks and experiment verification evidence. |
docs/results/gano-v3/split-counts.csv |
Publishes per-label partition sizes. |
docs/results/gano-v3/source.json |
Records dataset version, checksum, provenance and stated terms. |
docs/results/gano-v3/source-breakdown.json |
Summarizes test results by inferred source. |
docs/results/gano-v3/selection.json |
Records the validation-selected recipe and candidate scores. |
docs/results/gano-v3/runtime.json |
Records local CPU inference measurements. |
docs/results/gano-v3/review-decisions.json |
Publishes visual similarity-review decisions. |
docs/results/gano-v3/README.md |
Summarizes comparisons and held-out results. Nits: the confusion table truncates a tie without explanation; benchmark-only thread settings are attributed to training, and the benchmark script/invocation is not committed. |
docs/results/gano-v3/pretrained-source.json |
Records pretrained-weight origin and checksum. |
docs/results/gano-v3/per-label-metrics.csv |
Publishes label-level test scores. |
docs/results/gano-v3/model-provenance.json |
Records selected-model environment and artifact fingerprints. |
docs/results/gano-v3/model-config.json |
Preserves the selected training configuration. |
docs/results/gano-v3/metrics.json |
Publishes held-out metrics and uncertainty intervals. |
docs/results/gano-v3/classes.json |
Records the ordered output labels. |
docs/results/gano-v3/candidates/partial-fine-tuning/validation_metrics.json |
Publishes fine-tuning validation scores. |
docs/results/gano-v3/candidates/partial-fine-tuning/provenance.json |
Records candidate environment and parent-checkpoint provenance. |
docs/results/gano-v3/candidates/partial-fine-tuning/history.json |
Publishes structured fine-tuning history. |
docs/results/gano-v3/candidates/partial-fine-tuning/history.csv |
Publishes tabular fine-tuning history. |
docs/results/gano-v3/candidates/partial-fine-tuning/config.json |
Captures fine-tuning candidate settings. |
docs/results/gano-v3/candidates/frozen-no-augmentation/validation_metrics.json |
Publishes unaugmented candidate validation scores. |
docs/results/gano-v3/candidates/frozen-no-augmentation/provenance.json |
Records unaugmented candidate environment and checkpoint provenance. |
docs/results/gano-v3/candidates/frozen-no-augmentation/history.json |
Publishes structured unaugmented training history. |
docs/results/gano-v3/candidates/frozen-no-augmentation/history.csv |
Publishes tabular unaugmented training history. |
docs/results/gano-v3/candidates/frozen-no-augmentation/config.json |
Captures unaugmented candidate settings. |
docs/results/gano-v3/candidates/frozen-horizontal-flip/validation_metrics.json |
Publishes horizontal-flip candidate validation scores. |
docs/results/gano-v3/candidates/frozen-horizontal-flip/provenance.json |
Records horizontal-flip candidate environment and checkpoint provenance. |
docs/results/gano-v3/candidates/frozen-horizontal-flip/history.json |
Publishes structured horizontal-flip training history. |
docs/results/gano-v3/candidates/frozen-horizontal-flip/history.csv |
Publishes tabular horizontal-flip training history. |
docs/results/gano-v3/candidates/frozen-horizontal-flip/config.json |
Captures horizontal-flip candidate settings. |
docs/results/gano-v3/audit-summary.json |
Publishes aggregate audit counts and fingerprints. |
docs/model-card.md |
Documents measured performance, intended use and limitations. |
docs/experiments.md |
Provides experiment reproduction and reporting commands. |
docs/experiment-plan.md |
Defines fixed recipes and the evaluation protocol. |
docs/data.md |
Documents acquisition, auditing, grouping and leakage limitations. |
docs/architecture.md |
Describes audit, training, selection and reporting boundaries. |
CONTRIBUTING.md |
Updates development installation instructions. |
configs/partial-fine-tuning.json |
Defines the partial fine-tuning recipe. |
configs/frozen-no-augmentation.json |
Defines the unaugmented frozen baseline. |
configs/frozen-horizontal-flip.json |
Defines the frozen horizontal-flip recipe. |
CHANGELOG.md |
Records the measured-study milestone. |
.github/workflows/ci.yml |
Installs reporting dependencies for quality checks. |
.gitattributes |
Marks large evidence artifacts as generated. |
Review effort: Max
|
Addressed all six inline findings in 9d9c4fb, with preserved experiment evidence and documentation in 601033a. Each inline thread has a response explaining its fix and regression coverage. Also addressed the additional findings in the review overview:
Verification: 46 local tests pass, Ruff lint/format and documentation-link checks pass, and both Linux CI jobs are green. The original pre-test selection record, complete split manifest, every numeric metric and every prediction-file byte are unchanged. The original metric files are archived, and the stricter re-evaluation is recorded in review-integrity-verification.json. No retraining or test-driven model change was made. |



Adds the first measured real-data study to Cat Vision, with reproducible audit evidence, a fixed experiment plan, validation-only selection and an honest held-out report.
Measured result
The selected frozen MobileNetV2 + horizontal-flip recipe achieves 75.4% accuracy, 0.751 macro F1 and 93.7% top-3 accuracy on 836 held-out images. The 95% class-stratified group-bootstrap accuracy interval is 72.5–78.0%; the training-majority baseline scores 6.7%.
Three predefined recipes were compared on the same 841-image validation partition. Partial fine-tuning raised training accuracy to 92.8% but did not improve validation macro F1. Only the validation-selected checkpoint was evaluated on test, after a separate commit recorded the choice.
Data and engineering
Validation
git diff --check: pass.This is a single-seed, within-collection baseline. It does not establish independent phone-photo performance, complete animal-identity separation, calibrated scores or unknown/non-cat rejection. The historical 88.16% notebook result remains explicitly labeled training accuracy and is not used as a comparable benchmark.
Review follow-up
Addresses all six inline findings plus the review overview's class-map, initialization, allocation, accessibility, CLI and benchmark notes. Review decisions are bound to both source-image hashes; validation metrics are bound to model/config/manifest identities before ranking. The original pre-test selection and original raw metric files are preserved, and stricter re-evaluation reproduces every number and prediction byte. See
docs/results/gano-v3/review-integrity-verification.json.