Skip to content

Add audited model experiments and measured evaluation reports - #3

Merged
stefisha merged 10 commits into
mainfrom
feat/measured-baseline
Nov 3, 2026
Merged

stefisha merged 10 commits into
mainfrom
feat/measured-baseline

Conversation

@stefisha

@stefisha stefisha commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Adds the first measured real-data study to Cat Vision, with reproducible audit evidence, a fixed experiment plan, validation-only selection and an honest held-out report.

Measured result

The selected frozen MobileNetV2 + horizontal-flip recipe achieves 75.4% accuracy, 0.751 macro F1 and 93.7% top-3 accuracy on 836 held-out images. The 95% class-stratified group-bootstrap accuracy interval is 72.5–78.0%; the training-majority baseline scores 6.7%.

Three predefined recipes were compared on the same 841-image validation partition. Partial fine-tuning raised training accuracy to 92.8% but did not improve validation macro F1. Only the validation-selected checkpoint was evaluated on test, after a separate commit recorded the choice.

Data and engineering

  • Pin and checksum the original public Gano v3 archive; document its stated collection terms and source taxonomy, including Tuxedo as a coat pattern.
  • Decode 5,625 images, remove 52 redundant exact files, and review 29 additional similarity pairs: 28 photo-copy matches and one false match.
  • Freeze group-disjoint splits: 3,896 train / 841 validation / 836 test. Re-running audit/preparation reproduces the exact committed manifest.
  • Add audit, traced partial fine-tuning, validation evaluation, selection and report commands; preserve the original preprocessing and inference contracts.
  • Publish configs, histories, hashes, provenance, per-label scores, per-image predictions, confusion/learning/reliability figures and a small CPU runtime benchmark.
  • Generate standalone offline HTML reports, with photo thumbnails explicitly opt-in. Dataset photos and weight files stay outside Git.
  • Replace pending-result documentation with the measured study and model card. Large generated manifests/predictions are marked as generated; review decisions and summary metrics remain visible.

Validation

  • 46 local tests pass, including no-download TensorFlow round trips, fine-tuning/BatchNorm behavior, audit/grouping, bootstrap behavior, selection integrity, safe dataset acquisition, and report generation/path safety.
  • Ruff lint and formatting, local documentation links and git diff --check: pass.
  • Scientific figures rendered and visually inspected.
  • Extracted local model bundle successfully loads and predicts.
  • Linux quality and TensorFlow jobs pass for the final PR head.

This is a single-seed, within-collection baseline. It does not establish independent phone-photo performance, complete animal-identity separation, calibrated scores or unknown/non-cat rejection. The historical 88.16% notebook result remains explicitly labeled training accuracy and is not used as a comparable benchmark.

Review follow-up

Addresses all six inline findings plus the review overview's class-map, initialization, allocation, accessibility, CLI and benchmark notes. Review decisions are bound to both source-image hashes; validation metrics are bound to model/config/manifest identities before ranking. The original pre-test selection and original raw metric files are preserved, and stricter re-evaluation reproduces every number and prediction byte. See docs/results/gano-v3/review-integrity-verification.json.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Review decisions are not bound to image hashes, and validation scores are not bound to their checkpoint or manifest, risking cross-split photo leakage and incorrect model selection.

6 open issues
What changed in this PR

Adds Cat Vision’s first measured Gano v3 study, extending MobileNetV2 with dataset auditing, validation-only recipe selection and offline evaluation reports. The published results report 75.4% accuracy on 836 held-out images.

Changes:

  • Add checksum-pinned acquisition, reviewed similarity groups and group-disjoint dataset preparation.
  • Add configurable augmentation, traced fine-tuning, bootstrap intervals and offline reporting.
  • Publish experiment evidence and update documentation, regression tests, dependencies and CI.

This pull request requires changes. Critical issues remain in audit-decision freshness and validation-metric provenance, alongside the moderate issues and nits noted below.

File Reviewed changes and outstanding findings
uv.lock Locks the expanded audit and reporting dependencies.
tests/​test_selection.py Tests candidate ranking, tiebreaks and artifact checks.
tests/​test_report.py Tests report generation, escaping and image-path safeguards.
tests/​test_pipeline.py Extends fine-tuning and split-specific evaluation coverage.
tests/​test_metrics.py Tests reproducible grouped bootstrap behavior.
tests/​test_fetch.py Tests archive checksums and extraction safeguards.
tests/​test_audit.py Tests similarity reviews, grouping and audited preparation.
src/​cat_vision/​selection.py Adds validation-based selection. Critical: scores are not bound to the checkpoint/config/manifest that produced them. Moderate: the saved class mapping is not verified.
src/​cat_vision/​report.py Generates figures and offline HTML. Moderate: diagnostic plots lack accessible textual equivalents. Nit: the Tuxedo note appears for unrelated datasets.
src/​cat_vision/​pipeline.py Adds traced fine-tuning and expanded evaluation. Moderate: child initialization metadata can disagree with inherited weights. Nit: config errors omit the allowed optional keys.
src/​cat_vision/​model.py Makes horizontal-flip augmentation configurable.
src/​cat_vision/​metrics.py Implements class-stratified group-bootstrap intervals.
src/​cat_vision/​data.py Adds grouped preparation and manifest validation. Moderate: forced initial assignments can substantially oversize validation/test partitions.
src/​cat_vision/​cli.py Exposes audit, selection, reporting and new training/evaluation options. Nits: stale evaluation help and missing tests for the new CLI wiring.
src/​cat_vision/​audit.py Adds similarity screening and reviewed groups. Critical: filename-only decisions can silently reuse stale judgments after image contents change.
scripts/​fetch_gano.py Downloads and extracts the pinned archive. Moderate: incorrect downloads reach the final archive path before verification, preventing automatic retry.
README.md Presents measured results and updated usage commands.
pyproject.toml Adds audit dependencies and optional reporting dependencies.
Makefile Installs reporting dependencies for development.
docs/​verification.md Records software checks and experiment verification evidence.
docs/​results/​gano-v3/​split-counts.csv Publishes per-label partition sizes.
docs/​results/​gano-v3/​source.json Records dataset version, checksum, provenance and stated terms.
docs/​results/​gano-v3/​source-breakdown.json Summarizes test results by inferred source.
docs/​results/​gano-v3/​selection.json Records the validation-selected recipe and candidate scores.
docs/​results/​gano-v3/​runtime.json Records local CPU inference measurements.
docs/​results/​gano-v3/​review-decisions.json Publishes visual similarity-review decisions.
docs/​results/​gano-v3/​README.md Summarizes comparisons and held-out results. Nits: the confusion table truncates a tie without explanation; benchmark-only thread settings are attributed to training, and the benchmark script/invocation is not committed.
docs/​results/​gano-v3/​pretrained-source.json Records pretrained-weight origin and checksum.
docs/​results/​gano-v3/​per-label-metrics.csv Publishes label-level test scores.
docs/​results/​gano-v3/​model-provenance.json Records selected-model environment and artifact fingerprints.
docs/​results/​gano-v3/​model-config.json Preserves the selected training configuration.
docs/​results/​gano-v3/​metrics.json Publishes held-out metrics and uncertainty intervals.
docs/​results/​gano-v3/​classes.json Records the ordered output labels.
docs/​results/​gano-v3/​candidates/​partial-fine-tuning/​validation_metrics.json Publishes fine-tuning validation scores.
docs/​results/​gano-v3/​candidates/​partial-fine-tuning/​provenance.json Records candidate environment and parent-checkpoint provenance.
docs/​results/​gano-v3/​candidates/​partial-fine-tuning/​history.json Publishes structured fine-tuning history.
docs/​results/​gano-v3/​candidates/​partial-fine-tuning/​history.csv Publishes tabular fine-tuning history.
docs/​results/​gano-v3/​candidates/​partial-fine-tuning/​config.json Captures fine-tuning candidate settings.
docs/​results/​gano-v3/​candidates/​frozen-no-augmentation/​validation_metrics.json Publishes unaugmented candidate validation scores.
docs/​results/​gano-v3/​candidates/​frozen-no-augmentation/​provenance.json Records unaugmented candidate environment and checkpoint provenance.
docs/​results/​gano-v3/​candidates/​frozen-no-augmentation/​history.json Publishes structured unaugmented training history.
docs/​results/​gano-v3/​candidates/​frozen-no-augmentation/​history.csv Publishes tabular unaugmented training history.
docs/​results/​gano-v3/​candidates/​frozen-no-augmentation/​config.json Captures unaugmented candidate settings.
docs/​results/​gano-v3/​candidates/​frozen-horizontal-flip/​validation_metrics.json Publishes horizontal-flip candidate validation scores.
docs/​results/​gano-v3/​candidates/​frozen-horizontal-flip/​provenance.json Records horizontal-flip candidate environment and checkpoint provenance.
docs/​results/​gano-v3/​candidates/​frozen-horizontal-flip/​history.json Publishes structured horizontal-flip training history.
docs/​results/​gano-v3/​candidates/​frozen-horizontal-flip/​history.csv Publishes tabular horizontal-flip training history.
docs/​results/​gano-v3/​candidates/​frozen-horizontal-flip/​config.json Captures horizontal-flip candidate settings.
docs/​results/​gano-v3/​audit-summary.json Publishes aggregate audit counts and fingerprints.
docs/​model-card.md Documents measured performance, intended use and limitations.
docs/​experiments.md Provides experiment reproduction and reporting commands.
docs/​experiment-plan.md Defines fixed recipes and the evaluation protocol.
docs/​data.md Documents acquisition, auditing, grouping and leakage limitations.
docs/​architecture.md Describes audit, training, selection and reporting boundaries.
CONTRIBUTING.md Updates development installation instructions.
configs/​partial-fine-tuning.json Defines the partial fine-tuning recipe.
configs/​frozen-no-augmentation.json Defines the unaugmented frozen baseline.
configs/​frozen-horizontal-flip.json Defines the frozen horizontal-flip recipe.
CHANGELOG.md Records the measured-study milestone.
.github/​workflows/​ci.yml Installs reporting dependencies for quality checks.
.gitattributes Marks large evidence artifacts as generated.

Review effort: Max

Comment thread src/cat_vision/audit.py Resolved
Comment thread src/cat_vision/selection.py Resolved
Comment thread scripts/fetch_gano.py Resolved
Comment thread docs/results/gano-v3/README.md Resolved
Comment thread src/cat_vision/pipeline.py Resolved
Comment thread src/cat_vision/report.py Resolved
@stefisha

stefisha commented Nov 3, 2026

Copy link
Copy Markdown
Owner Author

Addressed all six inline findings in 9d9c4fb, with preserved experiment evidence and documentation in 601033a. Each inline thread has a response explaining its fix and regression coverage.

Also addressed the additional findings in the review overview:

  • Verify the saved class map during candidate selection and reject child initialization metadata that disagrees with inherited weights.
  • Repair unnecessarily oversized initial group holdouts through whole-group moves/swaps while preserving class coverage. The published Gano manifest is unchanged.
  • Provide accessible data tables for diagnostic plots and condition the Tuxedo note on the actual labels.
  • Correct evaluation help and test the new audit/prepare/train/evaluate/select/report CLI options from outside the checkout.
  • Commit the inference benchmark script and exact invocation; attribute its explicit thread settings to the benchmark rather than unrecorded historical training settings.

Verification: 46 local tests pass, Ruff lint/format and documentation-link checks pass, and both Linux CI jobs are green. The original pre-test selection record, complete split manifest, every numeric metric and every prediction-file byte are unchanged. The original metric files are archived, and the stricter re-evaluation is recorded in review-integrity-verification.json. No retraining or test-driven model change was made.

@stefisha
stefisha merged commit 5e7ab95 into main Nov 3, 2026
4 checks passed
@stefisha
stefisha deleted the feat/measured-baseline branch November 3, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants