Skip to content

benchmarks.md

Latest commit

 

History

History
91 lines (61 loc) · 3.17 KB

benchmarks.md

File metadata and controls

91 lines (61 loc) · 3.17 KB
Raw

Benchmark Guide

The repo has two benchmark layers:

  1. PyTorch checkpoint evaluation through models.training.evaluation
  2. Desktop enhancement benchmarks through benchmarks/scripts/* and tools/*

Recommended benchmark flow

1. Evaluate a checkpoint directly

python -m models.training.evaluation --pairs_csv datasets/manifests/powerpoint_raw_x2/val_pairs.csv --checkpoint models/span/education-finetuned/x2/best_model.pt --output_csv benchmarks/results/x2-eval.csv --output_summary_json benchmarks/results/x2-eval-summary.json

2. Benchmark desktop image quality

python benchmarks/scripts/benchmark_quality.py --reference_dir datasets/benchmark/custom-test-set/HR --input_dir datasets/benchmark/custom-test-set/LR_bicx2 --output_dir outputs/benchmarks/quality --quality_csv benchmarks/results/quality-metrics.csv --with_ocr

3. Benchmark desktop processing latency

python benchmarks/scripts/benchmark_processing_time.py --input_dir datasets/benchmark/custom-test-set/LR_bicx2 --output_dir outputs/benchmarks/processing --output_csv benchmarks/results/processing-times.csv

4. Benchmark a video path

python benchmarks/scripts/benchmark_video_quality.py --reference_video datasets/benchmark/video/reference.mp4 --input_video datasets/benchmark/video/input.mp4 --output_dir outputs/benchmarks/video --output_csv benchmarks/results/video-quality-metrics.csv

5. Initialize manual device CSVs

python benchmarks/scripts/init_device_benchmarks.py

This creates headers for:

  • battery-consumption.csv
  • gpu-performance.csv
  • thermal.csv

Desktop enhancement runner

The benchmark scripts above delegate actual image or video enhancement to:

tools/benchmark-runner/runner.py

That runner supports presets and emits per-file timing CSVs into the output directory.

Reporting and plots

Generate summary reports from benchmarks/results/:

python tools/reporting/report_generator.py --results_dir benchmarks/results --output_format markdown --output benchmarks/results/report.md

The repo also includes analysis helpers under benchmarks/analysis/:

  • calculate_metrics.py
  • compare_results.py
  • generate_plots.py
  • analyze_video_quality.py

Important output files

File What it contains
benchmarks/results/quality-metrics.csv image quality summary from desktop runner
benchmarks/results/processing-times.csv aggregate runtime summary
benchmarks/results/video-quality-metrics.csv video quality summary
benchmarks/results/*summary.json OCR or heldout summary files
outputs/benchmarks/*/inference-times.csv raw per-file timings from the desktop runner

Existing benchmark assets in the repo

The benchmarks/results/ folder already contains evaluation artifacts for multiple x2 and x3 runs, including OCR comparison CSV and JSON files. Use them as examples for naming and output format.

Related Docs