Skip to content

Fix exports from combined Lance datasets - #73

Merged
bghira merged 1 commit into
mainfrom
fix/export-logical-lance-shards
Sep 7, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/export-logical-lance-shards

Conversation

@bghira

@bghira bghira commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

CaptionFlow stores logical shard names in the combined Lance dataset shard column, but the exporter still enumerated the legacy physical default shard. This caused WebShart exports to target default.json and export zero rows.

This change:

  • discovers logical shard names from Lance
  • filters export queries by logical shard with parameterized SQL
  • preserves shard filters
  • adds regression coverage for full and filtered WebShart exports

Validation:

  • 494 tests passed
  • 100% changed-line coverage

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/caption_flow/storage/exporter.py 65.26% <100.00%> (+0.35%) ⬆️
src/caption_flow/storage/manager.py 74.22% <100.00%> (+0.45%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bghira
bghira merged commit 277b82e into main Sep 7, 2026
4 checks passed
@bghira
bghira deleted the fix/export-logical-lance-shards branch September 7, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant