Add Spark interop test for reading Puffin deletion vectors - #3476
Add Spark interop test for reading Puffin deletion vectors#3476moomindani wants to merge 3 commits into
Conversation
ebyhr
left a comment
ebyhr
left a comment
There was a problem hiding this comment.
Looks good to me except for comments.
- Move test_read_spark_written_puffin_dv from test_puffin_spark_interop.py to test_deletes.py; delete the old file - Replace run_spark_commands helper calls with direct spark.sql() calls - Tighten entries assertion from > 0 to == 1
Co-authored-by: Isaac
rambleraptor
left a comment
rambleraptor
left a comment
There was a problem hiding this comment.
This looks great. Thanks for doing this!
|
Hi @ebyhr, thanks for the review! I've addressed all your comments (moved the test to |
|
Gentle reminder — this has two approvals and CI is green. Could a committer help merge this when convenient? |
|
Following up on the reminder from three weeks ago — this is waiting on a committer rather than on further review. It's test-only: one file ( @Fokko this test covers the DV read path you added in #1516 — would you mind taking a look and merging if it looks good? Happy to rebase first if you'd prefer. |
3 participants
Rationale for this change
Extracted from #3474 per review feedback (#3474 (comment)): this integration test verifies that PyIceberg can read Puffin deletion vectors written by Spark, which holds independently of the
PuffinWriterchanges in that PR.Are these changes tested?
This PR is test-only. The test passed CI as part of #3474 (integration-test job) before being extracted.
Are there any user-facing changes?
No.