Add cross-language Split.serialize()#455
Conversation
XiaoHongbo-Hope
force-pushed
the
split_optimize
branch
8 times, most recently
from
a1c8b00 to
becb6aa
Compare
Let a non-Rust reader (e.g. pypaimon) rebuild its own DataSplit from a Rust-planned split and read the files directly, without re-running its own scan planning. to_dict() exposes bucket / bucket_path / total_buckets / partition / raw_convertible and, per data file, the fully-resolved file_path plus scalar metadata (schema_id, level, sequence numbers, first_row_id, write_cols, creation_time, ...) and aligned per-file deletion files. partition is the serialized BinaryRow, byte-identical to a manifest _PARTITION. Planning-only statistics (key/value stats, min/max key) are omitted since planning already happened. Tested via test_split_to_dict_exposes_fields and test_split_to_dict_partition_and_reads.
XiaoHongbo-Hope
force-pushed
the
split_optimize
branch
from
becb6aa to
2b534b3
Compare
|
Thanks @XiaoHongbo-Hope! A question on necessity first. The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a basic, opt-in path behind a config flag, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans and reads: PR3 already exposes
So: what's the use case for exposing split internals? If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward. |
Thanks, Junrui. Got your concern, will discuss with Jingsong and then back to you. |
|
@JunRuiLee Thanks Junrui. My view: the read side can be handled by adding more workers in production, but plan runs on the driver. And It's additive and opt-in — no storage-format change — and it doesn't block #413; once full-Rust read lands, this just becomes an alternate/interim path. |
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Do you need to use Python? Should we directly let AI see how to design cross language calls? For example, designing a cross language binary format for Split?
… binary) Expose a planned split as the standard Java DataSplit#serialize (version 8) binary instead of a Python-only dict, so any Paimon reader (pypaimon, Java) can rebuild it without re-planning. DataSplit::serialize() writes the v8 framing plus each DataFileMeta as a BinaryRow (nested SimpleStats, string arrays, inline-compact fields). Verified byte-identical to Paimon's compatibility/datasplit-v8 fixture (testdata/datasplit_v8.bin).
…writeUTF serialize() now errors instead of silently dropping row ranges: a split from a row-id / global-index / vector plan has no v8 representation, so serializing would widen it and read extra rows. write_java_utf now emits true Java modified UTF-8 (NUL -> C0 80, supplementary chars via surrogate pairs) with a 65535-byte length check, matching DataOutput#writeUTF, instead of raw UTF-8 that could silently corrupt or wrap.
|
Let's create an unified protocol: apache/paimon#8482 |
…h path Add serialize() to datafusion.pyi so type checkers and stub users see the new API. Add a unit test for the BinaryArray var-length (element > 7 bytes) branch, which the datasplit-v8 fixture (all elements inline) does not exercise.
Wrap DataSplit#serialize with the Java SplitSerializer frame (magic "SPLIT_V1" + version + DATA_SPLIT type id) so the bytes round-trip through the merged cross-language SplitSerializer (#8482); PySplit.serialize() now emits the framed form. Verified byte-exact against Java's split-v1-data golden, which also covers EMPTY_STATS, multiple data files, and a null-padded deletion list.
XiaoHongbo-Hope
changed the title
The stub said DataSplit v8 binary but serialize() now returns the SplitSerializer v1 frame wrapping a DataSplit v8 payload; correct the comment so downstream doesn't parse it as raw v8. Add a unit test for the BinaryArray multi-word null bitset (n > 32) path.
| /// v8 has no field for them, so silently dropping them would widen the split and read extra rows. | ||
| pub fn serialize(&self) -> crate::Result<Vec<u8>> { | ||
| if self.row_ranges.is_some() { | ||
| return Err(crate::Error::DataInvalid { |
There was a problem hiding this comment.
Maybe it is the time to create IndexedSplit.
…, supplementary-char test file_source now rejects out-of-TINYINT-range values via i8::try_from instead of silently wrapping. Add BinaryRowBuilder::build_row_data() (single alloc, self-documenting) replacing build_serialized()[4..].to_vec() in the DataFileMeta / SimpleStats row serialization. Test write_java_utf on a supplementary char (U+1F600 -> surrogate pair -> 6 bytes); note the golden files are Java-generated.
…rate testdata/ is excluded from the crate package (Cargo.toml), so the include_bytes! golden files were not published -- cargo test from the published crate would fail to compile. Move them to src/table/goldens/ (packaged, and not matched by the testdata/ exclude).
…ype 3) Instead of rejecting splits carrying row ranges, serialize them as the SplitSerializer IndexedSplit type (magic + version + DataSplit body + ranges + null scores), matching Java IndexedSplit#serialize. Verified byte-for-byte against the split-v1-indexed golden (up to its vector-scores tail, which the Rust split does not model).
3 participants
Purpose
Add Split.to_dict() to the Python bindings so a non-Rust reader (e.g. pypaimon) can rebuild its own split from a Rust-planned split and read the files directly — planning runs in Rust (the serial, driver-side bottleneck), reading stays in the existing reader, no re-planning.
Brief change log
PySplit::to_dict() exposes a planned split as a plain dict — bucket / paths / partition plus per-file metadata (file_path, schema_id, first_row_id, write_cols, …) and deletion files. Planning-only stats omitted.
Tests
test_split_to_dict_exposes_fields, test_split_to_dict_partition_and_reads.API and Format
Additive — new Split.to_dict() Python method. No storage-format change.
Documentation
Covered by the method docstring.