Immutable
release. Only release title and notes can be modified.
Added
WriteOptions.withGlobalDictMaxRetainedBytes(long)makes the writer's global-dictionary retained-memory budget (default 1 GB) configurable. (4cbc12dd)
Fixed
VortexWriterno longer buffers a global-dictionary candidate column's raw data for the whole file, so importing a huge Parquet file (e.g. an 18.5M-row dataset) no longer exhausts the heap; a column whose retained bytes exceed a fixed budget is demoted to per-chunk encoding. (fb4b6be0)- A chunked
Listcolumn spanning several flat chunks now decodes into a single stitched array instead of throwing a rawClassCastException; any other unhandled dtype now fails withVortexException. (#268) - A chunked
Utf8/Binarycolumn with an entirely-null chunk (NullArray) no longer throwschunk is not a VarBinArray; the chunk materializes as an all-null run. (#269) - A chunked
Listcolumn with an entirely-null chunk (NullArray) no longer throwschunk is not a ListArray; the chunk's rows become zero-length lists with out-of-band nulls. (#269) - Nullable low-cardinality columns now share one global dictionary across chunks instead of re-emitting a per-chunk dictionary. (5fe8b544)
MaskedEncodingEncoderencodes an all-valid or all-invalid validity bitmap asvortex.constantinstead of a raw per-row bitmap. (ecd47ead)MaskedEncodingEncodertriesvortex.sparsefor a mixed-validity bitmap and keeps it over the raw bitmap when smaller. (506d036f)SequenceEncodingEncoder.encodeCascadeno longer lets a full-data encode exception escape when a sampled prefix looked arithmetic but the full data wasn't. (506d036f)ConstantEncodingEncodernow acceptsDType.Bool(previouslyPrimitiveonly). (5379af66)RunEndEncodingEncodergains avortex.runendpath for Bool. (5379af66)fastlanes.rlenow supportsDType.Bool, completing Bool coverage across all general-purpose encodings (vortex.constant,vortex.sparse,vortex.runend,fastlanes.rle). (fe6f132b)Utf8/Binarycolumns now use cost-based encoding selection instead of first-match dispatch (Dict no longer always beats FSST regardless of size). (1bbc3549)FsstEncodingDecoderno longer throws on valid files with an all-default (empty)vortex.fsstmetadata segment. (1bbc3549)FsstEncodingEncoderwrites symbol tables in the order the Rust reference requires; previously an out-of-order table could fail to open in the JNI/Rust reader. (95e0dfb4)
Performance
FsstEncodingEncoder: iterative symbol-table training (variable-length 1–8 byte symbols, was a fixed 2-byte bigram scheme) and narrower metadata buffer ptypes. (95e0dfb4, 1bbc3549)FsstEncodingEncoder: non-boxing symbol-match lookup and stratified training-sample selection. (4a3fa0a9)VortexWriterbuffers global-dictionary candidates by capped cardinality (dedup map + ~2 B/row codes) instead of raw values, so a column whose distinct set grows past the cap mid-file demotes immediately instead of accumulating unbounded memory. (62bb851e)