Skip to content

v0.12.3

Latest

Choose a tag to compare

@github-actions github-actions released this 19 Jul 08:05
Immutable release. Only release title and notes can be modified.

Added

  • WriteOptions.withGlobalDictMaxRetainedBytes(long) makes the writer's global-dictionary retained-memory budget (default 1 GB) configurable. (4cbc12dd)

Fixed

  • VortexWriter no longer buffers a global-dictionary candidate column's raw data for the whole file, so importing a huge Parquet file (e.g. an 18.5M-row dataset) no longer exhausts the heap; a column whose retained bytes exceed a fixed budget is demoted to per-chunk encoding. (fb4b6be0)
  • A chunked List column spanning several flat chunks now decodes into a single stitched array instead of throwing a raw ClassCastException; any other unhandled dtype now fails with VortexException. (#268)
  • A chunked Utf8/Binary column with an entirely-null chunk (NullArray) no longer throws chunk is not a VarBinArray; the chunk materializes as an all-null run. (#269)
  • A chunked List column with an entirely-null chunk (NullArray) no longer throws chunk is not a ListArray; the chunk's rows become zero-length lists with out-of-band nulls. (#269)
  • Nullable low-cardinality columns now share one global dictionary across chunks instead of re-emitting a per-chunk dictionary. (5fe8b544)
  • MaskedEncodingEncoder encodes an all-valid or all-invalid validity bitmap as vortex.constant instead of a raw per-row bitmap. (ecd47ead)
  • MaskedEncodingEncoder tries vortex.sparse for a mixed-validity bitmap and keeps it over the raw bitmap when smaller. (506d036f)
  • SequenceEncodingEncoder.encodeCascade no longer lets a full-data encode exception escape when a sampled prefix looked arithmetic but the full data wasn't. (506d036f)
  • ConstantEncodingEncoder now accepts DType.Bool (previously Primitive only). (5379af66)
  • RunEndEncodingEncoder gains a vortex.runend path for Bool. (5379af66)
  • fastlanes.rle now supports DType.Bool, completing Bool coverage across all general-purpose encodings (vortex.constant, vortex.sparse, vortex.runend, fastlanes.rle). (fe6f132b)
  • Utf8/Binary columns now use cost-based encoding selection instead of first-match dispatch (Dict no longer always beats FSST regardless of size). (1bbc3549)
  • FsstEncodingDecoder no longer throws on valid files with an all-default (empty) vortex.fsst metadata segment. (1bbc3549)
  • FsstEncodingEncoder writes symbol tables in the order the Rust reference requires; previously an out-of-order table could fail to open in the JNI/Rust reader. (95e0dfb4)

Performance

  • FsstEncodingEncoder: iterative symbol-table training (variable-length 1–8 byte symbols, was a fixed 2-byte bigram scheme) and narrower metadata buffer ptypes. (95e0dfb4, 1bbc3549)
  • FsstEncodingEncoder: non-boxing symbol-match lookup and stratified training-sample selection. (4a3fa0a9)
  • VortexWriter buffers global-dictionary candidates by capped cardinality (dedup map + ~2 B/row codes) instead of raw values, so a column whose distinct set grows past the cap mid-file demotes immediately instead of accumulating unbounded memory. (62bb851e)