Statistices nested columns - #9775
Conversation
Store file-level statistics via a post-order walk of the DType tree, so nested struct fields get whole-file pruning instead of only top-level columns. Each leaf gets a stats entry, and each nullable struct gets a trailing null-count entry. A new `is_nested` footer flag keeps old files readable via the legacy top-level-only layout. Signed-off-by: "Thor" <thor.hansen@dash0.com>
Migrate Min/Max/Sum/NullCount/NaNCount/UncompressedSizeInBytes onto the same persistent AggregateFnRef/Accumulator framework the zoned layout's zone-map builder uses, removing the duplicate Stat-enum-keyed accumulation and its now-dead StatNameArrayBuilder. Utf8/Binary min/max truncation stays on the bespoke Precision-tracked path, since it needs per-value exactness that BoundedMax/BoundedMin can't express. Signed-off-by: "Thor" <thor.hansen@dash0.com>
Cover arbitrary-depth nesting, not just the single level exercised by the existing tests: postorder layout with mixed nullability across three levels, StatsNode accumulation contributing a null-count entry at each nullable level, and whole-file pruning resolving a three-level field path (x.y.z). Signed-off-by: "Thor" <thor.hansen@dash0.com>
Merging this PR will improve performance by 39.37%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.5 µs | 12.8 µs | +59.85% |
| ⚡ | Simulation | decompress[u64, (4000, 1024)] |
85.5 µs | 70.4 µs | +21.51% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing polarsignals:statistices-nested-columns (d29bb2a) with develop (a997c3f)
Footnotes
-
1471 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
I think for maximum compatibility this has to be a new top level field in https://github.com/vortex-data/vortex/pull/9775/changes#diff-9c5a36ce8f55480706984e8996f192cd3abbdd1b97afa6f702f646b3ce42e039L25. The problem you run into is that old readers already shipped code that make assumption about existing footer statistics. One issue is that they validate field count and if the count doesn't add up they will assume it's a corrupted file. The writer could optionally populate both of the segments and the new version will only ever read the new field. If you go that route you will see all the places that make assumptions about field stats that need fixing, i.e. datafusion, duckdb and jni |
2 participants
Summary
Being able to perform file level pruning on nested columns speeds up queries.
Changes
Adds post order stats to the file stats footer for nested columns. It adds a new
is_nested: bool = falseto the stats set flatbuffer for backward compatibility.Additionally it modifies the way the stats footer is accumulated to use the same AggRef functions that the zone maps use as well.
Only goofy thing it it also has to track truncatable stats separately because the aggregate partials don't have a way to express exactness.