Parquet statistics, dictionary and page-index filter fixes

[pmattione-nvidia]

Eight fixes and one new capability in the Parquet statistics, dictionary and page-index filters,
found while bringing the hybrid scan reader's row-group pruning to parity with parquet-mr on real
workloads. Four of them affect correctness: three could drop rows that match the filter, and two
read uninitialized or out-of-bounds memory.

Missing block synchronization in the dictionary page prune kernel

The kernel that decodes a dictionary page and probes it initializes the per-row-group result slots
with a strided loop across the block, then immediately has every thread begin decoding values.
Nothing separates the two, so a thread that has already moved on can write a result that another
thread of the same block then overwrites with its initial value.

Added the group.sync() between the initialization loop and the decode loop.

Empty dictionaries probed as though they held values

query_dictionaries decided whether a column chunk had a dictionary page by asking whether its hash
set occupied any slots. cuco rounds every capacity up to at least one bucket, so a chunk with no
dictionary page still has slots and failed that test. Its set was never built, so probing it reported
the literal as absent and the row group was pruned — rows that match the filter, silently dropped.

Emptiness is now read from the value count, which is zero exactly when there is no dictionary page,
and such a chunk takes the existing "skip the dictionary filter" path.

Page-level nullability statistics recorded inverted, and left uninitialized

The page statistics caster wrote false for a page whose null count equals its row count — an
entirely null page recorded as holding no nulls, the opposite of what the row-group caster records
and of what the filter expression reads. It also had no case for a null count of zero, so a page with
no nulls kept whatever byte the uninitialized value array happened to hold. The row-group caster had
a related gap: statistics carrying no null count at all left the entry uninitialized and marked
valid.

Both casters now cover all three states of the statistic and mark the entry null when the metadata
does not answer the question. The column is named all_null in the page path to match what it means,
which is true only when every value in the chunk or page is null, false when none are, and null when
only some are or the writer did not say.

Chunks holding nothing but nulls survived comparison predicates

A writer has no non-null value to compute min and max from for an entirely null chunk, so it omits
them. Every min/max comparison against a literal therefore evaluated to null, and a null verdict
keeps the chunk, so such a chunk was read for predicates that no null can satisfy. parquet-mr prunes
it.

Comparison predicates now request the nullability column and are wrapped in NOT(all_null) AND ...,
which is decisive exactly where min and max are absent. An existing test expectation moves from one
surviving row group to zero for col0 < 100 AND IS_NULL(col0), which no row can satisfy and which
now prunes the whole file.

One absent statistic disabled pruning for the rest of the expression

The statistics expression is a three-valued predicate in which null means "this metadata does not
say, keep the chunk". It was assembled with the plain logical connectives, which return null whenever
either side is null, so a single conjunct the metadata could not answer masked the verdict of every
conjunct it could — false AND unknown came out unknown and kept a chunk that one decisive conjunct
had already ruled out. The guard above made this reachable for any partly null chunk, whose
nullability statistic is unknown by construction.

The connectives are now null-aware throughout the tree, both the ones the converter introduces for
EQUAL and NOT_EQUAL and the ones the user's own expression contains, so a decisive conjunct
prunes whatever the others say. Answering "not entirely null" also takes all three states of the
nullability column rather than a plain NOT, whose null result is an unknown handed to a comparison
that is in fact decisive. This can only prune more, never less: a null in the statistics table is
missing metadata rather than a null in the data, and treating unknown as "keep" is the safe direction
either way. ParquetReaderTest.FilterNullableStats covers it over a file whose row groups take each
of the nullability statistic's three states.

Dictionary pruning gave up on files written without a page index

dictionary_pages_byte_ranges needed two things from metadata that a writer may leave out. Per-page
encoding_stats was required to prove no page fell back to a non-dictionary encoding and so holds
values the dictionary lacks, and the offset index was required to say where the dictionary page ends.
Neither is mandatory, and files that lack them — including a large share of production Parquet — got
no dictionary pruning at all.

Without encoding_stats, the chunk's encodings list is now consulted: PLAIN_DICTIONARY present
with nothing else beyond RLE and BIT_PACKED, which only ever encode levels, says every data page
was dictionary encoded. V2 chunks are still skipped, since RLE_DICTIONARY is listed for both
dictionary-encoded pages and a fallback's and only the per-page stats tell those apart. Without an
offset index, secondary_filters_byte_ranges returns a dictionary_page_range whose extent marks
the range as an upper bound on a page that may not be there; the caller caps what it spends with
dictionary_page_byte_ranges_to_read, measures the page actually read with dictionary_page_length,
and hands over a span trimmed to that page or an empty one.

Out-of-bounds decode for a chunk that claims dictionary encoding but has no dictionary page

A writer is permitted to describe a chunk as dictionary encoded and then write no dictionary page,
which the bounded ranges above make visible: what was read begins with a data page instead. The prune
kernels skip a page only when it has no values and the decompression step covers only dictionary
pages, so such a page's still-compressed bytes were decoded as dictionary values, bounded by its
uncompressed size and therefore past the end of the span.

decode_dictionary_page_headers now resets that page and clears the chunk's compressed pointer, size
and dictionary page count, leaving the chunk exactly as an empty span leaves it, so it is simply not
pruned with.

Java bindings for the dictionary page ranges

The Java API mirrored the old signature and returned dictionary page ranges as plain ByteRanges,
which cannot express a range that merely bounds a page, and so cannot be used against files without a
page index.

SecondaryFilterRanges now carries DictionaryPageRange, which pairs the byte range with its extent,
and HybridScanReader exposes capping a bounded range and measuring the page within what was read.
HybridScanReaderTest covers pruning against files written with and without a page index, including
a chunk whose dictionary page is absent.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.