Tell the compressor which serialized IDs the writer may emit - #9770
Tell the compressor which serialized IDs the writer may emit#9770mhk197 wants to merge 5 commits into
Conversation
Merging this PR will improve performance by 11.91%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
575.7 ns | 521.6 ns | +10.39% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/compressor-serialized-ids (94ab744) with develop (85b70cf)2
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
develop(5c8c397) during the generation of this report, so 85b70cf was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
CascadingCompressor carries the snapshot of serialized IDs the writer may emit, filled by the file writer from the enabled editions through BtrBlocksCompressorBuilder::allow_serialized_ids. A scheme whose encoding has more than one wire format picks its compression mode from it with allows_serialized_id, the newest permitted one; without a restriction every ID is allowed. No scheme consults the set yet. This is the mechanism docs/specs/editions.md describes under compression with replacement encodings (#9779). Signed-off-by: Matt Katz <mhkatz97@gmail.com>
85990c2 to
a49c384
Compare
| .with_btrblocks_builder( | ||
| BtrBlocksCompressorBuilder::default() | ||
| .retain_allowed_encodings(&allowed_array_encodings), | ||
| .retain_allowed_encodings(&allowed_array_encodings) |
There was a problem hiding this comment.
I think this has to be serialised ids
There was a problem hiding this comment.
I think we should somehow unify these 2 functions
There was a problem hiding this comment.
I guess we can enable a scheme if at least one of associated serialized ids is enabled?
There was a problem hiding this comment.
Ah actually I think this needs to be separate since there's a diff bw what ids MUST be allowed for a scheme to proc (all) vs what is optional (at least one).
FSST for example requires VarBin and FSST to be enabled. DBP requires at least one of v1 or v2.
Scheme::produced_encodings now names the serialized IDs a scheme may write its output under, oldest first. BtrBlocksCompressorBuilder::allow_serialized_ids replaces retain_allowed_encodings: it keeps a scheme when at least one of those IDs is permitted and hands the set to the compressor, so the writer makes one call from the serialized IDs its editions permit instead of mapping them back to in-memory encodings, which could not tell two wire formats of one encoding apart. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
The compressor seeds each root CompressorContext with its permitted serialized IDs and every descent inherits them, so a scheme asks compress_ctx.allows_serialized_id both while estimating and while compressing and picks the same mode in both. The per-compressor accessor goes; allowed_serialized_ids remains for inspection. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
|
Do you have an example of this being used? |
f5b732f to
c27a6d6
Compare
…ed ids Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Restore produced_encodings in feature-gated schemes and fix stale trait links. Document scheme ID resolution for Clippy. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
3 participants
Filtering compression schemes by in-memory encoding cannot distinguish multiple wire formats of the same encoding. The writer now passes its permitted serialized IDs to compressor configuration, which selects the newest compatible version of each scheme before matching, generating statistics, estimating, or compressing.
Scheme::produced_encodingsdeclares the serialized IDs that a scheme directly introduces; every declared ID must be allowed. This preserves requirements such as FSST needing bothvortex.fsstandvortex.varbin. Alternative versions are represented byScheme::predecessor: register only the newest version, fall back through its predecessor chain when its IDs are unavailable, and remove the chain when no version is eligible. Historical versions do not compete with their replacements during compression.BtrBlocksCompressorBuilder::allow_serialized_idsreplacesretain_allowed_encodingsand applies restrictions at build time, including to schemes added after the restriction. Both it andCascadingCompressor::with_allowed_serialized_idsintersect repeated restrictions. Registration order is preserved, and scheme IDs in a predecessor chain resolve to the selected version for exclusions andhas_schemechecks. Cycles and overlapping chains are rejected.The default file writer uses the serialized IDs of its enabled editions, or every registered serialized ID when edition enforcement is disabled. Custom write strategies retain their own configuration. Serialization still chooses the oldest wire form the resulting array fits and validates it against the writer's allowed IDs.
Existing production schemes have no predecessors, so this prepares version selection for consumers such as bitpacking (#9754) and decimal byte parts (#9759). The related edition specification is discussed in #9779.
Tests cover version selection before estimation, required output IDs, repeated restrictions, schemes added after restrictions, registration order, exclusions across versions, invalid predecessor chains, and writer round trips with unavailable encodings.
Validation:
cargo check -p vortex-btrblocks --all-features: passed.cargo nextest run -p vortex-compressor -p vortex-btrblocks -p vortex-file --all-features: 274 passed, 1 skipped.cargo test --doc -p vortex-compressor -p vortex-btrblocks --all-features: 4 passed.RUSTDOCFLAGS='-D warnings' cargo doc -p vortex-compressor -p vortex-btrblocks --all-features --no-deps: passed.cargo clippy -p vortex-compressor -p vortex-btrblocks --all-targets --all-features: passed without warnings.cargo +nightly fmt --all: passed.