Repository navigation
Add effect/encoding/Compression - #10657
effect-bot wants to merge 15 commits into
Conversation
🦋 Changeset detectedLatest commit: 3110798 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
Separate the two libzstd compatibility exceptions from RFC 8878 assertions, and document maxOutputSize as an output bound rather than a memory bound.
…ight-state branches
… conformance suite
|
Heads-up on the stack, nothing to do here. The Parquet reader (#10655) moved onto the Arrow stack, and #10666 adds |
Stack
This PR is part of the coordinated data-processing effort, EFF-3804. It's an independent foundation based on
main.effect/encoding/FlatBufferseffect/arroweffect/dataframemain:effect/encoding/Thrifteffect/encoding/CompressionLZ4_FRAME,ZSTD) are a later layer on Arrow and this module. That layer still needs user approval, see "Open decisions".Closes EFF-3821
Why
Parquet pages and Arrow IPC buffers are compressed with Snappy, gzip, LZ4 or Zstandard. This module puts those codecs in one place so format modules don't each carry their own copy.
The runtime's
DecompressionStreamdoesn't help here:Summary
maxOutputSizeis required. It bounds the decompressed output, not total memory: gzip's inflater uses a fixed 96 KiB workspace and Zstandard up to 128 KiB of literals plus tables, whatever the limit, and a host allocation failure is not turned into aCompressionError.expectedSizeis the exact size, when the container records it. Any other size fails withSizeMismatchas soon as the output passes it, and the output is allocated once. IfexpectedSizeis larger thanmaxOutputSize, the call fails withOutputLimitbefore decoding.Frame_Content_Sizeand the LZ4 frame content size.RangeError, because a bad option is a caller bug rather than bad data.Malformed.Malformed.internal/inflate.tsthateffect/encoding/Zipuses.Zip.tstointernal/crc32.ts, so both modules share one copy. Zip's behaviour is unchanged and its 25 tests pass.decompress(codec, …)dispatcher. An Arrow reader that only needs LZ4 and Zstd doesn't bundle Snappy and gzip.@since 4.1.5.Agreed with the Parquet reader (EFF-3820)
CompressionCodecenum to these functions and handlesUNCOMPRESSEDinline.BROTLI,LZOand the deprecated HadoopLZ4fail with Parquet's own Unsupported error.expectedSizefrom the page header. For DataPageV2 that's minus the level bytes. Parquet validates sizes taken from headers before calling.CompressionErrorinto its own error and keeps the original ascause.Spec-assertion tests
These follow the MCP conformance layout (
test/ai/McpServer/McpConformance/):test/encoding/Compression.test.tsruns one suite file per format fromtest/encoding/CompressionConformance/.describeoritcites its section, and test names state the assertion.generate.pyundertest/fixtures/compression/<codec>/, about 100 KB in total.format_description.txt@ 1.2.2lz4_Block_format.mdandlz4_Frame_format.md(1.6.4) @ v1.10.0CompressionStream("gzip")Not covered (summary; the full lists are in each suite header):
Unsupported.Unsupported.Window_Size, which §3.1.1.4 excludes ("must be smaller than") but libzstd's encoder emits;Truncated.Evidence
CompressionErrorvalues. Nothing threw, and the slowest input took 28 ms.expectedSizenode:zlibdoes 310–450)Open decisions (not settled by this PR)
maxOutputSize, and its defaults are its own decision.CompressionErrorinto its own error. Parquet's mapping is on EFF-3820.LZ4.Status
Review round 1 (EFF-3821): Zstd now rejects a Huffman weight stream that lacks the bits of its two initial FSE states (RFC 8878 §4.1, §4.2.1.2), with a valid control and two malformed cases; the resource wording is narrowed. Ready for re-review, then a simplification pass.