Skip to content

Add effect/encoding/Compression - #10657

Open
effect-bot wants to merge 15 commits into
mainfrom
feat/encoding-compression
Open

effect-bot wants to merge 15 commits into
mainfrom
feat/encoding-compression

Conversation

@effect-bot

@effect-bot effect-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Stack

This PR is part of the coordinated data-processing effort, EFF-3804. It's an independent foundation based on main.

Closes EFF-3821

Why

Parquet pages and Arrow IPC buffers are compressed with Snappy, gzip, LZ4 or Zstandard. This module puts those codecs in one place so format modules don't each carry their own copy.

The runtime's DecompressionStream doesn't help here:

  • It only covers gzip and deflate everywhere, plus brotli and zstd on some runtimes.
  • It's async and streaming, while a page is decoded as one unit inside a synchronous reader.

Summary

effect/encoding/Compression
  decompressSnappy    (input, options) -> Result<Uint8Array, CompressionError>   raw block format (Parquet SNAPPY)
  decompressGzip      (input, options) -> ...   RFC 1952, multi-member (Parquet GZIP)
  decompressLz4Block  (input, options) -> ...   LZ4 block (Parquet LZ4_RAW)
  decompressLz4Frame  (input, options) -> ...   LZ4 frame (Arrow LZ4_FRAME)
  decompressZstd      (input, options) -> ...   RFC 8878, no dictionaries (Parquet and Arrow ZSTD)
  DecompressOptions   { maxOutputSize: number; expectedSize?: number }
  CompressionError    { codec: "Snappy" | "Gzip" | "Lz4Block" | "Lz4Frame" | "Zstd",
                        kind: Malformed | Truncated | Checksum | SizeMismatch | OutputLimit | Unsupported, message }
  isCompressionError
  • Pure TypeScript, synchronous, decode only. Every function returns a new buffer that doesn't share memory with the input.
  • No default limit. maxOutputSize is required. It bounds the decompressed output, not total memory: gzip's inflater uses a fixed 96 KiB workspace and Zstandard up to 128 KiB of literals plus tables, whatever the limit, and a host allocation failure is not turned into a CompressionError.
  • expectedSize is the exact size, when the container records it. Any other size fails with SizeMismatch as soon as the output passes it, and the output is allocated once. If expectedSize is larger than maxOutputSize, the call fails with OutputLimit before decoding.
  • Sizes stored in the stream are checked before allocating. That covers the Snappy preamble, the zstd Frame_Content_Size and the LZ4 frame content size.
  • Option values must be non-negative safe integers. Otherwise the call throws a RangeError, because a bad option is a caller bug rather than bad data.
  • Strict decoding:
    • Checksums are always verified when present.
    • Bytes left after the end of the data are Malformed.
    • A size stored in the stream that disagrees with the output is Malformed.
  • Shared code with Zip:
    • gzip reuses the raw deflate decoder in internal/inflate.ts that effect/encoding/Zip uses.
    • CRC-32 moved from Zip.ts to internal/crc32.ts, so both modules share one copy. Zip's behaviour is unchanged and its 25 tests pass.
  • Per-codec functions, not a decompress(codec, …) dispatcher. An Arrow reader that only needs LZ4 and Zstd doesn't bundle Snappy and gzip.
  • Experimental, @since 4.1.5.

Agreed with the Parquet reader (EFF-3820)

  • Parquet maps its CompressionCodec enum to these functions and handles UNCOMPRESSED inline.
  • BROTLI, LZO and the deprecated Hadoop LZ4 fail with Parquet's own Unsupported error.
  • Parquet passes expectedSize from the page header. For DataPageV2 that's minus the level bytes. Parquet validates sizes taken from headers before calling.
  • Parquet maps CompressionError into its own error and keeps the original as cause.

Spec-assertion tests

These follow the MCP conformance layout (test/ai/McpServer/McpConformance/):

  • test/encoding/Compression.test.ts runs one suite file per format from test/encoding/CompressionConformance/.
  • Each suite file starts with the pinned spec and a Not covered list. That list covers unsupported, deferred and not-applicable assertions, and decoder policy that isn't a spec requirement.
  • Each describe or it cites its section, and test names state the assertion.
  • Resource limits are in their own "Resource limits (local policy)" blocks.
  • Fixtures from reference encoders are interoperability evidence alongside the assertion tests. They don't replace them. Each one is generated by a pinned generate.py under test/fixtures/compression/<codec>/, about 100 KB in total.
Suite Spec (pinned) Tests Reference used for fixtures
Snappy google/snappy format_description.txt @ 1.2.2 21 cramjam 2.11.0
Lz4 lz4 lz4_Block_format.md and lz4_Frame_format.md (1.6.4) @ v1.10.0 47 python-lz4 4.4.4 (liblz4 1.9.4)
Gzip RFC 1952 (and RFC 1951 block types) 31 CPython 3.13 zlib 1.3.2, runtime CompressionStream("gzip")
Zstd RFC 8878 81 python-zstandard 0.25.0 (libzstd 1.5.7)
Shared options (API contract) 2

Not covered (summary; the full lists are in each suite header):

  • Unsupported features:
    • Zstd dictionaries, which fail with Unsupported.
    • LZ4 dictionaries and the LZ4 legacy frame, both Unsupported.
    • The Snappy framing format.
  • Untestable: gzip ISIZE wrap-around, which needs more than 4 GiB of output.
  • Zstd compatibility exceptions (accepted to match libzstd 1.5.7, tested in their own "Compatibility exceptions (not RFC 8878 conformance)" block, decision pending on EFF-3821):
    • a match offset equal to Window_Size, which §3.1.1.4 excludes ("must be smaller than") but libzstd's encoder emits;
    • a sequences FSE table with a single symbol, which §4.1.1 and §3.1.1.3.2.1 exclude. A single-symbol Huffman weight table still fails.
  • Encoder-only requirements: excluded throughout.
  • Decoder choices that go beyond the spec:
    • Checksums are always verified.
    • Trailing bytes are rejected.
    • Empty input is Truncated.
    • LZ4 end-of-block rules 2 and 3 are accepted. The spec allows rejecting them but doesn't require it.
    • Offsets beyond the window are rejected, as RFC 8878 requires (libzstd doesn't check).

Evidence

  • Interoperability at scale (one-off checks, not committed): 5 to 7 MB of mixed data was compressed by each reference library and decoded byte-identical here:
    • 45 zstd files at levels -5 to 22, with long-distance matching, without content size, and multithreaded;
    • gzip at levels 1, 6 and 9, and multi-member;
    • LZ4 frames, linked and independent, with checksums;
    • LZ4 blocks;
    • Snappy.
  • libzstd agreement: libzstd decodes all 93 valid hand-crafted zstd frames to the same output, and rejects the malformed ones. The two exceptions are documented in the Zstd suite header.
  • Fuzzing: 45,000 randomly mutated Snappy, LZ4 and Zstd fixtures produced only CompressionError values. Nothing threw, and the slowest input took 28 ms.
  • Rough throughput (noisy machine, MB/s of output):
Codec MB/s
Snappy 190–210
LZ4 block 245, or 466 with expectedSize
LZ4 frame 300–350
gzip 140–230 (node:zlib does 310–450)
zstd 180–215
pnpm vitest --run test/encoding/Compression.test.ts      192 passed
pnpm vitest --run packages/effect/test/encoding/Zip.test.ts   25 passed
pnpm doctest --run packages/effect/src/encoding/Compression.ts   1 passed
pnpm check, pnpm lint, pnpm jsdocs --check                clean

Open decisions (not settled by this PR)

  • Arrow IPC compressed bodies: the decode boundary, the default resource limits and the error mapping are still waiting on user approval. This PR adds no Arrow code and assumes none of those policies.
  • Default limits: none here. Each consumer passes maxOutputSize, and its defaults are its own decision.
  • Error mapping: each format maps CompressionError into its own error. Parquet's mapping is on EFF-3820.
  • Codecs left out:
    • Brotli, which needs a ~122 KB static dictionary.
    • LZO, and Parquet's deprecated Hadoop-framed LZ4.
    • Zstd dictionaries.
    • Compression (writing), which no consumer needs yet. A Parquet or Arrow writer would need it.

Status

Review round 1 (EFF-3821): Zstd now rejects a Huffman weight stream that lacks the bits of its two initial FSE states (RFC 8878 §4.1, §4.2.1.2), with a valid control and two malformed cases; the resource wording is narrowed. Ready for re-review, then a simplification pass.

@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3110798

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-janitor effect-janitor Bot added the 4.0 label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 40.09 KB 40.09 KB 0.00 KB (0.00%)
basic.ts 7.31 KB 7.31 KB 0.00 KB (0.00%)
batching.ts 11.21 KB 11.21 KB 0.00 KB (0.00%)
brand.ts 7.00 KB 7.00 KB 0.00 KB (0.00%)
cache.ts 11.73 KB 11.73 KB 0.00 KB (0.00%)
config.ts 23.09 KB 23.09 KB 0.00 KB (0.00%)
differ.ts 22.20 KB 22.20 KB 0.00 KB (0.00%)
http-client.ts 27.45 KB 27.45 KB 0.00 KB (0.00%)
http-router.ts 37.56 KB 37.56 KB 0.00 KB (0.00%)
logger.ts 12.19 KB 12.19 KB 0.00 KB (0.00%)
metric.ts 9.65 KB 9.65 KB 0.00 KB (0.00%)
optic.ts 7.21 KB 7.21 KB 0.00 KB (0.00%)
pubsub.ts 16.21 KB 16.21 KB 0.00 KB (0.00%)
queue.ts 13.09 KB 13.09 KB 0.00 KB (0.00%)
schedule.ts 11.99 KB 11.99 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.29 KB 14.29 KB 0.00 KB (0.00%)
schema-binary.ts 42.45 KB 42.45 KB 0.00 KB (0.00%)
schema-class.ts 21.96 KB 21.96 KB 0.00 KB (0.00%)
schema-codegen.ts 30.70 KB 30.70 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.56 KB 30.56 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.26 KB 28.26 KB 0.00 KB (0.00%)
schema-string-transformation.ts 15.54 KB 15.54 KB 0.00 KB (0.00%)
schema-string.ts 12.50 KB 12.50 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.43 KB 16.43 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.63 KB 39.63 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 21.11 KB 21.11 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.28 KB 21.28 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.37 KB 21.37 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.42 KB 26.42 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.45 KB 21.45 KB 0.00 KB (0.00%)
schema.ts 21.05 KB 21.05 KB 0.00 KB (0.00%)
stm.ts 14.40 KB 14.40 KB 0.00 KB (0.00%)
stream.ts 10.56 KB 10.56 KB 0.00 KB (0.00%)

@effect-bot

Copy link
Copy Markdown
Contributor Author

Heads-up on the stack, nothing to do here. The Parquet reader (#10655) moved onto the Arrow stack, and #10666 adds ArrowParquet, which reads Parquet files as Arrow record batches. Both keep the boundary agreed here. They have no codecs of their own and pass the codecs option (Parquet.Decompress per codec) through, so this PR stays independent of them and based on main. internal/crc32.ts on the Parquet branch is still byte-identical to the file in this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant