Skip to content

Add effect/encoding/Parquet - #10655

Open
effect-bot wants to merge 35 commits into
feat/arrow-supportfrom
feat/encoding-parquet
Open

effect-bot wants to merge 35 commits into
feat/arrow-supportfrom
feat/encoding-parquet

Conversation

@effect-bot

@effect-bot effect-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds effect/encoding/Parquet, an experimental reader for Apache Parquet files. It reads the footer metadata and one column chunk at a time, into buffers laid out the way Arrow lays out arrays. This is part of the data-processing work in EFF-3804.

All Parquet metadata, page headers included, goes through the compact-protocol reader of #10652 (effect/encoding/Thrift).

Stack

main
├─ #10643 effect/encoding/FlatBuffers
│   └─ #10649 effect/arrow
│       ├─ #10641 effect/dataframe
│       └─ this PR: effect/encoding/Parquet (carries #10652 by merge)
│           └─ #10666 ArrowParquet, Parquet files as Arrow record batches
├─ #10652 effect/encoding/Thrift
└─ #10657 effect/encoding/Compression

This PR is now part of the Arrow stack. Its base is feat/arrow-support (it was feat/encoding-thrift), and the branch has feat/arrow-support merged in, so that #10666 can build Arrow record batches on top of both. No commit was rewritten. The Parquet module itself still has no Arrow dependency.

Reading the diff: until #10652 merges, the diff against feat/arrow-support also shows the Thrift files of #10652 (Thrift.ts, its test and test/fixtures/thrift) and a few files from a newer main that the Arrow chain doesn't have yet (packages/ai/openai, LayerRef). Neither is part of this PR. The Parquet change is Parquet.ts, internal/parquet/encodings.ts, internal/crc32.ts, Zip.ts, the barrel, Parquet.test.ts and test/fixtures/parquet.

Merge order: #10643, #10649 and #10652 first, then this PR, then #10666. #10657 is independent.

API

  • readMetadata(source, options?) returns the FileMetaData: the schema tree, one ColumnDescriptor per leaf (path, physical type, max levels, column order), the row groups, statistics and key/value metadata.
  • readColumn(source, metadata, { rowGroup, column, codecs?, ...limits }) returns a Column. It has one Layer per schema node on the path (validity bitmap for optional nodes, offsets for repeated ones) and the leaf Values with a slot for every null. That way the Arrow adapter in Add ArrowParquet to effect/arrow #10666 mostly wraps buffers.
  • Source is { size, read(offset, length) } returning an Effect, so in-memory bytes (fromBytes), files and HTTP range reads all work the same.
  • ParquetError has kinds Malformed, Unsupported, Limit and Codec. Bad options and out-of-range indexes are defects (Usage), like Thrift.

Supported scope (apache-parquet-format 2.14.0)

  • All eight physical types. Logical and converted types are decoded into the schema but not applied to values. A logical type that is unknown, or not defined for its physical type, is ignored, and so is the converted type next to it (LogicalTypes.md: converted types apply only when no logical type is present, and an ignored one leaves the physical type). A BYTE_ARRAY with an unknown logical type and UTF8 is read as bytes, not checked as UTF-8.
  • Negative num_rows on the file or a row group is Malformed.
  • CRCs are checked on the pages read to decode a chunk's values, skipped pages included. Pages after the last value, and the pages of a chunk with no values, are not read, so they aren't checked.
  • PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE booleans, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and BYTE_STREAM_SPLIT. Levels can be RLE or the deprecated BIT_PACKED.
  • Data pages v1 and v2 and dictionary pages. Index pages and unknown page types are skipped.
  • Any nesting of optional, required and repeated nodes.

The following fail with Unsupported: encrypted footers and columns, ALP, chunks in other files (file_path), and any compressed page without a matching Decompress.

Compression boundary

The module has no codecs. codecs maps a Parquet codec name to a Decompress function ((data, uncompressedSize) => Effect<Uint8Array, unknown>), and UNCOMPRESSED is built in. The shared decoders are planned as effect/encoding/Compression (EFF-3821, #10657). Wiring them in, and choosing defaults, is a follow-up. The boundary agreed there is already in place: page sizes are checked to be non-negative, at most maxPageSize, and consistent with the v2 level lengths before a codec is called. A codec failure becomes kind: "Codec", with the column path and page offset in the message and the original error as cause.

Untrusted input

Every offset and length is checked against the bytes that are actually there. maxFooterLength, maxDepth (at most 255, so levels fit a byte), maxPageSize, maxValues and maxBinaryLength cap allocations before they happen. maxBinaryLength also counts fixed-width (FIXED_LEN_BYTE_ARRAY, INT96) values, nulls included, and maxValues (default 2^24) also caps each page, dictionary pages included. Memory and time still grow with values times nesting depth, which the readColumn docs call out. These limits are kept separate from the spec checks.

Tests

Parquet.test.ts has 82 tests. PQ-LT-3, PQ-LT-5 and PQ-LT-12 cover the ignored and absent logical type next to a converted type, and PQ-META-3 covers negative row counts. pnpm check, pnpm lint and pnpm jsdocs --check pass. At c06994ca17, pnpm vitest --run over test/arrow and test/encoding runs 862 tests in 23 files.

  • packages/effect/test/encoding/Parquet.test.ts has one conformance test per spec assertion, named by id (PQ-FILE-1 and so on), on synthetic files built with the Thrift writer. test/fixtures/parquet/CONFORMANCE.md maps each id to the spec at the pinned tag. It also lists what is unsupported, what is deferred (page index, bloom filters, statistics semantics, logical type application) and what only applies to writers. This is not a full-conformance claim.
  • Nine pyarrow 26.0.1 fixtures (v1/v2, dictionary, every encoding, nested lists/maps/structs, GZIP, empty) are read back and compared with gold.json. They show interoperability and are not a substitute for the assertion tests.
  • A separate hostile input block covers limits and malformed input.

Closes EFF-3820

tim-smart and others added 19 commits November 15, 2026 03:36
Reads and writes the Apache Thrift compact protocol without an IDL.
Reader is a TProtocol-shaped cursor: readFieldBegin/readListBegin/
readMapBegin, scalar reads, skip(type), and readValue into a typed
Value tree. Every read is bounds-checked; lengths and container sizes
larger than the remaining input fail before allocating, nesting is
limited by maxDepth, and decode returns a Result with a ThriftError.
Binary values are views of the input. i64 reads as bigint, or as a
checked number with readI64Number. Writer mirrors the reader.

The fixtures hold 349 value trees with the bytes the official Python
thrift 0.25.0 TCompactProtocol writes for them, plus seven structs cut
from apache/parquet-testing files (footers, page headers, bloom filter
header, offset and column index). The tests check that our bytes are
identical, read every case back, and fuzz corrupted input.
Share one integer range check, decode strings with internal/utf8, drop the INVALID sentinel, the Reader's struct counter and the i64 bound constants, and build Writer container labels only on failure. Section banners go and the Reader and Writer docs no longer repeat their members.
Compare gold cases as Thrift.Value instead of converting back to JSON, use effect/encoding/Hex, and remove an identity helper and duplicated instructions from generate.py.
@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c06994c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/ai-openai Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…d reject negative row counts

LogicalTypes.md uses a ConvertedType only when no LogicalType is present,
and an ignored LogicalType leaves only the physical type, so UTF8 next to
an unknown or misplaced logical type no longer validates text. num_rows of
the file and of a row group must not be negative. The module docs now say
which pages have their CRC checked.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants