Repository navigation
Add effect/encoding/Parquet - #10655
Open
effect-bot wants to merge 35 commits into
Open
effect-bot wants to merge 35 commits into
effect-bot wants to merge 35 commits into
Conversation
Reads and writes the Apache Thrift compact protocol without an IDL. Reader is a TProtocol-shaped cursor: readFieldBegin/readListBegin/ readMapBegin, scalar reads, skip(type), and readValue into a typed Value tree. Every read is bounds-checked; lengths and container sizes larger than the remaining input fail before allocating, nesting is limited by maxDepth, and decode returns a Result with a ThriftError. Binary values are views of the input. i64 reads as bigint, or as a checked number with readI64Number. Writer mirrors the reader. The fixtures hold 349 value trees with the bytes the official Python thrift 0.25.0 TCompactProtocol writes for them, plus seven structs cut from apache/parquet-testing files (footers, page headers, bloom filter header, offset and column index). The tests check that our bytes are identical, read every case back, and fuzz corrupted input.
Share one integer range check, decode strings with internal/utf8, drop the INVALID sentinel, the Reader's struct counter and the i64 bound constants, and build Writer container labels only on failure. Section banners go and the Reader and Writer docs no longer repeat their members.
Compare gold cases as Thrift.Value instead of converting back to JSON, use effect/encoding/Hex, and remove an identity helper and duplicated instructions from generate.py.
🦋 Changeset detectedLatest commit: c06994c The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
effect-bot
changed the base branch from
feat/encoding-thrift
to
feat/arrow-support
November 16, 2026 10:33
This was referenced Nov 16, 2026
…d reject negative row counts LogicalTypes.md uses a ConvertedType only when no LogicalType is present, and an ignored LogicalType leaves only the physical type, so UTF8 next to an unknown or misplaced logical type no longer validates text. num_rows of the file and of a row group must not be negative. The module docs now say which pages have their CRC checked.
… early in annotations
…e, text and logical type wording
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
effect/encoding/Parquet, an experimental reader for Apache Parquet files. It reads the footer metadata and one column chunk at a time, into buffers laid out the way Arrow lays out arrays. This is part of the data-processing work in EFF-3804.All Parquet metadata, page headers included, goes through the compact-protocol reader of #10652 (
effect/encoding/Thrift).Stack
This PR is now part of the Arrow stack. Its base is
feat/arrow-support(it wasfeat/encoding-thrift), and the branch hasfeat/arrow-supportmerged in, so that #10666 can build Arrow record batches on top of both. No commit was rewritten. The Parquet module itself still has no Arrow dependency.Reading the diff: until #10652 merges, the diff against
feat/arrow-supportalso shows the Thrift files of #10652 (Thrift.ts, its test andtest/fixtures/thrift) and a few files from a newermainthat the Arrow chain doesn't have yet (packages/ai/openai,LayerRef). Neither is part of this PR. The Parquet change isParquet.ts,internal/parquet/encodings.ts,internal/crc32.ts,Zip.ts, the barrel,Parquet.test.tsandtest/fixtures/parquet.Merge order: #10643, #10649 and #10652 first, then this PR, then #10666. #10657 is independent.
API
readMetadata(source, options?)returns theFileMetaData: the schema tree, oneColumnDescriptorper leaf (path, physical type, max levels, column order), the row groups, statistics and key/value metadata.readColumn(source, metadata, { rowGroup, column, codecs?, ...limits })returns aColumn. It has oneLayerper schema node on the path (validity bitmap for optional nodes, offsets for repeated ones) and the leafValueswith a slot for every null. That way the Arrow adapter in Add ArrowParquet to effect/arrow #10666 mostly wraps buffers.Sourceis{ size, read(offset, length) }returning an Effect, so in-memory bytes (fromBytes), files and HTTP range reads all work the same.ParquetErrorhas kindsMalformed,Unsupported,LimitandCodec. Bad options and out-of-range indexes are defects (Usage), like Thrift.Supported scope (apache-parquet-format 2.14.0)
UTF8is read as bytes, not checked as UTF-8.num_rowson the file or a row group isMalformed.The following fail with
Unsupported: encrypted footers and columns, ALP, chunks in other files (file_path), and any compressed page without a matchingDecompress.Compression boundary
The module has no codecs.
codecsmaps a Parquet codec name to aDecompressfunction ((data, uncompressedSize) => Effect<Uint8Array, unknown>), andUNCOMPRESSEDis built in. The shared decoders are planned aseffect/encoding/Compression(EFF-3821, #10657). Wiring them in, and choosing defaults, is a follow-up. The boundary agreed there is already in place: page sizes are checked to be non-negative, at mostmaxPageSize, and consistent with the v2 level lengths before a codec is called. A codec failure becomeskind: "Codec", with the column path and page offset in the message and the original error ascause.Untrusted input
Every offset and length is checked against the bytes that are actually there.
maxFooterLength,maxDepth(at most 255, so levels fit a byte),maxPageSize,maxValuesandmaxBinaryLengthcap allocations before they happen.maxBinaryLengthalso counts fixed-width (FIXED_LEN_BYTE_ARRAY,INT96) values, nulls included, andmaxValues(default 2^24) also caps each page, dictionary pages included. Memory and time still grow with values times nesting depth, which thereadColumndocs call out. These limits are kept separate from the spec checks.Tests
Parquet.test.tshas 82 tests.PQ-LT-3,PQ-LT-5andPQ-LT-12cover the ignored and absent logical type next to a converted type, andPQ-META-3covers negative row counts.pnpm check,pnpm lintandpnpm jsdocs --checkpass. Atc06994ca17,pnpm vitest --runovertest/arrowandtest/encodingruns 862 tests in 23 files.packages/effect/test/encoding/Parquet.test.tshas one conformance test per spec assertion, named by id (PQ-FILE-1and so on), on synthetic files built with the Thrift writer.test/fixtures/parquet/CONFORMANCE.mdmaps each id to the spec at the pinned tag. It also lists what is unsupported, what is deferred (page index, bloom filters, statistics semantics, logical type application) and what only applies to writers. This is not a full-conformance claim.gold.json. They show interoperability and are not a substitute for the assertion tests.hostile inputblock covers limits and malformed input.Closes EFF-3820