Skip to content

Add ArrowParquet to effect/arrow - #10666

Open
effect-bot wants to merge 12 commits into
feat/encoding-parquetfrom
feat/arrow-parquet
Open

effect-bot wants to merge 12 commits into
feat/encoding-parquetfrom
feat/arrow-parquet

Conversation

@effect-bot

@effect-bot effect-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds ArrowParquet to effect/arrow. It reads a Parquet file as Arrow record batches, one ArrowTable.RecordBatch per row group. This is the Parquet to Arrow integration of EFF-3804.

Stack

main
├─ #10643 effect/encoding/FlatBuffers
│   └─ #10649 effect/arrow
│       ├─ #10641 effect/dataframe
│       └─ #10655 effect/encoding/Parquet (carries #10652 by merge)
│           └─ this PR: ArrowParquet
├─ #10652 effect/encoding/Thrift
└─ #10657 effect/encoding/Compression

The base is feat/encoding-parquet, which now has feat/arrow-support merged in, so this diff is the adapter alone. Thrift and Parquet decoding are used as they are and nothing is duplicated. Compressed pages go through the codecs option of Parquet.readColumn, which this module passes on, so #10657 stays optional.

API

  • schema(metadata, { columns? }) maps a Parquet schema to an ArrowSchema.Schema and returns a Result.
  • readRecordBatch(source, metadata, { rowGroup, columns?, codecs?, ...limits }) reads one row group as a record batch.
  • recordBatches(source, options?) is a Stream that reads the footer and then emits one batch per row group, in file order. A row group is read when its batch is pulled. Batches are not split or merged, and a file without row groups emits nothing.

columns selects top-level fields by name, in the order given, and only their column chunks are read. Failures of this module are Parquet.ParquetError, and the errors of the source pass through. A file that breaks the format is Malformed, a valid file that can't be mapped is Unsupported, and an unknown column name or row group index is a defect (Usage), like in Parquet and Thrift.

Mapping (parquet-format 2.14.0)

"Shared" means the Arrow array uses the buffer Parquet.readColumn returned. "Copy" means one new buffer per column chunk.

Parquet Arrow Values
BOOLEAN Bool shared
INT32, INTEGER(32) Int32, UInt32 shared
INT32 with INTEGER(8 or 16) Int8, Int16, UInt8, UInt16 copy, range checked
INT32 DATE, TIME(MILLIS) Date in days, Time in milliseconds shared
INT64, INTEGER(64) Int64, UInt64 shared
INT64 TIME(MICROS, NANOS) Time in that unit shared
INT64 TIMESTAMP(unit, isAdjustedToUTC) Timestamp in that unit, timezone "UTC" when adjusted shared
INT32, INT64 DECIMAL(p, s) Decimal(p, s) of 32 and 64 bits shared, precision checked
BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY DECIMAL(p, s) Decimal of 128 bits up to p = 38, of 256 bits up to 76 copy (big-endian to little-endian), width and precision checked
INT96 Timestamp in nanoseconds, no timezone copy
FLOAT, DOUBLE Float32, Float64 shared
BYTE_ARRAY, BSON Binary shared
STRING, ENUM Utf8 shared
JSON Utf8 with extension arrow.json shared
FIXED_LEN_BYTE_ARRAY(n) FixedSizeBinary(n) shared
UUID FixedSizeBinary(16) with extension arrow.uuid shared
FLOAT16 Float16 shared
UNKNOWN Null no buffer
group Struct children
LIST, MAP List, Map, with the backward-compatibility rules of LogicalTypes.md offsets shared
repeated field outside LIST and MAP required List of required elements offsets shared

A repeated map value is read as a list. The Maps text doesn't allow it, so this is a local extension and not MAP conformance (ARROW-PARQUET-010 labels it local).

Converted types map as the logical type the spec gives for them. Validity bitmaps and the offsets and bytes of BYTE_ARRAY values are always shared. A field is nullable when its Parquet field is OPTIONAL. A fieldId is kept as PARQUET:field_id, and the file's key/value metadata becomes the schema metadata. The annotations the Arrow type doesn't carry are kept as field metadata: PARQUET:logical_type is ENUM or BSON, and a Time has PARQUET:is_adjusted_to_utc (true or false). A converted type next to an ignored logical type is ignored too, so UTF8 beside an unknown logical type is Binary.

No value is created per row. There is no DateTime, Duration, BigDecimal or string on the path, and the builders of ArrowColumn are not used. The precision checks compare 32-bit limbs in place.

Ownership

A batch owns its buffers. Parquet.readColumn copies every value out of the pages and out of what a Decompress returns, so no buffer of a batch is a view of the source, and the source can be changed or reused once a batch is returned. That is a property of the current reader, not a documented contract of Parquet, so ARROW-PARQUET-017 guards it. The copies made here are bounded by maxValues, at most 32 bytes per slot.

Rejected

  • Malformed: a negative row count (from Parquet.readMetadata, so an empty projection can't give a batch of negative length), a narrow integer outside its bit width, a DECIMAL with more digits than its precision or wider than its Arrow type, an UNKNOWN column with a value, sibling columns that disagree about a shared struct or list (so children always have the length their parent says), and LIST or MAP groups without the structure the spec requires.
  • Unsupported, only when the column is selected: INTERVAL, VARIANT, GEOMETRY, GEOGRAPHY and FILE, a DECIMAL of more than 76 digits, a map without values, a group without fields, and an INT96 outside 64-bit nanoseconds.

All limits of the Parquet reader still apply (maxPageSize, maxValues, maxBinaryLength, maxFooterLength, maxDepth).

Choices to review

  • A map with keys but no value field is Unsupported. The spec allows reading it as a map of nulls, and Arrow C++ does.
  • DECIMAL on INT32 and INT64 becomes Decimal32 and Decimal64, which shares the buffer. Arrow C++ widens to Decimal128.
  • ARROW:schema is not interpreted. Types pyarrow restores from it come back as their Parquet storage: large_string as Utf8, a dictionary as its values.
  • ENUM, BSON and the isAdjustedToUTC of a TIME have no place in the Arrow type, so they are kept in the field metadata under PARQUET: keys of this module (Arrow C++ has no such keys). Times of day and dates are not range checked, and JSON is not parsed.
  • Not in this PR: keeping dictionary pages as Arrow dictionaries (needs a reader change), reading column chunks concurrently, selecting nested fields, and re-chunking batches.

Tests

  • packages/effect/test/arrow/ArrowParquet.test.ts has 26 tests, inventoried as ARROW-PARQUET-001 to 018 in test/arrow/CONFORMANCE.md with the spec requirement and the local policy of each row kept apart.
  • The nine pyarrow fixtures of Add effect/encoding/Parquet #10655 are read as batches and compared with gold.json. A new fixture logical.parquet with its own gold file covers the logical types the others lack. The existing fixtures and gold.json are unchanged.
  • Synthetic files cover what pyarrow doesn't write and every rejection. The builders and shared assertions of Parquet.test.ts live in test/utils/parquet.ts for that, with no assertion changed.
  • Batches survive an ArrowIpc stream round trip. I also read the IPC output of all ten fixtures in pyarrow 26.0.1 by hand (not committed): each passes validate(full=True) and equals pyarrow's own read of the Parquet file, apart from the three ARROW:schema type differences above.

At 2c865c1757: pnpm lint, pnpm check and pnpm jsdocs --check pass. pnpm vitest --run over test/arrow and test/encoding runs 888 tests in 24 files. pnpm doctest --run over src/arrow and Parquet.ts passes (13 in 6 files).

Closes EFF-3804
Closes EFF-3820

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2c865c1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

The Arrow types of these annotations are the same as those of STRING,
unannotated binary and the other TIME, so the field metadata now carries
the difference. Also cover the empty projection of a negative row count,
UTF8 next to an ignored logical type, and label the repeated MAP value as
a local extension.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant