Repository navigation
Add ArrowParquet to effect/arrow - #10666
Open
effect-bot wants to merge 12 commits into
Open
effect-bot wants to merge 12 commits into
effect-bot wants to merge 12 commits into
Conversation
🦋 Changeset detectedLatest commit: 2c865c1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This was referenced Nov 16, 2026
The Arrow types of these annotations are the same as those of STRING, unannotated binary and the other TIME, so the field metadata now carries the difference. Also cover the empty projection of a negative row count, UTF8 next to an ignored logical type, and label the repeated MAP value as a local extension.
…aves in one expression
… and shorten repeated fixture notes
…d a test local clearer names
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ArrowParquettoeffect/arrow. It reads a Parquet file as Arrow record batches, oneArrowTable.RecordBatchper row group. This is the Parquet to Arrow integration of EFF-3804.Stack
The base is
feat/encoding-parquet, which now hasfeat/arrow-supportmerged in, so this diff is the adapter alone. Thrift and Parquet decoding are used as they are and nothing is duplicated. Compressed pages go through thecodecsoption ofParquet.readColumn, which this module passes on, so #10657 stays optional.API
schema(metadata, { columns? })maps a Parquet schema to anArrowSchema.Schemaand returns aResult.readRecordBatch(source, metadata, { rowGroup, columns?, codecs?, ...limits })reads one row group as a record batch.recordBatches(source, options?)is aStreamthat reads the footer and then emits one batch per row group, in file order. A row group is read when its batch is pulled. Batches are not split or merged, and a file without row groups emits nothing.columnsselects top-level fields by name, in the order given, and only their column chunks are read. Failures of this module areParquet.ParquetError, and the errors of the source pass through. A file that breaks the format isMalformed, a valid file that can't be mapped isUnsupported, and an unknown column name or row group index is a defect (Usage), like inParquetandThrift.Mapping (parquet-format 2.14.0)
"Shared" means the Arrow array uses the buffer
Parquet.readColumnreturned. "Copy" means one new buffer per column chunk."UTC"when adjustedarrow.jsonarrow.uuidA repeated map value is read as a list. The Maps text doesn't allow it, so this is a local extension and not MAP conformance (
ARROW-PARQUET-010labels it local).Converted types map as the logical type the spec gives for them. Validity bitmaps and the offsets and bytes of BYTE_ARRAY values are always shared. A field is nullable when its Parquet field is OPTIONAL. A
fieldIdis kept asPARQUET:field_id, and the file's key/value metadata becomes the schema metadata. The annotations the Arrow type doesn't carry are kept as field metadata:PARQUET:logical_typeisENUMorBSON, and a Time hasPARQUET:is_adjusted_to_utc(trueorfalse). A converted type next to an ignored logical type is ignored too, soUTF8beside an unknown logical type is Binary.No value is created per row. There is no
DateTime,Duration,BigDecimalor string on the path, and the builders ofArrowColumnare not used. The precision checks compare 32-bit limbs in place.Ownership
A batch owns its buffers.
Parquet.readColumncopies every value out of the pages and out of what aDecompressreturns, so no buffer of a batch is a view of the source, and the source can be changed or reused once a batch is returned. That is a property of the current reader, not a documented contract ofParquet, soARROW-PARQUET-017guards it. The copies made here are bounded bymaxValues, at most 32 bytes per slot.Rejected
Malformed: a negative row count (fromParquet.readMetadata, so an empty projection can't give a batch of negative length), a narrow integer outside its bit width, a DECIMAL with more digits than its precision or wider than its Arrow type, an UNKNOWN column with a value, sibling columns that disagree about a shared struct or list (so children always have the length their parent says), and LIST or MAP groups without the structure the spec requires.Unsupported, only when the column is selected: INTERVAL, VARIANT, GEOMETRY, GEOGRAPHY and FILE, a DECIMAL of more than 76 digits, a map without values, a group without fields, and an INT96 outside 64-bit nanoseconds.All limits of the Parquet reader still apply (
maxPageSize,maxValues,maxBinaryLength,maxFooterLength,maxDepth).Choices to review
Unsupported. The spec allows reading it as a map of nulls, and Arrow C++ does.ARROW:schemais not interpreted. Types pyarrow restores from it come back as their Parquet storage:large_stringas Utf8, a dictionary as its values.isAdjustedToUTCof a TIME have no place in the Arrow type, so they are kept in the field metadata underPARQUET:keys of this module (Arrow C++ has no such keys). Times of day and dates are not range checked, and JSON is not parsed.Tests
packages/effect/test/arrow/ArrowParquet.test.tshas 26 tests, inventoried asARROW-PARQUET-001to018intest/arrow/CONFORMANCE.mdwith the spec requirement and the local policy of each row kept apart.gold.json. A new fixturelogical.parquetwith its own gold file covers the logical types the others lack. The existing fixtures andgold.jsonare unchanged.Parquet.test.tslive intest/utils/parquet.tsfor that, with no assertion changed.ArrowIpcstream round trip. I also read the IPC output of all ten fixtures in pyarrow 26.0.1 by hand (not committed): each passesvalidate(full=True)and equals pyarrow's own read of the Parquet file, apart from the threeARROW:schematype differences above.At
2c865c1757:pnpm lint,pnpm checkandpnpm jsdocs --checkpass.pnpm vitest --runovertest/arrowandtest/encodingruns 888 tests in 24 files.pnpm doctest --runoversrc/arrowandParquet.tspasses (13 in 6 files).Closes EFF-3804
Closes EFF-3820