Repository navigation
Add effect/arrow - #10649
Open
effect-bot wants to merge 37 commits into
Open
Add effect/arrow#10649effect-bot wants to merge 37 commits into
effect-bot wants to merge 37 commits into
Conversation
🦋 Changeset detectedLatest commit: 468005a The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This was referenced Nov 15, 2026
effect-bot
force-pushed
the
feat/arrow-support
branch
from
November 15, 2026 12:46
3760249 to
6e6a98a
Compare
effect-bot
force-pushed
the
feat/arrow-support
branch
2 times, most recently
from
November 15, 2026 19:47
18d2b91 to
142441c
Compare
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
Decode a pyarrow IPC stream (with a dictionary delta and sliced arrays) and file covering every supported type family, re-encode them, stream them through the channels in arbitrary chunks and reject malformed input. Build every type family with ArrowColumn.fromValues and check table construction and slicing.
Arrow's private FlatBuffers reader and builder are replaced by the public module. Malformed metadata now fails with an ArrowIpcError of kind Invalid whose cause is the FlatBuffersError, raised at the format.ts boundary instead of by a guard around the whole decoder.
Offsets may point at one field table or string many times, so a few hundred bytes of metadata could decode to an exponentially large schema. Like Arrow C++ (ARROW-11559), decoding now spends at most 8 units per metadata byte: one per field or key-value table and one per string character.
Only slots of valid parents and selected union members must respect a non-nullable field, which also locates struct and fixed-size list failures at the right slot.
Keys follow the field type: iterable lists and maps, holes, struct members by name or position, union members and extension storage, so distinct values are no longer merged. Non-array iterables are read once per build, so one-shot iterators can be keyed and then built.
Reading sets it as an own property instead of assigning the prototype, and building reads it only as an own property, so a missing member or column is null or missing rather than Object.prototype.
effect-bot
force-pushed
the
feat/arrow-support
branch
from
November 15, 2026 20:20
142441c to
cba4e4b
Compare
…er an invalid decimal type in a table build
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Each layer depends only on the ones below it. #10655 joined the stack so that #10666 can read Parquet files as Arrow record batches. It carries #10652 (
effect/encoding/Thrift) by merge until that lands onmain. #10657 supplies optional codecs through thecodecsoption of the Parquet reader. Arrow Flight and Protobuf / gRPC / ConnectRPC are out of scope.Summary
Experimental Apache Arrow modules, split out of #10641:
ArrowSchema: data types, fields and schemas.ArrowData,ArrowColumn: physical array data and typed column readers, plusfromValuesbuilders.ArrowTable: record batches and tables, built from rows or columns, withsliceandcolumn.ArrowIpc: IPC stream and file encoding and decoding, asResultfunctions and asChannels for chunked input.ArrowExtension: extension types.FlatBuffers boundary
IPC metadata is read and written with
effect/encoding/FlatBuffersfrom #10643. Arrow's private reader and writer are gone, and #10643 needed no changes.internal/format.tsis the only code that touches FlatBuffers. AFlatBuffersErrorfrom untrusted bytes becomes anArrowIpcErrorof kindInvalidwith the FlatBuffers error ascause.maxDepth: 128and a budget of 8 decoded units per metadata byte, as Arrow C++ does. Without the budget, a few hundred bytes of metadata that reuse one field table could expand into 100,000+ fields.validate(full=True)), checked once by hand and not committed.Review fixes
One commit per finding, each with a regression test that fails before the fix:
concat. The new fixturedeltas.arrowsis written by pyarrow and then patched to start each delta at offset 1.ArrowIpcErrorkindUnsupportedrather than quietly turning nulls into values.ArrowBuildError), the encoder and the decoder (ArrowIpcError).DateTimesupports reads as its stored integer instead of throwingIllegalArgumentError, and builders accept that integer back.ArrowBuildError.pathlocates a value from the top:[row, ...nested]forfromValuesand[row, column, ...nested]for tables, with the row counted across batches. The olditems/keys/valuessegments are gone.ArrowIpcdocs said writing needs unsliced tables. Sliced writes always worked, so the docs now describe that, and the round-trip tests cover them.Second review round
All four Arrow findings of the second review reproduced and are fixed, each with a regression test that fails before the fix:
__proto__is an own property of rows and structs, and building reads it only as an own property.Third review round
encodeand has any iterable read exactly once, and dictionary/run-end keying and construction both use that storage value. Before, keying encoded values a second time, so one-shot iterables inside run-end or dictionary extension columns read back empty.["a", , "b"]as utf8 used to read back["a", null, "ab"]).ArrowExtensiondocs promise oneencodecall per non-null value.Fourth review round
ArrowIpcmodule docs say decoding validates structure (offsets, indices, views, buffer sizes, type metadata), not stored values: a decimal with more digits than its precision decodes as stored. This is the accepted structural-only policy. Before, the docs said the input was validated "fully".Tests
test/arrow/ArrowIpc.test.ts,ArrowColumn.test.tsandArrowTable.test.ts, 19 tests. pyarrow 26 is the oracle.test/fixtures/arrow/generate.pyregenerates all three fixtures.pnpm check,pnpm lint,pnpm jsdocs --checkpnpm doctest --run packages/effect/src/arrow packages/effect/src/encoding/FlatBuffers.tspnpm vitest --run test/arrow test/encoding/FlatBuffers.test.ts(69 tests in 4 files at468005af8e)Closes EFF-3804
Chunked stream decoding
ArrowIpc.decodedecoded a message's FlatBuffers metadata again on every incoming chunk while it waited for the message body. It now decodes the metadata once and keeps it until the body is complete. Decoding the 11.6 KBtypes.arrowsfixture in 1-byte chunks went from about 1,000 ms to about 25 ms. Results are the same, so there is no new assertion.ARROW-IPC-003already covers bodies that span chunks.