Skip to content

Add effect/arrow - #10649

Open
effect-bot wants to merge 37 commits into
feat/encoding-flatbuffersfrom
feat/arrow-support
Open

effect-bot wants to merge 37 commits into
feat/encoding-flatbuffersfrom
feat/arrow-support

Conversation

@effect-bot

@effect-bot effect-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Stack

main
├─ #10643 effect/encoding/FlatBuffers
│   └─ this PR: effect/arrow
│       ├─ #10641 effect/dataframe
│       └─ #10655 effect/encoding/Parquet (carries #10652 by merge)
│           └─ #10666 ArrowParquet, Parquet files as Arrow record batches
├─ #10652 effect/encoding/Thrift
└─ #10657 effect/encoding/Compression

Each layer depends only on the ones below it. #10655 joined the stack so that #10666 can read Parquet files as Arrow record batches. It carries #10652 (effect/encoding/Thrift) by merge until that lands on main. #10657 supplies optional codecs through the codecs option of the Parquet reader. Arrow Flight and Protobuf / gRPC / ConnectRPC are out of scope.

Summary

Experimental Apache Arrow modules, split out of #10641:

  • ArrowSchema: data types, fields and schemas.
  • ArrowData, ArrowColumn: physical array data and typed column readers, plus fromValues builders.
  • ArrowTable: record batches and tables, built from rows or columns, with slice and column.
  • ArrowIpc: IPC stream and file encoding and decoding, as Result functions and as Channels for chunked input.
  • ArrowExtension: extension types.

FlatBuffers boundary

IPC metadata is read and written with effect/encoding/FlatBuffers from #10643. Arrow's private reader and writer are gone, and #10643 needed no changes.

  • internal/format.ts is the only code that touches FlatBuffers. A FlatBuffersError from untrusted bytes becomes an ArrowIpcError of kind Invalid with the FlatBuffers error as cause.
  • Decoding uses maxDepth: 128 and a budget of 8 decoded units per metadata byte, as Arrow C++ does. Without the budget, a few hundred bytes of metadata that reuse one field table could expand into 100,000+ fields.
  • Record batch bodies stay zero-copy views of the input. The module docs say so: don't mutate the input while you use the table.
  • Re-encoded fixtures read back in pyarrow 26 as equal tables (validate(full=True)), checked once by hand and not committed.

Review fixes

One commit per finding, each with a regression test that fails before the fix:

  • Dictionary and run-end dedup keys are injective, so nested values that print alike stay distinct.
  • Dictionary deltas whose offsets don't start at zero (string, binary, list and map) are rebased in one place in concat. The new fixture deltas.arrows is written by pyarrow and then patched to start each delta at offset 1.
  • Writing a union that has null slots of its own (V4 data) fails with ArrowIpcError kind Unsupported rather than quietly turning nulls into values.
  • Union type ids go through one check (unique, 0..127, one per member) in the builders (ArrowBuildError), the encoder and the decoder (ArrowIpcError).
  • Temporal values are range-checked against their storage width instead of wrapping.
  • A legal Date or Timestamp outside the range DateTime supports reads as its stored integer instead of throwing IllegalArgumentError, and builders accept that integer back.
  • ArrowBuildError.path locates a value from the top: [row, ...nested] for fromValues and [row, column, ...nested] for tables, with the row counted across batches. The old items/keys/values segments are gone.
  • The ArrowIpc docs said writing needs unsliced tables. Sliced writes always worked, so the docs now describe that, and the round-trip tests cover them.

Second review round

All four Arrow findings of the second review reproduced and are fixed, each with a regression test that fails before the fix:

  • Dictionary and run-end keys are now derived from the field type with the builder's own readers (item iteration with holes as null, map entries from any iterable, struct members, union choice). A value the builder can't read gets no key. The parallel serializer is gone.
  • Delta concat cuts both sides to the items their slots use, so an unused child suffix on either side no longer leaks (list, map and fixed-size list). The fixture generator now gives the first list dictionaries an unused trailing child item.
  • Nullability is checked only on the slots a parent uses, so a sparse union rejects a null in its selected non-nullable member. This also fixes error paths after null structs.
  • A field named __proto__ is an own property of rows and structs, and building reads it only as an own property.
  • The IPC docs say a written slice decodes to the slice's rows only, but its buffers can keep bytes from outside the slice, so slicing is not redaction.

Third review round

  • Building has one encoding boundary: each value goes through its extension's encode and has any iterable read exactly once, and dictionary/run-end keying and construction both use that storage value. Before, keying encoded values a second time, so one-shot iterables inside run-end or dictionary extension columns read back empty.
  • Holes in sparse input arrays read as nulls in every builder (["a", , "b"] as utf8 used to read back ["a", null, "ab"]).
  • The ArrowExtension docs promise one encode call per non-null value.

Fourth review round

  • The ArrowIpc module docs say decoding validates structure (offsets, indices, views, buffer sizes, type metadata), not stored values: a decimal with more digits than its precision decodes as stored. This is the accepted structural-only policy. Before, the docs said the input was validated "fully".

Tests

test/arrow/ArrowIpc.test.ts, ArrowColumn.test.ts and ArrowTable.test.ts, 19 tests. pyarrow 26 is the oracle. test/fixtures/arrow/generate.py regenerates all three fixtures.

  • pnpm check, pnpm lint, pnpm jsdocs --check
  • pnpm doctest --run packages/effect/src/arrow packages/effect/src/encoding/FlatBuffers.ts
  • pnpm vitest --run test/arrow test/encoding/FlatBuffers.test.ts (69 tests in 4 files at 468005af8e)

Closes EFF-3804

Chunked stream decoding

ArrowIpc.decode decoded a message's FlatBuffers metadata again on every incoming chunk while it waited for the message body. It now decodes the metadata once and keeps it until the body is complete. Decoding the 11.6 KB types.arrows fixture in 1-byte chunks went from about 1,000 ms to about 25 ms. Results are the same, so there is no new assertion. ARROW-IPC-003 already covers bodies that span chunks.

@changeset-bot

changeset-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 468005a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-bot
effect-bot force-pushed the feat/arrow-support branch 2 times, most recently from 18d2b91 to 142441c Compare November 15, 2026 19:47
fubhy and others added 23 commits November 15, 2026 20:18
Apache Arrow schemas, columns, record batches and tables with IPC stream and file codecs. Split out of #10641.
Decode a pyarrow IPC stream (with a dictionary delta and sliced arrays) and file covering every supported type family, re-encode them, stream them through the channels in arbitrary chunks and reject malformed input. Build every type family with ArrowColumn.fromValues and check table construction and slicing.
Arrow's private FlatBuffers reader and builder are replaced by the public module. Malformed metadata now fails with an ArrowIpcError of kind Invalid whose cause is the FlatBuffersError, raised at the format.ts boundary instead of by a guard around the whole decoder.
Offsets may point at one field table or string many times, so a few hundred bytes of metadata could decode to an exponentially large schema. Like Arrow C++ (ARROW-11559), decoding now spends at most 8 units per metadata byte: one per field or key-value table and one per string character.
Only slots of valid parents and selected union members must respect a non-nullable field, which also locates struct and fixed-size list failures at the right slot.
Keys follow the field type: iterable lists and maps, holes, struct members by name or position, union members and extension storage, so distinct values are no longer merged. Non-array iterables are read once per build, so one-shot iterators can be keyed and then built.
Reading sets it as an own property instead of assigning the prototype, and building reads it only as an own property, so a missing member or column is null or missing rather than Object.prototype.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants