Skip to content

Arrow write: one bad value inside a ColumnGroup aborts the whole export, even in Mode.LOYAL #2075

Description

@zaleslaw

One value that cannot be converted to the target Arrow type fails the entire write — no file is produced —
when it sits in a column inside a ColumnGroup. Mode.LOYAL is supposed to degrade such a value and report
it to mismatchSubscriber instead of aborting, and it does exactly that for a top-level column.

The reason is where the conversion happens: a ColumnGroup written into an ArrowType.Struct field has its
columns converted inside ArrowWriterImpl.infillVector, outside the try/catch in allocateVectorAndInfill
that applies ArrowWriter.Mode to a failed conversion. So the exception goes straight out of the writer and
the subscriber is never called.

val df = dataFrameOf(DataColumn.createColumnGroup("g", dataFrameOf("x")("abc", "def")))
val schema = Schema(listOf(Field("g", FieldType.nullable(ArrowType.Struct()),
    listOf(Field("x", FieldType.nullable(ArrowType.Int(32, true)), null)))))
df.arrowWriter(schema, ArrowWriter.Mode.LOYAL) { println(it) }
    .use { it.saveArrowFeatherToByteArray() }
// CellConversionException: Failed to convert 'abc' from kotlin.String to kotlin.Int in column 'x', row 0
// mismatchSubscriber: never called

Pre-existing and not specific to a vector type — the same happens for any child vector, e.g.
is IntVector -> column.convertToInt(). Noticed in #2063
(#2063 (comment)).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingfilesreading/writing from/to files

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions