One value that cannot be converted to the target Arrow type fails the entire write — no file is produced —
when it sits in a column inside a ColumnGroup. Mode.LOYAL is supposed to degrade such a value and report
it to mismatchSubscriber instead of aborting, and it does exactly that for a top-level column.
The reason is where the conversion happens: a ColumnGroup written into an ArrowType.Struct field has its
columns converted inside ArrowWriterImpl.infillVector, outside the try/catch in allocateVectorAndInfill
that applies ArrowWriter.Mode to a failed conversion. So the exception goes straight out of the writer and
the subscriber is never called.
val df = dataFrameOf(DataColumn.createColumnGroup("g", dataFrameOf("x")("abc", "def")))
val schema = Schema(listOf(Field("g", FieldType.nullable(ArrowType.Struct()),
listOf(Field("x", FieldType.nullable(ArrowType.Int(32, true)), null)))))
df.arrowWriter(schema, ArrowWriter.Mode.LOYAL) { println(it) }
.use { it.saveArrowFeatherToByteArray() }
// CellConversionException: Failed to convert 'abc' from kotlin.String to kotlin.Int in column 'x', row 0
// mismatchSubscriber: never called
Pre-existing and not specific to a vector type — the same happens for any child vector, e.g.
is IntVector -> column.convertToInt(). Noticed in #2063
(#2063 (comment)).
One value that cannot be converted to the target Arrow type fails the entire write — no file is produced —
when it sits in a column inside a
ColumnGroup.Mode.LOYALis supposed to degrade such a value and reportit to
mismatchSubscriberinstead of aborting, and it does exactly that for a top-level column.The reason is where the conversion happens: a
ColumnGroupwritten into anArrowType.Structfield has itscolumns converted inside
ArrowWriterImpl.infillVector, outside thetry/catchinallocateVectorAndInfillthat applies
ArrowWriter.Modeto a failed conversion. So the exception goes straight out of the writer andthe subscriber is never called.
Pre-existing and not specific to a vector type — the same happens for any child vector, e.g.
is IntVector -> column.convertToInt(). Noticed in #2063(#2063 (comment)).