Noticed that Total runtime of reading a bunch of parquet datasets goes from 55 sec in rc01 to 70 sec in dev.
Main difference comes from BitVectorHelper.getNullCount usage in new code. It actually goes over entire vector to count, even though no nulls are expected in StructVector according to schema
Extra check on StructVector nullability seems to achieve the same result, assuming we can trust file schema to not mislead us and not have sneaky nulls:
val nullMask = if (field.isNullable) vector.nullMaskOrNull(range) else null
then nullCount > 0 should not be needed then and reading time goes back to 55s.
Noticed that Total runtime of reading a bunch of parquet datasets goes from 55 sec in rc01 to 70 sec in dev.
Main difference comes from BitVectorHelper.getNullCount usage in new code. It actually goes over entire vector to count, even though no nulls are expected in StructVector according to schema
Extra check on StructVector nullability seems to achieve the same result, assuming we can trust file schema to not mislead us and not have sneaky nulls:
then
nullCount > 0should not be needed then and reading time goes back to 55s.