Skip to content

Guard BitVectorHelper.getNullCount in StructVector.nullMaskOrNull #2064

Description

@koperagen

Noticed that Total runtime of reading a bunch of parquet datasets goes from 55 sec in rc01 to 70 sec in dev.
Main difference comes from BitVectorHelper.getNullCount usage in new code. It actually goes over entire vector to count, even though no nulls are expected in StructVector according to schema
Extra check on StructVector nullability seems to achieve the same result, assuming we can trust file schema to not mislead us and not have sneaky nulls:

val nullMask = if (field.isNullable) vector.nullMaskOrNull(range) else null

then nullCount > 0 should not be needed then and reading time goes back to 55s.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

filesreading/writing from/to filesperformanceSomething related to how fast the library can handle data

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions