diff --git a/LogicalTypes.md b/LogicalTypes.md index 690ae3f56..fbe94c601 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -569,23 +569,24 @@ The sort order used for `BSON` is unsigned byte-wise comparison. ### VARIANT -`VARIANT` is used for a Variant value. It must annotate a group. The group must -contain a field named `metadata` and a field named `value`. Both fields must have -type `binary`, which is also called `BYTE_ARRAY` in the Parquet thrift definition. -The `VARIANT` annotated group can be used to store either an unshredded Variant -value, or a shredded Variant value. +`VARIANT` is used for a Variant value. It must annotate a group, +which can be used to store either an unshredded Variant value, or a +shredded Variant value. * The Variant group must be annotated with the `VARIANT` logical type, with the version number included in the declaration. -* Both fields `value` and `metadata` must be of type `binary` (called `BYTE_ARRAY` - in the Parquet thrift definition). +* The group must contain a field named `metadata` and at least one of `value` or + `typed_value`. +* Both `metadata` field and `value` field (if present) must have type `binary`, + which is also called `BYTE_ARRAY` in the Parquet thrift definition. * The `metadata` field is required and must be a valid Variant metadata component, as defined by the [Variant binary encoding specification](VariantEncoding.md). * When present, the `value` field must be a valid Variant value component, as defined by the [Variant binary encoding specification](VariantEncoding.md). * The `value` field is required for unshredded Variant values. -* The `value` field is optional and may be null only when parts of the Variant - value are shredded according to the [Variant shredding specification](VariantShredding.md). +* When the Variant value is shredded with `typed_value` according to the + [Variant shredding specification](VariantShredding.md), the `value` field may be annotated as `optional`, + or omitted if all values can be represented by `typed_value`. This is the expected representation of an unshredded Variant in Parquet: ``` diff --git a/VariantEncoding.md b/VariantEncoding.md index adfac53d6..9065b9747 100644 --- a/VariantEncoding.md +++ b/VariantEncoding.md @@ -47,8 +47,8 @@ A Variant value in Parquet is represented by a group with 2 fields, named `value * The Variant group must be annotated with the `VARIANT` logical type. * Both fields `value` and `metadata` must be of type `binary` (called `BYTE_ARRAY` in the Parquet thrift definition). * The `metadata` field is `required` and must be a valid Variant metadata, as defined below. -* The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns. -* When present, the `value` field must be a valid Variant value, as defined below. +* The `value` field must be annotated as `required` for unshredded Variant values. If the Variant value is [shredded](VariantShredding.md) as typed Parquet columns, then the `value` field may be annotated as `optional` or omitted entirely if ALL values can be represented in the typed column. +* When present, the `value` field must be a valid Variant value, as defined below. This is the expected unshredded representation in Parquet: diff --git a/VariantShredding.md b/VariantShredding.md index fe84044fd..ce3cd561c 100644 --- a/VariantShredding.md +++ b/VariantShredding.md @@ -41,6 +41,8 @@ All field names of a Variant, whether shredded or not, must be present in the me Variant values are stored in Parquet fields named `value`. Each `value` field may have an associated shredded field named `typed_value` that stores the value when it matches a specific type. When `typed_value` is present, readers **must** reconstruct shredded values according to this specification. +Each shredding field must contain `value` and/or `typed_value`. +When `value` is omitted, all values must be represented by `typed_value`. For example, a Variant field, `measurement` may be shredded as long values by adding `typed_value` with type `int64`: ``` @@ -62,7 +64,7 @@ The series of measurements `34, null, "n/a", 100` would be stored as: | "n/a" | `01 00` v1/empty | `13 6E 2F 61` (`n/a`) | null | | 100 | `01 00` v1/empty | null | `100` | -Both `value` and `typed_value` are optional fields used together to encode a single value. +When both `value` and `typed_value` are in the schema, they are optional fields used together to encode a single value. Values in the two fields must be interpreted according to the following table: | `value` | `typed_value` | Meaning | @@ -167,6 +169,7 @@ Each shredded field in the `typed_value` group is represented as a required grou The `value` field stores the value as Variant-encoded `binary` when the `typed_value` cannot represent the field. This layout enables readers to skip data based on the field statistics for `value` and `typed_value`. The `typed_value` field may be omitted when not shredding fields as a specific type. +The `value` field may be omitted when all values of that field can be represented by `typed_value`. The `value` column of a partially shredded object must never contain fields represented by the Parquet columns in `typed_value` (shredded fields). Readers may always assume that data is written correctly and that shredded fields in `typed_value` are not present in `value`. @@ -271,9 +274,9 @@ optional group event (VARIANT(1)) { # Data Skipping -Statistics for `typed_value` columns can be used for file, row group, or page skipping when `value` is always null (missing). +Statistics for `typed_value` columns can be used for file, row group, or page skipping when the `value` field is omitted from the schema or when `value` is always null (missing). -When the corresponding `value` column is all nulls, all values must be the shredded `typed_value` field's type. +When the corresponding `value` column is omitted or all nulls, all values must be the shredded `typed_value` field's type. Because the type is known, comparisons with values of that type are valid. `IS NULL`/`IS NOT NULL` and `IS NAN`/`IS NOT NAN` filter results are also valid.