GH-45937: [C++][Parquet] support to read, write and validate variant

[HuaHuaY]

Rationale for this change

This PR supports:

• Mapping between arrow extension variant type and Parquet variant type
• Reading and writing unshred and shredded variant arrays.
• Validate whether a variant array follows parquet format.

This PR does not support:

• Accumulate record batch and inferring typed_value shredded data from the value array

What changes are included in this PR?

• variant/format_internal.h:
  Define some type-related traits and concepts.

• variant/builder.h .cc:
  Add builders for encoded Variant values, arrays, nested objects and lists, and residual value arrays. This part shares a similar design with arrow-rs, but differs slightly because C++ lacks enum types that hold values. It employs a single-buffer builder design and also supports nested builder APIs. Some previous discussions: [C++][Parquet] Encoding tools for variant type #46555

• variant/decoding.h .cc:
  Add zero-copy views with shallow parsing and recursive validation.

• variant/shred.h .cc:
  Convert an unshredded VariantArray into a requested primitive, object, or list typed_value layout while retaining incompatible values in residual value.

• variant/unshred.h .cc:
  Reconstruct encoded Variant values from shredded storage.

• variant/validate.h .cc:
  Validate nested Variant arrays using strict writer rules or read-compatible rules. To avoid too much materialization of bitmaps, this module employs a two-pass design: first constructing the plan and finding variant, then executing it.

• variant/slot_internal.h .cc:
  Compile shredded storage layouts and process primitive, object, and list slots for validation and unshredding. This is a common module used by unshred and validate.

Some code use template <bool strict> because some schemas are read-compatible but do not support writing. For details, please see apache/parquet-format#591

Are these changes tested?

Yes.

Are there any user-facing changes?

ArrowReaderProperties adds set_variant_validation_enabled(). ArrowWriterProperties::Builder adds set_variant_validation_enabled(). Validation is enabled by default.

• GitHub Issue: [C++][Parquet] Add variant support #45937

[wgtmac]

It seems that some Arrow primitive types are missing from here: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings

[HuaHuaY]

This issue is still under discussion; those interested, please head over to #50622

[HuaHuaY]

@kou @raulcd We use C++20 to develop this repo now and MSVC has a new preprocesser to handle C++20 grammar like __VA_OPT__. Should we add this flag globally? We can try to add the flag in a new PR.

[raulcd]

Probably? I would try it on a different PR, as suggested, and investigate whether we can drop:

arrow/cpp/cmake_modules/SetupCxxFlags.cmake

Lines 221 to 236 in 374db36

	# Disable C5105 (macro expansion producing 'defined' has undefined
	# behavior) warning because there are codes that produce this
	# warning in Windows Kits. e.g.:
	#
	# #define _CRT_INTERNAL_NONSTDC_NAMES \
	# ( \
	# ( defined _CRT_DECLARE_NONSTDC_NAMES && _CRT_DECLARE_NONSTDC_NAMES) || \
	# (!defined _CRT_DECLARE_NONSTDC_NAMES && !__STDC__ ) \
	# )
	#
	# See also:
	# * C5105: https://docs.microsoft.com/en-US/cpp/error-messages/compiler-warnings/c5105
	# * Related reports:
	# * https://developercommunity.visualstudio.com/content/problem/387684/c5105-with-stdioh-and-experimentalpreprocessor.html
	# * https://developercommunity.visualstudio.com/content/problem/1249671/stdc17-generates-warning-compiling-windowsh.html
	set(CXX_COMMON_FLAGS "${CXX_COMMON_FLAGS} /wd5105")

[HuaHuaY]

@pitrou @emkornfield Can you help to review the API design? I’ve tried to keep the code concise, but this is still a huge PR. I’d like to focus on the overall API design for now, and then I’ll break this PR down into several smaller ones. I include brief introductions to each module in the PR description.

[emkornfield]

@pitrou @emkornfield Can you help to review the API design?

@HuaHuaY thanks for the PR. Which files contain the API design we should focus on?

[HuaHuaY]

Which files contain the API design we should focus on?

I'm sorry what I said earlier wasn't clear enough. I would like your help reviewing whether the implementation approach is appropriate and whether the provided interfaces are easy to use. As for specific code details, we can review those later in smaller, separate PRs.

You can start by reviewing these header files, and I consider them to be the parts directly used by the user:

• cpp/src/parquet/variant/format.h
• cpp/src/arrow/extension/parquet_variant.h
• cpp/src/parquet/variant/builder.h
• cpp/src/parquet/variant/shred.h and cpp/src/parquet/variant/unshred.h