From 1f7d5500c0f6dc46c6b079ec4df4f1561fa67646 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 09:27:15 +0800 Subject: [PATCH 01/20] [opt](parquet) Fuse fragmented nullable selection planning (#66397) - Fuse nullable definition-level runs with the row filter in one traversal. - Produce physical decode ranges, the selected NULL map, and selected value counts without first materializing and rescanning a row-wise selection map. - Reuse the existing selected-decoder strategies and nullable in-place expansion. - Restrict fusion to batches with at least 1,024 rows, at least 10% NULLs, and materially fragmented definition-level runs. No-NULL, low-NULL, clustered, nested, and non-expandable shapes keep the legacy path. The full benchmark matrix includes no-NULL, low-NULL, and clustered level plans as negative controls. Those shapes do not remove enough legacy work to guarantee a win, so this change deliberately leaves them unchanged. Decoder selection and encoding-specific materialization are not modified. - ASAN: `NativeNullableSelectionTest.*` and benchmark scenario tests: 16/16 passed. - ASAN: `ParquetV2NativeDecoderTest.*`: 118/118 passed. | Coverage | Legacy/fused pairs | Correctness | Regressions | Mean CPU change | Least improvement | |---|---:|---|---:|---:|---:| | Full scenario matrix | 100 | Identical ranges and NULL maps | N/A (includes negative controls) | N/A | N/A | | Production-eligible scenarios | 30 | Identical ranges and NULL maps | 0 | -42.68% | 8.08% | | Scenario | Repetitions | Legacy median CPU | Fused median CPU | CPU change | Legacy CV | Fused CV | |---|---:|---:|---:|---:|---:|---:| | 10% selectivity / 50% NULL, fragmented | 10 | 400,220 ns | 177,905 ns | -55.55% | 0.58% | 1.28% | | 99% selectivity / 50% NULL, high-selectivity boundary | 10 | 687,385 ns | 460,657 ns | -32.98% | 0.60% | 1.32% | The microbenchmark isolates nullable selection planning; it is not presented as an end-to-end query speedup. --- be/benchmark/parquet/AGENTS.md | 12 +- be/benchmark/parquet/README.md | 11 +- .../parquet/benchmark_parquet_kernels.hpp | 202 ++++++++++++++++++ .../parquet/parquet_benchmark_scenarios.h | 37 ++++ .../reader/native/column_chunk_reader.cpp | 59 +++++ .../reader/native/column_chunk_reader.h | 8 + .../parquet/reader/native/column_reader.cpp | 43 +++- .../parquet/reader/native/column_reader.h | 1 + .../parquet/reader/native/common.cpp | 128 +++++++++++ .../format_v2/parquet/reader/native/common.h | 11 + .../parquet_benchmark_scenarios_test.cpp | 26 +++ .../parquet/parquet_reader_control_test.cpp | 74 +++++++ 12 files changed, 601 insertions(+), 11 deletions(-) diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index 4c1d3cf4d5e197..4d8c0610f17b5d 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -51,7 +51,7 @@ be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetDecoder/' # currently 228 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetKernel/' # currently 92 + | grep -c '^ParquetKernel/' # currently 292 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetSelection/' # currently 25 @@ -146,13 +146,19 @@ cache to manufacture a cold run. | DELTA_LENGTH_BYTE_ARRAY | BYTE_ARRAY | | DELTA_BYTE_ARRAY | BYTE_ARRAY | -`ParquetKernel` contains 92 cases across six decode and selection stages: BYTE_STREAM_SPLIT, -DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, RAW_PREDICATE, and NESTED_SELECTION. It covers +`ParquetKernel` contains 292 cases across seven decode and selection stages: BYTE_STREAM_SPLIT, +DELTA_PREFIX_SUM, DICTIONARY_GATHER, NULLABLE_EXPAND, NULLABLE_SELECTION, RAW_PREDICATE, and +NESTED_SELECTION. It covers the applicable four- and eight-byte types, three dictionary working-set sizes, 0% through 90% null rates with both placement patterns, 0% through 100% raw-predicate selectivities, and 1%, 10%, and 50% nested parent-row selectivities with both placement patterns. Nested selection registers the legacy and fused implementations in the same binary and validates both against an independent source-level oracle before timing. +Nullable selection contributes 200 legacy/fused cases across five selectivities, five null rates, +and independent clustered or alternating selection/null placement. Each pair is validated for +identical physical ranges and null maps before timing. Treat no-NULL, low-NULL, and clustered +level-plan cases as negative controls: production fusion is gated to batches with at least 1,024 +rows, at least 10% NULLs, and materially fragmented definition-level runs. `ParquetSelection` contains 25 cases that isolate the selection-vector work used by Parquet predicate evaluation. It measures identity initialization, one raw-row filter, and two successive diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 156302b24238d1..891fd02d8375ed 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -45,14 +45,21 @@ be/output/lib/benchmark_test \ ## SIMD kernel cases -`ParquetKernel` isolates six decode and selection stages from reader setup and virtual consumer +`ParquetKernel` isolates seven decode and selection stages from reader setup and virtual consumer overhead: byte-stream-split transpose, delta prefix sum, numeric dictionary gather, nullable -expansion, raw predicate evaluation, and repeated-level sparse selection. It covers the applicable +expansion, nullable selection planning, raw predicate evaluation, and repeated-level sparse +selection. It covers the applicable 4-byte and 8-byte integer and floating-point physical types, raw-predicate selectivities from 0% through 100%, and nullable rates from 0% through 90% with clustered and alternating placement. Nested selection covers 1%, 10%, and 50% surviving parent rows with both placement patterns. Each nested-selection scenario registers both `impl_legacy` and `impl_fused`; both paths use the same source levels and are checked against an independent oracle before timing. +Nullable selection planning registers legacy and fused pairs across five selectivities, five null +rates, and independent clustered or alternating selection/null placement. Both implementations are +checked for identical physical ranges and null maps before timing. The full matrix also acts as a +negative control: production fusion is limited to batches with at least 1,024 rows, at least 10% +NULLs, and fragmented definition-level runs; no-NULL, low-NULL, and clustered pages retain the +legacy planner. Dictionary gather uses 32-, 4,096-, and 262,144-entry working sets to separate cache-resident and cache-miss-dominated behavior. diff --git a/be/benchmark/parquet/benchmark_parquet_kernels.hpp b/be/benchmark/parquet/benchmark_parquet_kernels.hpp index 619d58fd8cf8ac..de3a15c254b8db 100644 --- a/be/benchmark/parquet/benchmark_parquet_kernels.hpp +++ b/be/benchmark/parquet/benchmark_parquet_kernels.hpp @@ -63,6 +63,191 @@ struct NestedSelectionScratch { size_t ancestor_null_count = 0; }; +struct NullableSelectionScratch { + format::parquet::native::ColumnSelectVector legacy_selection; + ParquetSelection physical_selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; +}; + +inline void append_nullable_run(std::vector* runs, bool is_null, size_t run_length, + bool* previous_is_null) { + if (runs->empty()) { + if (is_null) { + runs->push_back(0); + } + } else if (*previous_is_null == is_null) { + runs->push_back(0); + } + while (run_length > USHRT_MAX) { + runs->push_back(USHRT_MAX); + runs->push_back(0); + run_length -= USHRT_MAX; + } + runs->push_back(static_cast(run_length)); + *previous_is_null = is_null; +} + +inline std::vector build_nullable_runs(const NullMap& nulls) { + std::vector runs; + bool previous_is_null = false; + size_t row = 0; + while (row < nulls.size()) { + const bool is_null = nulls[row] != 0; + const size_t begin = row++; + while (row < nulls.size() && (nulls[row] != 0) == is_null) { + ++row; + } + append_nullable_run(&runs, is_null, row - begin, &previous_is_null); + } + return runs; +} + +inline Status run_legacy_nullable_selection(NullableSelectionScratch* scratch, + const std::vector& null_runs, + size_t num_values, + format::parquet::native::FilterMap* filter) { + using ReadType = format::parquet::native::ColumnSelectVector::DataReadType; + scratch->output_nulls.clear(); + scratch->selected_nulls.clear(); + scratch->physical_selection.ranges.clear(); + scratch->physical_selection.total_values = 0; + scratch->physical_selection.selected_values = 0; + RETURN_IF_ERROR(scratch->legacy_selection.init(null_runs, num_values, &scratch->output_nulls, + filter, 0)); + scratch->num_filtered = scratch->legacy_selection.num_filtered(); + + size_t physical_cursor = 0; + ReadType type; + while (const size_t run_length = scratch->legacy_selection.get_next_run(&type)) { + switch (type) { + case ReadType::CONTENT: + if (!scratch->physical_selection.ranges.empty() && + scratch->physical_selection.ranges.back().first + + scratch->physical_selection.ranges.back().count == + physical_cursor) { + scratch->physical_selection.ranges.back().count += run_length; + } else { + scratch->physical_selection.ranges.push_back( + {.first = physical_cursor, .count = run_length}); + } + scratch->physical_selection.selected_values += run_length; + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 0); + physical_cursor += run_length; + break; + case ReadType::NULL_DATA: + scratch->selected_nulls.resize_fill(scratch->selected_nulls.size() + run_length, 1); + break; + case ReadType::FILTERED_CONTENT: + physical_cursor += run_length; + break; + case ReadType::FILTERED_NULL: + break; + } + } + scratch->physical_selection.total_values = physical_cursor; + return Status::OK(); +} + +inline Status run_nullable_selection_once(NullableSelectionScratch* scratch, + const std::vector& null_runs, size_t num_values, + size_t num_nulls, + format::parquet::native::FilterMap* filter, + NullableSelectionImplementation implementation) { + if (implementation == NullableSelectionImplementation::LEGACY) { + return run_legacy_nullable_selection(scratch, null_runs, num_values, filter); + } + scratch->output_nulls.clear(); + return format::parquet::native::build_filtered_nullable_selection( + null_runs, num_values, num_nulls, &scratch->output_nulls, filter, 0, + &scratch->physical_selection, &scratch->selected_nulls, &scratch->num_filtered); +} + +inline bool equal_selection(const ParquetSelection& lhs, const ParquetSelection& rhs) { + if (lhs.total_values != rhs.total_values || lhs.selected_values != rhs.selected_values || + lhs.ranges.size() != rhs.ranges.size()) { + return false; + } + for (size_t range = 0; range < lhs.ranges.size(); ++range) { + if (lhs.ranges[range].first != rhs.ranges[range].first || + lhs.ranges[range].count != rhs.ranges[range].count) { + return false; + } + } + return true; +} + +inline void run_nullable_selection_kernel(benchmark::State& state, + const NullableSelectionScenario& scenario) { + using format::parquet::native::FilterMap; + + std::vector filter_data(KERNEL_ROWS, 0); + const auto selected = make_selection_plan(KERNEL_ROWS, scenario.selectivity_percent, + scenario.selection_pattern); + visit_selected_rows(selected, [&](size_t row) { filter_data[row] = 1; }); + FilterMap filter; + auto status = filter.init(filter_data.data(), filter_data.size(), false); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + + NullMap nulls; + nulls.resize_fill(KERNEL_ROWS, 0); + const auto null_plan = + make_selection_plan(KERNEL_ROWS, scenario.null_percent, scenario.null_pattern); + visit_selected_rows(null_plan, [&](size_t row) { nulls[row] = 1; }); + const auto null_runs = build_nullable_runs(nulls); + + NullableSelectionScratch legacy; + NullableSelectionScratch fused; + status = run_nullable_selection_once(&legacy, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::LEGACY); + if (status.ok()) { + status = + run_nullable_selection_once(&fused, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, NullableSelectionImplementation::FUSED); + } + if (!status.ok() || !equal_selection(legacy.physical_selection, fused.physical_selection) || + legacy.output_nulls != fused.output_nulls || + legacy.selected_nulls != fused.selected_nulls || + legacy.num_filtered != fused.num_filtered) { + if (status.ok()) { + state.SkipWithError("nullable selection implementations disagree"); + } else { + state.SkipWithError(status.to_string().c_str()); + } + return; + } + + NullableSelectionScratch scratch; + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, null_plan.selected_rows, + &filter, scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + for (auto _ : state) { + status = run_nullable_selection_once(&scratch, null_runs, KERNEL_ROWS, + null_plan.selected_rows, &filter, + scenario.implementation); + if (!status.ok()) { + state.SkipWithError(status.to_string().c_str()); + return; + } + benchmark::DoNotOptimize(scratch.physical_selection.ranges.data()); + benchmark::DoNotOptimize(scratch.selected_nulls.data()); + benchmark::ClobberMemory(); + } + + state.SetItemsProcessed(static_cast(state.iterations()) * + static_cast(KERNEL_ROWS)); + state.counters["rows"] = static_cast(KERNEL_ROWS); + state.counters["selected_rows"] = static_cast(selected.selected_rows); + state.counters["null_rows"] = static_cast(null_plan.selected_rows); +} + inline NestedSelectionOracle build_nested_selection_oracle( const std::vector& repetition_levels, const std::vector& definition_levels, @@ -493,7 +678,24 @@ inline bool register_kernel_benchmarks() { return true; } +inline bool register_nullable_selection_benchmarks() { + for (const auto& scenario : nullable_selection_scenarios()) { + const std::string name = "ParquetKernel/nullable_selection/sel_" + + std::to_string(scenario.selectivity_percent) + "/null_" + + std::to_string(scenario.null_percent) + "/selection_" + + to_string(scenario.selection_pattern) + "/nulls_" + + to_string(scenario.null_pattern) + "/impl_" + + to_string(scenario.implementation); + benchmark::RegisterBenchmark(name.c_str(), [=](benchmark::State& state) { + run_nullable_selection_kernel(state, scenario); + })->Unit(benchmark::kNanosecond); + } + return true; +} + inline const bool KERNEL_BENCHMARKS_REGISTERED = register_kernel_benchmarks(); +inline const bool NULLABLE_SELECTION_BENCHMARKS_REGISTERED = + register_nullable_selection_benchmarks(); } // namespace detail } // namespace doris::parquet_benchmark diff --git a/be/benchmark/parquet/parquet_benchmark_scenarios.h b/be/benchmark/parquet/parquet_benchmark_scenarios.h index a9c58c15d8cff1..f7eddf0b11a5aa 100644 --- a/be/benchmark/parquet/parquet_benchmark_scenarios.h +++ b/be/benchmark/parquet/parquet_benchmark_scenarios.h @@ -55,6 +55,7 @@ enum class Kernel { NESTED_SELECTION }; enum class NestedSelectionImplementation { LEGACY, FUSED }; +enum class NullableSelectionImplementation { LEGACY, FUSED }; struct DecoderScenario { Encoding encoding; @@ -89,6 +90,14 @@ struct SelectionScenario { Pattern pattern; }; +struct NullableSelectionScenario { + int selectivity_percent; + int null_percent; + Pattern selection_pattern; + Pattern null_pattern; + NullableSelectionImplementation implementation; +}; + struct SelectionRange { size_t first; size_t count; @@ -177,6 +186,24 @@ inline std::vector selection_scenarios() { return scenarios; } +inline std::vector nullable_selection_scenarios() { + std::vector scenarios; + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + scenarios.push_back({selectivity, null_percent, selection_pattern, + null_pattern, implementation}); + } + } + } + } + } + return scenarios; +} + inline std::vector reader_scenarios() { std::vector scenarios; std::set::materialize_values( return Status::OK(); } +template +bool ColumnChunkReader::supports_fused_nullable_selection( + IColumn& column) const { + return visit_nullable_expandable_column(column, [](auto&) {}); +} + +template +Status ColumnChunkReader::materialize_fused_nullable_values( + MutableColumnPtr& doris_column, const DataTypeSerDe& serde, ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, size_t num_nulls, + const NullMap& selected_nulls) { + if (num_values == 0) { + return Status::OK(); + } + SCOPED_RAW_TIMER(&_chunk_statistics.decode_value_time); + DORIS_CHECK_GT(num_nulls, 0); + const size_t physical_values = num_values - num_nulls; + DORIS_CHECK_EQ(state.selection.total_values, physical_values); + DORIS_CHECK_LE(state.selection.selected_values, selected_nulls.size()); + if (UNLIKELY(_empty_value_section && physical_values != 0)) { + return Status::Corruption( + "Parquet definition levels require {} values from an empty value section", + physical_values); + } + if (UNLIKELY((doris_column->is_column_dictionary() || context.dictionary_index_only) && + !_has_dict && physical_values != 0)) { + return Status::IOError("Not dictionary coded"); + } + if (UNLIKELY(_remaining_num_values < num_values)) { + return Status::IOError("Decode too many values in current page"); + } + RETURN_IF_ERROR(translate_value_encoding(_current_encoding, &context.encoding)); + + ++_chunk_statistics.hybrid_selection_batches; + const auto status = decode_prepared_nullable_values(*doris_column, serde, *_page_decoder, + context, state, selected_nulls, + &_chunk_statistics.materialization_time); + _chunk_statistics.hybrid_selection_ranges += state.selection.ranges.size(); + RETURN_IF_ERROR(status); + _remaining_num_values -= num_values; + return Status::OK(); +} + template bool ColumnChunkReader::can_filter_fixed_width_values( const VExprSPtrs& conjuncts, int column_id, const DataTypeSerDe* serde, diff --git a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h index 35bf5200d7aebb..a50b7e4bc080ff 100644 --- a/be/src/format_v2/parquet/reader/native/column_chunk_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_chunk_reader.h @@ -177,6 +177,14 @@ class ColumnChunkReader { ParquetDecodeContext& context, ParquetMaterializationState& state, ColumnSelectVector& select_vector); + bool supports_fused_nullable_selection(IColumn& column) const; + + Status materialize_fused_nullable_values(MutableColumnPtr& doris_column, + const DataTypeSerDe& serde, + ParquetDecodeContext& context, + ParquetMaterializationState& state, size_t num_values, + size_t num_nulls, const NullMap& selected_nulls); + static bool supports_raw_fixed_filter_encoding(tparquet::Encoding::type encoding, tparquet::Type::type physical_type) { switch (encoding) { diff --git a/be/src/format_v2/parquet/reader/native/column_reader.cpp b/be/src/format_v2/parquet/reader/native/column_reader.cpp index 7cda926162cbd1..314a28690417e5 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native/column_reader.cpp @@ -809,6 +809,8 @@ void ScalarColumnReader::release_batch_scratch( max_retained_bytes); release_selection |= release_vector_if_oversized(&_materialization_state.selection.ranges, max_retained_bytes); + release_selection |= + release_filter_if_oversized(&_fused_nullable_selection_nulls, max_retained_bytes); release_selection |= release_filter_if_oversized(&_fixed_width_predicate_nulls, max_retained_bytes); release_selection |= @@ -842,6 +844,7 @@ void ScalarColumnReader::release_batch_scratch( release_selection |= release_vector_for_aggregate(&_nested_filter_map_data); release_selection |= release_vector_for_aggregate(&_materialization_state.dictionary_indices); release_selection |= release_vector_for_aggregate(&_materialization_state.selection.ranges); + release_selection |= release_filter_for_aggregate(&_fused_nullable_selection_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_nulls); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_matches); release_selection |= release_filter_for_aggregate(&_fixed_width_predicate_conversion_nulls); @@ -870,7 +873,8 @@ size_t ScalarColumnReader::retained_batch_scratch_b _def_levels.capacity() * sizeof(level_t) + _null_run_lengths.capacity() * sizeof(uint16_t) + _nested_filter_map_data.capacity() * sizeof(uint8_t) + - _fixed_width_predicate_nulls.capacity() + _fixed_width_predicate_matches.capacity() + + _fused_nullable_selection_nulls.capacity() + _fixed_width_predicate_nulls.capacity() + + _fixed_width_predicate_matches.capacity() + _fixed_width_predicate_conversion_nulls.capacity() + _materialization_state.dictionary_indices.capacity() * sizeof(uint32_t) + _materialization_state.selection.ranges.capacity() * sizeof(ParquetSelectionRange) + @@ -885,7 +889,8 @@ size_t ScalarColumnReader::active_batch_scratch_byt _serde == nullptr ? 0 : _serde->active_parquet_raw_predicate_scratch_bytes(); return decoder_bytes + serde_bytes + _rep_levels.size() * sizeof(level_t) + _def_levels.size() * sizeof(level_t) + _null_run_lengths.size() * sizeof(uint16_t) + - _nested_filter_map_data.size() * sizeof(uint8_t) + _fixed_width_predicate_nulls.size() + + _nested_filter_map_data.size() * sizeof(uint8_t) + + _fused_nullable_selection_nulls.size() + _fixed_width_predicate_nulls.size() + _fixed_width_predicate_matches.size() + _fixed_width_predicate_conversion_nulls.size() + _materialization_state.dictionary_indices.size() * sizeof(uint32_t) + _materialization_state.selection.ranges.size() * sizeof(ParquetSelectionRange) + @@ -902,6 +907,7 @@ void ScalarColumnReader::reserve_batch_scratch_for_ _nested_filter_map_data.reserve(elements); _materialization_state.dictionary_indices.reserve(elements); _materialization_state.selection.ranges.reserve(elements); + _fused_nullable_selection_nulls.reserve(elements); _ancestor_null_indices.reserve(elements); } @@ -965,6 +971,7 @@ Status ScalarColumnReader::_read_values(size_t num_ } MutableColumnPtr data_column; _null_run_lengths.clear(); + size_t num_nulls = 0; NullMap* map_data_column = nullptr; doris_column = IColumn::mutate(std::move(doris_column)); if (is_column_nullable(*doris_column)) { @@ -987,6 +994,9 @@ Status ScalarColumnReader::_read_values(size_t num_ } bool is_null = def_level < _field_schema->definition_level; + if (is_null) { + num_nulls += loop_read; + } if (!(prev_is_null ^ is_null)) { _null_run_lengths.emplace_back(0); } @@ -1016,10 +1026,26 @@ Status ScalarColumnReader::_read_values(size_t num_ } _null_run_lengths.emplace_back((u_short)remaining); } + const bool use_fused_nullable_selection = + map_data_column != nullptr && filter_map.has_filter() && num_nulls > 0 && + should_use_fused_nullable_selection(num_values, num_nulls, _null_run_lengths.size()) && + _chunk_reader->supports_fused_nullable_selection(*data_column); { SCOPED_RAW_TIMER(&_decode_null_map_time); - RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, - &filter_map, _filter_map_index)); + if (use_fused_nullable_selection) { + size_t num_filtered = 0; + // The fused path owns both the physical ranges and selected NULL layout. Restrict it + // to fragmented, materially nullable level plans: clustered, low-NULL, and no-NULL + // pages already collapse into a few cheap legacy runs, while fusing them adds planning + // branches without removing enough work to guarantee a win. + RETURN_IF_ERROR(build_filtered_nullable_selection( + _null_run_lengths, num_values, num_nulls, map_data_column, &filter_map, + _filter_map_index, &_materialization_state.selection, + &_fused_nullable_selection_nulls, &num_filtered)); + } else { + RETURN_IF_ERROR(_select_vector.init(_null_run_lengths, num_values, map_data_column, + &filter_map, _filter_map_index)); + } _filter_map_index += num_values; } DORIS_CHECK(_serde != nullptr); @@ -1030,8 +1056,13 @@ Status ScalarColumnReader::_read_values(size_t num_ conversion_failure_map(*_field_schema, type, _materialization_state.enable_strict_mode, map_data_column, &compatibility_scratch); const size_t materialization_start_row = data_column->size(); - const auto status = _chunk_reader->materialize_values(data_column, *_serde, _decode_context, - _materialization_state, _select_vector); + const auto status = + use_fused_nullable_selection + ? _chunk_reader->materialize_fused_nullable_values( + data_column, *_serde, _decode_context, _materialization_state, + num_values, num_nulls, _fused_nullable_selection_nulls) + : _chunk_reader->materialize_values(data_column, *_serde, _decode_context, + _materialization_state, _select_vector); _materialization_state.conversion_failure_null_map = nullptr; if (status.ok()) { mark_local_timestamp_defaults(*_field_schema, type, diff --git a/be/src/format_v2/parquet/reader/native/column_reader.h b/be/src/format_v2/parquet/reader/native/column_reader.h index ed8bc7d8f697bc..b43822cd1653b0 100644 --- a/be/src/format_v2/parquet/reader/native/column_reader.h +++ b/be/src/format_v2/parquet/reader/native/column_reader.h @@ -433,6 +433,7 @@ class ScalarColumnReader : public ColumnReader { std::vector _null_run_lengths; std::unordered_set _ancestor_null_indices; std::vector _nested_filter_map_data; + NullMap _fused_nullable_selection_nulls; NullMap _fixed_width_predicate_nulls; IColumn::Filter _fixed_width_predicate_matches; IColumn::Filter _fixed_width_predicate_conversion_nulls; diff --git a/be/src/format_v2/parquet/reader/native/common.cpp b/be/src/format_v2/parquet/reader/native/common.cpp index c50488575e4993..020f613c5d267f 100644 --- a/be/src/format_v2/parquet/reader/native/common.cpp +++ b/be/src/format_v2/parquet/reader/native/common.cpp @@ -17,6 +17,7 @@ #include "format_v2/parquet/reader/native/common.h" +#include #include #include "core/types.h" @@ -68,6 +69,133 @@ bool FilterMap::can_filter_all(size_t remaining_num_values, size_t filter_map_in remaining_num_values) == remaining_num_values; } +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, + size_t num_null_runs) { + constexpr size_t MIN_BATCH_VALUES = 1024; + constexpr size_t MIN_NULL_RUNS = 32; + constexpr size_t MAX_AVERAGE_NULL_RUN = 64; + constexpr size_t MIN_NULL_RATIO_DENOMINATOR = 10; + if (num_values < MIN_BATCH_VALUES || num_nulls < num_values / MIN_NULL_RATIO_DENOMINATOR) { + return false; + } + return num_null_runs >= std::max(MIN_NULL_RUNS, num_values / MAX_AVERAGE_NULL_RUN); +} + +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered) { + if (output_null_map == nullptr || filter_map == nullptr || selection == nullptr || + selected_nulls == nullptr || num_filtered == nullptr) { + return Status::InvalidArgument( + "Nullable selection planning requires non-null output state"); + } + if (!filter_map->has_filter()) { + return Status::InvalidArgument("Nullable selection planning requires a row filter"); + } + if (!filter_map->filter_all() && + (filter_map->filter_map_data() == nullptr || + filter_map_index + num_values > filter_map->filter_map_size())) { + return Status::InvalidArgument("Nullable selection filter range [{}, {}) exceeds size {}", + filter_map_index, filter_map_index + num_values, + filter_map->filter_map_size()); + } + if (num_nulls > num_values) { + return Status::InvalidArgument("Nullable selection has {} nulls for {} values", num_nulls, + num_values); + } + + selection->ranges.clear(); + selection->total_values = num_values - num_nulls; + selection->selected_values = 0; + selected_nulls->clear(); + *num_filtered = 0; + if (filter_map->filter_all()) { + *num_filtered = num_values; + return Status::OK(); + } + + selected_nulls->reserve(num_values); + const uint8_t* filter = filter_map->filter_map_data() + filter_map_index; + const auto select_physical_values = [&](size_t physical_index, size_t count) { + if (!selection->ranges.empty() && + selection->ranges.back().first + selection->ranges.back().count == physical_index) { + selection->ranges.back().count += count; + } else { + selection->ranges.push_back({.first = physical_index, .count = count}); + } + selection->selected_values += count; + }; + + if (num_nulls == 0) { + size_t row = 0; + while (row < num_values) { + const bool selected = filter[row] != 0; + const size_t run_start = row++; + while (row < num_values && (filter[row] != 0) == selected) { + ++row; + } + const size_t run_length = row - run_start; + if (selected) { + select_physical_values(run_start, run_length); + } else { + *num_filtered += run_length; + } + } + selected_nulls->resize_fill(selection->selected_values, 0); + } else { + size_t logical_index = 0; + size_t physical_index = 0; + size_t observed_nulls = 0; + bool is_null = false; + for (const size_t run_length : run_length_null_map) { + if (logical_index + run_length > num_values) { + return Status::InvalidArgument("Nullable selection run lengths exceed {} values", + num_values); + } + const size_t run_end = logical_index + run_length; + while (logical_index < run_end) { + const bool selected = filter[logical_index] != 0; + const size_t filter_run_start = logical_index++; + while (logical_index < run_end && (filter[logical_index] != 0) == selected) { + ++logical_index; + } + const size_t filter_run_length = logical_index - filter_run_start; + if (selected) { + selected_nulls->resize_fill(selected_nulls->size() + filter_run_length, + static_cast(is_null)); + if (!is_null) { + select_physical_values(physical_index, filter_run_length); + } + } else { + *num_filtered += filter_run_length; + } + if (!is_null) { + physical_index += filter_run_length; + } else { + observed_nulls += filter_run_length; + } + } + is_null = !is_null; + } + if (logical_index != num_values || observed_nulls != num_nulls || + physical_index != selection->total_values) { + return Status::InvalidArgument( + "Nullable selection level plan is inconsistent: values={}, nulls={}", + logical_index, observed_nulls); + } + } + + const size_t old_null_size = output_null_map->size(); + output_null_map->resize(old_null_size + selected_nulls->size()); + if (!selected_nulls->empty()) { + memcpy(output_null_map->data() + old_null_size, selected_nulls->data(), + selected_nulls->size()); + } + return Status::OK(); +} + Status FilterMap::generate_nested_filter_map(const std::vector& rep_levels, std::vector& nested_filter_map_data, std::unique_ptr* nested_filter_map, diff --git a/be/src/format_v2/parquet/reader/native/common.h b/be/src/format_v2/parquet/reader/native/common.h index eb6848ee299f10..bd687616f226ce 100644 --- a/be/src/format_v2/parquet/reader/native/common.h +++ b/be/src/format_v2/parquet/reader/native/common.h @@ -25,6 +25,7 @@ #include "common/status.h" #include "core/column/column_nullable.h" +#include "core/data_type_serde/parquet_decode_source.h" namespace doris::format::parquet::native { @@ -116,4 +117,14 @@ class ColumnSelectVector { size_t _read_index = 0; }; +Status build_filtered_nullable_selection(const std::vector& run_length_null_map, + size_t num_values, size_t num_nulls, + NullMap* output_null_map, FilterMap* filter_map, + size_t filter_map_index, ParquetSelection* selection, + NullMap* selected_nulls, size_t* num_filtered); + +// Fusion pays for its additional planning branches only when definition levels are materially +// nullable and fragmented. Keep compact/no-NULL batches on the run-oriented legacy path. +bool should_use_fused_nullable_selection(size_t num_values, size_t num_nulls, size_t num_null_runs); + } // namespace doris::format::parquet::native diff --git a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp index 2145b6ab60da43..fe1d871816ed26 100644 --- a/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp +++ b/be/test/format_v2/parquet/parquet_benchmark_scenarios_test.cpp @@ -147,6 +147,32 @@ TEST(ParquetBenchmarkScenariosTest, SelectionMatrixCoversIdentityAndSuccessiveCo } } +TEST(ParquetBenchmarkScenariosTest, NullableSelectionPairsLegacyAndFusedAcrossRowShapes) { + const auto scenarios = nullable_selection_scenarios(); + EXPECT_EQ(scenarios.size(), size_t {200}); + for (const int selectivity : {1, 10, 50, 90, 99}) { + for (const int null_percent : {0, 1, 10, 50, 90}) { + for (const auto selection_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto null_pattern : {Pattern::CLUSTERED, Pattern::ALTERNATING}) { + for (const auto implementation : {NullableSelectionImplementation::LEGACY, + NullableSelectionImplementation::FUSED}) { + EXPECT_TRUE(std::ranges::any_of( + scenarios, + [&](const NullableSelectionScenario& scenario) { + return scenario.selectivity_percent == selectivity && + scenario.null_percent == null_percent && + scenario.selection_pattern == selection_pattern && + scenario.null_pattern == null_pattern && + scenario.implementation == implementation; + })) + << "missing nullable selection comparison shape"; + } + } + } + } + } +} + TEST(ParquetBenchmarkScenariosTest, ReaderMatrixCoversNullableSparseAndProjectionAxes) { const auto scenarios = reader_scenarios(); // Keep the exact count aligned with the upstream complex-residual scenario retained by rebase. diff --git a/be/test/format_v2/parquet/parquet_reader_control_test.cpp b/be/test/format_v2/parquet/parquet_reader_control_test.cpp index e21439e885570d..37deb32d5bf37b 100644 --- a/be/test/format_v2/parquet/parquet_reader_control_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_control_test.cpp @@ -27,6 +27,7 @@ #include "core/column/column_string.h" #include "core/column/column_vector.h" #include "core/data_type/data_type_number.h" +#include "core/data_type_serde/parquet_decode_source.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_scan.h" #include "format_v2/parquet/reader/column_reader.h" @@ -186,6 +187,79 @@ TEST(SelectionVectorTest, IdentitySelectionDoesNotMaterializeFilter) { EXPECT_EQ(filter, nullptr); } +TEST(NativeNullableSelectionTest, BuildsPhysicalRangesAndSelectedNullsInOnePass) { + using native::FilterMap; + + const std::vector null_runs {2, 1, 3, 2, 2}; + const std::vector filter_data {1, 0, 1, 1, 0, 1, 1, 1, 0, 1}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls {1}; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(null_runs, filter_data.size(), 3, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 7); + EXPECT_EQ(selection.selected_values, 4); + ASSERT_EQ(selection.ranges.size(), 4); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 1); + EXPECT_EQ(selection.ranges[1].first, 2); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 4); + EXPECT_EQ(selection.ranges[2].count, 1); + EXPECT_EQ(selection.ranges[3].first, 6); + EXPECT_EQ(selection.ranges[3].count, 1); + EXPECT_EQ(selected_nulls, (NullMap {0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(output_nulls, (NullMap {1, 0, 1, 0, 0, 1, 1, 0})); + EXPECT_EQ(num_filtered, 3); +} + +TEST(NativeNullableSelectionTest, UsesDirectPhysicalCoordinatesWithoutNulls) { + using native::FilterMap; + + const std::vector no_nulls {10}; + const std::vector filter_data {1, 1, 0, 1, 0, 0, 1, 1, 1, 0}; + FilterMap filter; + ASSERT_TRUE(filter.init(filter_data.data(), filter_data.size(), false).ok()); + ParquetSelection selection; + NullMap output_nulls; + NullMap selected_nulls; + size_t num_filtered = 0; + + ASSERT_TRUE(native::build_filtered_nullable_selection(no_nulls, filter_data.size(), 0, + &output_nulls, &filter, 0, &selection, + &selected_nulls, &num_filtered) + .ok()); + + EXPECT_EQ(selection.total_values, 10); + EXPECT_EQ(selection.selected_values, 6); + ASSERT_EQ(selection.ranges.size(), 3); + EXPECT_EQ(selection.ranges[0].first, 0); + EXPECT_EQ(selection.ranges[0].count, 2); + EXPECT_EQ(selection.ranges[1].first, 3); + EXPECT_EQ(selection.ranges[1].count, 1); + EXPECT_EQ(selection.ranges[2].first, 6); + EXPECT_EQ(selection.ranges[2].count, 3); + EXPECT_EQ(selected_nulls, (NullMap {0, 0, 0, 0, 0, 0})); + EXPECT_EQ(output_nulls, selected_nulls); + EXPECT_EQ(num_filtered, 4); +} + +TEST(NativeNullableSelectionTest, EnablesFusionOnlyForMateriallyFragmentedNullableBatches) { + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 0, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 655, 1311)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(65536, 32768, 3)); + EXPECT_FALSE(native::should_use_fused_nullable_selection(512, 256, 512)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 6553, 13107)); + EXPECT_TRUE(native::should_use_fused_nullable_selection(65536, 32768, 65536)); +} + TEST(NativeNestedSelectionTest, BuildsSelectionAndCompactsSurvivingParentLevels) { using native::ColumnSelectVector; using native::FilterMap; From 5954b45f424ffabc21cd84ccd3cb1903c65f82d7 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 11:30:10 +0800 Subject: [PATCH 02/20] [feature](variant) Support reading Iceberg Variant from Parquet (#66302) Forward-port the Iceberg Variant Parquet reader to the plugin-driven connector on master while preserving mixed-version scan safety and delete-only merge behavior. --- be/src/core/block/block.cpp | 5 +- .../column/variant_v2/column_variant_v2.cpp | 172 ++- .../column/variant_v2/column_variant_v2.h | 63 +- .../column_variant_v2_read_view.cpp | 3 + be/src/exec/operator/file_scan_operator.cpp | 43 + be/src/exec/operator/scan_operator.h | 12 + be/src/exec/scan/access_path_parser.cpp | 95 +- be/src/exec/scan/access_path_parser.h | 6 + be/src/exec/scan/file_scanner_v2.cpp | 37 +- be/src/exec/scan/file_scanner_v2.h | 2 + be/src/exec/scan/scanner.cpp | 9 + be/src/exec/scan/scanner.h | 4 + be/src/exec/scan/split_source_connector.h | 17 + be/src/exec/sink/viceberg_merge_sink.cpp | 26 +- be/src/exec/sink/viceberg_merge_sink.h | 1 + .../function/function_variant_element_v2.cpp | 45 +- be/src/format_v2/column_data.h | 24 + be/src/format_v2/column_mapper.cpp | 373 ++++-- be/src/format_v2/column_mapper.h | 17 + be/src/format_v2/column_mapper_nested.cpp | 116 -- be/src/format_v2/column_mapper_nested.h | 2 - be/src/format_v2/file_reader.cpp | 8 + be/src/format_v2/file_reader.h | 76 +- .../format_v2/parquet/native_schema_desc.cpp | 434 +++++++ be/src/format_v2/parquet/native_schema_desc.h | 4 + .../format_v2/parquet/native_schema_node.cpp | 3 +- .../parquet/parquet_column_schema.cpp | 63 +- .../format_v2/parquet/parquet_column_schema.h | 4 + be/src/format_v2/parquet/parquet_profile.cpp | 20 + be/src/format_v2/parquet/parquet_profile.h | 14 + be/src/format_v2/parquet/parquet_reader.cpp | 123 ++ be/src/format_v2/parquet/parquet_reader.h | 9 + be/src/format_v2/parquet/parquet_scan.cpp | 29 +- .../format_v2/parquet/parquet_statistics.cpp | 458 ++++++- be/src/format_v2/parquet/parquet_statistics.h | 2 +- be/src/format_v2/parquet/parquet_type.h | 2 + .../parquet/reader/count_column_reader.cpp | 11 + .../parquet/reader/native_column_reader.cpp | 217 +++- .../parquet/reader/native_column_reader.h | 8 +- .../parquet/reader/variant_column_reader.cpp | 1112 +++++++++++++++++ .../parquet/reader/variant_column_reader.h | 67 + be/src/format_v2/schema_projection.cpp | 5 + be/src/format_v2/table/iceberg_reader.cpp | 88 ++ be/src/format_v2/table/iceberg_reader.h | 4 + be/src/format_v2/table_reader.h | 36 +- .../core/column/column_variant_v2_test.cpp | 27 + be/test/exec/scan/access_path_parser_test.cpp | 87 ++ be/test/exec/scan/file_scanner_v2_test.cpp | 34 + .../scan/scanner_late_arrival_rf_test.cpp | 49 +- .../exec/sink/viceberg_merge_sink_test.cpp | 38 + be/test/format_v2/column_mapper_test.cpp | 904 +++++--------- .../format_v2/parquet/parquet_reader_test.cpp | 570 +++++++++ .../format_v2/parquet/parquet_schema_test.cpp | 397 ++++++ .../parquet/parquet_statistics_test.cpp | 437 ++++++- .../parquet/variant_column_reader_test.cpp | 927 ++++++++++++++ .../format_v2/table/iceberg_reader_test.cpp | 39 + .../format_v2/table_reader_request_test.cpp | 26 + .../iceberg/IcebergScanPlanProvider.java | 13 + .../connector/iceberg/IcebergTypeMapping.java | 6 +- .../iceberg/IcebergWritePlanProvider.java | 28 +- .../iceberg/IcebergScanPlanProviderTest.java | 16 + .../iceberg/IcebergTypeMappingReadTest.java | 10 +- .../iceberg/IcebergWritePlanProviderTest.java | 13 + .../spi/handle/ConnectorWriteHandle.java | 8 + .../spi/scan/ConnectorScanPlanProvider.java | 11 + .../converter/ConnectorColumnConverter.java | 5 + .../datasource/scan/PluginDrivenScanNode.java | 63 + .../translator/PhysicalPlanTranslator.java | 12 +- ...nkToPhysicalExternalRowLevelMergeSink.java | 1 + .../AccessPathExpressionCollector.java | 9 + .../rewrite/AccessPathPlanCollector.java | 7 + .../rules/rewrite/NestedColumnPruning.java | 5 + .../rules/rewrite/SlotTypeReplacer.java | 5 + .../ExternalRowLevelMergePlanBuilder.java | 2 + .../ExternalRowLevelUpdatePlanBuilder.java | 1 + .../LogicalExternalRowLevelMergeSink.java | 32 +- .../PhysicalExternalRowLevelMergeSink.java | 64 +- .../doris/planner/PluginDrivenTableSink.java | 49 +- .../ConnectorColumnConverterTest.java | 7 + ...PluginDrivenScanNodeCompatibilityTest.java | 50 + .../rules/rewrite/PruneNestedColumnTest.java | 19 + .../rewrite/VariantPruningLogicTest.java | 19 + .../planner/PluginDrivenTableSinkTest.java | 14 + gensrc/thrift/DataSinks.thrift | 2 + .../iceberg/iceberg_variant_shredded.parquet | Bin 0 -> 34528 bytes ...-7100-4eb0-a42e-e52ddc62d9e3.metadata.json | 74 ++ ...958052-e154-425f-8850-f0011d0272c5-m0.avro | Bin 0 -> 7778 bytes ...-b7958052-e154-425f-8850-f0011d0272c5.avro | Bin 0 -> 4758 bytes .../iceberg/test_iceberg_varbinary.out | Bin 1626 -> 1648 bytes .../iceberg/test_iceberg_variant_read.out | 123 ++ .../iceberg/test_iceberg_varbinary.groovy | 20 +- .../iceberg/test_iceberg_variant_read.groovy | 551 ++++++++ .../variant_p0/variant_with_rowstore.groovy | 7 +- 93 files changed, 7646 insertions(+), 1004 deletions(-) create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.cpp create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.h create mode 100644 be/test/format_v2/parquet/variant_column_reader_test.cpp create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro create mode 100644 regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro create mode 100644 regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out create mode 100644 regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy diff --git a/be/src/core/block/block.cpp b/be/src/core/block/block.cpp index 7922aae07eeea0..2059466f9fa3a1 100644 --- a/be/src/core/block/block.cpp +++ b/be/src/core/block/block.cpp @@ -829,6 +829,7 @@ void Block::clear() { data.clear(); } +// Both clear paths must preserve shared children even when a composite column is top-level exclusive. void Block::clear_column_data(int64_t column_size) { SCOPED_SKIP_MEMORY_CHECK(); // data.size() greater than column_size, means here have some @@ -840,7 +841,7 @@ void Block::clear_column_data(int64_t column_size) { } for (auto& d : data) { if (d.column) { - if (d.column->is_exclusive()) { + if (is_recursively_exclusive(*d.column)) { d.column->assert_mutable()->clear(); } else { d.column = d.column->clone_empty(); @@ -855,7 +856,7 @@ void Block::clear_column_data(const std::vector& columns_to_clear) { DCHECK_LT(col, data.size()); auto& column = data[col].column; if (column) { - if (column->is_exclusive()) { + if (is_recursively_exclusive(*column)) { column->assert_mutable()->clear(); } else { column = column->clone_empty(); diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 6a74ac091ee5cb..0d5a7fb74bb6df 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -347,7 +347,8 @@ ColumnVariantV2::ColumnVariantV2(const ColumnVariantV2& other) _meta_ids(other._meta_ids), _values(other._values), _typed(other._typed), - _typed_type(other._typed_type) {} + _typed_type(other._typed_type), + _shredded(other._shredded) {} ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, DataTypePtr scalar_type) { @@ -359,6 +360,15 @@ ColumnVariantV2::MutablePtr ColumnVariantV2::create_typed(ColumnPtr column, return result; } +ColumnVariantV2::MutablePtr ColumnVariantV2::create_shredded( + std::shared_ptr state) { + DORIS_CHECK(state != nullptr) << "shredded ColumnVariantV2 state must not be null"; + auto result = ColumnVariantV2::create(); + result->_shredded = std::move(state); + result->_check_invariants(); + return result; +} + const IColumn& ColumnVariantV2::typed_column() const { DORIS_CHECK(_typed != nullptr) << "typed_column requires ColumnVariantV2 typed state"; return *_typed; @@ -369,7 +379,30 @@ const DataTypePtr& ColumnVariantV2::typed_type() const { return _typed_type; } +std::optional ColumnVariantV2::find_shredded_typed_value( + std::span path) const { + if (!_shredded) { + return std::nullopt; + } + return _shredded->find_typed_value(path); +} + void ColumnVariantV2::ensure_encoded() { + if (_shredded) { + const ColumnVariantV2& materialized = _shredded->materialized_column(); + DORIS_CHECK(!materialized.is_shredded()) + << "shredded state materializer returned another shredded column"; + // The shredded state may cache and share its canonical materialization across readers. + // Detach every mutable buffer before dropping that owner so later COW mutations stay legal. + _metadatas = materialized._metadatas->clone_resized(materialized._metadatas->size()); + _meta_ids = materialized._meta_ids->clone_resized(materialized._meta_ids->size()); + _values = materialized._values->clone_resized(materialized._values->size()); + _typed = materialized._typed == nullptr + ? nullptr + : materialized._typed->clone_resized(materialized._typed->size()); + _typed_type = materialized._typed_type; + _shredded.reset(); + } if (!_typed) { DCHECK(_typed_type == nullptr); return; @@ -394,6 +427,9 @@ void ColumnVariantV2::ensure_encoded() { } std::string ColumnVariantV2::get_name() const { + if (_shredded) { + return "variant_v2(shredded)"; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr); return "variant_v2(typed=" + _typed_type->get_name() + ")"; @@ -403,6 +439,9 @@ std::string ColumnVariantV2::get_name() const { } size_t ColumnVariantV2::size() const { + if (_shredded) { + return _shredded->size(); + } if (_typed) { DCHECK(_typed_type != nullptr); DCHECK(_metadatas->empty()); @@ -417,6 +456,9 @@ size_t ColumnVariantV2::size() const { } size_t ColumnVariantV2::byte_size() const { + if (_shredded) { + return _shredded->byte_size(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -428,6 +470,9 @@ size_t ColumnVariantV2::byte_size() const { } size_t ColumnVariantV2::allocated_bytes() const { + if (_shredded) { + return _shredded->allocated_bytes(); + } if (_typed) { DCHECK(_metadatas->empty()); DCHECK(_meta_ids->empty()); @@ -441,6 +486,9 @@ size_t ColumnVariantV2::allocated_bytes() const { bool ColumnVariantV2::has_enough_capacity(const IColumn& src) const { const auto& source = assert_cast(src); + if (_shredded || source._shredded) { + return false; + } if (static_cast(_typed) != static_cast(source._typed)) { return false; } @@ -460,6 +508,11 @@ bool ColumnVariantV2::structure_equals(const IColumn& rhs) const { } void ColumnVariantV2::sanity_check() const { + if (_shredded) { + _shredded->sanity_check(); + _check_invariants(); + return; + } if (_typed) { _typed->sanity_check(); } else { @@ -487,6 +540,10 @@ void ColumnVariantV2::sanity_check() const { } void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { + if (_shredded) { + _shredded->for_each_subcolumn(callback); + return; + } if (_typed) { callback(*static_cast(_typed)); } else { @@ -497,6 +554,11 @@ void ColumnVariantV2::for_each_subcolumn(ColumnCallback callback) const { } void ColumnVariantV2::mutate_subcolumns() { + if (_shredded) { + // Shredded state is immutable and reference-counted, so keep a partial leaf projection + // intact until an operation explicitly requires canonical bytes. + return; + } if (_typed) { mutate_subcolumn(_typed); } else { @@ -507,6 +569,11 @@ void ColumnVariantV2::mutate_subcolumns() { } void ColumnVariantV2::clear() { + if (_shredded) { + _shredded.reset(); + _check_invariants(); + return; + } if (_typed) { mutate_subcolumn(_typed); _typed->clear(); @@ -528,7 +595,7 @@ void ColumnVariantV2::clear() { // Validate the encoded batch before appending metadata, ids, and values. void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) const EncodedDataView& data) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -606,7 +673,7 @@ void ColumnVariantV2::insert_encoded_rows( // NOLINT(readability-function-size) } void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } DORIS_CHECK(_typed_type == nullptr) << "encoded state cannot retain a typed data type"; @@ -639,6 +706,9 @@ void ColumnVariantV2::insert_encoded_batch(const VariantBatchBuilder& block) { } VariantRef ColumnVariantV2::get_value_ref(size_t row) const { + if (_shredded) { + return _shredded->materialized_column().get_value_ref(row); + } DCHECK(!_typed); DCHECK(_typed_type == nullptr); DCHECK_LT(row, size()); @@ -672,7 +742,7 @@ void ColumnVariantV2::insert_many_defaults(size_t length) { return; } - if (_typed) { + if (_typed || _shredded) { ensure_encoded(); } @@ -724,6 +794,39 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) return; } + if (!_shredded && !_typed && empty() && _metadatas->empty() && source._shredded) { + // IColumn::cut() inserts into an empty clone. Select the physical tree directly because an + // incomplete leaf projection cannot be reconstructed merely to copy a row range. + _shredded = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + _check_invariants(); + return; + } + if (_shredded && source._shredded) { + auto selected_source = start == 0 && length == source.size() + ? source._shredded + : source._shredded->select_range(start, length); + // C++20 libc++ removed shared_ptr::unique(); use_count preserves the same COW invariant + // on every supported toolchain before mutating the format-owned state. + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + // A partial physical projection has no metadata/value pair to encode. Preserve that + // invariant by merging compatible scanner batches before the canonical fallback below. + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + } + if (_shredded) { + ensure_encoded(); + } + if (source._shredded) { + insert_range_from(source._shredded->materialized_column(), start, length); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_range_from(*source._typed, start, length); @@ -808,6 +911,21 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } + if (_shredded) { + ensure_encoded(); + } + if (!_typed && empty() && _metadatas->empty() && source._shredded) { + // Gather into the native shredded representation for the same reason as range selection: + // row selection does not require, and may not have, a complete logical Variant value. + _shredded = source._shredded->select_indices(indices_begin, indices_end); + _check_invariants(); + return; + } + if (source._shredded) { + insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); + return; + } + if (_typed && source._typed && exact_typed_identity(_typed_type, source._typed_type)) { mutate_subcolumn(_typed); _typed->insert_indices_from(*source._typed, indices_begin, indices_end); @@ -885,6 +1003,9 @@ void ColumnVariantV2::pop_back(size_t length) { if (length == 0) { return; } + if (_shredded) { + ensure_encoded(); + } if (_typed) { mutate_subcolumn(_typed); _typed->pop_back(length); @@ -1152,6 +1273,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma if (std::none_of(null_map, null_map + size(), [](uint8_t value) { return value != 0; })) { return; } + if (_shredded) { + ensure_encoded(); + } // Hash joins serialize the nested value even for a null-safe NULL key. Normalize those hidden // values to the canonical Variant default so build and probe keys compare byte-for-byte. @@ -1176,6 +1300,9 @@ void ColumnVariantV2::replace_column_null_data(const uint8_t* __restrict null_ma ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint) const { column_match_filter_size(size(), filter.size()); + if (_shredded) { + return ColumnVariantV2::create_shredded(_shredded->filter(filter, result_size_hint)); + } if (_typed) { ColumnPtr filtered = _typed->filter(filter, result_size_hint); auto result = ColumnVariantV2::create(); @@ -1201,6 +1328,13 @@ ColumnPtr ColumnVariantV2::filter(const Filter& filter, ssize_t result_size_hint size_t ColumnVariantV2::filter(const Filter& filter) { column_match_filter_size(size(), filter.size()); + if (_shredded) { + // Scanner-side compaction is a row-selection operation, not a request for canonical + // Variant bytes. Keep partial Parquet projections in their physical representation. + _shredded = _shredded->filter(filter, -1); + _check_invariants(); + return size(); + } if (_typed) { ColumnPtr filtered = static_cast(_typed)->filter(filter, -1); const size_t filtered_size = filtered->size(); @@ -1232,6 +1366,10 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } } + if (_shredded) { + return _shredded->materialized_column().permute(permutation, limit); + } + if (_typed) { MutableColumnPtr permuted = _typed->permute(permutation, result_size); auto result = ColumnVariantV2::create(); @@ -1257,6 +1395,20 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { + if (_shredded) { + if (new_size == 0) { + // Empty scanner placeholders carry no rows and therefore need no physical shredded + // state. Avoid forcing a partial leaf projection through full materialization. + return ColumnVariantV2::create(); + } + if (new_size == size()) { + auto result = ColumnVariantV2::create(); + result->_shredded = _shredded; + result->_check_invariants(); + return result; + } + return _shredded->materialized_column().clone_resized(new_size); + } if (_typed) { auto result = ColumnVariantV2::create(); if (new_size <= size()) { @@ -1295,6 +1447,9 @@ MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { void ColumnVariantV2::resize(size_t new_size) { const size_t old_size = size(); + if (_shredded && new_size != old_size) { + ensure_encoded(); + } if (_typed) { if (new_size == old_size) { return; @@ -1357,6 +1512,7 @@ void ColumnVariantV2::_adopt_state_from(ColumnVariantV2& replacement) { _values = std::move(replacement._values); _typed = std::move(replacement._typed); _typed_type = std::move(replacement._typed_type); + _shredded = std::move(replacement._shredded); _check_invariants(); } @@ -1368,6 +1524,14 @@ void ColumnVariantV2::_detach_metadata_for_write() { } void ColumnVariantV2::_check_invariants() const { + if (_shredded) { + DORIS_CHECK(_typed == nullptr) << "shredded state cannot contain a typed column"; + DORIS_CHECK(_typed_type == nullptr) << "shredded state cannot retain a typed data type"; + DORIS_CHECK(_metadatas->empty()) << "shredded state cannot contain encoded metadata"; + DORIS_CHECK(_meta_ids->empty()) << "shredded state cannot contain encoded metadata ids"; + DORIS_CHECK(_values->empty()) << "shredded state cannot contain encoded values"; + return; + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; const IColumn* typed_column = static_cast(_typed).get(); diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index e39bef3454719e..abe4df4b8223d4 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -19,6 +19,8 @@ #include #include +#include +#include #include #include @@ -36,9 +38,58 @@ namespace doris { class DataTypeVariantV2SerDe; class VariantBatchBuilder; +class ColumnVariantV2; -// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes or one nullable -// typed scalar column. Mixed operations materialize the typed state as encoded bytes on demand. +struct VariantShreddedPathSegment { + enum class Kind : uint8_t { OBJECT_KEY, ARRAY_INDEX }; + + Kind kind = Kind::OBJECT_KEY; + StringRef key; + int64_t index = 0; +}; + +struct VariantShreddedTypedValue { + // The state owns the same immutable column. Keeping a ColumnPtr here lets expression results + // retain the decoded leaf without copying it or depending on scanner lifetime. + ColumnPtr column; + DataTypePtr type; +}; + +// Format readers keep their native shredded representation behind this interface. Core Variant +// code sees only logical paths and an explicit late-materialization boundary. +class VariantShreddedState { +public: + virtual ~VariantShreddedState() = default; + + virtual size_t size() const = 0; + virtual size_t byte_size() const = 0; + virtual size_t allocated_bytes() const = 0; + virtual void sanity_check() const = 0; + // Shredded columns are immutable and shared. Expose their physical tree only through the + // immutable callback contract. + virtual void for_each_subcolumn(IColumn::ColumnCallback callback) const = 0; + // Row selection must remain in the native shredded representation. A scanner may compact a + // predicate column before every logical Variant value is available for materialization. + virtual std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const = 0; + virtual std::shared_ptr select_range(size_t start, + size_t length) const = 0; + virtual std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // Appends another state only when both format-owned physical layouts have identical semantics. + // An incompatible source must leave this state unchanged and return false. + virtual bool try_append(const VariantShreddedState& source) = 0; + virtual std::optional find_typed_value( + std::span path) const = 0; + + // The returned column is cached and owned by this state, so borrowed VariantRef values remain + // valid for the state lifetime. Implementations must not materialize before this is called. + virtual const ColumnVariantV2& materialized_column() const = 0; +}; + +// ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes, one nullable +// typed scalar column, or a format-owned shredded tree. Mixed operations materialize typed or +// shredded state as encoded bytes only when canonical row bytes are required. class ColumnVariantV2 final : public COWHelper { public: struct EncodedDataView { @@ -87,10 +138,14 @@ class ColumnVariantV2 final : public COWHelper { // The input must be an exact, non-Const ColumnNullable whose nested column matches the // non-nullable supported scalar type. static MutablePtr create_typed(ColumnPtr column, DataTypePtr scalar_type); + static MutablePtr create_shredded(std::shared_ptr state); bool is_typed() const noexcept { return _typed != nullptr; } + bool is_shredded() const noexcept { return _shredded != nullptr; } const IColumn& typed_column() const; const DataTypePtr& typed_type() const; + std::optional find_shredded_typed_value( + std::span path) const; void ensure_encoded(); ReadView read_view() const; @@ -197,6 +252,10 @@ class ColumnVariantV2 final : public COWHelper { // single type described by _typed_type. IColumn::WrappedPtr _typed; DataTypePtr _typed_type; + + // A non-null state owns the decoded format columns. Encoded and typed storage stay empty until + // an operation explicitly requests canonical Variant bytes. + std::shared_ptr _shredded; }; template diff --git a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp index 3ce60aed8dd7c7..5e276fd0279220 100644 --- a/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2_read_view.cpp @@ -89,6 +89,9 @@ const DataTypePtr& ColumnVariantV2::ReadView::typed_type() const { } ColumnVariantV2::ReadView ColumnVariantV2::read_view() const { + if (_shredded) { + return _shredded->materialized_column().read_view(); + } if (_typed) { DORIS_CHECK(_typed_type != nullptr) << "typed state requires a data type"; return {static_cast(_typed).get(), &_typed_type}; diff --git a/be/src/exec/operator/file_scan_operator.cpp b/be/src/exec/operator/file_scan_operator.cpp index 370f77bed8b1c8..5290e3078fb171 100644 --- a/be/src/exec/operator/file_scan_operator.cpp +++ b/be/src/exec/operator/file_scan_operator.cpp @@ -19,8 +19,14 @@ #include +#include #include +#include "core/assert_cast.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "exec/operator/olap_scan_operator.h" #include "exec/operator/scan_operator.h" #include "exec/scan/file_scanner.h" @@ -31,6 +37,29 @@ #include "storage/tablet/tablet_manager.h" namespace doris { +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +} // namespace PushDownType FileScanLocalState::_should_push_down_binary_predicate( VectorizedFnCall* fn_call, VExprContext* expr_ctx, Field& constant_val, @@ -163,6 +192,20 @@ Status FileScanLocalState::_init_scanners(std::list* scanners) { const bool use_file_scanner_v2 = _should_use_file_scanner_v2(state()->query_options(), is_load, *scan_params); _operator_profile->add_info_string("UseScannerV2", use_file_scanner_v2 ? "true" : "false"); + const auto* output_tuple_desc = state()->desc_tbl().get_tuple_descriptor(_output_tuple_id); + DORIS_CHECK(output_tuple_desc != nullptr); + const bool metadata_only_count = + is_count_star_pushdown() && _split_source->all_ranges_have_table_level_row_count(); + if (!is_load && !use_file_scanner_v2 && !metadata_only_count && + std::ranges::any_of(output_tuple_desc->slots(), [](const SlotDescriptor* slot) { + return contains_variant_type(slot->get_data_type_ptr()); + })) { + // A syntactic COUNT(*) alone is insufficient: every assigned range must prove that the + // legacy scanner will emit metadata counts without decoding a Variant carrier. + return Status::NotSupported( + "External VARIANT columns require FileScannerV2; the legacy file scanner does " + "not support VARIANT"); + } for (int i = 0; i < _max_scanners; ++i) { ScannerSPtr scanner; if (use_file_scanner_v2) { diff --git a/be/src/exec/operator/scan_operator.h b/be/src/exec/operator/scan_operator.h index e11eb506a772ab..cb9effb23882ce 100644 --- a/be/src/exec/operator/scan_operator.h +++ b/be/src/exec/operator/scan_operator.h @@ -77,6 +77,18 @@ class ScanLocalStateBase : public PipelineXLocalState<> { virtual TPushAggOp::type get_push_down_agg_type() = 0; virtual const std::optional>& get_push_down_count_slot_ids() const = 0; + static bool is_count_star_pushdown(TPushAggOp::type agg_type, + const std::optional>& count_slot_ids) { + // An absent argument field is an old plan with unknown semantics. Only an explicitly empty + // argument list proves COUNT(*)/COUNT(1) and permits placeholder slots to be ignored. + return agg_type == TPushAggOp::type::COUNT && count_slot_ids.has_value() && + count_slot_ids->empty(); + } + + bool is_count_star_pushdown() { + return is_count_star_pushdown(get_push_down_agg_type(), get_push_down_count_slot_ids()); + } + // If scan operator is serial operator(like topn), its real parallelism is 1. // Otherwise, its real parallelism is query_parallel_instance_num. // query_parallel_instance_num of olap table is usually equal to session var parallel_pipeline_task_num. diff --git a/be/src/exec/scan/access_path_parser.cpp b/be/src/exec/scan/access_path_parser.cpp index c294f86f10437d..0d0a5b0a547405 100644 --- a/be/src/exec/scan/access_path_parser.cpp +++ b/be/src/exec/scan/access_path_parser.cpp @@ -189,6 +189,20 @@ void insert_access_path(AccessPathNode* root, const std::vector& pa insert_access_path(&root->children[path[path_idx]], path, path_idx + 1); } +void collect_variant_access_paths(const AccessPathNode& node, std::vector* path, + std::vector>* result) { + DORIS_CHECK(path != nullptr && result != nullptr); + for (const auto& [segment, child] : node.children) { + path->push_back(segment); + if (child.project_all || child.children.empty()) { + result->push_back(*path); + } else { + collect_variant_access_paths(child, path, result); + } + path->pop_back(); + } +} + Status build_nested_children_from_access_node(format::ColumnDefinition* column, const DataTypePtr& type, const AccessPathNode& node, const std::string& path, @@ -445,6 +459,19 @@ Status build_nested_children_from_access_node(format::ColumnDefinition* column, return build_map_children_from_access_node( column, assert_cast(*nested_type), node, path, schema_column, prefer_exact_name_match); + case TYPE_VARIANT: { + // A Variant nested below STRUCT/ARRAY/MAP owns paths relative to this terminal. Keeping + // them on the nested ColumnDefinition lets ColumnMapper select the same physical leaves + // as a root Variant without flattening away the surrounding container. + column->variant_access_paths.clear(); + std::vector variant_path; + collect_variant_access_paths(node, &variant_path, &column->variant_access_paths); + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } default: return Status::NotSupported("AccessPathParser does not support access path {} for slot {}", path, column->name); @@ -461,6 +488,44 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, if (is_scanner_materialized_virtual_column(column->name)) { return Status::OK(); } + if (remove_nullable(column->type)->get_primitive_type() == TYPE_VARIANT) { + column->variant_access_paths.clear(); + for (const auto& access_path : access_paths) { + if (access_path.type != TAccessPathType::DATA || + !access_path.__isset.data_access_path) { + return Status::NotSupported( + "AccessPathParser only supports DATA access paths for Variant slot {}", + column->name); + } + const auto& path = access_path.data_access_path.path; + if (path.empty()) { + // Match the generic access-path tree: an empty DATA path denotes the whole slot + // and dominates every narrower Variant path in the same request. + column->variant_access_paths.clear(); + return Status::OK(); + } + int32_t top_level_id = -1; + if (to_lower(path.front()) != to_lower(column->name) && + (!parse_non_negative_int(path.front(), &top_level_id) || + !column->has_identifier_field_id() || + top_level_id != column->get_identifier_field_id())) { + return Status::NotSupported( + "AccessPathParser access path {} does not match Variant slot {}", + access_path_to_string(path), column->name); + } + if (path.size() == 1) { + // A whole-root access covers every subpath and must disable physical leaf pruning. + column->variant_access_paths.clear(); + return Status::OK(); + } + column->variant_access_paths.emplace_back(path.begin() + 1, path.end()); + } + std::ranges::sort(column->variant_access_paths); + column->variant_access_paths.erase(std::unique(column->variant_access_paths.begin(), + column->variant_access_paths.end()), + column->variant_access_paths.end()); + return Status::OK(); + } if (!is_complex_type(remove_nullable(column->type)->get_primitive_type())) { return Status::OK(); } @@ -506,8 +571,36 @@ Status AccessPathParser::build_nested_children(format::ColumnDefinition* column, bool prefer_exact_name_match) { DORIS_CHECK(column != nullptr); DORIS_CHECK(slot_desc != nullptr); - return build_nested_children(column, slot_desc->all_access_paths(), schema_column, + return build_nested_children(column, slot_desc->all_access_paths(), + slot_desc->predicate_access_paths(), schema_column, prefer_exact_name_match); } +Status AccessPathParser::build_nested_children( + format::ColumnDefinition* column, const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match) { + DORIS_CHECK(column != nullptr); + auto predicate_column = *column; + RETURN_IF_ERROR(build_nested_children(column, all_access_paths, schema_column, + prefer_exact_name_match)); + column->has_predicate_access_paths = !predicate_access_paths.empty(); + column->predicate_children.clear(); + column->predicate_variant_access_paths.clear(); + if (predicate_access_paths.empty()) { + return Status::OK(); + } + + predicate_column.children.clear(); + predicate_column.variant_access_paths.clear(); + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + RETURN_IF_ERROR(build_nested_children(&predicate_column, predicate_access_paths, schema_column, + prefer_exact_name_match)); + column->predicate_children = std::move(predicate_column.children); + column->predicate_variant_access_paths = std::move(predicate_column.variant_access_paths); + return Status::OK(); +} + } // namespace doris diff --git a/be/src/exec/scan/access_path_parser.h b/be/src/exec/scan/access_path_parser.h index 0be785a33906e8..650993aca08b27 100644 --- a/be/src/exec/scan/access_path_parser.h +++ b/be/src/exec/scan/access_path_parser.h @@ -38,6 +38,12 @@ class AccessPathParser { const std::vector& access_paths, const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); + + static Status build_nested_children( + format::ColumnDefinition* column, + const std::vector& all_access_paths, + const std::vector& predicate_access_paths, + const format::ColumnDefinition* schema_column, bool prefer_exact_name_match = true); }; } // namespace doris diff --git a/be/src/exec/scan/file_scanner_v2.cpp b/be/src/exec/scan/file_scanner_v2.cpp index 052d465daa2005..a0840b7ae50024 100644 --- a/be/src/exec/scan/file_scanner_v2.cpp +++ b/be/src/exec/scan/file_scanner_v2.cpp @@ -36,7 +36,10 @@ #include "core/block/column_with_type_and_name.h" #include "core/column/column.h" #include "core/data_type/data_type.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "core/data_type_serde/data_type_serde.h" #include "core/string_ref.h" #include "exec/common/util.hpp" @@ -79,6 +82,8 @@ namespace { constexpr int kIcebergPositionDeleteContent = 1; constexpr int kIcebergDeletionVectorContent = 3; +std::string table_format_name(const TFileRangeDesc& range); + std::string table_format_name(const TFileRangeDesc& range) { return range.__isset.table_format_params ? range.table_format_params.table_format_type : "NotSet"; @@ -89,6 +94,26 @@ TFileFormatType::type get_range_format_type(const TFileScanRangeParams& params, return range.__isset.format_type ? range.format_type : params.format_type; } +bool contains_variant_type(const DataTypePtr& input) { + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + bool is_supported_table_format(const TFileRangeDesc& range) { const auto table_format = table_format_name(range); if (table_format == "hudi" && range.__isset.table_format_params && @@ -486,6 +511,13 @@ Status FileScannerV2::_filter_output_block(Block* block) { _get_current_format_type()); } +bool FileScannerV2::_can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + // A Variant access expression is evaluated above the file reader. Keep each file-local + // shredded schema intact until that projection turns complete and leaf-only states into a + // common logical result column. + return !_has_variant_projection; +} + Status FileScannerV2::_contextualize_output_filter_status(Status status, TFileFormatType::type format_type) { if (!status.ok() && format_type == TFileFormatType::FORMAT_ORC) { @@ -672,8 +704,9 @@ Status FileScannerV2::_create_table_reader_for_format( Status FileScannerV2::_prepare_table_reader_split(const TFileRangeDesc& range, std::map partition_values) { + const auto format_type = get_range_format_type(*_params, range); format::FileFormat current_split_format; - RETURN_IF_ERROR(_to_file_format(get_range_format_type(*_params, range), ¤t_split_format)); + RETURN_IF_ERROR(_to_file_format(format_type, ¤t_split_format)); VExprContextSPtrs conjuncts; RETURN_IF_ERROR(_build_table_conjuncts(&conjuncts)); VExprContextSPtrs partition_prune_conjuncts; @@ -797,6 +830,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ _projected_columns.clear(); _projected_columns.reserve(_params->required_slots.size()); _need_global_rowid_column = false; + _has_variant_projection = false; format::ProjectedColumnBuildContext build_context { .scan_params = _params, .range = &_current_range, @@ -814,6 +848,7 @@ Status FileScannerV2::_build_projected_columns(const format::TableReader& table_ slot_info.slot_id); } auto column = _build_table_column(it->second); + _has_variant_projection = _has_variant_projection || contains_variant_type(column.type); build_context.slot_desc = it->second; if (column.name.starts_with(BeConsts::GLOBAL_ROWID_COL)) { _need_global_rowid_column = true; diff --git a/be/src/exec/scan/file_scanner_v2.h b/be/src/exec/scan/file_scanner_v2.h index 03e5f4d6bbc1a0..3ccc4e075ef209 100644 --- a/be/src/exec/scan/file_scanner_v2.h +++ b/be/src/exec/scan/file_scanner_v2.h @@ -114,6 +114,7 @@ class FileScannerV2 final : public Scanner { protected: Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) override; + bool _can_merge_padding_blocks(const Block& left, const Block& right) const override; Status _filter_output_block(Block* block) override; void _collect_profile_before_close() override; bool _should_update_load_counters() const override; @@ -190,6 +191,7 @@ class FileScannerV2 final : public Scanner { // the reader is format-specific, so it is rebuilt whenever this stops matching the range. std::string _table_reader_format; std::vector _projected_columns; + bool _has_variant_projection = false; // File formats without embedded schema, such as CSV, still need the FE slot descriptors in // file-column order. This mirrors old FileScanner::_file_slot_descs and is passed only to // readers that cannot derive their schema from file metadata. diff --git a/be/src/exec/scan/scanner.cpp b/be/src/exec/scan/scanner.cpp index cbdf5fa7eeec55..e30543b9343613 100644 --- a/be/src/exec/scan/scanner.cpp +++ b/be/src/exec/scan/scanner.cpp @@ -88,6 +88,15 @@ Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool auto& row_descriptor = _local_state->_parent->row_descriptor(); if (_output_row_descriptor) { _origin_block.clear_column_data(row_descriptor.num_materialized_slots()); + if (!_can_merge_padding_blocks(_padding_block, _origin_block)) { + DORIS_CHECK(_padding_block.empty()) + << "padding policy must remain stable for one scanner"; + // Some physical columns carry file-local state that an upper projection must consume + // before the next split is read. Padding those blocks first would make correctness + // depend on whether two file tails happen to share one output batch. + RETURN_IF_ERROR(get_block(state, &_origin_block, eos)); + return _do_projections(&_origin_block, block); + } const auto min_batch_size = std::max(state->batch_size() / 2, 1); const auto block_max_bytes = state->preferred_block_size_bytes(); while (_padding_block.rows() < min_batch_size && _padding_block.bytes() < block_max_bytes && diff --git a/be/src/exec/scan/scanner.h b/be/src/exec/scan/scanner.h index 1bf4cdb5bb2ea1..75e583ecdfce61 100644 --- a/be/src/exec/scan/scanner.h +++ b/be/src/exec/scan/scanner.h @@ -123,6 +123,10 @@ class Scanner { // Subclass should implement this to return data. virtual Status _get_block_impl(RuntimeState* state, Block* block, bool* eof) = 0; + virtual bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const { + return true; + } + Status _merge_padding_block() { if (_padding_block.empty()) { _padding_block.swap(_origin_block); diff --git a/be/src/exec/scan/split_source_connector.h b/be/src/exec/scan/split_source_connector.h index 5926baff303cbf..f13190cf0641c2 100644 --- a/be/src/exec/scan/split_source_connector.h +++ b/be/src/exec/scan/split_source_connector.h @@ -17,6 +17,8 @@ #pragma once +#include + #include "common/config.h" #include "core/custom_allocator.h" #include "runtime/runtime_state.h" @@ -45,6 +47,8 @@ class SplitSourceConnector { virtual TFileScanRangeParams* get_params() = 0; + virtual bool all_ranges_have_table_level_row_count() const { return false; } + protected: template , typename V2 = std::vector> requires(std::is_same_v, @@ -125,6 +129,19 @@ class LocalSplitSourceConnector : public SplitSourceConnector { throw Exception( Status::FatalError("Unreachable, params is got by file_scan_range_params_map")); } + + bool all_ranges_have_table_level_row_count() const override { + // Every assigned range must carry a proven count; one fallback range would still require + // decoding the projected carrier through the selected scanner. + return !_scan_ranges.empty() && std::ranges::all_of(_scan_ranges, [](const auto& params) { + const auto& ranges = params.scan_range.ext_scan_range.file_scan_range.ranges; + return !ranges.empty() && std::ranges::all_of(ranges, [](const auto& range) { + return range.__isset.table_format_params && + range.table_format_params.__isset.table_level_row_count && + range.table_format_params.table_level_row_count >= 0; + }); + }); + } }; /** diff --git a/be/src/exec/sink/viceberg_merge_sink.cpp b/be/src/exec/sink/viceberg_merge_sink.cpp index 5ff5a0a1f28150..5008b217228cd2 100644 --- a/be/src/exec/sink/viceberg_merge_sink.cpp +++ b/be/src/exec/sink/viceberg_merge_sink.cpp @@ -55,13 +55,15 @@ VIcebergMergeSink::~VIcebergMergeSink() = default; Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& row_desc) { RETURN_IF_ERROR(_build_inner_sinks()); - _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, - nullptr, nullptr); - _table_writer->defer_file_cleanup_until_outer_close(); + if (_writes_data_files) { + _table_writer = std::make_unique(_table_sink, _table_output_expr_ctxs, + nullptr, nullptr); + _table_writer->defer_file_cleanup_until_outer_close(); + RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); + } _delete_writer = std::make_unique(_delete_sink, _delete_output_expr_ctxs, nullptr, nullptr); _delete_writer->defer_file_cleanup_until_outer_close(); - RETURN_IF_ERROR(_table_writer->init_properties(pool, row_desc)); RETURN_IF_ERROR(_delete_writer->init_properties(pool)); return Status::OK(); } @@ -88,10 +90,13 @@ Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { RETURN_IF_ERROR(_prepare_output_layout()); - RuntimeProfile* table_profile = profile->create_child("IcebergMergeTableWriter", true, true); RuntimeProfile* delete_profile = profile->create_child("IcebergMergeDeleteWriter", true, true); - RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + if (_table_writer) { + RuntimeProfile* table_profile = + profile->create_child("IcebergMergeTableWriter", true, true); + RETURN_IF_ERROR(_table_writer->open(state, table_profile)); + } RETURN_IF_ERROR(_delete_writer->open(state, delete_profile)); return Status::OK(); @@ -153,6 +158,13 @@ Status VIcebergMergeSink::write(RuntimeState* state, Block& block) { _delete_row_count += delete_rows; _insert_row_count += insert_rows; + // A delete-only plan deliberately omits the data writer so Variant target schemas never enter + // the unsupported Iceberg data-write path. Reject a mismatched FE plan before dereferencing it. + if (has_insert && !_writes_data_files) { + return Status::InternalError( + "Iceberg delete-only merge sink received a data insert operation"); + } + bool skip_io = false; #ifdef BE_TEST skip_io = _skip_io; @@ -338,6 +350,8 @@ Status VIcebergMergeSink::_build_inner_sinks() { } const auto& merge_sink = _t_sink.iceberg_merge_sink; + // An old FE cannot produce delete-only plans, so an unset flag retains its data-writer path. + _writes_data_files = !merge_sink.__isset.writes_data_files || merge_sink.writes_data_files; // Missing means an old FE plan, which predates SQL MERGE cardinality validation. _require_merge_cardinality_check = merge_sink.__isset.require_merge_cardinality_check && merge_sink.require_merge_cardinality_check; diff --git a/be/src/exec/sink/viceberg_merge_sink.h b/be/src/exec/sink/viceberg_merge_sink.h index f3733a3318230e..88e9ef89121b74 100644 --- a/be/src/exec/sink/viceberg_merge_sink.h +++ b/be/src/exec/sink/viceberg_merge_sink.h @@ -78,6 +78,7 @@ class VIcebergMergeSink final : public AsyncResultWriter { std::vector _data_column_indices; std::map _matched_row_positions; size_t _matched_row_id_state_size = sizeof(std::map); + bool _writes_data_files = true; bool _require_merge_cardinality_check = false; VExprContextSPtrs _table_output_expr_ctxs; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 644a033f4af356..90863fe86c5c72 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -131,6 +131,42 @@ Status extract_encoded_variant_element(const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, std::span outer_nulls, ColumnPtr* output); +std::optional extract_shredded_typed_variant_element( + const ColumnVariantV2& source, const ResolvedVariantElementV2Path& path, + std::span outer_nulls) { + DorisVector shredded_path; + shredded_path.reserve(path.size()); + for (size_t position = 0; position < path.size(); ++position) { + VariantShreddedPathSegment segment; + if (path.kind_at(position) == VariantElementV2PathSegment::Kind::OBJECT_KEY) { + segment.kind = VariantShreddedPathSegment::Kind::OBJECT_KEY; + segment.key = path.object_key_at(position); + } else { + segment.kind = VariantShreddedPathSegment::Kind::ARRAY_INDEX; + segment.index = path.array_index_at(position); + } + shredded_path.push_back(segment); + } + + auto match = source.find_shredded_typed_value(shredded_path); + if (!match.has_value()) { + return std::nullopt; + } + const auto& leaf = assert_cast(*match->column); + auto nulls = leaf.get_null_map_column().clone_resized(source.size()); + auto& null_data = assert_cast(*nulls).get_data(); + for (size_t row = 0; row < source.size(); ++row) { + null_data[row] = + static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); + } + + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null + // map is produced here, so predicates and casts can consume the leaf without reconstructing + // canonical Variant rows. + auto values = ColumnVariantV2::create_typed(match->column, match->type); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + Status make_all_null_variant_element_result(size_t rows, ColumnPtr* output); } // namespace @@ -217,7 +253,14 @@ Status extract_variant_element_v2(const ColumnVariantV2& source, ColumnPtr candidate; try { - if (!source.is_typed()) { + if (source.is_shredded()) { + if (auto typed = extract_shredded_typed_variant_element(source, path, outer_nulls)) { + candidate = std::move(*typed); + } else { + RETURN_IF_ERROR( + extract_encoded_variant_element(source, path, outer_nulls, &candidate)); + } + } else if (!source.is_typed()) { RETURN_IF_ERROR(extract_encoded_variant_element(source, path, outer_nulls, &candidate)); } else { // A typed Variant is one scalar root value per row. String payloads are strings, not diff --git a/be/src/format_v2/column_data.h b/be/src/format_v2/column_data.h index 41593fa157dacb..61907512dcf7ea 100644 --- a/be/src/format_v2/column_data.h +++ b/be/src/format_v2/column_data.h @@ -261,6 +261,17 @@ struct ColumnDefinition { // Full table-schema identity subtree before access-path pruning. ID-less physical complex // wrappers must be discovered from this view without adding unrequested children to output. std::vector identity_children {}; + // Logical object-key paths requested from a Variant column. An empty collection means the + // whole Variant is required; non-empty paths may be resolved to format-specific shredded + // physical children after the per-file schema is known. + std::vector> variant_access_paths {}; + // Predicate access paths are kept separately from the final union projection. File Scanner V2 + // can lower this smaller semantic tree to an eager predicate projection while deferring the + // final children until rows survive. The flag distinguishes no predicate metadata from a + // whole-root predicate, whose child/path collections are intentionally empty. + bool has_predicate_access_paths = false; + std::vector predicate_children {}; + std::vector> predicate_variant_access_paths {}; // Expression used to materialize missing/default/generated values when the column is not read // directly from the file. VExprContextSPtr default_expr = nullptr; @@ -375,6 +386,19 @@ struct LocalColumnIndex { std::string debug_string() const; }; +inline bool same_local_column_index(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { + if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || + lhs.children.size() != rhs.children.size()) { + return false; + } + for (size_t i = 0; i < lhs.children.size(); ++i) { + if (!same_local_column_index(lhs.children[i], rhs.children[i])) { + return false; + } + } + return true; +} + inline bool is_full_projection(const LocalColumnIndex* projection) { return projection == nullptr || projection->project_all_children; } diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index 9f433cc853f61a..b9488b5f78f2ab 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -18,6 +18,7 @@ #include "format_v2/column_mapper.h" #include +#include #include #include #include @@ -1289,6 +1290,12 @@ static bool needs_projected_file_type_rebuild(const ColumnMapping& mapping) { remove_nullable(mapping.table_type)->get_primitive_type()) { return true; } + if (remove_nullable(mapping.file_type)->get_primitive_type() == TYPE_STRUCT && + mapping.child_mappings.size() != mapping.original_file_children.size()) { + // A predicate access path keeps the parent Struct type but intentionally carries only the + // referenced child descriptors; type equality alone must not restore the pruned siblings. + return true; + } if (!mapping.table_type->equals(*mapping.file_type)) { return true; } @@ -1657,17 +1664,24 @@ static bool has_projected_file_children(const ColumnMapping& mapping) { return false; } -static bool needs_nested_file_projection(const ColumnMapping& mapping) { +static bool needs_nested_file_projection(const ColumnMapping& mapping, + bool include_variant_access_paths = false) { if (has_projected_file_children(mapping)) { // Return True if the projected child column is missing / re-ordered return true; } - return std::ranges::any_of(mapping.child_mappings, [](const ColumnMapping& child_mapping) { - return needs_nested_file_projection(child_mapping); - }); + if (include_variant_access_paths && !mapping.variant_access_paths.empty()) { + return true; + } + return std::ranges::any_of( + mapping.child_mappings, [include_variant_access_paths](const ColumnMapping& child) { + return needs_nested_file_projection(child, include_variant_access_paths); + }); } -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection); +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection = false); // Build the projected file children/type according to the pruned complex projection. For example, // if we have a struct column `s` with children `id` and `name`, and the projection only keeps @@ -1705,11 +1719,15 @@ static Status rebuild_projected_file_children_and_type( // projected output shape; file readers still read full keys to construct ColumnMap offsets and keep // key semantics unchanged. If a caller tries to project only/prune the key child, the common schema // projection helper rejects it. -static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { +static Status build_complex_projection(const ColumnMapping& mapping, LocalColumnIndex* projection, + bool enable_variant_leaf_projection) { if (projection == nullptr) { return Status::InvalidArgument("projection is null"); } DORIS_CHECK(mapping.file_local_id.has_value()); + if (enable_variant_leaf_projection && build_variant_projection(mapping, projection)) { + return Status::OK(); + } *projection = LocalColumnIndex::local(*mapping.file_local_id); projection->timestamp_is_adjusted_to_utc = mapping.timestamp_is_adjusted_to_utc; projection->project_all_children = mapping.child_mappings.empty(); @@ -1724,7 +1742,8 @@ static Status build_complex_projection(const ColumnMapping& mapping, LocalColumn } for (const auto* child_mapping : present_children) { LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection)); + RETURN_IF_ERROR(build_complex_projection(*child_mapping, &child_projection, + enable_variant_leaf_projection)); projection->children.push_back(std::move(child_projection)); } if (!projection->project_all_children && projection->children.empty()) { @@ -1759,9 +1778,6 @@ static void attach_timestamp_semantics(const ColumnMapping& mapping, LocalColumn attach_timestamp_semantics(child_mapping, &*child_it); } } - -using FilterProjectionMap = std::map; - // Update the mapping's file type according to the projection, and determine whether the projection // is trivial (i.e. the projected file type is the same as the table type, so no need to // rematerialize the complex value back to table layout after reading from file). @@ -1786,51 +1802,148 @@ static Status apply_projection_to_mapping_file_type(const LocalColumnIndex& proj return Status::OK(); } -static Status merge_filter_projection(const FilterProjectionMap* filter_projections, - LocalColumnIndex* projection) { - DORIS_CHECK(projection != nullptr); - if (filter_projections == nullptr) { - return Status::OK(); +static const ColumnDefinition* find_file_child_by_name( + const std::vector& children, std::string_view name) { + const auto child_it = std::ranges::find_if( + children, [name](const ColumnDefinition& child) { return child.name == name; }); + return child_it == children.end() ? nullptr : &*child_it; +} + +static bool variant_leaf_type_preserves_physical_identity(const ColumnDefinition& leaf) { + if (!leaf.children.empty() || leaf.type == nullptr) { + return false; } - const auto filter_projection_it = filter_projections->find(projection->column_id()); - if (filter_projection_it == filter_projections->end()) { - return Status::OK(); + // ColumnDefinition does not transport Parquet's raw-binary/UUID and timestamp-unit tags. + // Limit direct leaves to identities fully described by the Doris scalar type; every ambiguous + // identity must retain the complete wrapper so reconstruction can inspect its physical schema. + switch (remove_nullable(leaf.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; } - // Merge predicate-only nested paths into the root projection that is about to be scanned. - // Example: `SELECT s.a WHERE s.b > 1` first builds the output projection `s -> a` from - // ColumnMapping, while build_nested_struct_filter_projection_map() records `s -> b`. This merge - // produces one file scan projection `s -> a,b`. - RETURN_IF_ERROR(merge_local_column_index(projection, filter_projection_it->second)); - return Status::OK(); } -static bool table_root_is_map(const ColumnMapping& mapping) { - if (mapping.table_type == nullptr) { +static bool build_variant_leaf_path_projection(const ColumnMapping& mapping, + const std::vector& path, + LocalColumnIndex* root_projection) { + DORIS_CHECK(root_projection != nullptr); + const auto is_numeric_selector = [](std::string_view value) { + if (value.empty()) { + return false; + } + const size_t digits_begin = value.front() == '+' || value.front() == '-' ? 1 : 0; + return digits_begin < value.size() && + std::ranges::all_of(value.substr(digits_begin), + [](unsigned char c) { return std::isdigit(c); }); + }; + if (path.size() != 1 || path[0].empty() || path[0] == "NULL" || + path[0].find('.') != std::string::npos || is_numeric_selector(path[0]) || + !mapping.file_local_id.has_value()) { + // Thrift currently carries access paths as strings without segment-kind or escaping + // metadata. Signed numeric tokens are therefore also ambiguous between an array selector + // and an object key, so only a single unambiguous key can be mapped losslessly to a leaf. return false; } - return remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_MAP; + *root_projection = LocalColumnIndex::partial_local(*mapping.file_local_id); + const auto* root_typed = find_file_child_by_name(mapping.original_file_children, "typed_value"); + if (root_typed == nullptr || root_typed->children.empty() || root_typed->type == nullptr || + remove_nullable(root_typed->type)->get_primitive_type() != TYPE_STRUCT) { + return false; + } + root_projection->children.push_back( + LocalColumnIndex::partial_local(root_typed->file_local_id())); + auto* current_projection = &root_projection->children.back(); + const auto* typed_children = &root_typed->children; + for (size_t position = 0; position < path.size(); ++position) { + const auto* wrapper = find_file_child_by_name(*typed_children, path[position]); + if (wrapper == nullptr) { + return false; + } + current_projection->children.push_back( + LocalColumnIndex::partial_local(wrapper->file_local_id())); + current_projection = ¤t_projection->children.back(); + const auto* typed = find_file_child_by_name(wrapper->children, "typed_value"); + if (typed == nullptr) { + return false; + } + auto typed_projection = LocalColumnIndex::partial_local(typed->file_local_id()); + const bool leaf = position + 1 == path.size(); + if (leaf) { + // Only primitive typed values can be returned as a direct vector. Complex shredded + // values still need their wrapper shape and therefore keep the full Variant fallback. + if (!variant_leaf_type_preserves_physical_identity(*typed)) { + return false; + } + typed_projection.project_all_children = true; + } + current_projection->children.push_back(std::move(typed_projection)); + current_projection = ¤t_projection->children.back(); + typed_children = &typed->children; + } + return true; } -static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, - bool is_predicate_column, bool force_full_complex_scan_projection, - const FilterProjectionMap* filter_projections = nullptr) { +static bool build_variant_projection(const ColumnMapping& mapping, LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (mapping.table_type == nullptr || mapping.variant_access_paths.empty() || + remove_nullable(mapping.table_type)->get_primitive_type() != TYPE_VARIANT) { + return false; + } + std::optional merged; + for (const auto& path : mapping.variant_access_paths) { + LocalColumnIndex path_projection; + if (!build_variant_leaf_path_projection(mapping, path, &path_projection)) { + return false; + } + if (!merged.has_value()) { + merged = std::move(path_projection); + } else if (!merge_local_column_index(&*merged, path_projection).ok()) { + return false; + } + } + if (!merged.has_value()) { + return false; + } + *projection = std::move(*merged); + return true; +} + +static Status build_scan_projection(ColumnMapping* mapping, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection, + LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); const auto file_column_id = LocalColumnId(mapping->file_local_id.value()); - LocalColumnIndex projection = LocalColumnIndex::top_level(file_column_id); - projection.timestamp_is_adjusted_to_utc = mapping->timestamp_is_adjusted_to_utc; + *projection = LocalColumnIndex::top_level(file_column_id); // Columnar readers can turn a complex mapping into a nested file projection, but // row-oriented readers must scan the full top-level complex field because all children are // encoded in the same text cell. - if (!force_full_complex_scan_projection && needs_nested_file_projection(*mapping)) { - RETURN_IF_ERROR(build_complex_projection(*mapping, &projection)); - } - if (is_predicate_column && !force_full_complex_scan_projection) { - DCHECK(filter_projections != nullptr); - // If a projected complex root is also used by a predicate, rebuild the predicate scan - // projection from the output mapping before merging predicate-only children. For - // `SELECT s.a WHERE s.b > 1`, build_complex_projection() produces `s -> a` and - // merge_filter_projection() adds `s -> b`, so the predicate column reads both children. - RETURN_IF_ERROR(merge_filter_projection(filter_projections, &projection)); + if (enable_variant_leaf_projection && !force_full_complex_scan_projection && + build_variant_projection(*mapping, projection)) { + // The per-file Parquet reader will validate residual-value statistics before honoring this + // physical leaf projection; unsafe files atomically fall back to the complete Variant. + } else if (!force_full_complex_scan_projection && + needs_nested_file_projection(*mapping, enable_variant_leaf_projection)) { + RETURN_IF_ERROR( + build_complex_projection(*mapping, projection, enable_variant_leaf_projection)); } + return Status::OK(); +} + +static Status add_scan_column(FileScanRequest* file_request, ColumnMapping* mapping, + bool is_predicate_column, bool force_full_complex_scan_projection, + bool enable_variant_leaf_projection) { + LocalColumnIndex projection; + RETURN_IF_ERROR(build_scan_projection(mapping, force_full_complex_scan_projection, + enable_variant_leaf_projection, &projection)); attach_timestamp_semantics(*mapping, &projection); FileScanRequestBuilder builder(file_request); if (is_predicate_column) { @@ -1852,18 +1965,21 @@ static const LocalColumnIndex* find_scan_projection( // mapping.file_type/projected_file_children from the original file schema to the exact shape that // FileReader will return. // -// Example: for `SELECT s.a WHERE s.b > 1`, add_scan_column() keeps only one predicate scan -// projection `s -> a,b`. Applying that projection changes the mapping's file type from the full -// file struct `s` to the projected file struct `s`, so later filter rewrite and -// TableReader final materialization use the same column shape as the file-local block. +// Applying the selected projection changes a mapping's file type to the exact nested shape exposed +// by FileReader, so later filter rewrite and TableReader materialization agree with the file block. static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& file_request, - ColumnMapping* mapping) { + ColumnMapping* mapping, + bool predicate_mapping = false) { DORIS_CHECK(mapping != nullptr); DORIS_CHECK(mapping->file_local_id.has_value()); const auto file_column_id = LocalColumnId(*mapping->file_local_id); - // Predicate columns are the actual scan projection when a column is used by row-level filters: - // add_scan_column() removes the duplicate non-predicate projection in that case. - const auto* projection = find_scan_projection(file_request.predicate_columns, file_column_id); + const LocalColumnIndex* projection = nullptr; + if (!predicate_mapping && file_request.has_deferred_non_predicate_column(file_column_id)) { + projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); + } + if (projection == nullptr) { + projection = find_scan_projection(file_request.predicate_columns, file_column_id); + } if (projection == nullptr) { projection = find_scan_projection(file_request.non_predicate_columns, file_column_id); } @@ -1871,76 +1987,6 @@ static Status apply_scan_projection_to_mapping_file_type(const FileScanRequest& return apply_projection_to_mapping_file_type(*projection, mapping); } -// Build extra scan projections required only by row-level filters on nested struct children. -// -// Example: for `SELECT s.a FROM t WHERE s.b.c > 1`, the output projection may only contain `s.a`, -// but the file reader must also read `s.b.c` to evaluate the predicate. This function collects the -// table-side filter path, resolves it through ColumnMapping first, and records the corresponding -// file-side projection in filter_projections. This keeps renamed fields consistent between the scan -// projection and row-level conjunct rewrite. Example: -// table filter path: s -> renamed_b -> c -// old file path: s -> b -> c -// recorded path: s -> b -> c -// When add_scan_column() adds the same root as a predicate column, it rebuilds that root from the -// output mapping, merges this filter-only projection into it, and removes the duplicate -// non-predicate root entry. -static Status build_nested_struct_filter_projection_map( - const std::vector& table_filters, const std::vector& mappings, - FilterProjectionMap* filter_projections) { - DORIS_CHECK(filter_projections != nullptr); - filter_projections->clear(); - for (const auto& table_filter : table_filters) { - if (table_filter.conjunct == nullptr) { - continue; - } - // Collect all nested struct paths in the table filter. For example, for - // `s.id > 5 AND element_at(s, 'renamed_name') = 'abc'`, collect the table paths - // `s -> id` and `s -> renamed_name`, then resolve each one to its file-side projection. - std::vector paths; - collect_nested_struct_paths(table_filter.conjunct->root(), &paths); - for (const auto& path : paths) { - auto mapping_it = std::ranges::find_if(mappings, [&](const ColumnMapping& mapping) { - return mapping.global_index == path.root_global_index; - }); - if (mapping_it == mappings.end() || !mapping_it->file_local_id.has_value() || - path.selectors.empty()) { - continue; - } - - ResolvedNestedStructPath resolved; - LocalColumnIndex root_projection; - if (!resolve_nested_struct_path_for_file(path, mappings, &resolved)) { - if (!table_root_is_map(*mapping_it)) { - continue; - } - // Direct map value filters such as `m.value.a > 1` need the value leaf for row - // evaluation even when the query only projects another value child. This is only a - // scan projection fallback; complex map/array expressions are still not rewritten - // into file-local conjuncts. - LocalColumnIndex child_projection; - RETURN_IF_ERROR(build_file_child_projection_from_schema( - mapping_it->original_file_children, path.selectors, &child_projection)); - if (child_projection.local_id() < 0) { - continue; - } - root_projection = LocalColumnIndex::partial_local(*mapping_it->file_local_id); - root_projection.children.push_back(std::move(child_projection)); - } else { - root_projection = std::move(resolved.file_projection); - } - auto filter_projection_it = filter_projections->find(root_projection.column_id()); - if (filter_projection_it == filter_projections->end()) { - filter_projections->emplace(root_projection.column_id(), - std::move(root_projection)); - continue; - } - RETURN_IF_ERROR( - merge_local_column_index(&filter_projection_it->second, root_projection)); - } - } - return Status::OK(); -} - static void rebuild_projection(ColumnMapping* mapping, LocalIndex block_position) { DORIS_CHECK(mapping->file_local_id.has_value()); if (mapping->is_trivial || needs_complex_rematerialize(*mapping)) { @@ -2033,6 +2079,7 @@ Status TableColumnMapper::_create_mapping_for_column(const ColumnDefinition& tab mapping->global_index = global_index; mapping->table_column_name = table_column.name; mapping->table_type = table_column.type; + mapping->variant_access_paths = table_column.variant_access_paths; // Row-lineage names are Iceberg metadata contracts, not reserved names in generic Hive, // Hudi, or Paimon schemas. Only the Iceberg reader may opt into virtual synthesis. const auto row_lineage_type = @@ -2157,6 +2204,19 @@ Status TableColumnMapper::create_mapping(const std::vector& pr RETURN_IF_ERROR(_create_mapping_for_column(projected_columns[column_idx], GlobalIndex(column_idx), &mapping)); _mappings.push_back(std::move(mapping)); + if (enable_independent_predicate_projection() && + projected_columns[column_idx].has_predicate_access_paths) { + auto predicate_column = projected_columns[column_idx]; + predicate_column.children = predicate_column.predicate_children; + predicate_column.variant_access_paths = predicate_column.predicate_variant_access_paths; + predicate_column.has_predicate_access_paths = false; + predicate_column.predicate_children.clear(); + predicate_column.predicate_variant_access_paths.clear(); + ColumnMapping predicate_mapping; + RETURN_IF_ERROR(_create_mapping_for_column(predicate_column, GlobalIndex(column_idx), + &predicate_mapping)); + _predicate_mappings.push_back(std::move(predicate_mapping)); + } } return Status::OK(); } @@ -2164,7 +2224,13 @@ Status TableColumnMapper::create_mapping(const std::vector& pr std::vector TableColumnMapper::_filter_visible_mappings() const { std::vector mappings; mappings.reserve(_mappings.size() + _hidden_mappings.size()); - mappings.insert(mappings.end(), _mappings.begin(), _mappings.end()); + for (const auto& mapping : _mappings) { + const auto predicate_it = std::ranges::find_if( + _predicate_mappings, [&](const ColumnMapping& predicate_mapping) { + return predicate_mapping.global_index == mapping.global_index; + }); + mappings.push_back(predicate_it == _predicate_mappings.end() ? mapping : *predicate_it); + } mappings.insert(mappings.end(), _hidden_mappings.begin(), _hidden_mappings.end()); return mappings; } @@ -2206,6 +2272,7 @@ Status TableColumnMapper::create_scan_request( // continues to address the same physical column. file_request->local_positions = *fixed_local_positions; } + file_request->non_predicate_positions.clear(); file_request->conjuncts.clear(); file_request->delete_conjuncts.clear(); _filter_entries.clear(); @@ -2228,7 +2295,8 @@ Status TableColumnMapper::create_scan_request( } if (!used_by_filter || !enable_lazy_materialization()) { RETURN_IF_ERROR(add_scan_column(file_request, mapping, false, - force_full_complex_scan_projection())); + force_full_complex_scan_projection(), + enable_variant_leaf_projection())); } } } @@ -2267,12 +2335,12 @@ Status TableColumnMapper::create_scan_request( if (!mapping.file_local_id.has_value()) { continue; } - auto position_it = - file_request->local_positions.find(LocalColumnId(*mapping.file_local_id)); + const auto local_id = LocalColumnId(*mapping.file_local_id); + const auto position_it = file_request->local_positions.find(local_id); DORIS_CHECK(position_it != file_request->local_positions.end()) << file_request->local_positions.size() << " " << *mapping.file_local_id << " " << mapping.file_column_name; - rebuild_projection(&mapping, position_it->second); + rebuild_projection(&mapping, file_request->non_predicate_position(local_id)); } return Status::OK(); } @@ -2286,7 +2354,19 @@ ColumnMapping* TableColumnMapper::_find_mapping(GlobalIndex global_index) { return nullptr; } +ColumnMapping* TableColumnMapper::_find_predicate_mapping(GlobalIndex global_index) { + for (auto& mapping : _predicate_mappings) { + if (mapping.global_index == global_index) { + return &mapping; + } + } + return nullptr; +} + ColumnMapping* TableColumnMapper::_find_filter_mapping(GlobalIndex global_index) { + if (auto* mapping = _find_predicate_mapping(global_index); mapping != nullptr) { + return mapping; + } if (auto* mapping = _find_mapping(global_index); mapping != nullptr) { return mapping; } @@ -2302,10 +2382,7 @@ Status TableColumnMapper::localize_filters(const std::vector& table FileScanRequest* file_request, RuntimeState* runtime_state) { std::set localized_predicate_columns; - FilterProjectionMap filter_projections; auto filter_mappings = _filter_visible_mappings(); - RETURN_IF_ERROR(build_nested_struct_filter_projection_map(table_filters, filter_mappings, - &filter_projections)); for (const auto& table_filter : table_filters) { for (const auto& global_index : table_filter.global_indices) { auto* mapping = _find_filter_mapping(global_index); @@ -2313,9 +2390,29 @@ Status TableColumnMapper::localize_filters(const std::vector& table !filter_conversion_has_local_source(mapping->filter_conversion)) { continue; } + // Nested eager projection is an FE contract. Without predicate_access_paths the + // all-access-path mapping is read as one unit instead of inferring another subtree + // from VExpr and risking a shape that disagrees with final materialization. RETURN_IF_ERROR(add_scan_column(file_request, mapping, enable_lazy_materialization(), force_full_complex_scan_projection(), - &filter_projections)); + enable_variant_leaf_projection())); + auto* output_mapping = _find_mapping(global_index); + if (!enable_independent_predicate_projection() || output_mapping == nullptr || + mapping == output_mapping || !output_mapping->file_local_id.has_value()) { + continue; + } + LocalColumnIndex output_projection; + RETURN_IF_ERROR( + build_scan_projection(output_mapping, force_full_complex_scan_projection(), + enable_variant_leaf_projection(), &output_projection)); + const auto* predicate_projection = find_scan_projection(file_request->predicate_columns, + output_projection.column_id()); + DORIS_CHECK(predicate_projection != nullptr); + if (!same_local_column_index(*predicate_projection, output_projection)) { + FileScanRequestBuilder builder(file_request); + RETURN_IF_ERROR( + builder.add_deferred_non_predicate_column(std::move(output_projection))); + } } } // Rebuild the file type for every scan-local mapping before expression rewrite. Predicate-only @@ -2326,6 +2423,13 @@ Status TableColumnMapper::localize_filters(const std::vector& table RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping)); } } + for (auto& mapping : _predicate_mappings) { + if (mapping.file_local_id.has_value() && + file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { + RETURN_IF_ERROR(apply_scan_projection_to_mapping_file_type(*file_request, &mapping, + /*predicate_mapping=*/true)); + } + } for (auto& mapping : _hidden_mappings) { if (mapping.file_local_id.has_value() && file_request->local_positions.contains(LocalColumnId(*mapping.file_local_id))) { @@ -2408,14 +2512,15 @@ Status TableColumnMapper::localize_filters(const std::vector& table // Candidate columns are added before expression rewriting because their file-block positions // are needed to localize slot refs. If rewriting rejects every filter that references a visible - // column, move its already-merged output/filter projection to the lazy non-predicate set - // instead of forcing it through the eager predicate path. + // column, move its all-access-path projection to the lazy non-predicate set instead of forcing + // it through the eager predicate path. for (auto& mapping : _mappings) { if (!mapping.file_local_id.has_value()) { continue; } const auto local_id = LocalColumnId(*mapping.file_local_id); - if (localized_predicate_columns.contains(local_id)) { + if (localized_predicate_columns.contains(local_id) || + file_request->has_deferred_non_predicate_column(local_id)) { continue; } const auto predicate_it = std::ranges::find_if( @@ -2460,6 +2565,9 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c mapping->projected_file_children = file_field.children; mapping->timestamp_is_adjusted_to_utc = file_field.timestamp_is_adjusted_to_utc; mapping->file_type = file_field.type; + // Access paths are relative to the Variant terminal, so recursive complex mappings must carry + // them instead of leaving them only on the top-level table column. + mapping->variant_access_paths = table_column.variant_access_paths; mapping->is_trivial = mapping_can_use_file_column_directly(*mapping); mapping->filter_conversion = direct_filter_conversion(*mapping); mapping->child_mappings.clear(); @@ -2525,6 +2633,7 @@ Status TableColumnMapper::_create_direct_mapping(const ColumnDefinition& table_c child_mapping.table_type = table_child.type; child_mapping.file_type = table_child.type; child_mapping.default_expr = table_child.default_expr; + child_mapping.variant_access_paths = table_child.variant_access_paths; child_mapping.filter_conversion = FilterConversionType::FINALIZE_ONLY; mapping->child_mappings.push_back(std::move(child_mapping)); continue; diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index eb652833be81a9..fe9ac2ebb56c00 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -129,6 +129,8 @@ struct ColumnMapping { // file-local block layout when projection, predicate-only children, and schema evolution mix. std::vector projected_file_children; std::optional timestamp_is_adjusted_to_utc = std::nullopt; + // Table-side Variant object-key paths retained until the physical shredding schema is known. + std::vector> variant_access_paths; // Split/file-local constant entry when this mapping is produced from partition/default/virtual // expression instead of physical file data. std::optional constant_index; @@ -214,6 +216,7 @@ class TableColumnMapper { RuntimeState* runtime_state = nullptr); void clear() { _mappings.clear(); + _predicate_mappings.clear(); _hidden_mappings.clear(); _constant_map.clear(); _filter_entries.clear(); @@ -234,6 +237,12 @@ class TableColumnMapper { // delimited text field. They must scan the whole complex top-level field and let TableReader // rematerialize the requested table child after row-level filters have run. virtual bool force_full_complex_scan_projection() const { return false; } + // Only Parquet currently has a Variant physical shredding schema and a reader that can + // validate residual-value completeness before honoring a typed-leaf projection. + virtual bool enable_variant_leaf_projection() const { return false; } + // Parquet can keep two independent readers/cursors for the same complex root: a narrow eager + // predicate subtree and the final output subtree materialized only for surviving rows. + virtual bool enable_independent_predicate_projection() const { return false; } const ColumnDefinition* _find_file_field( const ColumnDefinition& table_column, @@ -256,6 +265,7 @@ class TableColumnMapper { std::vector _filter_visible_mappings() const; ColumnMapping* _find_mapping(GlobalIndex global_index); + ColumnMapping* _find_predicate_mapping(GlobalIndex global_index); ColumnMapping* _find_filter_mapping(GlobalIndex global_index); TableColumnMapperOptions _options; @@ -263,6 +273,9 @@ class TableColumnMapper { // describes how to get one table/global column from file-local sources, and carries metadata // for filter localization and result finalize. std::vector _mappings; + // Optional mappings built from SlotDescriptor::predicate_access_paths. They deliberately keep + // a different file type/projection shape from the final output mappings above. + std::vector _predicate_mappings; // Predicate-only top-level columns are not output projection columns, so keep their mappings // here. They are visible only to filter localization and file-reader predicate construction. std::vector _hidden_mappings; @@ -279,6 +292,10 @@ class TableColumnMapper { class ParquetColumnMapper final : public TableColumnMapper { public: using TableColumnMapper::TableColumnMapper; + +protected: + bool enable_variant_leaf_projection() const override { return true; } + bool enable_independent_predicate_projection() const override { return true; } }; // Mapper for readers that always materialize every required file column before filtering. The diff --git a/be/src/format_v2/column_mapper_nested.cpp b/be/src/format_v2/column_mapper_nested.cpp index fefcdf26f12e1d..61e04a0b7301de 100644 --- a/be/src/format_v2/column_mapper_nested.cpp +++ b/be/src/format_v2/column_mapper_nested.cpp @@ -30,87 +30,12 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/primitive_type.h" #include "exprs/vexpr.h" -#include "format_v2/expr/cast.h" #include "gen_cpp/Exprs_types.h" namespace doris::format { namespace { -static bool is_cast_expr(const VExprSPtr& expr) { - return dynamic_cast(expr.get()) != nullptr; -} - -static bool is_signed_integer_type(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - case TYPE_SMALLINT: - case TYPE_INT: - case TYPE_BIGINT: - case TYPE_LARGEINT: - return true; - default: - return false; - } -} - -static int primitive_integer_width(PrimitiveType type) { - switch (type) { - case TYPE_TINYINT: - return 1; - case TYPE_SMALLINT: - return 2; - case TYPE_INT: - return 4; - case TYPE_BIGINT: - return 8; - case TYPE_LARGEINT: - return 16; - default: - return 0; - } -} - -static bool is_decimal_type(PrimitiveType type) { - switch (type) { - case TYPE_DECIMAL32: - case TYPE_DECIMAL64: - case TYPE_DECIMALV2: - case TYPE_DECIMAL128I: - case TYPE_DECIMAL256: - return true; - default: - return false; - } -} - -static bool is_order_preserving_safe_cast(const DataTypePtr& from_type, - const DataTypePtr& to_type) { - if (from_type == nullptr || to_type == nullptr) { - return false; - } - const auto from_nested_type = remove_nullable(from_type); - const auto to_nested_type = remove_nullable(to_type); - if (from_nested_type->equals(*to_nested_type)) { - return true; - } - - const auto from_primitive_type = from_nested_type->get_primitive_type(); - const auto to_primitive_type = to_nested_type->get_primitive_type(); - if (is_signed_integer_type(from_primitive_type) && is_signed_integer_type(to_primitive_type)) { - return primitive_integer_width(to_primitive_type) >= - primitive_integer_width(from_primitive_type); - } - if (from_primitive_type == TYPE_FLOAT && to_primitive_type == TYPE_DOUBLE) { - return true; - } - if (is_decimal_type(from_primitive_type) && is_decimal_type(to_primitive_type)) { - return from_nested_type->get_scale() == to_nested_type->get_scale() && - to_nested_type->get_precision() >= from_nested_type->get_precision(); - } - return false; -} - static bool parse_struct_child_selector(const VExprSPtr& expr, StructChildSelector* selector) { DORIS_CHECK(selector != nullptr); if (expr == nullptr || !expr->is_literal()) { @@ -191,27 +116,6 @@ static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* return true; } -static bool extract_nested_struct_path_for_pruning(const VExprSPtr& expr, NestedStructPath* path) { - DORIS_CHECK(path != nullptr); - // Simple `ELEMENT_AT` - if (extract_nested_struct_path(expr, path)) { - return true; - } - - // `ELEMENT_AT` with `CAST` - if (!is_cast_expr(expr) || expr->get_num_children() != 1) { - return false; - } - const auto& child = expr->children()[0]; - if (!is_order_preserving_safe_cast(child->data_type(), expr->data_type())) { - return false; - } - // A safe widening cast is null-preserving and keeps the comparison ordering of the nested - // primitive leaf, so file-layer pruning can target the original leaf statistics. The row-level - // filter still evaluates the original cast expression after read. - return extract_nested_struct_path_for_pruning(child, path); -} - static const ColumnDefinition* resolve_file_child(const std::vector& children, const StructChildSelector& selector) { if (selector.by_name) { @@ -493,26 +397,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, return resolve_nested_struct_path_for_file(path, mappings, resolved, true); } -// Collect nested struct leaf references that can be turned into file-reader projections. For -// example, from `s.a > 1 AND element_at(s, 'b') = 2`, this records two paths rooted at `s`: -// `s -> a` and `s -> b`. Non-struct expressions are traversed recursively, while a recognized -// struct path is emitted once so the caller can merge it into the scan projection for that -// top-level file column. -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths) { - DORIS_CHECK(paths != nullptr); - if (expr == nullptr) { - return; - } - NestedStructPath path; - if (extract_nested_struct_path_for_pruning(expr, &path)) { - paths->push_back(std::move(path)); - return; - } - for (const auto& child : expr->children()) { - collect_nested_struct_paths(child, paths); - } -} - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings) { std::vector result; diff --git a/be/src/format_v2/column_mapper_nested.h b/be/src/format_v2/column_mapper_nested.h index 7b2e8cb1513cb4..ab96512e1709ed 100644 --- a/be/src/format_v2/column_mapper_nested.h +++ b/be/src/format_v2/column_mapper_nested.h @@ -86,8 +86,6 @@ bool resolve_nested_struct_expr_for_file(const VExprSPtr& expr, const std::vector& mappings, ResolvedNestedStructPath* resolved); -void collect_nested_struct_paths(const VExprSPtr& expr, std::vector* paths); - std::vector present_child_mappings_in_file_order( const std::vector& child_mappings); diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 9bbf7c3a8fc066..a2ca4894044404 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -65,6 +65,14 @@ std::string FileScanRequest::debug_string() const { } out << column_id << ":" << block_position; } + out << "}, non_predicate_positions={"; + position_idx = 0; + for (const auto& [column_id, block_position] : non_predicate_positions) { + if (position_idx++ > 0) { + out << ", "; + } + out << column_id << ":" << block_position; + } out << "}, conjunct_count=" << conjuncts.size() << ", delete_conjunct_count=" << delete_conjuncts.size() << ", count_star_placeholder_columns={"; diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 65a2d03417c605..0256b8c1eebcb2 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -68,8 +68,8 @@ struct FileScanRequest { // Columns that must be read before row-level filtering. They are materialized eagerly because // conjuncts/delete_conjuncts need them to decide the selected rows. std::vector predicate_columns; - // Columns read after row-level filtering. Predicate columns are also available for output and - // should not be duplicated here. + // Columns read after row-level filtering. A complex root may intentionally also appear in + // predicate_columns when its eager predicate subtree is smaller than its final output subtree. std::vector non_predicate_columns; // Predicate columns introduced only to evaluate hidden filter slots. Their values are dead // after all file-local predicates run, although the shared file block still needs row-shaped @@ -77,6 +77,10 @@ struct FileScanRequest { std::vector predicate_only_columns; // file-local column id -> file-local output block position. std::map local_positions; + // Optional output position for a root that has independent eager-predicate and deferred-output + // projections. local_positions continues to identify the position referenced by localized + // predicate expressions. + std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; // Delete predicates converted to file-local expressions. A TRUE result means that the row is @@ -98,6 +102,26 @@ struct FileScanRequest { bool is_predicate_only(LocalColumnId column_id) const { return std::ranges::find(predicate_only_columns, column_id) != predicate_only_columns.end(); } + + LocalIndex non_predicate_position(LocalColumnId column_id) const { + const auto it = non_predicate_positions.find(column_id); + return it == non_predicate_positions.end() ? local_positions.at(column_id) : it->second; + } + + bool has_deferred_non_predicate_column(LocalColumnId column_id) const { + return non_predicate_positions.contains(column_id); + } + + size_t block_column_count() const { + size_t count = 0; + for (const auto& [_, position] : local_positions) { + count = std::max(count, position.value() + 1); + } + for (const auto& [_, position] : non_predicate_positions) { + count = std::max(count, position.value() + 1); + } + return count; + } }; // Helper for constructing the scan-column layout in FileScanRequest. @@ -105,9 +129,10 @@ struct FileScanRequest { // as Parquet can read predicate columns first, filter rows, and then lazily read the remaining // projected columns. The two lists still share one file-local output block, whose positions are // stored in local_positions. This builder centralizes the mechanical rules for that shared layout: -// - each root file column gets one stable block position; +// - each root file column gets one stable predicate block position; // - predicate columns dominate non-predicate columns because they are already returned in the file // block and can be reused for final materialization; +// - a smaller complex predicate subtree may get a second deferred output position; // - repeated nested projections for the same root are merged instead of duplicated. // TableColumnMapper should still own table-to-file semantic resolution. This helper only owns the // FileScanRequest layout contract after a file-local projection has been produced. @@ -127,6 +152,38 @@ class FileScanRequestBuilder { /*is_predicate_column=*/false); } + Status add_deferred_non_predicate_column(LocalColumnIndex projection) { + const auto file_column_id = projection.column_id(); + DORIS_CHECK(file_column_id != LocalColumnId::invalid()); + DORIS_CHECK(_request->local_positions.contains(file_column_id)); + DORIS_CHECK(std::ranges::any_of(_request->predicate_columns, + [&](const LocalColumnIndex& predicate) { + return predicate.column_id() == file_column_id; + })); + + if (!_request->non_predicate_positions.contains(file_column_id)) { + _request->non_predicate_positions.emplace(file_column_id, + _next_block_position(*_request)); + } + _sort_projection_children_by_file_id(&projection); + auto existing = std::ranges::find_if(_request->non_predicate_columns, + [&](const LocalColumnIndex& output) { + return output.column_id() == file_column_id; + }); + if (existing == _request->non_predicate_columns.end()) { + _request->non_predicate_columns.push_back(std::move(projection)); + } else { + RETURN_IF_ERROR(merge_local_column_index(&*existing, projection)); + _sort_projection_children_by_file_id(&*existing); + } + if (!_request->is_predicate_only(file_column_id)) { + // The eager complex value has a different physical shape from the final value and + // must never leak into table materialization after its predicates have run. + _request->predicate_only_columns.push_back(file_column_id); + } + return Status::OK(); + } + Status add_predicate_column(LocalColumnId column_id) { return add_predicate_column(LocalColumnIndex::top_level(column_id)); } @@ -141,6 +198,9 @@ class FileScanRequestBuilder { for (const auto& [_, block_position] : request.local_positions) { next_position = std::max(next_position, block_position.value() + 1); } + for (const auto& [_, block_position] : request.non_predicate_positions) { + next_position = std::max(next_position, block_position.value() + 1); + } return LocalIndex(next_position); } @@ -164,9 +224,11 @@ class FileScanRequestBuilder { const auto file_column_id = projection.column_id(); DORIS_CHECK(file_column_id != LocalColumnId::invalid()); if (!is_predicate_column && - std::ranges::find_if(_request->predicate_columns, [&](const LocalColumnIndex& p) { - return p.column_id() == file_column_id; - }) != _request->predicate_columns.end()) { + std::ranges::find_if(_request->predicate_columns, + [&](const LocalColumnIndex& p) { + return p.column_id() == file_column_id; + }) != _request->predicate_columns.end() && + !_request->has_deferred_non_predicate_column(file_column_id)) { return Status::OK(); } if (!_request->local_positions.contains(file_column_id)) { @@ -184,7 +246,7 @@ class FileScanRequestBuilder { _sort_projection_children_by_file_id(&*existing_projection_it); } - if (is_predicate_column) { + if (is_predicate_column && !_request->has_deferred_non_predicate_column(file_column_id)) { auto it = std::ranges::find_if( _request->non_predicate_columns, [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp b/be/src/format_v2/parquet/native_schema_desc.cpp index 9eba6d8feaa55a..b56afb6f7170fb 100644 --- a/be/src/format_v2/parquet/native_schema_desc.cpp +++ b/be/src/format_v2/parquet/native_schema_desc.cpp @@ -20,7 +20,9 @@ #include #include +#include #include +#include #include #include "common/cast_set.h" @@ -55,6 +57,412 @@ static bool is_map_node(const tparquet::SchemaElement& schema) { (schema.__isset.logicalType && schema.logicalType.__isset.MAP); } +static bool is_variant_node(const tparquet::SchemaElement& schema) { + return schema.__isset.logicalType && schema.logicalType.__isset.VARIANT; +} + +enum class VariantPrimitiveAnnotation : uint8_t { + NONE, + INT8, + INT16, + DECIMAL, + DATE, + TIME_MICROS, + TIMESTAMP_MICROS, + TIMESTAMP_NANOS, + STRING, + UUID, + UNSUPPORTED, +}; + +static VariantPrimitiveAnnotation variant_logical_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.logicalType) { + return VariantPrimitiveAnnotation::NONE; + } + const auto& logical = schema.logicalType; + if (logical.__isset.INTEGER) { + if (!logical.INTEGER.isSigned) { + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.INTEGER.bitWidth == 8) { + return VariantPrimitiveAnnotation::INT8; + } + if (logical.INTEGER.bitWidth == 16) { + return VariantPrimitiveAnnotation::INT16; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.DECIMAL) { + return VariantPrimitiveAnnotation::DECIMAL; + } + if (logical.__isset.DATE) { + return VariantPrimitiveAnnotation::DATE; + } + if (logical.__isset.TIME) { + return !logical.TIME.isAdjustedToUTC && logical.TIME.unit.__isset.MICROS + ? VariantPrimitiveAnnotation::TIME_MICROS + : VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.TIMESTAMP) { + if (logical.TIMESTAMP.unit.__isset.MICROS) { + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + } + if (logical.TIMESTAMP.unit.__isset.NANOS) { + return VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + } + return VariantPrimitiveAnnotation::UNSUPPORTED; + } + if (logical.__isset.STRING) { + return VariantPrimitiveAnnotation::STRING; + } + if (logical.__isset.UUID) { + return VariantPrimitiveAnnotation::UUID; + } + const bool empty = !logical.__isset.MAP && !logical.__isset.LIST && !logical.__isset.ENUM && + !logical.__isset.UNKNOWN && !logical.__isset.JSON && !logical.__isset.BSON && + !logical.__isset.FLOAT16 && !logical.__isset.GEOMETRY && + !logical.__isset.GEOGRAPHY && !logical.__isset.VARIANT; + return empty ? VariantPrimitiveAnnotation::NONE : VariantPrimitiveAnnotation::UNSUPPORTED; +} + +static VariantPrimitiveAnnotation variant_converted_annotation( + const tparquet::SchemaElement& schema) { + if (!schema.__isset.converted_type) { + return VariantPrimitiveAnnotation::NONE; + } + switch (schema.converted_type) { + case tparquet::ConvertedType::INT_8: + return VariantPrimitiveAnnotation::INT8; + case tparquet::ConvertedType::INT_16: + return VariantPrimitiveAnnotation::INT16; + case tparquet::ConvertedType::DECIMAL: + return VariantPrimitiveAnnotation::DECIMAL; + case tparquet::ConvertedType::DATE: + return VariantPrimitiveAnnotation::DATE; + case tparquet::ConvertedType::TIME_MICROS: + return VariantPrimitiveAnnotation::TIME_MICROS; + case tparquet::ConvertedType::TIMESTAMP_MICROS: + return VariantPrimitiveAnnotation::TIMESTAMP_MICROS; + case tparquet::ConvertedType::UTF8: + return VariantPrimitiveAnnotation::STRING; + default: + return VariantPrimitiveAnnotation::UNSUPPORTED; + } +} + +static Status validate_variant_decimal(const tparquet::SchemaElement& schema, + tparquet::Type::type physical_type) { + int32_t precision = -1; + int32_t scale = -1; + if (schema.__isset.logicalType && schema.logicalType.__isset.DECIMAL) { + precision = schema.logicalType.DECIMAL.precision; + scale = schema.logicalType.DECIMAL.scale; + if ((schema.__isset.precision && schema.precision != precision) || + (schema.__isset.scale && schema.scale != scale)) { + return Status::Corruption( + "Parquet Variant DECIMAL logical and converted parameters disagree"); + } + } else if (schema.__isset.precision && schema.__isset.scale) { + precision = schema.precision; + scale = schema.scale; + } + if (precision <= 0 || precision > 38 || scale < 0 || scale > precision) { + return Status::Corruption("Parquet Variant DECIMAL({}, {}) is invalid", precision, scale); + } + + if ((physical_type == tparquet::Type::INT32 && precision > 9) || + (physical_type == tparquet::Type::INT64 && (precision < 10 || precision > 18)) || + ((physical_type == tparquet::Type::BYTE_ARRAY || + physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) && + precision < 19)) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not match physical type {}", precision, + physical_type); + } + if (physical_type == tparquet::Type::FIXED_LEN_BYTE_ARRAY) { + static constexpr int32_t MAX_PRECISION_BY_LENGTH[] = {2, 4, 6, 9, 11, 14, 16, 18, + 21, 23, 26, 28, 31, 33, 35, 38}; + if (!schema.__isset.type_length || schema.type_length <= 0 || schema.type_length > 16 || + precision > MAX_PRECISION_BY_LENGTH[schema.type_length - 1]) { + return Status::Corruption( + "Parquet Variant DECIMAL precision {} does not fit fixed length {}", precision, + schema.__isset.type_length ? schema.type_length : -1); + } + } + return Status::OK(); +} + +static Status validate_variant_primitive_type(const NativeFieldSchema& typed) { + const auto& schema = typed.parquet_schema; + auto logical = variant_logical_annotation(schema); + auto converted = variant_converted_annotation(schema); + if (logical == VariantPrimitiveAnnotation::UNSUPPORTED || + converted == VariantPrimitiveAnnotation::UNSUPPORTED || + (logical != VariantPrimitiveAnnotation::NONE && + converted != VariantPrimitiveAnnotation::NONE && logical != converted)) { + return Status::Corruption( + "Parquet Variant typed value {} has an unsupported logical annotation", typed.name); + } + const auto annotation = logical != VariantPrimitiveAnnotation::NONE ? logical : converted; + const auto physical = schema.type; + bool valid = false; + switch (physical) { + case tparquet::Type::BOOLEAN: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::INT32: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::INT8 || + annotation == VariantPrimitiveAnnotation::INT16 || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::DATE; + break; + case tparquet::Type::INT64: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::DECIMAL || + annotation == VariantPrimitiveAnnotation::TIME_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_MICROS || + annotation == VariantPrimitiveAnnotation::TIMESTAMP_NANOS; + break; + case tparquet::Type::FLOAT: + case tparquet::Type::DOUBLE: + valid = annotation == VariantPrimitiveAnnotation::NONE; + break; + case tparquet::Type::BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::STRING || + annotation == VariantPrimitiveAnnotation::DECIMAL; + break; + case tparquet::Type::FIXED_LEN_BYTE_ARRAY: + valid = annotation == VariantPrimitiveAnnotation::DECIMAL || + (annotation == VariantPrimitiveAnnotation::UUID && schema.__isset.type_length && + schema.type_length == 16); + break; + default: + valid = false; + break; + } + if (!valid) { + return Status::Corruption( + "Parquet Variant typed value {} has unsupported physical/logical type pair", + typed.name); + } + if (annotation == VariantPrimitiveAnnotation::DECIMAL) { + RETURN_IF_ERROR(validate_variant_decimal(schema, physical)); + } + return Status::OK(); +} + +class ScopedBoolOverride { +public: + ScopedBoolOverride(bool& target, bool value) : _target(target), _original(target) { + _target = value; + } + ~ScopedBoolOverride() { _target = _original; } + +private: + bool& _target; + bool _original; +}; + +static Status validate_variant_layout(const tparquet::SchemaElement& group_schema, + const NativeFieldSchema& group_field) { + const auto& annotation = group_schema.logicalType.VARIANT; + if (annotation.__isset.specification_version && annotation.specification_version != 1) { + return Status::NotSupported("Parquet Variant specification version {} is not supported", + annotation.specification_version); + } + if (group_field.children.size() < 2 || group_field.children.size() > 3) { + return Status::Corruption( + "Parquet Variant {} must contain metadata, value, and optional typed_value", + group_schema.name); + } + + const NativeFieldSchema* metadata = nullptr; + const NativeFieldSchema* value = nullptr; + const NativeFieldSchema* typed_value = nullptr; + for (const auto& child : group_field.children) { + const NativeFieldSchema** target = nullptr; + if (child.name == "metadata") { + target = &metadata; + } else if (child.name == "value") { + target = &value; + } else if (child.name == "typed_value") { + target = &typed_value; + } else { + return Status::Corruption("Parquet Variant {} has unexpected child {}", + group_schema.name, child.name); + } + if (*target != nullptr) { + return Status::Corruption("Parquet Variant {} has duplicate child {}", + group_schema.name, child.name); + } + *target = &child; + } + if (metadata == nullptr || value == nullptr) { + return Status::Corruption("Parquet Variant {} requires metadata and value children", + group_schema.name); + } + if (!metadata->children.empty() || metadata->physical_type != tparquet::Type::BYTE_ARRAY || + metadata->parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant {} metadata must be a required BYTE_ARRAY", + group_schema.name); + } + const auto expected_value_repetition = typed_value == nullptr + ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL; + // SQL nullability belongs to the outer Variant group. Only shredding makes value optional, + // because typed_value may carry all or part of the logical value instead. + if (!value->children.empty() || value->physical_type != tparquet::Type::BYTE_ARRAY || + value->parquet_schema.repetition_type != expected_value_repetition) { + return Status::Corruption("Parquet Variant {} value must be a {} BYTE_ARRAY", + group_schema.name, + typed_value == nullptr ? "required" : "optional"); + } + if (typed_value != nullptr && + typed_value->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant {} typed_value must be optional", + group_schema.name); + } + + enum class WrapperContext : uint8_t { OBJECT_FIELD, ARRAY_ELEMENT }; + std::function validate_typed_value; + std::function validate_wrapper; + validate_wrapper = [&](const NativeFieldSchema& wrapper, WrapperContext context) -> Status { + if (!wrapper.parquet_schema.__isset.repetition_type || + wrapper.parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + return Status::Corruption("Parquet Variant shredded wrapper {} must be required", + wrapper.name); + } + const NativeFieldSchema* fallback = nullptr; + const NativeFieldSchema* typed = nullptr; + for (const auto& child : wrapper.children) { + if (child.name == "value") { + if (fallback != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate value child", wrapper.name); + } + fallback = &child; + } else if (child.name == "typed_value") { + if (typed != nullptr) { + return Status::Corruption( + "Parquet Variant wrapper {} has duplicate typed_value child", + wrapper.name); + } + typed = &child; + } else { + return Status::Corruption("Parquet Variant wrapper {} has unexpected child {}", + wrapper.name, child.name); + } + } + if (fallback == nullptr && typed == nullptr) { + return Status::Corruption( + "Parquet Variant shredded wrapper {} requires at least one of value or " + "typed_value", + wrapper.name); + } + // Object fields always retain the fallback value carrier; only typed_value is optional. + // Array elements may omit either carrier when every element uses the remaining one. + if (context == WrapperContext::OBJECT_FIELD && fallback == nullptr) { + return Status::Corruption( + "Parquet Variant object wrapper {} requires an optional value child", + wrapper.name); + } + if (fallback != nullptr && + (!fallback->children.empty() || fallback->physical_type != tparquet::Type::BYTE_ARRAY || + !fallback->parquet_schema.__isset.repetition_type || + fallback->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL)) { + return Status::Corruption( + "Parquet Variant wrapper {} value must be an optional BYTE_ARRAY", + wrapper.name); + } + if (typed != nullptr) { + if (!typed->parquet_schema.__isset.repetition_type || + typed->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { + return Status::Corruption("Parquet Variant wrapper {} typed_value must be optional", + wrapper.name); + } + return validate_typed_value(*typed); + } + return Status::OK(); + }; + validate_typed_value = [&](const NativeFieldSchema& typed) -> Status { + if (!typed.unsupported_reason.empty()) { + return Status::NotSupported("Parquet Variant typed value {} is not supported: {}", + typed.name, typed.unsupported_reason); + } + if (typed.children.empty()) { + const auto& physical = typed.parquet_schema; + if (physical.__isset.logicalType && physical.logicalType.__isset.INTEGER && + !physical.logicalType.INTEGER.isSigned) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.converted_type && + (physical.converted_type == tparquet::ConvertedType::UINT_8 || + physical.converted_type == tparquet::ConvertedType::UINT_16 || + physical.converted_type == tparquet::ConvertedType::UINT_32 || + physical.converted_type == tparquet::ConvertedType::UINT_64)) { + return Status::Corruption( + "Parquet Variant unsigned integers are not valid typed values"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIME) { + const auto& time = physical.logicalType.TIME; + // Variant v1 has one canonical TIME representation: local wall-clock MICROS. + // Accepting adjusted or lower-precision forms would make projection-dependent + // reconstruction disagree with the canonical Variant value. + if (time.isAdjustedToUTC) { + return Status::Corruption( + "Parquet Variant TIME must have isAdjustedToUTC=false"); + } + if (!time.unit.__isset.MICROS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + } + if (physical.__isset.converted_type && + physical.converted_type == tparquet::ConvertedType::TIME_MILLIS) { + return Status::Corruption( + "Parquet Variant TIME(MILLIS) is not supported; use TIME(MICROS)"); + } + if (physical.__isset.logicalType && physical.logicalType.__isset.TIMESTAMP && + physical.logicalType.TIMESTAMP.unit.__isset.NANOS) { + // Reject at schema open so full reconstruction and direct typed-leaf access have + // the same precision contract instead of diverging after projection planning. + return Status::NotSupported("Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + // Preserve precise diagnostics above, then enforce the complete Variant matrix before + // generic Parquet inference can re-encode a value with another logical identity. + RETURN_IF_ERROR(validate_variant_primitive_type(typed)); + return Status::OK(); + } + + const PrimitiveType primitive = remove_nullable(typed.data_type)->get_primitive_type(); + if (primitive == TYPE_STRUCT) { + std::unordered_set field_names; + for (const auto& child : typed.children) { + if (!field_names.insert(child.name).second) { + // Name lookup selects one physical wrapper, so duplicates would otherwise make + // full reconstruction and leaf projection observe different logical values. + return Status::Corruption( + "Parquet Variant object has duplicate shredded field {}", child.name); + } + RETURN_IF_ERROR(validate_wrapper(child, WrapperContext::OBJECT_FIELD)); + } + return Status::OK(); + } + if (primitive == TYPE_ARRAY && typed.children.size() == 1) { + return validate_wrapper(typed.children[0], WrapperContext::ARRAY_ELEMENT); + } + return Status::Corruption("Invalid Parquet Variant typed_value schema {}", typed.name); + }; + if (typed_value != nullptr) { + RETURN_IF_ERROR(validate_typed_value(*typed_value)); + } + return Status::OK(); +} + static bool has_primitive_only_annotation(const tparquet::SchemaElement& schema) { if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; @@ -265,6 +673,11 @@ Status NativeFieldDescriptor::parse_node_field( // nested structure or nullable list return parse_group_field(t_schemas, curr_pos, node_field); } + if (is_variant_node(t_schema)) { + return Status::InvalidArgument( + "Parquet Variant logical type requires a group node, got primitive {}", + t_schema.name); + } if (is_repeated_node(t_schema)) { // repeated (LIST) // produce required list @@ -517,6 +930,27 @@ Status NativeFieldDescriptor::parse_group_field( const std::vector& t_schemas, size_t curr_pos, NativeFieldSchema* group_field) { auto& group_schema = t_schemas[curr_pos]; + group_field->parquet_schema = group_schema; + if (is_variant_node(group_schema)) { + if (is_repeated_node(group_schema)) { + // A repeated annotated group needs an ARRAY carrier and Dremel-level remapping; treating + // it as a scalar Variant would expose the wrong row shape. + return Status::NotSupported("repeated Parquet Variant group {} is not supported", + group_schema.name); + } + // UTC-adjusted timestamps inside Variant carry an instant, independent of the catalog's + // presentation mapping. Parsing them as DATETIMEV2 would apply the session timezone and + // lose that instant when the shredded value is re-encoded into ColumnVariantV2. + { + ScopedBoolOverride timestamp_tz_mapping(_enable_mapping_timestamp_tz, true); + RETURN_IF_ERROR(parse_struct_field(t_schemas, curr_pos, group_field)); + } + RETURN_IF_ERROR(validate_variant_layout(group_schema, *group_field)); + group_field->variant_physical_type = group_field->data_type; + // Native page readers dispatch groups from data_type, so preserve the physical STRUCT + // here. The public Parquet schema maps it to logical Variant without losing this shape. + return Status::OK(); + } if ((group_schema.__isset.logicalType && group_schema.logicalType.__isset.ENUM) || (group_schema.__isset.converted_type && group_schema.converted_type == tparquet::ConvertedType::ENUM)) { diff --git a/be/src/format_v2/parquet/native_schema_desc.h b/be/src/format_v2/parquet/native_schema_desc.h index dfb669559fc71a..918be6d2c65c58 100644 --- a/be/src/format_v2/parquet/native_schema_desc.h +++ b/be/src/format_v2/parquet/native_schema_desc.h @@ -47,6 +47,10 @@ struct NativeFieldSchema { // Used to identify whether this field is a nested field. DataTypePtr data_type; + + // VARIANT is logically exposed as DataTypeVariantV2, while native page readers still need the + // physical STRUCT shape formed by metadata/value/typed_value. + DataTypePtr variant_physical_type; // Schema construction keeps a physical fallback so unprojected columns and metadata-only // queries remain readable, while projection validation reports the original logical failure. std::string unsupported_reason; diff --git a/be/src/format_v2/parquet/native_schema_node.cpp b/be/src/format_v2/parquet/native_schema_node.cpp index 052df93e4b4143..e1f573951fbbfc 100644 --- a/be/src/format_v2/parquet/native_schema_node.cpp +++ b/be/src/format_v2/parquet/native_schema_node.cpp @@ -64,7 +64,8 @@ Status build_native_schema_node(const DataTypePtr& projected_type, const auto type = remove_nullable(projected_type); switch (type->get_primitive_type()) { case TYPE_STRUCT: { - if (file_schema.kind != ParquetColumnSchemaKind::STRUCT) { + if (file_schema.kind != ParquetColumnSchemaKind::STRUCT && + file_schema.kind != ParquetColumnSchemaKind::VARIANT) { return Status::Corruption("Parquet column {} is not a STRUCT", file_schema.name); } const auto* struct_type = assert_cast(type.get()); diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 7e541b57da2780..623f1ece3dd409 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -20,7 +20,11 @@ #include #include +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/parquet_type.h" @@ -63,7 +67,11 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc result->fixed_length = schema.__isset.type_length ? schema.type_length : -1; if (schema.__isset.logicalType) { const auto& logical = schema.logicalType; - if (logical.__isset.DECIMAL) { + if (logical.__isset.STRING) { + result->is_string_annotation = true; + } else if (logical.__isset.UUID) { + result->is_uuid = true; + } else if (logical.__isset.DECIMAL) { result->is_decimal = true; result->decimal_precision = logical.DECIMAL.precision; result->decimal_scale = logical.DECIMAL.scale; @@ -87,6 +95,9 @@ void fill_native_type_descriptor(const NativeFieldSchema& field, ParquetTypeDesc } } else if (schema.__isset.converted_type) { switch (schema.converted_type) { + case tparquet::ConvertedType::UTF8: + result->is_string_annotation = true; + break; case tparquet::ConvertedType::DECIMAL: result->is_decimal = true; result->decimal_precision = schema.__isset.precision ? schema.precision : -1; @@ -167,13 +178,31 @@ void propagate_native_max_levels(ParquetColumnSchema* schema) { } } +bool contains_variant_node(const ParquetColumnSchema& schema) { + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + return true; + } + return std::ranges::any_of(schema.children, [](const auto& child) { + DORIS_CHECK(child != nullptr); + return contains_variant_node(*child); + }); +} + std::unique_ptr build_native_node_schema(const NativeFieldSchema& field, int32_t local_id) { auto result = std::make_unique(); result->local_id = local_id; result->parquet_field_id = field.field_id; result->name = field.name; - result->type = field.data_type; + result->variant_physical_type = field.variant_physical_type; + if (field.variant_physical_type != nullptr) { + DataTypePtr variant_type = std::make_shared(); + result->type = field.variant_physical_type->is_nullable() + ? make_nullable(std::move(variant_type)) + : std::move(variant_type); + } else { + result->type = field.data_type; + } result->definition_level = field.definition_level; result->repetition_level = field.repetition_level; result->max_definition_level = field.definition_level; @@ -191,7 +220,9 @@ std::unique_ptr build_native_node_schema(const NativeFieldS fill_native_type_descriptor(field, &result->type_descriptor); return result; } - if (primitive_type == TYPE_ARRAY) { + if (field.variant_physical_type != nullptr) { + result->kind = ParquetColumnSchemaKind::VARIANT; + } else if (primitive_type == TYPE_ARRAY) { result->kind = ParquetColumnSchemaKind::LIST; } else if (primitive_type == TYPE_MAP) { result->kind = ParquetColumnSchemaKind::MAP; @@ -203,6 +234,32 @@ std::unique_ptr build_native_node_schema(const NativeFieldS result->children.push_back( build_native_node_schema(field.children[child_idx], cast_set(child_idx))); } + // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild + // every enclosing complex type so file-block columns keep the same logical shape as readers. + if (result->kind != ParquetColumnSchemaKind::VARIANT && contains_variant_node(*result)) { + DataTypePtr logical_type; + if (result->kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(result->children.size() == 1); + logical_type = std::make_shared(result->children[0]->type); + } else if (result->kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(result->children.size() == 2); + logical_type = std::make_shared(make_nullable(result->children[0]->type), + make_nullable(result->children[1]->type)); + } else { + DataTypes child_types; + Strings child_names; + child_types.reserve(result->children.size()); + child_names.reserve(result->children.size()); + for (const auto& child : result->children) { + child_types.push_back(child->type); + child_names.push_back(child->name); + } + logical_type = std::make_shared(std::move(child_types), + std::move(child_names)); + } + result->type = result->type->is_nullable() ? make_nullable(std::move(logical_type)) + : std::move(logical_type); + } propagate_native_max_levels(result.get()); return result; } diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index 5f7dadf0492451..ad92af269770f5 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -33,6 +33,7 @@ enum class ParquetColumnSchemaKind { STRUCT, // Parquet group with STRUCT semantics LIST, // Parquet group with LIST semantics MAP, // Parquet group with MAP semantics + VARIANT, // Parquet Variant logical group }; // ============================================================================ @@ -48,6 +49,9 @@ struct ParquetColumnSchema { DataTypePtr type = nullptr; std::optional timestamp_is_adjusted_to_utc = std::nullopt; + // Set only for VARIANT. The public file type is DataTypeVariantV2, while this type describes + // the metadata/value/typed_value STRUCT consumed by the native decoder. + DataTypePtr variant_physical_type = nullptr; int leaf_column_id = -1; diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index e6717505cea898..5fd3100a2e5bcf 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -56,6 +56,8 @@ void ParquetProfile::init(RuntimeProfile* profile) { parquet_profile, 1); filtered_page_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "FilteredRowsByPage", TUnit::UNIT, parquet_profile, 1); + variant_leaf_projections = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantLeafProjections", + TUnit::UNIT, parquet_profile, 1); pages_skipped_by_data_page_filter = ADD_CHILD_COUNTER_WITH_LEVEL( profile, "PagesSkippedByDataPageFilter", TUnit::UNIT, parquet_profile, 1); data_page_filter_skip_bytes = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "DataPageFilterSkipBytes", @@ -88,6 +90,18 @@ void ParquetProfile::init(RuntimeProfile* profile) { ADD_CHILD_TIMER_WITH_LEVEL(profile, "LevelOnlySkipTime", parquet_profile, 1); materialization_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "MaterializationTime", parquet_profile, 1); + variant_reconstruction_time = + ADD_CHILD_TIMER_WITH_LEVEL(profile, "VariantReconstructionTime", parquet_profile, 1); + variant_reconstructed_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantReconstructedRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantDirectLeafRows", + TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_path_misses = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafPathMisses", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_residual_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafResidualFallbacks", TUnit::UNIT, parquet_profile, 1); + variant_direct_leaf_unsupported_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "VariantDirectLeafUnsupportedFallbacks", TUnit::UNIT, parquet_profile, 1); hybrid_selection_batches = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionBatches", TUnit::UNIT, parquet_profile, 1); hybrid_selection_ranges = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionRanges", @@ -298,6 +312,12 @@ ParquetColumnReaderProfile ParquetProfile::column_reader_profile() const { .level_only_read_time = level_only_read_time, .level_only_skip_time = level_only_skip_time, .materialization_time = materialization_time, + .variant_reconstruction_time = variant_reconstruction_time, + .variant_reconstructed_rows = variant_reconstructed_rows, + .variant_direct_leaf_rows = variant_direct_leaf_rows, + .variant_direct_leaf_path_misses = variant_direct_leaf_path_misses, + .variant_direct_leaf_residual_fallbacks = variant_direct_leaf_residual_fallbacks, + .variant_direct_leaf_unsupported_fallbacks = variant_direct_leaf_unsupported_fallbacks, .hybrid_selection_batches = hybrid_selection_batches, .hybrid_selection_ranges = hybrid_selection_ranges, .hybrid_selection_null_fallback_batches = hybrid_selection_null_fallback_batches, diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index 57c03c79336f06..ed1faa8f935134 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -38,6 +38,12 @@ struct ParquetColumnReaderProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns) + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; @@ -145,6 +151,8 @@ struct ParquetProfile { RuntimeProfile::Counter* selected_row_ranges = nullptr; RuntimeProfile::Counter* filtered_group_rows = nullptr; RuntimeProfile::Counter* filtered_page_rows = nullptr; + // File-level Variant access paths that safely retained a physical typed-leaf projection. + RuntimeProfile::Counter* variant_leaf_projections = nullptr; // ======== Page Skip ======== RuntimeProfile::Counter* pages_skipped_by_data_page_filter = nullptr; @@ -166,6 +174,12 @@ struct ParquetProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; + RuntimeProfile::Counter* variant_reconstruction_time = nullptr; + RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; + RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index 0e934d0c9bbe45..47a907d0a638ce 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -59,6 +60,117 @@ struct ParquetReaderScanState { bool enable_strict_mode = false; }; +const ParquetColumnSchema* projected_schema_child(const ParquetColumnSchema& schema, + int32_t local_id) { + const auto child_it = std::ranges::find_if( + schema.children, [local_id](const auto& child) { return child->local_id == local_id; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +const ParquetColumnSchema* schema_child_by_name(const ParquetColumnSchema& schema, + std::string_view name) { + const auto child_it = std::ranges::find_if( + schema.children, [name](const auto& child) { return child->name == name; }); + return child_it == schema.children.end() ? nullptr : child_it->get(); +} + +bool collect_variant_residual_leaf_ids(const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection, + std::vector* residual_leaf_ids) { + DORIS_CHECK(residual_leaf_ids != nullptr); + const auto* value = schema_child_by_name(schema, "value"); + const auto* typed_value = schema_child_by_name(schema, "typed_value"); + if (value != nullptr && typed_value != nullptr) { + if (value->kind != ParquetColumnSchemaKind::PRIMITIVE || value->leaf_column_id < 0) { + return false; + } + residual_leaf_ids->push_back(value->leaf_column_id); + } + for (const auto& child_projection : projection.children) { + const auto* child = projected_schema_child(schema, child_projection.local_id()); + if (child == nullptr || + !collect_variant_residual_leaf_ids(*child, child_projection, residual_leaf_ids)) { + return false; + } + } + return true; +} + +bool detail::variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT || schema.max_repetition_level != 0 || + !format::is_partial_projection(&projection)) { + return false; + } + std::vector residual_leaf_ids; + if (!collect_variant_residual_leaf_ids(schema, projection, &residual_leaf_ids)) { + return false; + } + std::ranges::sort(residual_leaf_ids); + residual_leaf_ids.erase(std::unique(residual_leaf_ids.begin(), residual_leaf_ids.end()), + residual_leaf_ids.end()); + for (const auto& row_group : metadata.row_groups) { + for (const int leaf_id : residual_leaf_ids) { + if (leaf_id < 0 || leaf_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[leaf_id]; + if (!chunk.__isset.meta_data || !chunk.meta_data.__isset.statistics || + !chunk.meta_data.statistics.__isset.null_count || + chunk.meta_data.statistics.null_count != row_group.num_rows) { + return false; + } + } + } + return true; +} + +size_t detail::finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection) { + DORIS_CHECK(projection != nullptr); + if (!format::is_partial_projection(projection)) { + return 0; + } + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + if (variant_projection_is_fully_shredded(metadata, schema, *projection)) { + return 1; + } + // Unknown residual completeness must restore this Variant wrapper atomically. For a + // repeated ancestor, footer null_count is in the leaf-value domain rather than Variant + // instances, so variant_projection_is_fully_shredded() deliberately takes this fallback. + projection->project_all_children = true; + projection->children.clear(); + return 0; + } + + size_t retained = 0; + for (auto& child_projection : projection->children) { + const auto* child_schema = projected_schema_child(schema, child_projection.local_id()); + DORIS_CHECK(child_schema != nullptr); + retained += finalize_variant_leaf_projection(metadata, *child_schema, &child_projection); + } + return retained; +} + +size_t finalize_variant_leaf_projections( + const NativeParquetMetadata& metadata, + const std::vector>& file_schema, + std::vector* projections) { + DORIS_CHECK(projections != nullptr); + size_t retained = 0; + for (auto& projection : *projections) { + const int32_t local_id = projection.local_id(); + if (local_id < 0 || local_id >= static_cast(file_schema.size())) { + continue; + } + retained += detail::finalize_variant_leaf_projection(metadata.to_thrift(), + *file_schema[local_id], &projection); + } + return retained; +} + Status validate_all_projected_leaves_supported(const ParquetColumnSchema& column_schema) { if (column_schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { if (!column_schema.type_descriptor.unsupported_reason.empty()) { @@ -496,6 +608,17 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); + const size_t retained_variant_leaf_projections = + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->predicate_columns) + + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->non_predicate_columns); + if (_parquet_profile.variant_leaf_projections != nullptr) { + COUNTER_UPDATE(_parquet_profile.variant_leaf_projections, + retained_variant_leaf_projections); + } RETURN_IF_ERROR(format::FileReader::open(std::move(request))); // `local_positions.empty()` means all columns are needed by table reader diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index 323415ed5611f5..5a1ebd64614b01 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -37,6 +37,15 @@ namespace doris::format::parquet { struct ParquetReaderScanState; +namespace detail { +bool variant_projection_is_fully_shredded(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + const format::LocalColumnIndex& projection); +size_t finalize_variant_leaf_projection(const tparquet::FileMetaData& metadata, + const ParquetColumnSchema& schema, + format::LocalColumnIndex* projection); +} // namespace detail + // ============================================================================ // ============================================================================ // init() -> get_schema() -> open(request) -> get_block() [loop] -> close() diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index a9154021999a4f..15b8bd90195599 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -303,7 +303,7 @@ void materialize_count_star_placeholders(const format::FileScanRequest& request, if (!request.is_count_star_placeholder(column.column_id())) { continue; } - const auto block_position = request.local_positions.at(column.column_id()).value(); + const auto block_position = request.non_predicate_position(column.column_id()).value(); auto placeholder = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK(placeholder->empty()); placeholder->insert_many_defaults(rows); @@ -519,8 +519,9 @@ Status finalize_native_row_group_read_plan( std::vector page_selected_ranges; std::map page_skip_plans; RETURN_IF_ERROR(select_row_group_ranges_by_native_page_index( - thrift, page_indexes, file_schema, request, row_group_plan->row_group_rows, - &page_selected_ranges, &page_skip_plans, pruning_stats, timezone, runtime_state)); + thrift, thrift.row_groups[row_group_plan->row_group_id], page_indexes, file_schema, + request, row_group_plan->row_group_rows, &page_selected_ranges, &page_skip_plans, + pruning_stats, timezone, runtime_state)); row_group_plan->selected_ranges = intersect_row_ranges(row_group_plan->selected_ranges, page_selected_ranges); row_group_plan->page_skip_plans = std::move(page_skip_plans); @@ -1220,9 +1221,17 @@ Status ParquetScanScheduler::open_next_row_group( RETURN_IF_ERROR(detail::build_native_prefetch_ranges( thrift_metadata, file_schema, request_scan_columns(request), row_group_idx, file_context.native_file->size(), compat.parquet_816_padding, &native_ranges)); - _current_merge_range_active = file_context.set_native_random_access_ranges( - native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, - _merge_read_slice_size); + if (request.non_predicate_positions.empty()) { + _current_merge_range_active = file_context.set_native_random_access_ranges( + native_ranges, detail::average_prefetch_range_size(native_ranges), _profile, + _merge_read_slice_size); + } else { + // Independent predicate/output readers may revisit the same physical leaf at different + // cursors. MergeRangeFileReader has one consumptive cache per range, so use the random + // access reader for this layout instead of sharing one sequential range cache. + _current_merge_range_active = file_context.set_native_random_access_ranges( + {}, 0, _profile, _merge_read_slice_size); + } for (const auto& col : request.predicate_columns) { const auto local_id = col.column_id(); @@ -2670,9 +2679,7 @@ Status ParquetScanScheduler::read_current_row_group_batch( // selection vector. This also merges pending range gaps with fully filtered batches. RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); DCHECK_EQ(file_block->get_by_position(block_position).type->get_primitive_type(), column_reader->type()->get_primitive_type()) @@ -2740,9 +2747,7 @@ Status ParquetScanScheduler::materialize_pending_predicate_batch( SCOPED_TIMER(_scan_profile.column_read_time); RETURN_IF_ERROR(flush_pending_non_predicate_skip_rows()); for (const auto& [fid, column_reader] : _current_non_predicate_columns) { - auto position_it = request.local_positions.find(fid); - DORIS_CHECK(position_it != request.local_positions.end()); - const auto block_position = position_it->second.value(); + const auto block_position = request.non_predicate_position(fid).value(); auto column = file_block->get_by_position(block_position).column->assert_mutable(); [[maybe_unused]] const auto old_size = column->size(); RETURN_IF_ERROR(column_reader->select(_pending_output_selection, diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index e6f42c59ce61ea..1d10e4cb2c002f 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -37,7 +37,10 @@ #include "core/data_type_serde/data_type_serde.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vectorized_fn_call.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" +#include "exprs/vslot_ref.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" #include "format_v2/parquet/reader/native/block_split_bloom_filter.h" @@ -456,6 +459,322 @@ std::optional file_column_id_by_block_position( return std::nullopt; } +enum class VariantComparisonOp { EQ, NE, LT, LE, GT, GE }; + +struct VariantShreddedPredicate { + int slot_index = -1; + std::vector path; + DataTypePtr comparison_type; + DataTypePtr literal_type; + Field literal; + VariantComparisonOp op = VariantComparisonOp::EQ; +}; + +std::string callable_name(const VExprSPtr& expr) { + if (const auto function = std::dynamic_pointer_cast(expr); + function != nullptr) { + return function->function_name(); + } + return expr == nullptr ? std::string {} : expr->expr_name(); +} + +std::optional variant_comparison_op(std::string_view name) { + if (name == "eq") { + return VariantComparisonOp::EQ; + } + if (name == "ne") { + return VariantComparisonOp::NE; + } + if (name == "lt") { + return VariantComparisonOp::LT; + } + if (name == "le") { + return VariantComparisonOp::LE; + } + if (name == "gt") { + return VariantComparisonOp::GT; + } + if (name == "ge") { + return VariantComparisonOp::GE; + } + return std::nullopt; +} + +VariantComparisonOp reverse_variant_comparison(VariantComparisonOp op) { + switch (op) { + case VariantComparisonOp::EQ: + case VariantComparisonOp::NE: + return op; + case VariantComparisonOp::LT: + return VariantComparisonOp::GT; + case VariantComparisonOp::LE: + return VariantComparisonOp::GE; + case VariantComparisonOp::GT: + return VariantComparisonOp::LT; + case VariantComparisonOp::GE: + return VariantComparisonOp::LE; + } + __builtin_unreachable(); +} + +std::optional> variant_literal(const VExprSPtr& expr) { + const auto literal = std::dynamic_pointer_cast(expr); + if (literal == nullptr || !literal->get_column_ptr() || literal->get_column_ptr()->empty()) { + return std::nullopt; + } + Field value; + literal->get_column_ptr()->get(0, value); + if (value.is_null()) { + return std::nullopt; + } + return std::make_pair(std::move(value), literal->get_data_type()); +} + +std::optional extract_variant_shredded_predicate( + const VExprContextSPtr& conjunct) { + if (conjunct == nullptr || conjunct->root() == nullptr || + conjunct->root()->get_num_children() != 2) { + return std::nullopt; + } + auto op = variant_comparison_op(callable_name(conjunct->root())); + if (!op.has_value()) { + return std::nullopt; + } + + VExprSPtr value_expr; + std::optional> literal; + if ((literal = variant_literal(conjunct->root()->get_child(1))).has_value()) { + value_expr = conjunct->root()->get_child(0); + } else if ((literal = variant_literal(conjunct->root()->get_child(0))).has_value()) { + value_expr = conjunct->root()->get_child(1); + op = reverse_variant_comparison(*op); + } else { + return std::nullopt; + } + + const auto comparison_type = value_expr->data_type(); + while (value_expr->node_type() == TExprNodeType::CAST_EXPR && + value_expr->get_num_children() == 1) { + if (!expr_zonemap::data_types_compatible(value_expr->data_type(), comparison_type)) { + // Every removed cast must preserve the comparison domain. Otherwise bounds for the + // raw typed leaf could skip rows whose value changes in an intermediate narrowing cast. + return std::nullopt; + } + value_expr = value_expr->get_child(0); + } + + std::vector reverse_path; + while (callable_name(value_expr) == "element_at" && value_expr->get_num_children() == 2) { + const auto key = variant_literal(value_expr->get_child(1)); + if (!key.has_value() || key->first.get_type() != TYPE_STRING) { + // Repeated array shredding has no single scalar page range, so only object keys are + // eligible for this file-level optimization. + return std::nullopt; + } + reverse_path.push_back(key->first.get()); + value_expr = value_expr->get_child(0); + } + const auto slot = std::dynamic_pointer_cast(value_expr); + if (slot == nullptr || reverse_path.empty() || comparison_type == nullptr || + remove_nullable(slot->data_type())->get_primitive_type() != TYPE_VARIANT || + !expr_zonemap::data_types_compatible(comparison_type, literal->second)) { + return std::nullopt; + } + std::ranges::reverse(reverse_path); + return VariantShreddedPredicate {.slot_index = slot->column_id(), + .path = std::move(reverse_path), + .comparison_type = comparison_type, + .literal_type = literal->second, + .literal = std::move(literal->first), + .op = *op}; +} + +bool has_variant_shredded_filter(const format::FileScanRequest& request) { + return std::ranges::any_of(request.conjuncts, [](const auto& conjunct) { + return extract_variant_shredded_predicate(conjunct).has_value(); + }); +} + +const ParquetColumnSchema* child_named(const ParquetColumnSchema& parent, std::string_view name) { + const auto it = std::ranges::find_if(parent.children, [&](const auto& child) { + return child != nullptr && child->name == name; + }); + return it == parent.children.end() ? nullptr : it->get(); +} + +struct ResolvedVariantShredding { + const ParquetColumnSchema* fallback_value = nullptr; + const ParquetColumnSchema* typed_value = nullptr; +}; + +bool metadata_cast_is_order_preserving(const DataTypePtr& source, const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return true; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + const auto source_primitive = source_type->get_primitive_type(); + const auto target_primitive = target_type->get_primitive_type(); + // Metadata bounds may cross only exact widening domains. This mirrors the residual CAST while + // excluding rounding, overflow, and narrowing cases that could reverse a pruning decision. + if (source_primitive == TYPE_FLOAT && target_primitive == TYPE_DOUBLE) { + return true; + } + if (is_int(source_primitive) && source_primitive != TYPE_LARGEINT && + is_decimalv3(target_primitive)) { + const uint32_t required_integer_digits = source_primitive == TYPE_TINYINT ? 3 + : source_primitive == TYPE_SMALLINT ? 5 + : source_primitive == TYPE_INT ? 10 + : 19; + return target_type->get_precision() >= target_type->get_scale() && + target_type->get_precision() - target_type->get_scale() >= required_integer_digits; + } + if (is_decimalv3(source_primitive) && is_decimalv3(target_primitive)) { + const uint32_t source_integer_digits = + source_type->get_precision() - source_type->get_scale(); + const uint32_t target_integer_digits = + target_type->get_precision() - target_type->get_scale(); + return target_integer_digits >= source_integer_digits && + target_type->get_scale() >= source_type->get_scale(); + } + return false; +} + +std::optional cast_metadata_field(const Field& value, const DataTypePtr& source, + const DataTypePtr& target) { + if (expr_zonemap::data_types_compatible(source, target)) { + return value; + } + const auto source_type = remove_nullable(source); + const auto target_type = remove_nullable(target); + if (source_type->get_primitive_type() == TYPE_FLOAT && + target_type->get_primitive_type() == TYPE_DOUBLE) { + return Field::create_field(static_cast(value.get())); + } + try { + auto source_column = source_type->create_column(); + source_column->insert(value); + DataTypeSerDe::FormatOptions options = DataTypeSerDe::get_default_format_options(); + options.converted_from_string = true; + std::string text = source_type->to_string(*source_column, 0, options); + StringRef input(text.data(), text.size()); + auto target_column = target_type->create_column(); + if (!target_type->get_serde() + ->from_string_strict_mode(input, *target_column, options) + .ok() || + target_column->size() != 1) { + return std::nullopt; + } + Field result; + target_column->get(0, result); + return result; + } catch (...) { + return std::nullopt; + } +} + +std::optional normalize_variant_statistics( + const VariantShreddedPredicate& predicate, const ParquetColumnSchema& typed_value, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_min_max || + expr_zonemap::data_types_compatible(typed_value.type, predicate.comparison_type)) { + return statistics; + } + auto min_value = + cast_metadata_field(statistics.min_value, typed_value.type, predicate.comparison_type); + auto max_value = + cast_metadata_field(statistics.max_value, typed_value.type, predicate.comparison_type); + if (!min_value.has_value() || !max_value.has_value()) { + return std::nullopt; + } + auto normalized = statistics; + normalized.min_value = std::move(*min_value); + normalized.max_value = std::move(*max_value); + return normalized; +} + +std::optional resolve_variant_shredding( + const std::vector>& file_schema, + const format::FileScanRequest& request, const VariantShreddedPredicate& predicate) { + const auto local_id = file_column_id_by_block_position(request, predicate.slot_index); + if (!local_id.has_value() || local_id->value() < 0 || + local_id->value() >= static_cast(file_schema.size())) { + return std::nullopt; + } + const ParquetColumnSchema* wrapper = file_schema[local_id->value()].get(); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::VARIANT) { + return std::nullopt; + } + for (const auto& component : predicate.path) { + const auto* typed_object = child_named(*wrapper, "typed_value"); + if (typed_object == nullptr || typed_object->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + wrapper = child_named(*typed_object, component); + if (wrapper == nullptr || wrapper->kind != ParquetColumnSchemaKind::STRUCT) { + return std::nullopt; + } + } + const auto* fallback = child_named(*wrapper, "value"); + const auto* typed = child_named(*wrapper, "typed_value"); + const auto typed_primitive = typed == nullptr || typed->type == nullptr + ? INVALID_TYPE + : remove_nullable(typed->type)->get_primitive_type(); + if (fallback == nullptr || typed == nullptr || + fallback->kind != ParquetColumnSchemaKind::PRIMITIVE || + typed->kind != ParquetColumnSchemaKind::PRIMITIVE || typed->max_repetition_level != 0 || + // Parquet float statistics do not prove that a page contains no NaN. Min/max pruning in + // the presence of NaN is not order preserving, so keep those pages until such proof exists. + typed_primitive == TYPE_FLOAT || typed_primitive == TYPE_DOUBLE || + !metadata_cast_is_order_preserving(typed->type, predicate.comparison_type) || + !expr_zonemap::data_types_compatible(predicate.comparison_type, predicate.literal_type)) { + return std::nullopt; + } + return ResolvedVariantShredding {.fallback_value = fallback, .typed_value = typed}; +} + +bool fallback_is_all_null(const tparquet::RowGroup& row_group, + const ParquetColumnSchema& fallback) { + if (fallback.max_repetition_level != 0 || fallback.leaf_column_id < 0 || + fallback.leaf_column_id >= static_cast(row_group.columns.size())) { + return false; + } + const auto& chunk = row_group.columns[fallback.leaf_column_id]; + return row_group.num_rows >= 0 && chunk.__isset.meta_data && + chunk.meta_data.num_values == row_group.num_rows && chunk.meta_data.__isset.statistics && + chunk.meta_data.statistics.__isset.null_count && + chunk.meta_data.statistics.null_count == chunk.meta_data.num_values; +} + +bool variant_statistics_exclude(const VariantShreddedPredicate& predicate, + const ParquetColumnStatistics& statistics) { + if (!statistics.has_any_statistics()) { + return false; + } + if (!statistics.has_not_null) { + return true; + } + if (!statistics.has_min_max) { + return false; + } + const auto& literal = predicate.literal; + switch (predicate.op) { + case VariantComparisonOp::EQ: + return literal < statistics.min_value || statistics.max_value < literal; + case VariantComparisonOp::NE: + return statistics.min_value == literal && statistics.max_value == literal; + case VariantComparisonOp::LT: + return statistics.min_value >= literal; + case VariantComparisonOp::LE: + return statistics.min_value > literal; + case VariantComparisonOp::GT: + return statistics.max_value <= literal; + case VariantComparisonOp::GE: + return statistics.max_value < literal; + } + __builtin_unreachable(); +} + bool has_expr_zonemap_filter(const format::FileScanRequest& request, const RuntimeState*) { // FileScannerV2 metadata pruning is a fixed part of its scan pipeline and must not inherit // the legacy scanner's expression ZoneMap session gate. @@ -467,7 +786,7 @@ bool has_expr_zonemap_filter(const format::FileScanRequest& request, const Runti return true; } } - return false; + return has_variant_shredded_filter(request); } std::set collect_expr_zonemap_slot_indexes(const VExprContextSPtrs& conjuncts) { @@ -621,9 +940,11 @@ void collect_filtered_leaf_ids(const ParquetColumnSchema& column_schema, if (!format::is_child_projected(projection, child_schema->local_id)) { continue; } - collect_filtered_leaf_ids(*child_schema, - format::find_child_projection(projection, child_schema->local_id), - leaf_column_ids); + // The leaf set must match the physical projection. A complete Variant projection naturally + // reaches every sibling; a validated typed-leaf projection reads only retained children. + const auto* child_projection = + format::find_child_projection(projection, child_schema->local_id); + collect_filtered_leaf_ids(*child_schema, child_projection, leaf_column_ids); } } @@ -631,7 +952,21 @@ bool native_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_sc DORIS_CHECK(column_schema.type != nullptr); // Raw VARBINARY file slots may feed table-side STRING casts. Footer/page metadata is still in // the pre-cast domain, so using it for a rewritten table predicate can cause false negatives. - return remove_nullable(column_schema.type)->get_primitive_type() != TYPE_VARBINARY; + if (remove_nullable(column_schema.type)->get_primitive_type() == TYPE_VARBINARY) { + return false; + } + // UUID readers render canonical text, so their physical 16-byte bounds are not STRING bounds. + return !column_schema.type_descriptor.is_uuid; +} + +bool variant_metadata_predicate_is_type_safe(const ParquetColumnSchema& column_schema) { + if (!native_metadata_predicate_is_type_safe(column_schema)) { + return false; + } + const auto& descriptor = column_schema.type_descriptor; + // An ordinary raw-binary STRING slot preserves its bytes, but Variant reconstruction renders + // the binary identity before the residual STRING cast and therefore changes the domain. + return !descriptor.is_string_like || descriptor.is_string_annotation; } bool check_native_statistics(const tparquet::FileMetaData& metadata, @@ -679,6 +1014,49 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, return result == ZoneMapFilterResult::kNoMatch; } +bool check_shredded_variant_statistics( + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, + const std::vector>& file_schema, + const format::FileScanRequest& request, const cctz::time_zone* timezone) { + for (const auto& conjunct : request.conjuncts) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + shredding->typed_value->leaf_column_id >= static_cast(row_group.columns.size()) || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto& chunk = row_group.columns[shredding->typed_value->leaf_column_id]; + if (!chunk.__isset.meta_data) { + continue; + } + const auto& column_metadata = chunk.meta_data; + if (column_metadata.num_values != row_group.num_rows) { + continue; + } + std::optional safe_statistics; + if (column_metadata.__isset.statistics) { + safe_statistics = detail::sanitize_native_footer_statistics( + shredding->typed_value->type_descriptor, column_metadata.statistics, true); + } + const auto statistics = ParquetStatisticsUtils::TransformColumnStatistics( + *shredding->typed_value, safe_statistics.has_value() ? &*safe_statistics : nullptr, + column_metadata.num_values, timezone); + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (normalized.has_value() && variant_statistics_exclude(*predicate, *normalized)) { + return true; + } + } + return false; +} + bool is_native_dictionary_data_encoding(tparquet::Encoding::type encoding) { return encoding == tparquet::Encoding::PLAIN_DICTIONARY || encoding == tparquet::Encoding::RLE_DICTIONARY; @@ -928,8 +1306,10 @@ Status select_row_groups_by_metadata( ParquetRowGroupPruneReason prune_reason = ParquetRowGroupPruneReason::NONE; if (probe_mode != ParquetMetadataProbeMode::EXPENSIVE_ONLY && has_expr_zonemap_filter(request, runtime_state) && - check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, - timezone)) { + (check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, + timezone) || + check_shredded_variant_statistics(metadata, row_group, file_schema, request, + timezone))) { prune_reason = ParquetRowGroupPruneReason::STATISTICS; } if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY && @@ -1034,11 +1414,16 @@ void collect_leaf_schemas(const ParquetColumnSchema& column_schema, return; } for (const auto& child_schema : column_schema.children) { - if (!format::is_child_projected(projection, child_schema->local_id)) { + if (column_schema.kind != ParquetColumnSchemaKind::VARIANT && + !format::is_child_projected(projection, child_schema->local_id)) { continue; } + // A logical Variant projection materializes every physical sibling; build skip plans for + // that identical leaf set so shredded columns cannot drift to different row positions. const auto* child_projection = - format::find_child_projection(projection, child_schema->local_id); + column_schema.kind == ParquetColumnSchemaKind::VARIANT + ? nullptr + : format::find_child_projection(projection, child_schema->local_id); collect_leaf_schemas(*child_schema, child_projection, leaf_schemas); } } @@ -1243,7 +1628,7 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t } // namespace Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, @@ -1329,6 +1714,59 @@ Status select_row_group_ranges_by_native_page_index( } } + for (const auto& conjunct : request.conjuncts) { + const auto predicate = extract_variant_shredded_predicate(conjunct); + if (!predicate.has_value()) { + continue; + } + const auto shredding = resolve_variant_shredding(file_schema, request, *predicate); + if (!shredding.has_value() || shredding->typed_value->leaf_column_id < 0 || + !fallback_is_all_null(row_group, *shredding->fallback_value) || + !variant_metadata_predicate_is_type_safe(*shredding->typed_value) || + !detail::has_supported_type_defined_order(metadata, + shredding->typed_value->leaf_column_id)) { + continue; + } + const auto index_it = page_indexes.find(shredding->typed_value->leaf_column_id); + if (index_it == page_indexes.end()) { + continue; + } + const auto& indexes = index_it->second; + std::vector filter_ranges; + bool usable = true; + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *shredding->typed_value, + page_idx, page_range.length, &statistics, timezone)) { + usable = false; + break; + } + const auto normalized = + normalize_variant_statistics(*predicate, *shredding->typed_value, statistics); + if (!normalized.has_value()) { + usable = false; + break; + } + if (!variant_statistics_exclude(*predicate, *normalized)) { + append_row_range(page_range, &filter_ranges); + } + } + if (!usable) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, filter_ranges); + if (selected_ranges->empty()) { + if (pruning_stats != nullptr) { + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + if (page_skip_plans != nullptr) { std::vector leaves; collect_request_leaf_schemas(file_schema, request, &leaves); diff --git a/be/src/format_v2/parquet/parquet_statistics.h b/be/src/format_v2/parquet/parquet_statistics.h index 72381548656f9d..611ec5abe97ac2 100644 --- a/be/src/format_v2/parquet/parquet_statistics.h +++ b/be/src/format_v2/parquet/parquet_statistics.h @@ -145,7 +145,7 @@ Status select_row_groups_by_metadata( ParquetMetadataProbeMode probe_mode = ParquetMetadataProbeMode::ALL); Status select_row_group_ranges_by_native_page_index( - const tparquet::FileMetaData& metadata, + const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::unordered_map& page_indexes, const std::vector>& file_schema, const format::FileScanRequest& request, int64_t row_group_rows, diff --git a/be/src/format_v2/parquet/parquet_type.h b/be/src/format_v2/parquet/parquet_type.h index 7ab3d2b3b39d8d..f06f4c25dd8ba2 100644 --- a/be/src/format_v2/parquet/parquet_type.h +++ b/be/src/format_v2/parquet/parquet_type.h @@ -67,6 +67,8 @@ struct ParquetTypeDescriptor { bool is_timestamp = false; // whether this is a timestamp type bool timestamp_is_adjusted_to_utc = false; // whether the timestamp is UTC-normalized bool is_string_like = false; // binary type that is neither decimal nor FLOAT16 + bool is_string_annotation = false; // STRING logical type, distinct from raw BINARY + bool is_uuid = false; // UUID logical type over FIXED_LEN_BYTE_ARRAY(16) std::string unsupported_reason; // non-empty when this Parquet logical type is unsupported }; diff --git a/be/src/format_v2/parquet/reader/count_column_reader.cpp b/be/src/format_v2/parquet/reader/count_column_reader.cpp index 171107107f1a38..8be684d7235957 100644 --- a/be/src/format_v2/parquet/reader/count_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/count_column_reader.cpp @@ -69,6 +69,17 @@ Status find_count_leaf(const ParquetColumnSchema& schema, // reading a potentially huge value BYTE_ARRAY for COUNT(map_col). DORIS_CHECK(!schema.children.empty()); return find_count_leaf(*schema.children.front(), nullptr, leaf); + case ParquetColumnSchemaKind::VARIANT: { + // The required metadata leaf is present exactly when the enclosing Variant group is + // present, so its levels preserve COUNT(variant_col) SQL-null semantics without decoding + // any Variant payload. + const auto metadata = std::ranges::find_if( + schema.children, [](const auto& child) { return child->name == "metadata"; }); + if (metadata == schema.children.end()) { + return Status::Corruption("Parquet Variant {} has no metadata column", schema.name); + } + return find_count_leaf(**metadata, nullptr, leaf); + } } return Status::InternalError("Unknown Parquet schema kind for column {}", schema.name); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index debc59155811b4..7352cc2d58b2bf 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -42,6 +42,7 @@ #include "format_v2/column_data.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_file_context.h" +#include "format_v2/parquet/reader/variant_column_reader.h" #include "runtime/runtime_state.h" namespace doris::format::parquet { @@ -50,25 +51,57 @@ namespace { constexpr size_t MAX_RETAINED_BATCH_SCRATCH_BYTES = 4UL << 20; DataTypePtr projected_type(const ParquetColumnSchema& schema, - const format::LocalColumnIndex* projection) { - if (!format::is_partial_projection(projection)) { - return schema.type; - } + const format::LocalColumnIndex* projection, bool physical_variant) { switch (schema.kind) { case ParquetColumnSchemaKind::PRIMITIVE: return schema.type; + case ParquetColumnSchemaKind::VARIANT: + DORIS_CHECK(schema.variant_physical_type != nullptr); + if (!physical_variant || !format::is_partial_projection(projection)) { + return physical_variant ? schema.variant_physical_type : schema.type; + } + { + DataTypes child_types; + Strings child_names; + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + DataTypePtr type = std::make_shared(std::move(child_types), + std::move(child_names)); + return schema.variant_physical_type->is_nullable() ? make_nullable(std::move(type)) + : std::move(type); + } case ParquetColumnSchemaKind::STRUCT: { DataTypes child_types; Strings child_names; - child_types.reserve(projection->children.size()); - child_names.reserve(projection->children.size()); - for (const auto& child_projection : projection->children) { - const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { - return child->local_id == child_projection.local_id(); - }); - DORIS_CHECK(child_it != schema.children.end()); - child_types.push_back(make_nullable(projected_type(**child_it, &child_projection))); - child_names.push_back((*child_it)->name); + if (format::is_partial_projection(projection)) { + child_types.reserve(projection->children.size()); + child_names.reserve(projection->children.size()); + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + child_types.push_back(make_nullable( + projected_type(**child_it, &child_projection, physical_variant))); + child_names.push_back((*child_it)->name); + } + } else { + child_types.reserve(schema.children.size()); + child_names.reserve(schema.children.size()); + for (const auto& child : schema.children) { + child_types.push_back( + make_nullable(projected_type(*child, nullptr, physical_variant))); + child_names.push_back(child->name); + } } DataTypePtr type = std::make_shared(child_types, child_names); return schema.type->is_nullable() ? make_nullable(type) : type; @@ -76,20 +109,25 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, case ParquetColumnSchemaKind::LIST: { DORIS_CHECK(schema.children.size() == 1); const auto* child_projection = - format::find_child_projection(projection, schema.children[0]->local_id); - DORIS_CHECK(child_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || child_projection != nullptr); DataTypePtr type = std::make_shared( - projected_type(*schema.children[0], child_projection)); + projected_type(*schema.children[0], child_projection, physical_variant)); return schema.type->is_nullable() ? make_nullable(type) : type; } case ParquetColumnSchemaKind::MAP: { DORIS_CHECK(schema.children.size() == 2); const auto* value_projection = - format::find_child_projection(projection, schema.children[1]->local_id); - DORIS_CHECK(value_projection != nullptr); + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + DORIS_CHECK(!format::is_partial_projection(projection) || value_projection != nullptr); DataTypePtr type = std::make_shared( - make_nullable(schema.children[0]->type), - make_nullable(projected_type(*schema.children[1], value_projection))); + make_nullable(projected_type(*schema.children[0], nullptr, physical_variant)), + make_nullable( + projected_type(*schema.children[1], value_projection, physical_variant))); return schema.type->is_nullable() ? make_nullable(type) : type; } } @@ -97,6 +135,64 @@ DataTypePtr projected_type(const ParquetColumnSchema& schema, return nullptr; } +std::unique_ptr build_variant_plan( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + auto plan = std::make_unique(); + plan->schema = &schema; + if (schema.kind == ParquetColumnSchemaKind::VARIANT) { + plan->contains_variant = true; + if (projection != nullptr) { + plan->variant_projection = *projection; + } + plan->variant_state_schema = create_variant_state_schema(schema, projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::PRIMITIVE) { + return plan; + } + + auto append_child = [&](const ParquetColumnSchema& child, + const format::LocalColumnIndex* child_projection) { + auto child_plan = build_variant_plan(child, child_projection); + plan->contains_variant = plan->contains_variant || child_plan->contains_variant; + plan->children.push_back(std::move(child_plan)); + }; + if (schema.kind == ParquetColumnSchemaKind::STRUCT && + format::is_partial_projection(projection)) { + for (const auto& child_projection : projection->children) { + const auto child_it = std::ranges::find_if(schema.children, [&](const auto& child) { + return child->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child_it != schema.children.end()); + append_child(**child_it, &child_projection); + } + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(schema.children.size() == 1); + const auto* child_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[0]->local_id) + : nullptr; + append_child(*schema.children[0], child_projection); + return plan; + } + if (schema.kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(schema.children.size() == 2); + append_child(*schema.children[0], nullptr); + const auto* value_projection = + format::is_partial_projection(projection) + ? format::find_child_projection(projection, schema.children[1]->local_id) + : nullptr; + append_child(*schema.children[1], value_projection); + return plan; + } + for (const auto& child : schema.children) { + append_child(*child, nullptr); + } + return plan; +} + const NativeFieldSchema* find_child_field(const NativeFieldSchema& parent, const ParquetColumnSchema& child) { auto field_it = std::ranges::find_if(parent.children, [&](const NativeFieldSchema& field) { @@ -108,7 +204,11 @@ const NativeFieldSchema* find_child_field(const NativeFieldSchema& parent, Status sync_native_field_types(const ParquetColumnSchema& schema, NativeFieldSchema* field) { DORIS_CHECK(field != nullptr); - field->data_type = schema.type; + // Variant's public type is logical; the native decoder must retain the physical shredded + // struct while timestamp semantics are copied into its descendants. + field->data_type = schema.kind == ParquetColumnSchemaKind::VARIANT + ? schema.variant_physical_type + : schema.type; for (const auto& child_schema : schema.children) { auto child_it = std::ranges::find_if(field->children, [&](const NativeFieldSchema& child) { return (child_schema->parquet_field_id >= 0 && @@ -167,10 +267,13 @@ void collect_projected_ids(const ParquetColumnSchema& schema, } // namespace -NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, - DataTypePtr projected_type, +NativeColumnReader::NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile) - : ParquetColumnReader(schema, std::move(projected_type), profile), + : ParquetColumnReader(schema, std::move(logical_type), profile), + _native_type(std::move(native_type)), + _variant_plan(std::move(variant_plan)), _nested(schema.kind != ParquetColumnSchemaKind::PRIMITIVE) {} NativeColumnReader::~NativeColumnReader() { @@ -212,19 +315,20 @@ Status NativeColumnReader::create( column_schema.local_id, metadata_field->name, column_schema.name); } - auto type = projected_type(column_schema, projection); - auto native_reader = std::unique_ptr( - new NativeColumnReader(column_schema, type, profile)); + auto logical_type = projected_type(column_schema, projection, false); + auto native_type = projected_type(column_schema, projection, true); + auto variant_plan = build_variant_plan(column_schema, projection); + auto native_reader = std::unique_ptr(new NativeColumnReader( + column_schema, std::move(logical_type), native_type, std::move(variant_plan), profile)); // Footer metadata is cached and shared across scans. Keep per-request timestamp semantics on a // reader-owned copy so mixed Paimon TIMESTAMP/TIMESTAMP_LTZ columns cannot contaminate it. native_reader->_native_field_schema = *metadata_field; RETURN_IF_ERROR(sync_native_field_types(column_schema, &native_reader->_native_field_schema)); auto* field = &native_reader->_native_field_schema; std::shared_ptr schema_node; - RETURN_IF_ERROR(build_native_schema_node(type, column_schema, &schema_node)); + RETURN_IF_ERROR(build_native_schema_node(native_type, column_schema, &schema_node)); std::set projected_ids; collect_projected_ids(column_schema, projection, *field, &projected_ids); - RETURN_IF_ERROR(native_reader->init( std::move(file), metadata, row_group_id, field, std::move(schema_node), std::move(projected_ids), selected_ranges, offset_indexes, timezone, @@ -289,7 +393,10 @@ Status NativeColumnReader::init( runtime_state != nullptr && runtime_state->enable_strict_mode(), int96_timezone_override)); DORIS_CHECK(_native_reader != nullptr); - _skip_column = _type->create_column(); + _skip_column = _native_type->create_column(); + if (_variant_plan->contains_variant) { + _variant_physical_column = _native_type->create_column(); + } return Status::OK(); } @@ -309,15 +416,22 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ native::FilterMap filter; RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); _native_reader->reset_filter_map_index(); - ColumnPtr native_column(std::move(column)); + const bool materialize_variant = + !dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type); + if (materialize_variant) { + _variant_physical_column->clear(); + } + ColumnPtr native_column = materialize_variant ? ColumnPtr(std::move(_variant_physical_column)) + : ColumnPtr(std::move(column)); bool eof = false; int64_t native_calls = 0; int64_t consecutive_empty_calls = 0; while (*rows_read < rows && !eof) { ++native_calls; size_t loop_rows = 0; + const DataTypePtr& decoder_type = materialize_variant ? _native_type : output_type; RETURN_IF_ERROR(_native_reader->read_column_data( - native_column, output_type, _schema_node, filter, + native_column, decoder_type, _schema_node, filter, static_cast(rows - *rows_read), &loop_rows, &eof, dictionary_ids)); if (loop_rows == 0 && !eof) { // A selected RowRanges plan may reject the current data page completely. V1 advances @@ -325,7 +439,11 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ // next page. Bound consecutive empty transitions by the Row Group row count to retain // a deterministic corruption exit if a decoder ever stops advancing. if (++consecutive_empty_calls > _row_group_rows + 1) { - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + } else { + column = IColumn::mutate(std::move(native_column)); + } return Status::Corruption("Native parquet reader made no progress for column {}", _name); } @@ -334,7 +452,20 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ consecutive_empty_calls = 0; *rows_read += static_cast(loop_rows); } - column = IColumn::mutate(std::move(native_column)); + if (materialize_variant) { + if (*rows_read != rows) { + _variant_physical_column = IColumn::mutate(std::move(native_column)); + return Status::Corruption("Native parquet reader returned {} rows, expected {} for {}", + *rows_read, rows, _name); + } + // The shredded state owns this decoded batch. Replace scanner scratch before handing the + // pointer off so typed path expressions can retain its physical leaves without a copy. + _variant_physical_column = _native_type->create_column(); + RETURN_IF_ERROR(materialize_variant_columns(*_variant_plan, std::move(native_column), + column, _profile)); + } else { + column = IColumn::mutate(std::move(native_column)); + } if (_profile.native_read_calls != nullptr) { COUNTER_UPDATE(_profile.native_read_calls, native_calls); } @@ -573,7 +704,7 @@ Status NativeColumnReader::skip(int64_t rows) { _filter_scratch.assign(static_cast(selected_rows), 0); int64_t rows_read = 0; RETURN_IF_ERROR(read_with_filter(selected_rows, _filter_scratch.data(), true, _skip_column, - _type, false, &rows_read)); + _native_type, false, &rows_read)); DORIS_CHECK(_skip_column->empty()); DORIS_CHECK(rows_read == selected_rows); _logical_row_position += rows_read; @@ -614,6 +745,11 @@ Status NativeColumnReader::select_with_dictionary_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); + if (_variant_plan->contains_variant) { + row_filter->clear(); + *used_filter = false; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); *used_filter = false; *survivor_count = 0; @@ -765,6 +901,13 @@ Status NativeColumnReader::select_with_fixed_width_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(used_filter != nullptr); DORIS_CHECK(execution_kind != nullptr); + if (_variant_plan->contains_variant) { + // Direct fixed-width evaluation cannot preserve a Variant physical subtree's row shape. + row_filter->clear(); + *used_filter = false; + *execution_kind = DirectPredicateExecutionKind::NONE; + return Status::OK(); + } RETURN_IF_ERROR(validate_selected_span(batch_rows)); const uint8_t* filter_data = nullptr; RETURN_IF_ERROR(selection.materialize_filter(selected_rows, batch_rows, &filter_data)); @@ -895,6 +1038,10 @@ bool NativeColumnReader::crossed_page_since_last_batch() { Result NativeColumnReader::dictionary_values() { DORIS_CHECK(_native_reader != nullptr); + if (_variant_plan->contains_variant) { + return ResultError( + Status::NotSupported("Parquet Variant columns do not expose dictionary values")); + } return _native_reader->dictionary_values(_type); } diff --git a/be/src/format_v2/parquet/reader/native_column_reader.h b/be/src/format_v2/parquet/reader/native_column_reader.h index e3ff5293ea62c0..91f63a79f02644 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.h +++ b/be/src/format_v2/parquet/reader/native_column_reader.h @@ -44,6 +44,7 @@ struct IOContext; namespace doris::format::parquet { class NativeParquetMetadata; +struct VariantMaterializationNode; namespace detail { inline constexpr int64_t MAX_NATIVE_LAZY_SKIP_ROWS = std::numeric_limits::max(); @@ -104,7 +105,9 @@ class NativeColumnReader final : public ParquetColumnReader { Result dictionary_values() override; private: - NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr projected_type, + NativeColumnReader(const ParquetColumnSchema& schema, DataTypePtr logical_type, + DataTypePtr native_type, + std::unique_ptr variant_plan, ParquetColumnReaderProfile profile); Status init(io::FileReaderSPtr file, const NativeParquetMetadata* metadata, int row_group_id, @@ -146,6 +149,9 @@ class NativeColumnReader final : public ParquetColumnReader { const std::unordered_map* _offset_indexes = nullptr; std::shared_ptr _schema_node; std::unique_ptr _native_reader; + DataTypePtr _native_type; + std::unique_ptr _variant_plan; + MutableColumnPtr _variant_physical_column; std::unique_ptr _page_cache_runtime_state; std::vector _selected_ranges; size_t _selected_range_idx = 0; diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp new file mode 100644 index 00000000000000..4827f951c94fe5 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -0,0 +1,1112 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common/exception.h" +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" +#include "core/column/column_nullable.h" +#include "core/column/column_struct.h" +#include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/column/variant_v2/column_variant_v2_typed_column.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_metadata.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +struct Cell { + const IColumn* column = nullptr; + bool is_null = false; +}; + +Cell cell_at(const IColumn& column, size_t row) { + if (row >= column.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant row {} exceeds column size {}", row, + column.size()); + } + if (const auto* nullable = check_and_get_column(column)) { + return {.column = &nullable->get_nested_column(), + .is_null = nullable->get_null_map_data()[row] != 0}; + } + return {.column = &column, .is_null = false}; +} + +const ParquetColumnSchema* find_child(const ParquetColumnSchema& schema, std::string_view name, + size_t* index) { + for (size_t i = 0; i < schema.children.size(); ++i) { + if (schema.children[i]->name == name) { + if (index != nullptr) { + *index = i; + } + return schema.children[i].get(); + } + } + return nullptr; +} + +Cell struct_child_at(const ParquetColumnSchema& schema, const IColumn& physical, size_t row, + std::string_view name, const ParquetColumnSchema** child_schema) { + const auto& structure = assert_cast(physical); + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure.tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} has no physical child {}", + schema.name, name); + } + if (child_schema != nullptr) { + *child_schema = child; + } + return cell_at(structure.get_column(index), row); +} + +uint8_t decimal_width(int precision) { + if (precision <= 0 || precision > 38) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant decimal precision {} is outside [1, 38]", precision); + } + return precision <= 9 ? 4 : (precision <= 18 ? 8 : 16); +} + +uint8_t integer_width(const ParquetColumnSchema& schema, PrimitiveType type) { + if (schema.type_descriptor.is_unsigned_integer) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Unsigned integers are not valid Parquet Variant typed values"); + } + if (schema.type_descriptor.integer_bit_width > 0) { + switch (schema.type_descriptor.integer_bit_width) { + case 8: + return 1; + case 16: + return 2; + case 32: + return 4; + case 64: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer width {}", + schema.type_descriptor.integer_bit_width); + } + } + switch (type) { + case TYPE_TINYINT: + return 1; + case TYPE_SMALLINT: + return 2; + case TYPE_INT: + return 4; + case TYPE_BIGINT: + return 8; + default: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant integer type {}", type); + } +} + +void append_typed_scalar(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantBatchBuilder::Row& builder) { + const PrimitiveType type = remove_nullable(schema.type)->get_primitive_type(); + switch (type) { + case TYPE_BOOLEAN: + builder.add_bool(assert_cast(column).get_data()[row] != 0); + return; + case TYPE_TINYINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_SMALLINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_INT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_BIGINT: + builder.add_scalar( + VariantScalarRef::integer(assert_cast(column).get_data()[row], + integer_width(schema, type))); + return; + case TYPE_FLOAT: + builder.add_float(assert_cast(column).get_data()[row]); + return; + case TYPE_DOUBLE: + builder.add_double(assert_cast(column).get_data()[row]); + return; + case TYPE_DECIMAL128I: { + const auto value = assert_cast(column).get_data()[row].value; + builder.add_decimal(value, static_cast(schema.type_descriptor.decimal_scale), + decimal_width(schema.type_descriptor.decimal_precision)); + return; + } + case TYPE_TIMEV2: { + const double seconds = assert_cast(column).get_data()[row]; + if (!std::isfinite(seconds) || + std::abs(seconds) > static_cast(std::numeric_limits::max()) / 1e6) { + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant TIME value"); + } + builder.add_time_ntz_micros(static_cast(std::llround(seconds * 1e6))); + return; + } + case TYPE_DATETIMEV2: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + // Native DATETIMEV2 is microsecond based. Reject before returning a silently truncated + // value; a raw INT64 nanos decoder can be added independently. + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), + schema.type_descriptor.timestamp_is_adjusted_to_utc); + return; + } + case TYPE_TIMESTAMPTZ: { + if (schema.type_descriptor.time_unit == ParquetTimeUnit::NANOS) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant TIMESTAMP(NANOS) is not supported"); + } + const auto& value = assert_cast(column).get_data()[row]; + builder.add_timestamp_micros( + variant_timestamp_micros(value, row, "Parquet Variant TIMESTAMP"), true); + return; + } + case TYPE_VARBINARY: { + const StringRef value = column.get_data_at(row); + if (!schema.type_descriptor.is_uuid) { + builder.add_binary(value); + return; + } + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + return; + } + case TYPE_STRING: { + const StringRef value = column.get_data_at(row); + if (schema.type_descriptor.is_uuid) { + if (value.size != 16) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant UUID has {} bytes instead of 16", value.size); + } + std::array uuid {}; + std::memcpy(uuid.data(), value.data, uuid.size()); + builder.add_uuid(uuid); + } else if (schema.type_descriptor.is_string_annotation) { + builder.add_string(value); + } else { + builder.add_binary(value); + } + return; + } + default: + if (!is_supported_variant_typed_identity(type)) { + throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, + "Parquet Variant typed value {} is not supported", + remove_nullable(schema.type)->get_name()); + } + dispatch_variant_typed_column( + column, type, [&](const auto& typed_column) { + with_variant_typed_scalar( + typed_column, row, + static_cast(remove_nullable(schema.type)->get_scale()), + [&](const VariantScalarRef& scalar) { builder.add_scalar(scalar); }); + }); + } +} + +enum class WrapperContext { ROOT, ARRAY_ELEMENT, OBJECT_FIELD }; + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context); + +void append_typed_value(const ParquetColumnSchema& schema, const IColumn& column, size_t row, + VariantMetadataRef metadata, const VariantRef* residual, + VariantBatchBuilder::Row& builder) { + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant scalar typed_value cannot have residual value bytes"); + } + append_typed_scalar(schema, column, row, builder); + return; + case ParquetColumnSchemaKind::STRUCT: { + if (static_cast(residual) && residual->basic_type() != VariantBasicType::OBJECT) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object typed_value has non-object residual value"); + } + const auto& structure = assert_cast(column); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant object {} physical field count mismatch", schema.name); + } + auto object = builder.start_object(); + if (static_cast(residual)) { + for (uint32_t i = 0; i < residual->num_elements(); ++i) { + uint32_t field_id = 0; + const VariantRef child = residual->object_value_at(i, &field_id); + object.add_key(residual->metadata.key_at(field_id)); + builder.add_value(child); + } + } + for (size_t i = 0; i < schema.children.size(); ++i) { + const auto& child_schema = *schema.children[i]; + const Cell child = cell_at(structure.get_column(i), row); + if (child.is_null) { + // Shredded object fields are optional wrapper groups. A missing group means the + // key is absent, which differs from a present wrapper encoding a Variant null. + continue; + } + // A null/null wrapper means this object field is absent. Delay add_key until its + // presence is known so absent shredded fields do not turn into Variant nulls. + size_t value_index = 0; + const auto* value_schema = find_child(child_schema, "value", &value_index); + const auto& child_struct = assert_cast(*child.column); + const bool value_present = value_schema != nullptr && + !cell_at(child_struct.get_column(value_index), row).is_null; + size_t typed_index = 0; + const auto* typed_schema = find_child(child_schema, "typed_value", &typed_index); + const bool typed_present = typed_schema != nullptr && + !cell_at(child_struct.get_column(typed_index), row).is_null; + if (!value_present && !typed_present) { + continue; + } + object.add_key(StringRef(child_schema.name)); + (void)append_wrapper(child_schema, *child.column, row, metadata, builder, + WrapperContext::OBJECT_FIELD); + } + object.finish(); + return; + } + case ParquetColumnSchemaKind::LIST: { + if (static_cast(residual)) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array typed_value cannot have residual value bytes"); + } + if (schema.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant array {} has invalid element schema", schema.name); + } + const auto& array = assert_cast(column); + const size_t begin = array.offset_at(static_cast(row)); + const size_t end = array.get_offsets()[row]; + auto scope = builder.start_array(); + for (size_t element = begin; element < end; ++element) { + const Cell cell = cell_at(array.get_data(), element); + if (cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant shredded array element wrapper is null"); + } + (void)append_wrapper(*schema.children[0], *cell.column, element, metadata, builder, + WrapperContext::ARRAY_ELEMENT); + } + scope.finish(); + return; + } + case ParquetColumnSchemaKind::MAP: + case ParquetColumnSchemaKind::VARIANT: + throw Exception(ErrorCode::CORRUPTION, "Invalid Parquet Variant typed_value schema {}", + schema.name); + } +} + +bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, size_t row, + VariantMetadataRef metadata, VariantBatchBuilder::Row& builder, + WrapperContext context) { + Cell value; + if (find_child(schema, "value", nullptr) != nullptr) { + value = struct_child_at(schema, wrapper, row, "value", nullptr); + } else { + value.is_null = true; + } + const ParquetColumnSchema* typed_schema = nullptr; + Cell typed; + if (find_child(schema, "typed_value", nullptr) != nullptr) { + typed = struct_child_at(schema, wrapper, row, "typed_value", &typed_schema); + } else { + typed.is_null = true; + } + + if (find_child(schema, "value", nullptr) == nullptr && typed_schema == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant wrapper {} has neither value nor typed_value", + schema.name); + } + if (value.is_null && typed.is_null) { + if (context == WrapperContext::OBJECT_FIELD) { + return false; + } + if (context == WrapperContext::ARRAY_ELEMENT) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant array element is missing"); + } + builder.add_null(); + return true; + } + + VariantRef residual {.metadata = metadata, .value = {}}; + if (!value.is_null) { + residual.value = value.column->get_data_at(row); + } + if (typed.is_null) { + builder.add_value(residual); + return true; + } + append_typed_value(*typed_schema, *typed.column, row, metadata, + value.is_null ? nullptr : &residual, builder); + return true; +} + +void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrapper, + const ColumnNullable* outer_nullable, size_t begin, size_t end, + ColumnVariantV2& variants) { + try { + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = end - begin}); + for (size_t row = begin; row < end; ++row) { + auto output_row = builder.begin_row(); + if (outer_nullable != nullptr && outer_nullable->get_null_map_data()[row] != 0) { + output_row.add_null(); + output_row.finish(); + continue; + } + const Cell metadata_cell = struct_child_at(schema, wrapper, row, "metadata", nullptr); + if (metadata_cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} has null metadata at row {}", schema.name, row); + } + const StringRef metadata_bytes = metadata_cell.column->get_data_at(row); + const VariantMetadataRef metadata {metadata_bytes.data, metadata_bytes.size}; + metadata.validate(); + (void)append_wrapper(schema, wrapper, row, metadata, output_row, WrapperContext::ROOT); + output_row.finish(); + } + VariantBatchBuilder batch = builder.finish_batch(); + variants.insert_encoded_batch(batch); + } catch (...) { + if (end - begin <= 1) { + throw; + } + // A single builder has one metadata dictionary. If heterogeneous file rows cannot fit in + // that dictionary, split without changing the destination column's already-valid batches. + // Corrupt input still reaches a one-row range and propagates its original exception. + const size_t middle = begin + (end - begin) / 2; + encode_variant_range(schema, wrapper, outer_nullable, begin, middle, variants); + encode_variant_range(schema, wrapper, outer_nullable, middle, end, variants); + } +} + +ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& schema, + const IColumn& physical) { + if (schema.kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema.name); + } + const auto* outer_nullable = check_and_get_column(physical); + const IColumn& wrapper = + outer_nullable == nullptr ? physical : outer_nullable->get_nested_column(); + const auto& structure = assert_cast(wrapper); + if (structure.tuple_size() != schema.children.size()) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} physical field count mismatch", + schema.name); + } + + auto variants = ColumnVariantV2::create(); + constexpr size_t MAX_RECONSTRUCTION_BATCH_ROWS = 4096; + for (size_t begin = 0; begin < physical.size(); begin += MAX_RECONSTRUCTION_BATCH_ROWS) { + encode_variant_range(schema, wrapper, outer_nullable, begin, + std::min(physical.size(), begin + MAX_RECONSTRUCTION_BATCH_ROWS), + *variants); + } + return variants; +} + +std::unique_ptr clone_schema( + const ParquetColumnSchema& source, const format::LocalColumnIndex* projection = nullptr) { + auto result = std::make_unique(); + result->local_id = source.local_id; + result->parquet_field_id = source.parquet_field_id; + result->name = source.name; + result->type = source.type; + result->variant_physical_type = source.variant_physical_type; + result->leaf_column_id = source.leaf_column_id; + result->type_descriptor = source.type_descriptor; + result->kind = source.kind; + result->max_definition_level = source.max_definition_level; + result->max_repetition_level = source.max_repetition_level; + result->nullable_definition_level = source.nullable_definition_level; + result->definition_level = source.definition_level; + result->repetition_level = source.repetition_level; + result->repeated_ancestor_definition_level = source.repeated_ancestor_definition_level; + result->repeated_repetition_level = source.repeated_repetition_level; + const bool partial = format::is_partial_projection(projection); + result->children.reserve(partial ? projection->children.size() : source.children.size()); + if (partial) { + // NativeColumnReader emits a partial STRUCT in projection order, so the retained schema + // must use that same order or field names will address the wrong physical tuple element. + for (const auto& child_projection : projection->children) { + const auto child = std::ranges::find_if(source.children, [&](const auto& candidate) { + return candidate->local_id == child_projection.local_id(); + }); + DORIS_CHECK(child != source.children.end()); + result->children.push_back(clone_schema(**child, &child_projection)); + } + } else { + for (const auto& child : source.children) { + result->children.push_back(clone_schema(*child)); + } + } + return result; +} + +ColumnPtr unwrap_nullable(ColumnPtr column) { + if (const auto* nullable = check_and_get_column(*column)) { + return nullable->get_nested_column_ptr(); + } + return column; +} + +ColumnPtr struct_child(const ParquetColumnSchema& schema, ColumnPtr column, std::string_view name, + const ParquetColumnSchema** child_schema) { + column = unwrap_nullable(std::move(column)); + const auto* structure = check_and_get_column(*column); + if (structure == nullptr) { + return nullptr; + } + size_t index = 0; + const auto* child = find_child(schema, name, &index); + if (child == nullptr || index >= structure->tuple_size()) { + return nullptr; + } + if (child_schema != nullptr) { + *child_schema = child; + } + return structure->get_column_ptr(index); +} + +bool has_present_value(const ColumnPtr& column) { + if (const auto* nullable = check_and_get_column(*column)) { + return std::ranges::any_of(nullable->get_null_map_data(), + [](uint8_t is_null) { return is_null == 0; }); + } + return !column->empty(); +} + +bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { + if (schema.type == nullptr || schema.kind != ParquetColumnSchemaKind::PRIMITIVE) { + return false; + } + // ColumnVariantV2 typed state carries only a Doris type. Binary/UUID annotations, temporal + // units, and other Parquet-only identity must therefore reconstruct canonical Variant bytes. + switch (remove_nullable(schema.type)->get_primitive_type()) { + case TYPE_BOOLEAN: + case TYPE_TINYINT: + case TYPE_SMALLINT: + case TYPE_INT: + case TYPE_BIGINT: + case TYPE_FLOAT: + case TYPE_DOUBLE: + case TYPE_DECIMAL128I: + case TYPE_DATEV2: + return true; + default: + return false; + } +} + +bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { + return (!left && !right) || (left && right && left->equals(*right)); +} + +bool same_type_descriptor(const ParquetTypeDescriptor& left, const ParquetTypeDescriptor& right) { + return same_data_type(left.doris_type, right.doris_type) && + same_data_type(left.physical_doris_type, right.physical_doris_type) && + left.extra_type_info == right.extra_type_info && left.time_unit == right.time_unit && + left.physical_type == right.physical_type && + left.integer_bit_width == right.integer_bit_width && + left.decimal_precision == right.decimal_precision && + left.decimal_scale == right.decimal_scale && left.fixed_length == right.fixed_length && + left.is_unsigned_integer == right.is_unsigned_integer && + left.is_decimal == right.is_decimal && left.is_timestamp == right.is_timestamp && + left.timestamp_is_adjusted_to_utc == right.timestamp_is_adjusted_to_utc && + left.is_string_like == right.is_string_like && + left.is_string_annotation == right.is_string_annotation && + left.is_uuid == right.is_uuid && left.unsupported_reason == right.unsupported_reason; +} + +bool same_shredded_schema(const ParquetColumnSchema& left, const ParquetColumnSchema& right) { + if (left.name != right.name || left.kind != right.kind || + !same_data_type(left.type, right.type) || + !same_type_descriptor(left.type_descriptor, right.type_descriptor) || + left.children.size() != right.children.size()) { + return false; + } + for (size_t i = 0; i < left.children.size(); ++i) { + if (!same_shredded_schema(*left.children[i], *right.children[i])) { + return false; + } + } + return true; +} + +void append_compatible_column(IColumn& output, const IColumn& converted); +void validate_compatible_column(const IColumn& output, const IColumn& converted); + +class ParquetVariantShreddedState final : public VariantShreddedState { +public: + ParquetVariantShreddedState(std::shared_ptr schema, + ColumnPtr physical, bool complete, + ParquetColumnReaderProfile profile = {}) + : _schema(std::move(schema)), + _physical(std::move(physical)), + _complete(complete), + _profile(profile) { + DORIS_CHECK(_schema != nullptr && static_cast(_physical)); + const ColumnPtr wrapper = unwrap_nullable(_physical); + const auto* structure = check_and_get_column(*wrapper); + if (structure == nullptr || structure->tuple_size() != _schema->children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} physical field count mismatch", _schema->name); + } + } + + size_t size() const override { return _physical->size(); } + size_t byte_size() const override { + std::lock_guard lock(_materialization_lock); + return _physical->byte_size() + (_materialized ? _materialized->byte_size() : 0); + } + size_t allocated_bytes() const override { + std::lock_guard lock(_materialization_lock); + return _physical->allocated_bytes() + + (_materialized ? _materialized->allocated_bytes() : 0); + } + void sanity_check() const override { _physical->sanity_check(); } + + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { + callback(*_physical); + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t result_size_hint) const override { + // Compact the decoded physical tree directly. In particular, a leaf-only projection has + // no metadata/value columns from which a canonical Variant could be reconstructed. + // The projection schema is immutable and reader-scoped, so derived selections share it + // instead of cloning the whole shredded tree for every filter operation. + return std::make_shared( + _schema, _physical->filter(filter, result_size_hint), _complete, _profile); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + return std::make_shared(_schema, _physical->cut(start, length), + _complete, _profile); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + MutableColumnPtr selected = _physical->clone_empty(); + selected->insert_indices_from(*_physical, indices_begin, indices_end); + return std::make_shared(_schema, std::move(selected), + _complete, _profile); + } + + bool try_append(const VariantShreddedState& source) override { + const auto* parquet_source = dynamic_cast(&source); + if (parquet_source == nullptr || _complete != parquet_source->_complete || + !same_shredded_schema(*_schema, *parquet_source->_schema)) { + return false; + } + validate_compatible_column(*_physical, *parquet_source->_physical); + auto mutable_physical = IColumn::mutate(std::move(_physical)); + append_compatible_column(*mutable_physical, *parquet_source->_physical); + _physical = std::move(mutable_physical); + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + return true; + } + + std::optional find_typed_value( + std::span path) const override { + auto path_miss = [&]() -> std::optional { + update_counter(_profile.variant_direct_leaf_path_misses, 1); + return std::nullopt; + }; + if (path.empty()) { + return path_miss(); + } + + const ParquetColumnSchema* typed_schema = nullptr; + ColumnPtr typed = struct_child(*_schema, _physical, "typed_value", &typed_schema); + if (!typed || typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + + for (size_t position = 0; position < path.size(); ++position) { + if (path[position].kind != VariantShreddedPathSegment::Kind::OBJECT_KEY) { + return path_miss(); + } + + const std::string_view key(path[position].key.data, path[position].key.size); + const ParquetColumnSchema* wrapper_schema = nullptr; + ColumnPtr wrapper = struct_child(*typed_schema, typed, key, &wrapper_schema); + if (!wrapper) { + return path_miss(); + } + + if (ColumnPtr residual = struct_child(*wrapper_schema, wrapper, "value", nullptr); + static_cast(residual) && has_present_value(residual)) { + // A residual value can contribute data to the same logical object. Reconstructing + // is required in that case; returning only the typed leaf would drop information. + update_counter(_profile.variant_direct_leaf_residual_fallbacks, 1); + return std::nullopt; + } + + typed = struct_child(*wrapper_schema, wrapper, "typed_value", &typed_schema); + if (!typed) { + return path_miss(); + } + if (position + 1 == path.size()) { + if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || + check_and_get_column(*typed) == nullptr || + !supports_direct_typed_variant_state(*typed_schema)) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue {.column = std::move(typed), + .type = remove_nullable(typed_schema->type)}; + } + if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return path_miss(); + } + } + return std::nullopt; + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_complete) { + throw Exception( + ErrorCode::INTERNAL_ERROR, + "A projected Parquet Variant can only serve its validated shredded leaves"); + } + if (!_materialized) { + SCOPED_TIMER(_profile.variant_reconstruction_time); + _materialized = encode_variant_column(*_schema, *_physical); + update_counter(_profile.variant_reconstructed_rows, + static_cast(_physical->size())); + } + return *_materialized; + } + +private: + static void update_counter(RuntimeProfile::Counter* counter, int64_t value) { + if (counter != nullptr) { + COUNTER_UPDATE(counter, value); + } + } + + std::shared_ptr _schema; + ColumnPtr _physical; + bool _complete = true; + ParquetColumnReaderProfile _profile; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; +}; + +MutableColumnPtr build_variant_column(std::shared_ptr schema, + ColumnPtr physical, bool complete, + const ParquetColumnReaderProfile& profile) { + DORIS_CHECK(schema != nullptr); + if (schema->kind != ParquetColumnSchemaKind::VARIANT) { + throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", + schema->name); + } + + const auto* outer_nullable = check_and_get_column(*physical); + MutableColumnPtr variants = + ColumnVariantV2::create_shredded(std::make_shared( + std::move(schema), physical, complete, profile)); + if (outer_nullable == nullptr) { + return variants; + } + auto nulls = outer_nullable->get_null_map_column().clone_resized(physical->size()); + return ColumnNullable::create(std::move(variants), std::move(nulls)); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile); + +ColumnPtr transform_non_nullable(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + const auto& schema = *plan.schema; + switch (schema.kind) { + case ParquetColumnSchemaKind::PRIMITIVE: + return physical; + case ParquetColumnSchemaKind::VARIANT: + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema(schema, plan.variant_projection + ? &*plan.variant_projection + : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + case ParquetColumnSchemaKind::STRUCT: { + const auto& structure = assert_cast(*physical); + if (structure.tuple_size() != plan.children.size()) { + throw Exception(ErrorCode::CORRUPTION, + "Projected Parquet STRUCT {} field count mismatch", schema.name); + } + Columns fields; + fields.reserve(plan.children.size()); + for (size_t i = 0; i < plan.children.size(); ++i) { + fields.push_back( + transform_node(*plan.children[i], structure.get_column_ptr(i), profile)); + } + return ColumnStruct::create(std::move(fields)); + } + case ParquetColumnSchemaKind::LIST: { + const auto& array = assert_cast(*physical); + if (plan.children.size() != 1) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet ARRAY plan is invalid"); + } + auto values = transform_node(*plan.children[0], array.get_data_ptr(), profile); + return ColumnArray::create(std::move(values), array.get_offsets_ptr()); + } + case ParquetColumnSchemaKind::MAP: { + const auto& map = assert_cast(*physical); + if (plan.children.size() != 2) { + throw Exception(ErrorCode::CORRUPTION, "Projected Parquet MAP plan is invalid"); + } + auto keys = transform_node(*plan.children[0], map.get_keys_ptr(), profile); + auto values = transform_node(*plan.children[1], map.get_values_ptr(), profile); + return ColumnMap::create(std::move(keys), std::move(values), map.get_offsets_ptr()); + } + } + throw Exception(ErrorCode::INTERNAL_ERROR, "Unknown Parquet schema kind"); +} + +ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physical, + const ParquetColumnReaderProfile& profile) { + if (plan.schema == nullptr) { + throw Exception(ErrorCode::INTERNAL_ERROR, "Parquet Variant materialization plan is null"); + } + if (plan.schema->kind == ParquetColumnSchemaKind::VARIANT) { + return build_variant_column( + plan.variant_state_schema + ? plan.variant_state_schema + : create_variant_state_schema( + *plan.schema, + plan.variant_projection ? &*plan.variant_projection : nullptr), + std::move(physical), + !format::is_partial_projection(plan.variant_projection ? &*plan.variant_projection + : nullptr), + profile); + } + if (const auto* nullable = check_and_get_column(*physical)) { + auto nested = transform_non_nullable(plan, nullable->get_nested_column_ptr(), profile); + return ColumnNullable::create(std::move(nested), nullable->get_null_map_column_ptr()); + } + return transform_non_nullable(plan, std::move(physical), profile); +} + +void append_compatible_column(IColumn& output, const IColumn& converted) { + if (auto* output_nullable = check_and_get_column(output)) { + if (const auto* converted_nullable = check_and_get_column(converted)) { + append_compatible_column(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + output_nullable->get_null_map_column().insert_range_from( + converted_nullable->get_null_map_column(), 0, converted.size()); + } else { + append_compatible_column(output_nullable->get_nested_column(), converted); + // External slots and nested Iceberg fields may remain nullable even when one file's + // physical node is required. Preserve that destination invariant with non-null bits. + output_nullable->push_false_to_nullmap(converted.size()); + } + return; + } + + if (const auto* converted_nullable = check_and_get_column(converted)) { + // Parquet writers may encode an Iceberg required field as optional. It can populate a + // non-nullable destination only when this batch proves that every value is present. + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + append_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + + if (auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + append_compatible_column(output_struct->get_column(i), converted_struct->get_column(i)); + } + return; + } + + if (auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + const size_t element_base = output_array->get_data().size(); + append_compatible_column(output_array->get_data(), converted_array->get_data()); + auto& output_offsets = output_array->get_offsets(); + output_offsets.reserve(output_offsets.size() + converted_array->size()); + for (const auto offset : converted_array->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + return; + } + + if (auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + const size_t element_base = output_map->get_keys().size(); + append_compatible_column(output_map->get_keys(), converted_map->get_keys()); + append_compatible_column(output_map->get_values(), converted_map->get_values()); + auto& output_offsets = output_map->get_offsets(); + output_offsets.reserve(output_offsets.size() + converted_map->size()); + for (const auto offset : converted_map->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + return; + } + + if (auto* output_variant = check_and_get_column(output)) { + const auto* converted_variant = check_and_get_column(converted); + if (converted_variant == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + output_variant->insert_range_from(*converted_variant, 0, converted_variant->size()); + return; + } + + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } + output.insert_range_from(converted, 0, converted.size()); +} + +void validate_compatible_column(const IColumn& output, const IColumn& converted) { + if (const auto* output_nullable = check_and_get_column(output)) { + if (const auto* converted_nullable = check_and_get_column(converted)) { + validate_compatible_column(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } else { + validate_compatible_column(output_nullable->get_nested_column(), converted); + } + return; + } + if (const auto* converted_nullable = check_and_get_column(converted)) { + if (converted_nullable->has_null()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced null data for a " + "non-nullable destination"); + } + validate_compatible_column(output, converted_nullable->get_nested_column()); + return; + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible STRUCT"); + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + validate_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + return; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + if (converted_array == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible ARRAY"); + } + validate_compatible_column(output_array->get_data(), converted_array->get_data()); + return; + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + if (converted_map == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible MAP"); + } + validate_compatible_column(output_map->get_keys(), converted_map->get_keys()); + validate_compatible_column(output_map->get_values(), converted_map->get_values()); + return; + } + if (check_and_get_column(output) != nullptr) { + if (check_and_get_column(converted) == nullptr) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced an incompatible column"); + } + return; + } + if (output.get_name() != converted.get_name()) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant materialization produced {} for {} destination", + converted.get_name(), output.get_name()); + } +} + +bool has_exact_column_shape(const IColumn& output, const IColumn& converted) { + const auto* output_nullable = check_and_get_column(output); + const auto* converted_nullable = check_and_get_column(converted); + if (output_nullable != nullptr || converted_nullable != nullptr) { + return output_nullable != nullptr && converted_nullable != nullptr && + has_exact_column_shape(output_nullable->get_nested_column(), + converted_nullable->get_nested_column()); + } + if (const auto* output_struct = check_and_get_column(output)) { + const auto* converted_struct = check_and_get_column(converted); + if (converted_struct == nullptr || + output_struct->tuple_size() != converted_struct->tuple_size()) { + return false; + } + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + if (!has_exact_column_shape(output_struct->get_column(i), + converted_struct->get_column(i))) { + return false; + } + } + return true; + } + if (const auto* output_array = check_and_get_column(output)) { + const auto* converted_array = check_and_get_column(converted); + return converted_array != nullptr && + has_exact_column_shape(output_array->get_data(), converted_array->get_data()); + } + if (const auto* output_map = check_and_get_column(output)) { + const auto* converted_map = check_and_get_column(converted); + return converted_map != nullptr && + has_exact_column_shape(output_map->get_keys(), converted_map->get_keys()) && + has_exact_column_shape(output_map->get_values(), converted_map->get_values()); + } + if (check_and_get_column(output) != nullptr) { + return check_and_get_column(converted) != nullptr; + } + return output.get_name() == converted.get_name(); +} + +void append_materialized_column(MutableColumnPtr& output, ColumnPtr converted) { + // Validate the complete destination shape before mutation. This preserves atomic failures + // without copying a full scratch batch, while an empty exact-shape output can adopt the tree. + validate_compatible_column(*output, *converted); + if (output->empty() && has_exact_column_shape(*output, *converted)) { + if (converted->is_exclusive()) { + // The transformed tree consumed the decoder tree and is recursively exclusive. Keep + // primitive siblings and their buffers intact instead of recursively COW-cloning them. + output = converted->assert_mutable(); + return; + } + output = IColumn::mutate(std::move(converted)); + return; + } + append_compatible_column(*output, *converted); +} + +} // namespace + +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection) { + return std::shared_ptr(clone_schema(schema, projection)); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_rows(schema, physical.get_ptr(), output, profile); +} + +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = build_variant_column(create_variant_state_schema(schema), + std::move(physical), true, profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + return materialize_variant_columns(plan, physical.get_ptr(), output, profile); +} + +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile) { + if (!output) { + return Status::InvalidArgument("Parquet Variant output column is null"); + } + RETURN_IF_CATCH_EXCEPTION({ + auto converted = transform_node(plan, std::move(physical), profile); + append_materialized_column(output, std::move(converted)); + }); + return Status::OK(); +} + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.h b/be/src/format_v2/parquet/reader/variant_column_reader.h new file mode 100644 index 00000000000000..53cc53983181c8 --- /dev/null +++ b/be/src/format_v2/parquet/reader/variant_column_reader.h @@ -0,0 +1,67 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include "common/status.h" +#include "core/column/column.h" +#include "format_v2/column_data.h" +#include "format_v2/parquet/parquet_profile.h" + +namespace doris::format::parquet { + +struct ParquetColumnSchema; + +// Projection-aligned view of the file schema. Complex nodes contain only the children decoded by +// NativeColumnReader. A VARIANT node may own a validated fully-shredded physical leaf projection. +struct VariantMaterializationNode { + const ParquetColumnSchema* schema = nullptr; + std::vector> children; + bool contains_variant = false; + std::optional variant_projection; + std::shared_ptr variant_state_schema; +}; + +// Builds the immutable schema retained by a shredded state in the exact order of the decoded +// physical projection. +std::shared_ptr create_variant_state_schema( + const ParquetColumnSchema& schema, const format::LocalColumnIndex* projection = nullptr); + +// Converts one physical Parquet Variant wrapper column to ColumnVariantV2 and appends it to output. +// SQL NULL is represented by the wrapper's outer null map; a present wrapper with neither value nor +// typed_value is the Variant null value. +Status materialize_variant_rows(const ParquetColumnSchema& schema, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_rows(const ParquetColumnSchema& schema, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +// Recursively replaces projected VARIANT nodes inside STRUCT/LIST/MAP columns while preserving the +// surrounding column shape, offsets, and null maps. The destination is unchanged on decode errors. +Status materialize_variant_columns(const VariantMaterializationNode& plan, const IColumn& physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); +Status materialize_variant_columns(const VariantMaterializationNode& plan, ColumnPtr physical, + MutableColumnPtr& output, + const ParquetColumnReaderProfile& profile = {}); + +} // namespace doris::format::parquet diff --git a/be/src/format_v2/schema_projection.cpp b/be/src/format_v2/schema_projection.cpp index 342f4c91898c92..d7b2a359131ab7 100644 --- a/be/src/format_v2/schema_projection.cpp +++ b/be/src/format_v2/schema_projection.cpp @@ -88,6 +88,11 @@ Status rebuild_semantic_projected_type(const DataTypePtr& original_type, nested_projected_type = std::make_shared(key_type, value_type); break; } + case TYPE_VARIANT: + // Variant children describe a format-specific physical shredding carrier, not the public + // logical type. Pruning those children must keep the file block exposed as Variant. + *projected_type = original_type; + return Status::OK(); default: return Status::InvalidArgument("Cannot project children from non-complex type {}", original_type->get_name()); diff --git a/be/src/format_v2/table/iceberg_reader.cpp b/be/src/format_v2/table/iceberg_reader.cpp index 097e6bab111091..1fc36a57c1ab69 100644 --- a/be/src/format_v2/table/iceberg_reader.cpp +++ b/be/src/format_v2/table/iceberg_reader.cpp @@ -63,6 +63,94 @@ namespace doris::format::iceberg { static constexpr const char* ROW_LINEAGE_ROW_ID = "_row_id"; static constexpr int32_t ROW_LINEAGE_ROW_ID_FIELD_ID = 2147483540; +namespace { + +bool contains_variant_type(const DataTypePtr& input) { + if (input == nullptr) { + return false; + } + const auto type = remove_nullable(input); + switch (type->get_primitive_type()) { + case TYPE_VARIANT: + return true; + case TYPE_ARRAY: + return contains_variant_type(assert_cast(*type).get_nested_type()); + case TYPE_MAP: { + const auto& map = assert_cast(*type); + return contains_variant_type(map.get_key_type()) || + contains_variant_type(map.get_value_type()); + } + case TYPE_STRUCT: + return std::ranges::any_of(assert_cast(*type).get_elements(), + contains_variant_type); + default: + return false; + } +} + +bool mapping_reads_variant(const format::ColumnMapping& mapping) { + if (!mapping.file_local_id.has_value()) { + return false; + } + if (contains_variant_type(mapping.original_file_type)) { + return true; + } + if (mapping.table_type != nullptr && + remove_nullable(mapping.table_type)->get_primitive_type() == TYPE_VARIANT) { + return true; + } + return std::ranges::any_of(mapping.child_mappings, mapping_reads_variant); +} + +const char* file_format_name(FileFormat format) { + switch (format) { + case FileFormat::PARQUET: + return "PARQUET"; + case FileFormat::ORC: + return "ORC"; + case FileFormat::CSV: + return "CSV"; + case FileFormat::JSON: + return "JSON"; + case FileFormat::TEXT: + return "TEXT"; + case FileFormat::JNI: + return "JNI"; + case FileFormat::NATIVE: + return "NATIVE"; + case FileFormat::ARROW: + return "ARROW"; + case FileFormat::WAL: + return "WAL"; + } + return "UNKNOWN"; +} + +} // namespace + +Status IcebergTableReader::validate_variant_file_mappings( + FileFormat format, const std::vector& mappings) { + if (format == FileFormat::PARQUET || !std::ranges::any_of(mappings, mapping_reads_variant)) { + return Status::OK(); + } + // Gate on a physical mapping, not the table schema: an older ORC/Avro file may legitimately + // omit a Variant field added by schema evolution, in which case the mapper synthesizes NULL. + return Status::NotSupported( + "Iceberg Variant is supported only for Parquet files in FileScannerV2; file format {} " + "(including ORC/Avro readers) is not supported", + file_format_name(format)); +} + +Status IcebergTableReader::validate_file_mapping(const format::TableColumnMapper& mapper) const { + if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && + _push_down_count_columns->empty()) { + // COUNT(*) may retain an arbitrary minimum-width slot, but that carrier is never a + // semantic physical read and must not trigger the Variant file-format capability gate. + return Status::OK(); + } + return validate_variant_file_mappings(_format, mapper.mappings()); +} + template static std::string join_values_for_debug(const std::vector& values) { std::ostringstream out; diff --git a/be/src/format_v2/table/iceberg_reader.h b/be/src/format_v2/table/iceberg_reader.h index 2768e4cd3e8c73..4118655dca46f0 100644 --- a/be/src/format_v2/table/iceberg_reader.h +++ b/be/src/format_v2/table/iceberg_reader.h @@ -49,6 +49,8 @@ namespace doris::format::iceberg { class IcebergTableReader : public format::TableReader { public: ~IcebergTableReader() override = default; + static Status validate_variant_file_mappings( + FileFormat format, const std::vector& mappings); Status init(format::TableReadOptions&& options) override { RETURN_IF_ERROR(format::TableReader::init(std::move(options))); _mapper_options.mode = format::TableColumnMappingMode::BY_FIELD_ID; @@ -73,6 +75,8 @@ class IcebergTableReader : public format::TableReader { } protected: + Status validate_file_mapping(const format::TableColumnMapper& mapper) const override; + void configure_mapper_options(format::TableColumnMapperOptions* options) const override { options->enable_row_lineage_virtual_columns = true; options->allow_idless_complex_wrapper_projection = diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index e95d8219576bce..0ffaa06a9d15b2 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -414,6 +414,7 @@ class TableReader { DORIS_CHECK(file_schema != nullptr); return Status::OK(); } + virtual Status validate_file_mapping(const TableColumnMapper&) const { return Status::OK(); } // Open the concrete reader for the current split/task and build the file-local scan request. virtual Status open_reader() { @@ -457,6 +458,7 @@ class TableReader { RETURN_IF_ERROR(close_current_reader()); return Status::OK(); } + RETURN_IF_ERROR(validate_file_mapping(*_data_reader.column_mapper)); // COUNT(*) has no semantic column argument, but Nereids retains a minimum-width scan slot // so the scan node still has an output tuple. Record only the current non-predicate file // columns before table-format hooks add row-position or equality-delete dependencies. This @@ -479,40 +481,34 @@ class TableReader { _data_reader.file_block_layout.clear(); _data_reader.block_template.clear(); _file_scan_request.reset(); - _data_reader.file_block_layout.resize(file_request->local_positions.size()); + _data_reader.file_block_layout.resize(file_request->block_column_count()); // 4. Build file block layout from file schema and column mapping. The layout describes // the block returned by file reader before table-column materialization. - for (const auto& [file_column_id, block_position] : file_request->local_positions) { + auto add_file_block_column = [&](const LocalColumnIndex& projection, + LocalIndex block_position) -> Status { DORIS_CHECK(block_position.value() < _data_reader.file_block_layout.size()); + const auto file_column_id = projection.column_id(); const auto* field = _find_column_definition(_data_reader.file_schema, file_column_id); DORIS_CHECK(field != nullptr); ColumnDefinition projected_field; - { - auto it = std::find_if( - file_request->non_predicate_columns.begin(), - file_request->non_predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->non_predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } - { - auto it = std::find_if( - file_request->predicate_columns.begin(), - file_request->predicate_columns.end(), - [&](const LocalColumnIndex& p) { return p.column_id() == file_column_id; }); - if (it != file_request->predicate_columns.end()) { - RETURN_IF_ERROR(project_column_definition(*field, *it, &projected_field)); - } - } + RETURN_IF_ERROR(project_column_definition(*field, projection, &projected_field)); _data_reader.file_block_layout[block_position.value()] = { .file_column_id = file_column_id, .name = projected_field.name, .type = projected_field.type, }; DORIS_CHECK(_data_reader.file_block_layout[block_position.value()].type != nullptr); + return Status::OK(); + }; + for (const auto& projection : file_request->predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->local_positions.at(projection.column_id()))); + } + for (const auto& projection : file_request->non_predicate_columns) { + RETURN_IF_ERROR(add_file_block_column( + projection, file_request->non_predicate_position(projection.column_id()))); } // 5. Prepare block template from file block layout. The block template stores the block diff --git a/be/test/core/column/column_variant_v2_test.cpp b/be/test/core/column/column_variant_v2_test.cpp index 2665304cfa0601..9937555b96d7c8 100644 --- a/be/test/core/column/column_variant_v2_test.cpp +++ b/be/test/core/column/column_variant_v2_test.cpp @@ -34,6 +34,7 @@ #include "common/exception.h" #include "core/arena.h" #include "core/assert_cast.h" +#include "core/block/block.h" #include "core/column/column_const.h" #include "core/column/column_decimal.h" #include "core/column/column_nullable.h" @@ -51,6 +52,7 @@ #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" #include "core/data_type/data_type_timestamptz.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/value/decimalv2_value.h" #include "core/value/ipv4_value.h" #include "core/value/ipv6_value.h" @@ -1788,6 +1790,31 @@ TEST(ColumnVariantV2Test, CowDetachAndClear) { EXPECT_EQ(a.num_elements(), 2); } +TEST(ColumnVariantV2Test, BlockClearDetachesSharedEncodedSubcolumns) { + for (bool clear_selected_only : {false, true}) { + SCOPED_TRACE(clear_selected_only); + auto column = ColumnVariantV2::create(); + insert_encoded_field(*column, encode_json(R"({"a":[1,2]})")); + const std::vector shared_subcolumns = subcolumns(*column); + + Block block; + block.insert({std::move(column), std::make_shared(), "v"}); + ASSERT_TRUE(block.get_by_position(0).column->is_exclusive()); + + if (clear_selected_only) { + block.clear_column_data(std::vector {0}); + } else { + block.clear_column_data(); + } + + EXPECT_EQ(block.get_by_position(0).column->size(), 0); + ASSERT_EQ(shared_subcolumns.size(), 3); + EXPECT_EQ(shared_subcolumns[0]->size(), 1); + EXPECT_EQ(shared_subcolumns[1]->size(), 1); + EXPECT_EQ(shared_subcolumns[2]->size(), 1); + } +} + TEST(ColumnVariantV2Test, EncodedRowCountInvariant) { EXPECT_DEATH( { diff --git a/be/test/exec/scan/access_path_parser_test.cpp b/be/test/exec/scan/access_path_parser_test.cpp index f72da2a69b0b56..23b95d54c0d7b6 100644 --- a/be/test/exec/scan/access_path_parser_test.cpp +++ b/be/test/exec/scan/access_path_parser_test.cpp @@ -32,6 +32,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" @@ -130,6 +131,92 @@ TEST(AccessPathParserTest, IgnoresPrimitiveColumnsAndScannerVirtualColumns) { EXPECT_TRUE(rowid.children.empty()); } +TEST(AccessPathParserTest, PreservesVariantObjectKeysForPhysicalShreddingProjection) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({"100", "nested", "leaf"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(variant.variant_access_paths, + (std::vector> {{"nested", "leaf"}, {"typed_col"}})); + + status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"100"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + + status = AccessPathParser::build_nested_children( + &variant, + std::vector {data_access_path({"100", "typed_col"}), + data_access_path({})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); +} + +TEST(AccessPathParserTest, SeparatesFinalAndPredicateComplexAccessPaths) { + auto variant = root_column(100, "v", std::make_shared()); + auto status = AccessPathParser::build_nested_children( + &variant, std::vector {data_access_path({"v"})}, + std::vector {data_access_path({"v", "n"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(variant.variant_access_paths.empty()); + ASSERT_TRUE(variant.has_predicate_access_paths); + EXPECT_EQ(variant.predicate_variant_access_paths, + (std::vector> {{"n"}})); + + auto int_type = std::make_shared(); + auto struct_type = + std::make_shared(DataTypes {int_type, int_type}, Strings {"a", "b"}); + auto structure = root_column(101, "s", struct_type); + status = AccessPathParser::build_nested_children( + &structure, std::vector {data_access_path({"s"})}, + std::vector {data_access_path({"s", "b"})}, nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 2); + ASSERT_TRUE(structure.has_predicate_access_paths); + ASSERT_EQ(structure.predicate_children.size(), 1); + EXPECT_EQ(structure.predicate_children[0].name, "b"); +} + +TEST(AccessPathParserTest, PreservesVariantPathsNestedInComplexColumns) { + auto variant_type = std::make_shared(); + + auto struct_type = + std::make_shared(DataTypes {variant_type}, Strings {"payload"}); + auto structure = root_column(100, "s", struct_type); + auto status = AccessPathParser::build_nested_children( + &structure, + std::vector {data_access_path({"s", "payload", "typed_col"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(structure.children.size(), 1); + EXPECT_EQ(structure.children[0].variant_access_paths, + (std::vector> {{"typed_col"}})); + + auto array = root_column(101, "items", std::make_shared(variant_type)); + status = AccessPathParser::build_nested_children( + &array, std::vector {data_access_path({"items", "*", "kind"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(array.children.size(), 1); + EXPECT_EQ(array.children[0].variant_access_paths, + (std::vector> {{"kind"}})); + + auto map = root_column( + 102, "attrs", + std::make_shared(std::make_shared(), variant_type)); + status = AccessPathParser::build_nested_children( + &map, std::vector {data_access_path({"attrs", "*", "enabled"})}, + nullptr); + ASSERT_TRUE(status.ok()) << status; + const auto* value = find_child_by_name(map, "value"); + ASSERT_NE(value, nullptr); + EXPECT_EQ(value->variant_access_paths, (std::vector> {{"enabled"}})); +} + // Scenario: reject unsupported top-level inputs before recursive type parsing, including META // paths, missing DATA payloads, and access paths whose root does not match the projected slot. TEST(AccessPathParserTest, RejectsUnsupportedTopLevelAccessPathInputs) { diff --git a/be/test/exec/scan/file_scanner_v2_test.cpp b/be/test/exec/scan/file_scanner_v2_test.cpp index 3506e5db27ae8a..fa167d2c7dce4a 100644 --- a/be/test/exec/scan/file_scanner_v2_test.cpp +++ b/be/test/exec/scan/file_scanner_v2_test.cpp @@ -117,6 +117,16 @@ TEST(FileScannerTest, V1CountPushdownRequiresExplicitCountStarArguments) { TPushAggOp::type::MINMAX, std::nullopt)); } +TEST(FileScannerTest, CountStarPlaceholderIsNotASemanticProjection) { + EXPECT_TRUE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, + std::vector {7})); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::COUNT, std::nullopt)); + EXPECT_FALSE(ScanLocalStateBase::is_count_star_pushdown(TPushAggOp::type::MINMAX, + std::vector {})); +} + TEST(FileScannerV2Test, AdaptiveBatchSizeRunsForCountFallbackOnly) { EXPECT_TRUE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, false)); EXPECT_FALSE(FileScannerV2::TEST_should_run_adaptive_batch_size(true, true)); @@ -456,6 +466,30 @@ TEST(FileScannerV2Test, FileScanLocalStateSelectsV2ForSupportedQueriesOnly) { EXPECT_FALSE(FileScanLocalState::TEST_should_use_file_scanner_v2(query_options, false, params)); } +TEST(FileScannerV2Test, LegacyCountExemptionRequiresMetadataCountOnEveryRange) { + auto scan_range = [](std::optional row_count) { + TScanRangeParams params; + auto& file_range = params.scan_range.ext_scan_range.file_scan_range; + TFileRangeDesc range; + if (row_count.has_value()) { + TTableFormatFileDesc table_format; + table_format.__set_table_level_row_count(*row_count); + range.__set_table_format_params(table_format); + } + file_range.ranges.push_back(std::move(range)); + return params; + }; + + LocalSplitSourceConnector proven({scan_range(4), scan_range(0)}, 2); + EXPECT_TRUE(proven.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector missing({scan_range(4), scan_range(std::nullopt)}, 2); + EXPECT_FALSE(missing.all_ranges_have_table_level_row_count()); + + LocalSplitSourceConnector invalid({scan_range(4), scan_range(-1)}, 2); + EXPECT_FALSE(invalid.all_ranges_have_table_level_row_count()); +} + TEST(FileScannerV2Test, JniCompatibilityShapesUseV2Scanner) { TQueryOptions query_options; query_options.__set_enable_file_scanner_v2(true); diff --git a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp index 960eba0f9c1bd9..23b7cd815c5f54 100644 --- a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp +++ b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp @@ -44,12 +44,16 @@ namespace doris { class TestScanner final : public Scanner { public: TestScanner(RuntimeState* state, ScanLocalStateBase* local_state, int64_t limit, - RuntimeProfile* profile) - : Scanner(state, local_state, limit, profile) {} + RuntimeProfile* profile, bool allow_padding = true) + : Scanner(state, local_state, limit, profile), _allow_padding(allow_padding) {} void add_block(Block block) { _blocks.push_back(std::move(block)); } protected: + bool _can_merge_padding_blocks(const Block& /*left*/, const Block& /*right*/) const override { + return _allow_padding; + } + Status _get_block_impl(RuntimeState* /*state*/, Block* block, bool* eof) override { if (_blocks.empty()) { *eof = true; @@ -62,6 +66,7 @@ class TestScanner final : public Scanner { } private: + bool _allow_padding = true; std::list _blocks; }; @@ -210,4 +215,44 @@ TEST(ScannerProjectionTest, publishes_shared_column_and_reuses_output_block) { EXPECT_EQ(output.get_by_position(0).column->get_int(1), 4); } +TEST(ScannerProjectionTest, projects_incompatible_blocks_before_reading_the_next_block) { + ObjectPool pool; + auto data_type = std::make_shared(); + auto row_descriptor = MockRowDescriptor({data_type}, &pool); + + MockRuntimeState state; + state._batch_size = 8; + + auto op = std::make_shared(); + op->_row_descriptor = row_descriptor; + op->_output_row_descriptor = + std::make_unique(std::vector {data_type}, &pool); + op->_output_tuple_desc = op->_output_row_descriptor->tuple_descriptors()[0]; + + auto local_state = std::make_shared(&state, op.get()); + local_state->_projections = MockSlotRef::create_mock_contexts(0, data_type); + + RuntimeProfile profile("scanner"); + TestScanner scanner(&state, local_state.get(), -1, &profile, false); + ASSERT_TRUE(scanner.init(&state, {}).ok()); + scanner.add_block(ColumnHelper::create_block({0, 1})); + scanner.add_block(ColumnHelper::create_block({2, 3, 4})); + + Block first_output; + bool eos = false; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &first_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(first_output.rows(), 2); + + Block second_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &second_output, &eos).ok()); + EXPECT_FALSE(eos); + EXPECT_EQ(second_output.rows(), 3); + + Block final_output; + ASSERT_TRUE(scanner.get_block_after_projects(&state, &final_output, &eos).ok()); + EXPECT_TRUE(eos); + EXPECT_EQ(final_output.rows(), 0); +} + } // namespace doris diff --git a/be/test/exec/sink/viceberg_merge_sink_test.cpp b/be/test/exec/sink/viceberg_merge_sink_test.cpp index eb7c0159d5fd38..d1da8fb7bfe1d5 100644 --- a/be/test/exec/sink/viceberg_merge_sink_test.cpp +++ b/be/test/exec/sink/viceberg_merge_sink_test.cpp @@ -33,6 +33,7 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "exec/sink/sink_common.h" #include "exec/sink/viceberg_delete_sink.h" #include "exec/sink/writer/iceberg/viceberg_table_writer.h" #include "exprs/vexpr_context.h" @@ -200,6 +201,43 @@ TEST_F(VIcebergMergeSinkTest, TestUpdateProducesDeleteAndInsert) { ASSERT_TRUE(sink->close(Status::OK()).ok()); } +TEST_F(VIcebergMergeSinkTest, TestDeleteOnlySkipsVariantDataWriter) { + ObjectPool pool; + MockRuntimeState state; + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + TDataSink t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + t_sink.iceberg_merge_sink.__set_schema_json( + "{\"type\":\"struct\",\"schema-id\":0,\"fields\":[" + "{\"id\":1,\"name\":\"payload\",\"required\":false,\"type\":\"variant\"}" + "]}"); + + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + sink->set_skip_io(true); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + EXPECT_EQ(nullptr, sink->_table_writer); + RuntimeProfile profile("iceberg_merge_sink"); + ASSERT_TRUE(sink->open(&state, &profile).ok()); + + // Delete-only plans must never use the insert opcode, which intentionally requires a data writer. + Block block = build_block_with_ops({kDeleteOperation}); + Status status = sink->write(&state, block); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(1, sink->_delete_row_count); + EXPECT_EQ(0, sink->_insert_row_count); + + ASSERT_TRUE(sink->close(Status::OK()).ok()); +} + TEST_F(VIcebergMergeSinkTest, TestMissingOperationColumn) { ObjectPool pool; MockRuntimeState state; diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index e6884f8fd81e06..0ef554fc7932f0 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -39,6 +39,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exprs/vectorized_fn_call.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" @@ -65,18 +66,10 @@ DataTypePtr i64() { return std::make_shared(); } -DataTypePtr f32() { - return std::make_shared(); -} - DataTypePtr f64() { return std::make_shared(); } -DataTypePtr dec32(uint32_t precision, uint32_t scale) { - return std::make_shared(precision, scale); -} - DataTypePtr str() { return std::make_shared(); } @@ -93,6 +86,10 @@ DataTypePtr u8() { return std::make_shared(); } +DataTypePtr variant_v2() { + return std::make_shared(); +} + ColumnDefinition field_id_col(const std::string& name, int32_t field_id, DataTypePtr type, int32_t local_id = -1) { ColumnDefinition column; @@ -518,14 +515,6 @@ VExprSPtr like_expr(const VExprSPtr& left, const std::string& pattern) { return expr; } -VExprSPtr struct_element_by_selector(const VExprSPtr& parent, DataTypePtr child_type, - const VExprSPtr& selector) { - auto expr = std::make_shared("struct_element", std::move(child_type)); - expr->add_child(parent); - expr->add_child(selector); - return expr; -} - VExprSPtr int_gt(const VExprSPtr& left, int32_t value) { auto expr = std::make_shared("gt", u8(), TExprNodeType::BINARY_PRED, TExprOpcode::GT); @@ -543,58 +532,12 @@ VExprSPtr binary_predicate(TExprOpcode::type opcode, const VExprSPtr& left, return expr; } -VExprSPtr in_predicate(const VExprSPtr& probe, const DataTypePtr& literal_type, - const std::vector& values) { - auto expr = std::make_shared("in", u8(), TExprNodeType::IN_PRED); - expr->add_child(probe); - for (const auto& value : values) { - expr->add_child(literal(literal_type, value)); - } - return expr; -} - -VExprSPtr null_predicate(const VExprSPtr& child, bool is_null) { - auto expr = - std::make_shared(is_null ? "is_null_pred" : "is_not_null_pred", u8()); - expr->add_child(child); - return expr; -} - VExprSPtr cast_expr(const VExprSPtr& child, DataTypePtr target_type) { auto expr = Cast::create_shared(std::move(target_type)); expr->add_child(child); return expr; } -VExprSPtr compound_predicate(TExprOpcode::type opcode, const VExprSPtr& left, - const VExprSPtr& right) { - auto expr = std::make_shared("compound", u8(), TExprNodeType::COMPOUND_PRED, - opcode); - expr->add_child(left); - expr->add_child(right); - return expr; -} - -std::vector collect_paths(const VExprSPtr& expr) { - std::vector paths; - collect_nested_struct_paths(expr, &paths); - return paths; -} - -void expect_name_selector(const StructChildSelector& selector, const std::string& name) { - EXPECT_TRUE(selector.by_name); - EXPECT_EQ(selector.name, name); -} - -void expect_ordinal_selector(const StructChildSelector& selector, size_t ordinal) { - EXPECT_FALSE(selector.by_name); - EXPECT_EQ(selector.ordinal, ordinal); -} - -void expect_path_root(const NestedStructPath& path, size_t global_index) { - EXPECT_EQ(path.root_global_index, GlobalIndex(global_index)); -} - class ColumnMapperCastTest : public testing::Test { protected: void SetUp() override { state.set_enable_strict_cast(true); } @@ -890,225 +833,6 @@ TEST(ColumnMapperNestedHelperTest, BuildsProjectionByNameAndOrdinalSelectors) { EXPECT_EQ(ordinal_projection.children[0].local_id(), 0); } -// ---------------------------------------------------------------------- -// collect_nested_struct_paths() helper tests. -// These tests assert the entry helper for nested scan projection: it only discovers -// table-side struct paths. Later localization decides how to add scan projections. -// ---------------------------------------------------------------------- - -TEST(ColumnMapperCollectNestedStructPathsTest, CollectsNameOrdinalAndBooleanSelectors) { - const auto leaf_type = i32(); - const auto inner_type = - std::make_shared(DataTypes {leaf_type, leaf_type}, Strings {"x", "y"}); - const auto root_type = std::make_shared(DataTypes {inner_type, leaf_type}, - Strings {"nested", "missing"}); - const auto root = table_slot(0, 3, root_type, "s"); - - const auto nested_by_ordinal = struct_element_by_selector( - struct_element_by_selector(root, inner_type, - literal(i32(), Field::create_field(1))), - leaf_type, literal(i32(), Field::create_field(2))); - auto paths = collect_paths(nested_by_ordinal); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 3); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_ordinal_selector(paths[0].selectors[0], 1); - expect_ordinal_selector(paths[0].selectors[1], 2); - - const std::vector positive_ordinal_selectors = { - literal(std::make_shared(), - Field::create_field(static_cast(1))), - literal(std::make_shared(), - Field::create_field(static_cast(2))), - literal(i32(), Field::create_field(3)), - literal(i64(), Field::create_field(4)), - literal(u8(), Field::create_field(true)), - }; - for (size_t idx = 0; idx < positive_ordinal_selectors.size(); ++idx) { - const auto selected = - struct_element_by_selector(root, leaf_type, positive_ordinal_selectors[idx]); - paths = collect_paths(selected); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_ordinal_selector(paths[0].selectors[0], idx == 4 ? 1 : idx + 1); - } - - paths = collect_paths(struct_element(root, leaf_type, "missing")); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "missing"); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, IgnoresInvalidSelectorsAndNonPathRoots) { - const auto leaf_type = i32(); - const auto root_type = std::make_shared(DataTypes {leaf_type}, Strings {"a"}); - const auto root = table_slot(0, 0, root_type, "s"); - - const std::vector invalid_selectors = { - literal(i32(), Field::create_field(0)), - literal(i32(), Field::create_field(-1)), - literal(u8(), Field::create_field(false)), - literal(f32(), Field::create_field(1.0F)), - literal(f64(), Field::create_field(1.0)), - table_slot(1, 1, i32(), "selector"), - }; - for (const auto& selector : invalid_selectors) { - EXPECT_TRUE(collect_paths(struct_element_by_selector(root, leaf_type, selector)).empty()); - } - - auto wrong_arity = std::make_shared("struct_element", leaf_type); - wrong_arity->add_child(root); - EXPECT_TRUE(collect_paths(wrong_arity).empty()); - - auto not_struct_element = std::make_shared("other_function", leaf_type); - not_struct_element->add_child(root); - not_struct_element->add_child(literal(str(), Field::create_field("a"))); - EXPECT_TRUE(collect_paths(not_struct_element).empty()); - - EXPECT_TRUE(collect_paths(struct_element(literal(str(), Field::create_field("x")), - leaf_type, "a")) - .empty()); - EXPECT_TRUE(collect_paths(nullptr).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, RecursesThroughExpressionsAndKeepsCompletePath) { - const auto leaf_type = i32(); - const auto inner_type = std::make_shared(DataTypes {leaf_type}, Strings {"b"}); - const auto root_type = - std::make_shared(DataTypes {inner_type, leaf_type}, Strings {"a", "c"}); - const auto root = table_slot(0, 2, root_type, "s"); - const auto path_a = struct_element_by_selector( - root, inner_type, literal(str(), Field::create_field("a"))); - const auto path_ab = struct_element_by_selector( - path_a, leaf_type, literal(str(), Field::create_field("b"))); - const auto path_c = struct_element_by_selector( - root, leaf_type, literal(str(), Field::create_field("c"))); - - auto paths = collect_paths(binary_predicate( - TExprOpcode::GT, path_ab, literal(leaf_type, Field::create_field(1)))); - ASSERT_EQ(paths.size(), 1); - expect_path_root(paths[0], 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - - paths = collect_paths(compound_predicate( - TExprOpcode::COMPOUND_OR, - binary_predicate(TExprOpcode::GT, path_ab, - literal(leaf_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_c, - literal(leaf_type, Field::create_field(2))))); - ASSERT_EQ(paths.size(), 2); - ASSERT_EQ(paths[0].selectors.size(), 2); - ASSERT_EQ(paths[1].selectors.size(), 1); - expect_name_selector(paths[0].selectors[0], "a"); - expect_name_selector(paths[0].selectors[1], "b"); - expect_name_selector(paths[1].selectors[0], "c"); - - auto fn = std::make_shared("fn", leaf_type); - fn->add_child(path_ab); - fn->add_child(table_slot(3, 4, leaf_type, "other")); - paths = collect_paths(fn); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); - - auto if_expr = std::make_shared("if", leaf_type); - if_expr->add_child(literal(u8(), Field::create_field(true))); - if_expr->add_child(path_ab); - if_expr->add_child(path_c); - paths = collect_paths(if_expr); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(compound_predicate(TExprOpcode::COMPOUND_AND, path_ab, path_ab)); - ASSERT_EQ(paths.size(), 2); - - paths = collect_paths(path_ab); - ASSERT_EQ(paths.size(), 1); - ASSERT_EQ(paths[0].selectors.size(), 2); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, CastBehaviorSeparatesProjectionAndPruningRules) { - const auto int_type = i32(); - const auto bigint_type = i64(); - const auto float_type = f32(); - const auto double_type = f64(); - const auto decimal_small = dec32(8, 2); - const auto decimal_wide = dec32(9, 2); - const auto decimal_changed_scale = dec32(9, 3); - - const auto root_type = std::make_shared( - DataTypes {int_type, float_type, decimal_small}, Strings {"i", "f", "d"}); - const auto root = table_slot(0, 0, root_type, "s"); - const auto int_path = struct_element(root, int_type, "i"); - const auto float_path = struct_element(root, float_type, "f"); - const auto decimal_path = struct_element(root, decimal_small, "d"); - - auto paths = collect_paths(cast_expr(int_path, bigint_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(float_path, double_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "f"); - - paths = collect_paths(cast_expr(decimal_path, decimal_wide)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - paths = collect_paths( - cast_expr(struct_element(root, make_nullable(int_type), "i"), make_nullable(int_type))); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - // Unsafe casts are not accepted as pruning paths, but collect_nested_struct_paths() still - // recurses into children so scan projection can read the column needed by row-level filters. - paths = collect_paths(cast_expr(struct_element(root, bigint_type, "i"), int_type)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "i"); - - paths = collect_paths(cast_expr(decimal_path, decimal_changed_scale)); - ASSERT_EQ(paths.size(), 1); - expect_name_selector(paths[0].selectors[0], "d"); - - EXPECT_TRUE(collect_paths(cast_expr(table_slot(1, 1, int_type, "plain"), bigint_type)).empty()); -} - -TEST(ColumnMapperCollectNestedStructPathsTest, ProjectionMergeKeepsFilterOnlyPathAndDeduplicates) { - const auto int_type = i32(); - const auto string_type = str(); - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", int_type); - auto table_output = struct_name_col("s", {table_a}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", int_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b, name_col("c", string_type, 2)}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_output}, {}, {file_struct}).ok()); - - const auto path_b = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "b"); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - binary_predicate(TExprOpcode::GT, path_b, - literal(int_type, Field::create_field(1))), - binary_predicate(TExprOpcode::LT, path_b, - literal(int_type, Field::create_field(10)))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_output}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - // Scenario: row-oriented readers such as CSV/Text cannot lazy-read predicate columns separately. // For a complex root that is both projected and referenced by a filter, the materialized mapper // keeps one non-predicate scan entry and asks the reader to read the full top-level struct. @@ -2441,41 +2165,6 @@ TEST(ColumnMapperLocalizeFiltersTest, ConstantFilterBuildsEntryWithoutFileScanCo mapper.mappings()[0].constant_index); } -TEST(ColumnMapperLocalizeFiltersTest, NestedFilterOnlyChildMergesIntoPredicateProjection) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.localize_filters({filter}, &request).ok()); - - EXPECT_TRUE(request.non_predicate_columns.empty()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - ASSERT_FALSE(request.predicate_columns[0].project_all_children); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); - ASSERT_EQ(request.local_positions.size(), 1); - EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); - ASSERT_TRUE(mapper.filter_entries().at(GlobalIndex(0)).is_local()); - EXPECT_EQ(mapper.filter_entries().at(GlobalIndex(0)).local_index(), LocalIndex(0)); -} - TEST(ColumnMapperLocalizeFiltersTest, PreservesExistingScanStateWhenAddingPredicateColumn) { const auto int_type = i32(); const std::vector table_schema = { @@ -2574,24 +2263,24 @@ TEST(ColumnMapperScanRequestTest, OrdinaryPredicateSlotRetainsPayloadForScannerB EXPECT_TRUE(request.predicate_only_columns.empty()); } -TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { +TEST(ColumnMapperScanRequestTest, StructAllAccessPathsAreEagerWithoutPredicateMapping) { const auto int_type = i32(); const auto string_type = str(); auto table_a = name_col("a", int_type); auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); + auto table_struct = struct_name_col("s", {table_a, table_b}); auto file_a = name_col("a", int_type, 0); auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); + auto file_c = name_col("c", int_type, 2); + auto file_struct = struct_name_col("s", {file_a, file_b, file_c}, 5); TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"), 10); + auto filter_expr = + int_gt(struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "a"), 10); TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), .global_indices = {GlobalIndex(0)}}; @@ -2605,159 +2294,6 @@ TEST(ColumnMapperScanRequestTest, StructOutputAndFilterOnlyChildAreMerged) { EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); } -TEST(ColumnMapperScanRequestTest, RenamedNestedPredicateTargetsMappedFileChild) { - const auto int_type = i32(); - - auto table_a = field_id_col("a", 1, int_type); - auto table_renamed_b = field_id_col("renamed_b", 2, int_type); - auto table_struct = struct_col("s", 10, {table_a, table_renamed_b}); - auto file_a = field_id_col("a", 1, int_type, 0); - auto file_b = field_id_col("b", 2, int_type, 1); - auto file_struct = struct_col("s", 10, {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - auto filter_expr = int_gt( - struct_element(table_slot(0, 0, table_struct.type, "s"), int_type, "renamed_b"), 10); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedInNullAndReverseComparisonFiltersAreMerged) { - const auto int_type = i32(); - const auto string_type = str(); - - auto table_a = name_col("a", int_type); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = struct_name_col("s", {table_a, table_b}); - - auto file_a = name_col("a", int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct.type, "s"), int_type, "a"); - auto in_filter = - in_predicate(nested_a, int_type, - {Field::create_field(5), Field::create_field(7)}); - auto reverse_filter = binary_predicate( - TExprOpcode::LT, literal(int_type, Field::create_field(3)), nested_a); - auto null_filter = null_predicate(nested_a, true); - auto not_null_filter = null_predicate(nested_a, false); - auto filter_expr = compound_predicate( - TExprOpcode::COMPOUND_AND, - compound_predicate(TExprOpcode::COMPOUND_AND, in_filter, reverse_filter), - compound_predicate(TExprOpcode::COMPOUND_AND, null_filter, not_null_filter)); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, NestedPredicateFilterThroughSafeCast) { - const auto file_int_type = i32(); - const auto table_bigint_type = i64(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_bigint_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_int_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_int_type, "a"); - auto filter_expr = - binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_bigint_type), - literal(table_bigint_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - -TEST(ColumnMapperScanRequestTest, UnsafeCastDoesNotBuildNestedPredicateFilter) { - const auto file_bigint_type = i64(); - const auto table_int_type = i32(); - const auto string_type = str(); - - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - auto full_table_struct = std::make_shared( - DataTypes {table_int_type, string_type}, Strings {"a", "b"}); - - auto file_a = name_col("a", file_bigint_type, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_a = - struct_element(table_slot(0, 0, full_table_struct, "s"), file_bigint_type, "a"); - auto filter_expr = binary_predicate(TExprOpcode::GT, cast_expr(nested_a, table_int_type), - literal(table_int_type, Field::create_field(5))); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(5)); - EXPECT_EQ(projection_ids(request.predicate_columns[0].children), std::vector({0, 1})); -} - -TEST(ColumnMapperScanRequestTest, DeepNestedPredicateTargetsLeafPath) { - const auto id_type = i32(); - const auto name_type = str(); - const auto string_type = str(); - auto table_b = name_col("b", string_type); - auto table_struct = struct_name_col("s", {table_b}); - - auto full_table_inner_type = - std::make_shared(DataTypes {id_type, name_type}, Strings {"id", "n"}); - auto full_table_struct_type = std::make_shared( - DataTypes {full_table_inner_type, string_type}, Strings {"a", "b"}); - - auto file_id = name_col("id", id_type, 0); - auto file_name = name_col("n", name_type, 1); - auto file_a = struct_name_col("a", {file_id, file_name}, 0); - auto file_b = name_col("b", string_type, 1); - auto file_struct = struct_name_col("s", {file_a, file_b}, 5); - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); - - const auto nested_id = - struct_element(struct_element(table_slot(0, 0, full_table_struct_type, "s"), - full_table_inner_type, "a"), - id_type, "id"); - auto filter_expr = - in_predicate(nested_id, id_type, - {Field::create_field(5), Field::create_field(7)}); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); -} - TEST(ColumnMapperScanRequestTest, ArrayStructProjectionPrunesElementChildren) { const auto int_type = i32(); const auto string_type = str(); @@ -2910,51 +2446,6 @@ TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) EXPECT_TRUE(request.non_predicate_columns[0].children.empty()); } -// Scenario: a map value struct projects child `b`, while a row filter reads value child `a`. -// The filter is too complex to become a file-local nested predicate. Lazy demotion must move the -// merged projection to the non-predicate set without dropping either physical value child. -TEST(ColumnMapperScanRequestTest, MapFilterOnlyValueChildMergesWithOutputProjection) { - const auto key_type = i32(); - const auto int_type = i32(); - const auto string_type = str(); - - auto table_value_b = name_col("b", string_type); - auto table_value = struct_name_col("value", {table_value_b}); - auto table_map = map_col("m", -1, {table_value}, key_type, table_value.type); - set_name_identifiers(&table_map, 0); - - auto file_key = name_col("key", key_type, 0); - auto file_value_a = name_col("a", int_type, 0); - auto file_value_b = name_col("b", string_type, 1); - auto file_value = struct_name_col("value", {file_value_a, file_value_b}, 1); - auto file_map = map_col("m", -1, {file_key, file_value}, key_type, file_value.type, 0); - set_name_identifiers(&file_map, 0); - - auto full_value_type = - std::make_shared(DataTypes {int_type, string_type}, Strings {"a", "b"}); - auto full_map_type = std::make_shared(key_type, full_value_type); - auto value_expr = - struct_element(table_slot(0, 0, full_map_type, "m"), full_value_type, "value"); - auto filter_expr = int_gt(struct_element(value_expr, int_type, "a"), 5); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); - ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {table_map}, &request).ok()); - - EXPECT_TRUE(request.predicate_columns.empty()); - ASSERT_EQ(request.non_predicate_columns.size(), 1); - const auto& projection = request.non_predicate_columns[0]; - EXPECT_EQ(projection.column_id(), LocalColumnId(0)); - ASSERT_FALSE(projection.project_all_children); - ASSERT_EQ(projection.children.size(), 1); - EXPECT_EQ(projection.children[0].local_id(), 1); - EXPECT_EQ(projection_ids(projection.children[0].children), std::vector({0, 1})); -} - // Scenario: when projected struct children are an in-order prefix of the file struct, the mapper can // read those physical children directly without rebuilding the file-side complex type. TEST(ColumnMapperScanRequestTest, MatchingProjectedStructDoesNotNeedComplexRematerialize) { @@ -3019,9 +2510,10 @@ TEST(ColumnMapperScanRequestTest, RenameOnlyProjectedStructDoesNotRebuildFilePro EXPECT_TRUE(mapper.mappings()[0].is_trivial); } -// Scenario: a row filter references an unprojected struct child, so the predicate projection is -// merged with the output projection and the mapper rebuilds the projected file struct type. -TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFileType) { +// Scenario: FE access paths are the sole contract for nested predicate projection. If a filter +// references a Struct child absent from all_access_paths and no predicate_access_paths were sent, +// File Scanner V2 must not infer and append that child from the expression. +TEST(ColumnMapperScanRequestTest, MissingPredicateAccessPathsDoNotInferStructProjection) { const auto int_type = i32(); const auto string_type = str(); @@ -3047,19 +2539,18 @@ TEST(ColumnMapperScanRequestTest, PredicateProjectionRebuildsProjectedStructFile FileScanRequest request; ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_TRUE(request.non_predicate_columns.empty()); - const auto& projection = request.predicate_columns[0]; + EXPECT_TRUE(request.predicate_columns.empty()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& projection = request.non_predicate_columns[0]; EXPECT_FALSE(projection.project_all_children); - EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1, 2})); + EXPECT_EQ(projection_ids(projection.children), std::vector({0, 1})); const auto* mapped_type = assert_cast( remove_nullable(mapper.mappings()[0].file_type).get()); - ASSERT_EQ(mapped_type->get_elements().size(), 3); + ASSERT_EQ(mapped_type->get_elements().size(), 2); EXPECT_EQ(mapped_type->get_element_name(0), "a"); EXPECT_EQ(mapped_type->get_element_name(1), "b"); - EXPECT_EQ(mapped_type->get_element_name(2), "c"); - EXPECT_FALSE(mapper.mappings()[0].is_trivial); + EXPECT_TRUE(request.conjuncts.empty()); } // Scenario: Paimon projects one struct child but filters on an unprojected TIMESTAMP_LTZ(9) @@ -3504,63 +2995,6 @@ TEST_F(ColumnMapperCastTest, NestedElementAtInPredicateUsesAllOrNothingLiteralRe EXPECT_TRUE(fallback_root->children()[2]->data_type()->equals(*table_bigint_type)); } -// Scenario: output projection reads one struct child while the row filter reads a different nested -// struct child. File-local conjunct rewrite must use the merged scan projection type. In the SQL -// shape below, `SELECT element_at(s, 'c') WHERE element_at(element_at(s, 'b'), 'cc') LIKE ...` -// reads file children `b.cc` and `c`; the localized inner `element_at(s, 'b')` returns -// `Struct(cc)`, not the full old file child `Struct(cc, new_dd)`. -TEST(ColumnMapperScanRequestTest, NestedElementAtConjunctUsesMergedScanProjectionChildType) { - const auto string_type = str(); - const auto int_type = i32(); - - auto table_cc = field_id_col("cc", 23, string_type); - auto table_new_dd = field_id_col("new_dd", 24, int_type); - auto table_b = struct_col("b", 20, {table_cc, table_new_dd}); - auto table_c = field_id_col("c", 25, string_type); - auto full_table_struct = struct_col("struct_column2", 19, {table_b, table_c}); - auto projected_table_struct = struct_col("struct_column2", 19, {table_c}); - - auto file_cc = field_id_col("cc", 23, string_type, 0); - auto file_new_dd = field_id_col("new_dd", 24, int_type, 1); - auto file_b = struct_col("b", 20, {file_cc, file_new_dd}, 0); - auto file_c = field_id_col("c", 25, string_type, 1); - auto file_struct = struct_col("new_struct_column", 19, {file_b, file_c}, 10); - - const auto table_slot_expr = table_slot(0, 0, full_table_struct.type, "struct_column2"); - const auto table_parent_expr = element_at(table_slot_expr, table_b.type, "b"); - const auto table_leaf_expr = element_at(table_parent_expr, string_type, "cc"); - auto filter_expr = like_expr(table_leaf_expr, "NestedC%"); - TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), - .global_indices = {GlobalIndex(0)}}; - - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); - ASSERT_TRUE(mapper.create_mapping({projected_table_struct}, {}, {file_struct}).ok()); - - FileScanRequest request; - ASSERT_TRUE(mapper.create_scan_request({filter}, {projected_table_struct}, &request).ok()); - ASSERT_EQ(request.conjuncts.size(), 1); - ASSERT_EQ(request.predicate_columns.size(), 1); - EXPECT_EQ(request.predicate_columns[0].column_id(), LocalColumnId(10)); - - const auto& localized_leaf = request.conjuncts[0]->root()->children()[0]; - ASSERT_EQ(localized_leaf->expr_name(), "element_at"); - const auto& localized_parent = localized_leaf->children()[0]; - ASSERT_EQ(localized_parent->expr_name(), "element_at"); - - const auto* localized_slot = - assert_cast(localized_parent->children()[0].get()); - EXPECT_EQ(localized_slot->column_name(), "new_struct_column"); - // The scan projection keeps the top-level file column id above, while the localized conjunct - // executes on the file-reader Block. The VSlotRef column id is therefore the block position of - // `new_struct_column` in this request, not the file schema id 10. - EXPECT_EQ(localized_slot->column_id(), 0); - - const auto* localized_parent_type = assert_cast( - remove_nullable(localized_parent->data_type()).get()); - ASSERT_EQ(localized_parent_type->get_elements().size(), 1); - EXPECT_EQ(localized_parent_type->get_element_name(0), "cc"); -} - // Scenario: struct child access through a computed map/array parent is not localized as a file // conjunct, because the projected value struct can have a different physical child order. TEST(ColumnMapperScanRequestTest, MapValuesStructChildConjunctStaysTableLevel) { @@ -4660,5 +4094,301 @@ TEST_F(ColumnMapperCastTest, ColumnMapperKeepsTableSlotIdWhenFileBlockPositionCh conjunct->close(); } +TEST(ColumnMapperTest, VariantAccessPathProjectsOnlyPhysicalTypedLeaf) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + EXPECT_EQ(root.children[0].local_id(), 2); + ASSERT_EQ(root.children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].local_id(), 0); + ASSERT_EQ(root.children[0].children[0].children.size(), 1); + EXPECT_EQ(root.children[0].children[0].children[0].local_id(), 1); + EXPECT_TRUE(root.children[0].children[0].children[0].project_all_children); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredStructOutputProjection) { + auto table_a = field_id_col("a", 2, i64()); + auto table_b = field_id_col("b", 3, i64()); + auto table_struct = struct_col("s", 1, {table_a, table_b}); + table_struct.has_predicate_access_paths = true; + table_struct.predicate_children = {table_b}; + + auto file_a = field_id_col("a", 2, i64(), 0); + auto file_b = field_id_col("b", 3, i64(), 1); + auto file_struct = struct_col("s", 1, {file_a, file_b}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto b = struct_element(table_slot(0, 0, table_struct.type, "s"), i64(), "b"); + auto predicate = binary_predicate(TExprOpcode::GT, b, + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1) << request.debug_string(); + ASSERT_EQ(request.predicate_columns[0].children.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children[0].local_id(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredVariantRootProjection) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.has_predicate_access_paths = true; + table_variant.predicate_variant_access_paths = {{"typed_col"}}; + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + + auto typed_col = + element_at(table_slot(0, 0, table_variant.type, "v"), variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_variant}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_FALSE(request.predicate_columns[0].project_all_children); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); + EXPECT_EQ(request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); +} + +TEST(ColumnMapperTest, NestedVariantAccessPathProjectsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_struct}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root = request.non_predicate_columns[0]; + ASSERT_FALSE(root.project_all_children); + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + EXPECT_EQ(variant.local_id(), 0); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].local_id(), 0); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, NestedVariantAllAccessPathKeepsPhysicalTypedLeaf) { + auto table_variant = field_id_col("payload", 2, variant_v2()); + table_variant.variant_access_paths = {{"typed_col"}}; + auto table_struct = struct_col("info", 1, {table_variant}); + + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("payload", 2, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + auto file_struct = struct_col("info", 1, {std::move(file_variant)}, 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_struct}, {}, {file_struct}).ok()); + + auto payload = + struct_element(table_slot(0, 0, table_struct.type, "info"), variant_v2(), "payload"); + auto typed_col = element_at(payload, variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i32()), + literal(i32(), Field::create_field(0))); + TableFilter filter {.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(0)}}; + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_struct}, &request).ok()); + ASSERT_EQ(request.predicate_columns.size(), 1); + const auto& root = request.predicate_columns[0]; + ASSERT_EQ(root.children.size(), 1); + const auto& variant = root.children[0]; + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); +} + +TEST(ColumnMapperTest, ArrayAndMapNestedVariantPathsReachPhysicalTypedLeaf) { + auto make_file_variant = [](std::string name, int32_t field_id, int32_t local_id) { + auto wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed = struct_name_col("typed_value", {std::move(wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed)}; + return variant; + }; + auto assert_variant_leaf = [](const LocalColumnIndex& variant) { + ASSERT_FALSE(variant.project_all_children); + ASSERT_EQ(variant.children.size(), 1); + EXPECT_EQ(variant.children[0].local_id(), 2); + ASSERT_EQ(variant.children[0].children.size(), 1); + ASSERT_EQ(variant.children[0].children[0].children.size(), 1); + EXPECT_EQ(variant.children[0].children[0].children[0].local_id(), 1); + }; + + { + auto table_element = field_id_col("element", 2, variant_v2()); + table_element.variant_access_paths = {{"typed_col"}}; + auto table_array = array_col("items", 1, table_element); + auto file_array = array_col("items", 1, make_file_variant("element", 2, 0), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_array}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + ASSERT_EQ(request.non_predicate_columns[0].children.size(), 1); + assert_variant_leaf(request.non_predicate_columns[0].children[0]); + } + + { + auto table_key = field_id_col("key", 2, str()); + auto table_value = field_id_col("value", 3, variant_v2()); + table_value.variant_access_paths = {{"typed_col"}}; + auto table_map = map_col("attributes", 1, {table_key, table_value}, str(), variant_v2()); + auto file_key = field_id_col("key", 2, str(), 0); + auto file_value = make_file_variant("value", 3, 1); + auto file_map = map_col("attributes", 1, {file_key, file_value}, str(), variant_v2(), 0); + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_map}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& children = request.non_predicate_columns[0].children; + const auto value_it = std::ranges::find_if( + children, [](const LocalColumnIndex& child) { return child.local_id() == 1; }); + ASSERT_NE(value_it, children.end()); + assert_variant_leaf(*value_it); + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresLosslessObjectPath) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto dotted_wrapper = struct_name_col( + "a.b", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 1); + auto numeric_wrapper = struct_name_col( + "1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 2); + auto negative_numeric_wrapper = struct_name_col( + "-1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 3); + auto positive_numeric_wrapper = struct_name_col( + "+1", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 4); + auto null_wrapper = struct_name_col( + "NULL", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 5); + auto typed_value = + struct_name_col("typed_value", + {std::move(field_wrapper), std::move(dotted_wrapper), + std::move(numeric_wrapper), std::move(negative_numeric_wrapper), + std::move(positive_numeric_wrapper), std::move(null_wrapper)}, + 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + for (const std::vector& path : + {std::vector {"a.b"}, std::vector {"a", "b"}, + std::vector {"1"}, std::vector {"-1"}, + std::vector {"+1"}, std::vector {"NULL"}}) { + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {path}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children) + << "unsafe Variant path must fall back to the complete physical subtree"; + } +} + +TEST(ColumnMapperTest, VariantLeafProjectionRequiresObjectTypedValue) { + auto element_wrapper = struct_name_col( + "element", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, 0); + auto array_typed_value = array_col("typed_value", -1, std::move(element_wrapper), 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(array_typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"element"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + +TEST(ColumnMapperTest, VariantLeafProjectionDeclinesAmbiguousPrimitiveIdentity) { + auto field_wrapper = struct_name_col( + "binary_col", + {name_col("value", varbinary(), 0), name_col("typed_value", varbinary(), 1)}, 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto file_variant = field_id_col("v", 10, variant_v2(), 0); + file_variant.children = {name_col("metadata", varbinary(), 0), + name_col("value", varbinary(), 1), std::move(typed_value)}; + + auto table_variant = field_id_col("v", 10, variant_v2()); + table_variant.variant_access_paths = {{"binary_col"}}; + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(mapper.create_mapping({table_variant}, {}, {file_variant}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({}, {table_variant}, &request).ok()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); +} + } // namespace } // namespace doris::format diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index be6837755b1abf..d20a3fe8e942ce 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -48,6 +48,7 @@ #include "core/column/column_string.h" #include "core/column/column_struct.h" #include "core/column/column_vector.h" +#include "core/column/variant_v2/column_variant_v2.h" #include "core/data_type/data_type_array.h" #include "core/data_type/data_type_date_or_datetime_v2.h" #include "core/data_type/data_type_factory.hpp" @@ -56,11 +57,13 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "core/field.h" #include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/column_mapper.h" #include "format_v2/expr/delete_predicate.h" @@ -154,6 +157,132 @@ class Int32GreaterThanExpr final : public VExpr { const std::string _expr_name = "Int32GreaterThanExpr"; }; +class VariantPathMetadataExpr : public VExpr { +public: + VariantPathMetadataExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathMetadataExpr is not executable"); + } + +private: + std::string _name; +}; + +class VariantInt32PathGreaterThanExpr final : public VariantPathMetadataExpr { +public: + VariantInt32PathGreaterThanExpr(int column_id, std::string key, int32_t value) + : VariantPathMetadataExpr("gt", std::make_shared(), + TExprNodeType::BINARY_PRED), + _column_id(column_id), + _key(std::move(key)), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef(_key)}}; + const auto typed = variants.find_shredded_typed_value(path); + if (!typed.has_value()) { + return Status::InternalError("Expected the projected Variant typed leaf"); + } + const auto& typed_nullable = assert_cast(*typed->column); + const auto& values = + assert_cast(typed_nullable.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && + !typed_nullable.is_null_at(input_row) && values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + +private: + int _column_id; + std::string _key; + int32_t _value; +}; + +VExprContextSPtr create_variant_int32_path_greater_than_conjunct(int column_id, std::string key, + int32_t value) { + auto slot = VSlotRef::create_shared(0, column_id, -1, + make_nullable(std::make_shared()), "v"); + auto key_literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(key)); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key_literal); + auto cast = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(std::make_shared(), + Field::create_field(value)); + auto gt = std::make_shared(column_id, std::move(key), value); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +class StructInt32ChildGreaterThanExpr final : public VExpr { +public: + StructInt32ChildGreaterThanExpr(int column_id, int32_t value) + : VExpr(std::make_shared(), false), + _column_id(column_id), + _value(value) {} + + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, + size_t count, ColumnPtr& result_column) const override { + const auto& nullable = + assert_cast(*block->get_by_position(_column_id).column); + const auto& structure = assert_cast(nullable.get_nested_column()); + const auto& child = assert_cast(structure.get_column(0)); + const auto& values = assert_cast(child.get_nested_column()).get_data(); + auto result = ColumnUInt8::create(); + auto& output = result->get_data(); + output.resize(count); + for (size_t row = 0; row < count; ++row) { + const size_t input_row = selector == nullptr ? row : (*selector)[row]; + output[row] = !nullable.is_null_at(input_row) && !child.is_null_at(input_row) && + values[input_row] > _value; + } + result_column = std::move(result); + return Status::OK(); + } + + const std::string& expr_name() const override { return _expr_name; } + + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_column_id); + } + +private: + int _column_id; + int32_t _value; + const std::string _expr_name = "StructInt32ChildGreaterThanExpr"; +}; + +VExprContextSPtr create_struct_int32_child_greater_than_conjunct(int column_id, int32_t value) { + auto context = VExprContext::create_shared( + std::make_shared(column_id, value)); + context->_prepared = true; + context->_opened = true; + return context; +} + class Int32DictionaryEqualsExpr final : public VExpr { public: Int32DictionaryEqualsExpr(int column_id, int32_t value) @@ -1647,6 +1776,447 @@ TEST_F(NewParquetReaderTest, CreatesParquetColumnMapper) { ASSERT_NE(dynamic_cast(mapper.get()), nullptr); } +TEST(ParquetVariantProjectionTest, ResidualStatisticsGuardPhysicalLeafProjection) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("v", 0, ParquetColumnSchemaKind::VARIANT); + root->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + root->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto root_typed = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + root_typed->children.push_back(std::move(wrapper)); + root->children.push_back(std::move(root_typed)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + EXPECT_TRUE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); + metadata.row_groups[0].columns[2].meta_data.__isset.statistics = false; + EXPECT_FALSE(format::parquet::detail::variant_projection_is_fully_shredded(metadata, *root, + projection)); +} + +TEST(ParquetVariantProjectionTest, FinalizesNestedVariantProjectionRecursively) { + using format::parquet::ParquetColumnSchema; + using format::parquet::ParquetColumnSchemaKind; + auto node = [](std::string name, int32_t local_id, ParquetColumnSchemaKind kind, + int leaf_id = -1) { + auto result = std::make_unique(); + result->name = std::move(name); + result->local_id = local_id; + result->kind = kind; + result->leaf_column_id = leaf_id; + return result; + }; + auto root = node("info", 0, ParquetColumnSchemaKind::STRUCT); + auto variant = node("payload", 0, ParquetColumnSchemaKind::VARIANT); + variant->children.push_back(node("metadata", 0, ParquetColumnSchemaKind::PRIMITIVE, 0)); + variant->children.push_back(node("value", 1, ParquetColumnSchemaKind::PRIMITIVE, 1)); + auto typed_object = node("typed_value", 2, ParquetColumnSchemaKind::STRUCT); + auto wrapper = node("n", 0, ParquetColumnSchemaKind::STRUCT); + wrapper->children.push_back(node("value", 0, ParquetColumnSchemaKind::PRIMITIVE, 2)); + wrapper->children.push_back(node("typed_value", 1, ParquetColumnSchemaKind::PRIMITIVE, 3)); + typed_object->children.push_back(std::move(wrapper)); + variant->children.push_back(std::move(typed_object)); + root->children.push_back(std::move(variant)); + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.back().children.push_back( + format::LocalColumnIndex::local(1)); + + tparquet::RowGroup row_group; + row_group.__set_num_rows(10); + for (int leaf = 0; leaf < 4; ++leaf) { + tparquet::Statistics statistics; + statistics.__set_null_count(leaf == 1 || leaf == 2 ? 10 : 0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(std::move(column_metadata)); + row_group.columns.push_back(std::move(chunk)); + } + tparquet::FileMetaData metadata; + metadata.row_groups.push_back(row_group); + + EXPECT_EQ( + format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &projection), + 1); + EXPECT_FALSE(projection.children[0].project_all_children); + + auto fallback = projection; + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(9); + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &fallback), + 0); + EXPECT_TRUE(fallback.children[0].project_all_children); + + auto repeated = projection; + root->children[0]->max_repetition_level = 1; + EXPECT_EQ(format::parquet::detail::finalize_variant_leaf_projection(metadata, *root, &repeated), + 0); + EXPECT_TRUE(repeated.children[0].project_all_children); +} + +TEST_F(NewParquetReaderTest, ReadsFullyShreddedVariantTypedLeafProjection) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_typed_leaf_projection"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + ASSERT_EQ(remove_nullable(schema[1].type)->get_primitive_type(), TYPE_VARIANT); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(root_typed->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(std::move(projection)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + ASSERT_TRUE(reader->open(request).ok()); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v"}); + size_t rows = 0; + bool eof = false; + while (!eof) { + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + } + ASSERT_EQ(rows, 4096); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("n")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->type->get_primitive_type(), TYPE_INT); + EXPECT_EQ(match->column->size(), rows); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), 0); + const std::array missing_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("missing")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(missing_path).has_value()); + ASSERT_NE(profile.get_counter("VariantDirectLeafPathMisses"), nullptr); + EXPECT_EQ(profile.get_counter("VariantDirectLeafPathMisses")->value(), 1); + const auto first_value = + assert_cast( + assert_cast(*match->column).get_nested_column()) + .get_data()[0]; + + IColumn::Filter keep(rows, 0); + keep[0] = 1; + const ColumnPtr filtered = variants.filter(keep, 1); + const auto& filtered_variants = assert_cast(*filtered); + ASSERT_TRUE(filtered_variants.is_shredded()); + const auto filtered_match = filtered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(filtered_match.has_value()); + EXPECT_EQ(filtered_match->column->size(), 1); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + first_value); + EXPECT_TRUE(variants.clone_resized(0)->empty()); + auto mutable_filtered = variants.clone_resized(variants.size()); + EXPECT_EQ(mutable_filtered->filter(keep), 1); + EXPECT_TRUE(assert_cast(*mutable_filtered).is_shredded()); + + // TableReader detaches mapped output columns before upper expressions run. Detachment must + // preserve an incomplete leaf projection because it has no canonical Variant to materialize. + auto detached = IColumn::mutate(block.get_by_position(0).column); + const auto& detached_variants = assert_cast( + assert_cast(*detached).get_nested_column()); + ASSERT_TRUE(detached_variants.is_shredded()); + ASSERT_TRUE(detached_variants.find_shredded_typed_value(path).has_value()); + + // Adaptive predicate probing cuts retained output columns into proper subsets. Keep that row + // selection in the physical shredded state as well. + const ColumnPtr sliced = variants.cut(1, 2); + const auto& sliced_variants = assert_cast(*sliced); + ASSERT_TRUE(sliced_variants.is_shredded()); + const auto sliced_match = sliced_variants.find_shredded_typed_value(path); + ASSERT_TRUE(sliced_match.has_value()); + ASSERT_EQ(sliced_match->column->size(), 2); + EXPECT_EQ(assert_cast( + assert_cast(*sliced_match->column).get_nested_column()) + .get_data()[0], + first_value + 1); + + const std::array indices {2, 0}; + MutableColumnPtr gathered = variants.clone_empty(); + gathered->insert_indices_from(variants, indices.data(), indices.data() + indices.size()); + const auto& gathered_variants = assert_cast(*gathered); + ASSERT_TRUE(gathered_variants.is_shredded()); + const auto gathered_match = gathered_variants.find_shredded_typed_value(path); + ASSERT_TRUE(gathered_match.has_value()); + ASSERT_EQ(gathered_match->column->size(), indices.size()); + EXPECT_EQ( + assert_cast( + assert_cast(*gathered_match->column).get_nested_column()) + .get_data()[0], + first_value + 2); +} + +TEST_F(NewParquetReaderTest, ShreddedVariantPredicateUsesTypedLeafPageIndexWithRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_page_pruning_with_root_output"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + // The root output deliberately retains the complete wrapper; its predicate may still use the + // typed leaf's page index without converting the output to a leaf-only Variant projection. + request->predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(1, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("FilteredRowsByPage"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByPage")->value(), 0); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 0); + ASSERT_NE(profile.get_counter("VariantDirectLeafRows"), nullptr); + EXPECT_GT(profile.get_counter("VariantDirectLeafRows")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); + ASSERT_NE(profile.get_counter("VariantReconstructionTime"), nullptr); + EXPECT_GT(profile.get_counter("VariantReconstructionTime")->value(), 0); +} + +TEST_F(NewParquetReaderTest, ReadsVariantPredicateLeafBeforeDeferredRootOutput) { + const char* source_root = std::getenv("ROOT"); + ASSERT_NE(source_root, nullptr); + _file_path = std::string(source_root) + + "/regression-test/data/external_table_p0/iceberg/" + "iceberg_variant_shredded.parquet"; + ASSERT_TRUE(std::filesystem::exists(_file_path)); + + RuntimeProfile profile("variant_predicate_leaf_deferred_root"); + auto reader = create_reader(0, -1, &profile); + reader->set_batch_size(1024); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const format::ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* root_typed = find_child(schema[1].children, "typed_value"); + ASSERT_NE(root_typed, nullptr); + const auto* n_wrapper = find_child(root_typed->children, "n"); + ASSERT_NE(n_wrapper, nullptr); + const auto* n_typed = find_child(n_wrapper->children, "typed_value"); + ASSERT_NE(n_typed, nullptr); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[1].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::partial_local(root_typed->local_id)); + predicate_projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(n_wrapper->local_id)); + predicate_projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(n_typed->local_id)); + + auto request = std::make_shared(); + request->predicate_columns.push_back(std::move(predicate_projection)); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[1].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[1].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[1].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_variant_int32_path_greater_than_conjunct(0, "n", 3000)); + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + while (!eof) { + Block block; + block.insert({schema[1].type->create_column(), schema[1].type, "v_predicate"}); + block.insert({schema[1].type->create_column(), schema[1].type, "v_output"}); + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + ASSERT_EQ(block.get_by_position(0).column->size(), batch_rows); + ASSERT_EQ(block.get_by_position(1).column->size(), batch_rows); + if (batch_rows > 0) { + const auto& nullable = + assert_cast(*block.get_by_position(1).column); + auto canonical = IColumn::mutate(nullable.get_nested_column_ptr()); + assert_cast(*canonical).ensure_encoded(); + } + } + EXPECT_EQ(rows, 1095); + ASSERT_NE(profile.get_counter("VariantLeafProjections"), nullptr); + EXPECT_EQ(profile.get_counter("VariantLeafProjections")->value(), 1); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); + ASSERT_NE(profile.get_counter("VariantReconstructedRows"), nullptr); + EXPECT_EQ(profile.get_counter("VariantReconstructedRows")->value(), rows); +} + +TEST_F(NewParquetReaderTest, ReadsStructPredicateChildBeforeDeferredRootOutput) { + write_struct_filter_parquet_file(_file_path); + RuntimeProfile profile("struct_predicate_child_deferred_root"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 1); + ASSERT_EQ(schema[0].children.size(), 2); + + auto predicate_projection = format::LocalColumnIndex::partial_local(schema[0].local_id); + predicate_projection.children.push_back( + format::LocalColumnIndex::local(schema[0].children[0].local_id)); + auto request = std::make_shared(); + request->predicate_columns.push_back(predicate_projection); + request->non_predicate_columns.push_back( + format::LocalColumnIndex::top_level(format::LocalColumnId(schema[0].local_id))); + request->predicate_only_columns.push_back(format::LocalColumnId(schema[0].local_id)); + request->local_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(0)); + request->non_predicate_positions.emplace(format::LocalColumnId(schema[0].local_id), + format::LocalIndex(1)); + request->conjuncts.push_back(create_struct_int32_child_greater_than_conjunct(0, 2)); + ASSERT_TRUE(reader->open(request).ok()); + + format::ColumnDefinition predicate_field; + ASSERT_TRUE(format::project_column_definition(schema[0], predicate_projection, &predicate_field) + .ok()); + size_t total_rows = 0; + std::vector names; + bool eof = false; + while (!eof) { + Block block; + block.insert({predicate_field.type->create_column(), predicate_field.type, "s_predicate"}); + block.insert({schema[0].type->create_column(), schema[0].type, "s_output"}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + total_rows += rows; + ASSERT_EQ(block.get_by_position(0).column->size(), rows); + ASSERT_EQ(block.get_by_position(1).column->size(), rows); + const auto& output_nullable = + assert_cast(*block.get_by_position(1).column); + const auto& output_struct = + assert_cast(output_nullable.get_nested_column()); + ASSERT_EQ(output_struct.tuple_size(), 2); + const auto& name_nullable = assert_cast(output_struct.get_column(1)); + const auto& name_values = + assert_cast(name_nullable.get_nested_column()); + for (size_t row = 0; row < rows; ++row) { + names.push_back(name_values.get_data_at(row).to_string()); + } + } + EXPECT_EQ(total_rows, 2); + EXPECT_EQ(names, (std::vector {"ten", "eleven"})); + ASSERT_NE(profile.get_counter("FilteredRowsByLazyRead"), nullptr); + EXPECT_GT(profile.get_counter("FilteredRowsByLazyRead")->value(), 0); +} + TEST_F(NewParquetReaderTest, CountComplexColumnUsesShapeOnlyPath) { write_nullable_map_parquet_file(_file_path); RuntimeProfile profile("count_map_shape_only_path"); diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index 74781c0e532b31..9ecc1edf6fc582 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -28,6 +29,7 @@ #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/data_type/primitive_type.h" #include "format_v2/parquet/native_schema_desc.h" #include "format_v2/parquet/native_schema_node.h" @@ -35,6 +37,401 @@ #include "format_v2/parquet/parquet_file_context.h" namespace doris::format::parquet { +namespace { + +std::vector unshredded_variant_schema( + std::optional specification_version = 1) { + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + + tparquet::SchemaElement variant; + variant.__set_name("payload"); + variant.__set_num_children(2); + variant.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant.__set_logicalType(tparquet::LogicalType()); + variant.logicalType.__set_VARIANT(tparquet::VariantType()); + if (specification_version.has_value()) { + variant.logicalType.VARIANT.__set_specification_version(*specification_version); + } + + tparquet::SchemaElement metadata; + metadata.__set_name("metadata"); + metadata.__set_type(tparquet::Type::BYTE_ARRAY); + metadata.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + return {root, variant, metadata, value}; +} + +std::vector struct_with_variant_schema() { + auto variant_fields = unshredded_variant_schema(); + variant_fields[0].__set_name("info"); + variant_fields[0].__set_num_children(2); + variant_fields[0].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + variant_fields[1].__set_name("payload"); + + tparquet::SchemaElement root; + root.__set_name("schema"); + root.__set_num_children(1); + tparquet::SchemaElement label; + label.__set_name("label"); + label.__set_type(tparquet::Type::BYTE_ARRAY); + label.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + return {root, + variant_fields[0], + label, + variant_fields[1], + variant_fields[2], + variant_fields[3]}; +} + +std::vector shredded_object_variant_schema(bool signed_integer = true, + bool required_wrapper = true) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_object; + typed_object.__set_name("typed_value"); + typed_object.__set_num_children(1); + typed_object.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement field_wrapper; + field_wrapper.__set_name("field"); + field_wrapper.__set_num_children(2); + field_wrapper.__set_repetition_type(required_wrapper ? tparquet::FieldRepetitionType::REQUIRED + : tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement fallback; + fallback.__set_name("value"); + fallback.__set_type(tparquet::Type::BYTE_ARRAY); + fallback.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_integer; + typed_integer.__set_name("typed_value"); + typed_integer.__set_type(tparquet::Type::INT32); + typed_integer.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + if (!signed_integer) { + typed_integer.__set_logicalType(tparquet::LogicalType()); + typed_integer.logicalType.__set_INTEGER(tparquet::IntType()); + typed_integer.logicalType.INTEGER.__set_bitWidth(32); + typed_integer.logicalType.INTEGER.__set_isSigned(false); + } + schema.insert(schema.end(), {typed_object, field_wrapper, fallback, typed_integer}); + return schema; +} + +std::vector shredded_primitive_variant_schema( + tparquet::SchemaElement typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_name("typed_value"); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_time_variant_schema(bool adjusted_to_utc, + bool millis) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(millis ? tparquet::Type::INT32 : tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIME(tparquet::TimeType()); + typed_value.logicalType.TIME.__set_isAdjustedToUTC(adjusted_to_utc); + typed_value.logicalType.TIME.__set_unit(tparquet::TimeUnit()); + if (millis) { + typed_value.logicalType.TIME.unit.__set_MILLIS(tparquet::MilliSeconds()); + } else { + typed_value.logicalType.TIME.unit.__set_MICROS(tparquet::MicroSeconds()); + } + schema.push_back(std::move(typed_value)); + return schema; +} + +std::vector shredded_array_variant_schema(bool include_value, + bool include_typed_value) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + + tparquet::SchemaElement typed_array; + typed_array.__set_name("typed_value"); + typed_array.__set_num_children(1); + typed_array.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_array.__set_converted_type(tparquet::ConvertedType::LIST); + tparquet::SchemaElement list; + list.__set_name("list"); + list.__set_num_children(1); + list.__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + tparquet::SchemaElement element; + element.__set_name("element"); + element.__set_num_children(static_cast(include_value) + + static_cast(include_typed_value)); + element.__set_repetition_type(tparquet::FieldRepetitionType::REQUIRED); + schema.insert(schema.end(), {typed_array, list, element}); + if (include_value) { + tparquet::SchemaElement value; + value.__set_name("value"); + value.__set_type(tparquet::Type::BYTE_ARRAY); + value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(value)); + } + if (include_typed_value) { + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT32); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + schema.push_back(std::move(typed_value)); + } + return schema; +} + +} // namespace + +TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { + for (const auto version : {std::optional {}, std::optional {1}}) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(unshredded_variant_schema(version)).ok()); + descriptor.assign_ids(); + + const auto* native_variant = descriptor.get_column(0); + ASSERT_NE(native_variant, nullptr); + // Native readers must keep seeing the physical group. The logical Variant mapping belongs + // to ParquetColumnSchema and must not make this group look like an unindexed scalar leaf. + EXPECT_EQ(remove_nullable(native_variant->data_type)->get_primitive_type(), TYPE_STRUCT); + ASSERT_EQ(native_variant->children.size(), 2); + EXPECT_EQ(native_variant->children[0].physical_column_index, 0); + EXPECT_EQ(native_variant->children[1].physical_column_index, 1); + EXPECT_EQ(descriptor.physical_fields_size(), 2); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); + EXPECT_NE(typeid_cast(remove_nullable(fields[0]->type).get()), + nullptr); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_EQ(fields[0]->children[0]->name, "metadata"); + EXPECT_EQ(fields[0]->children[1]->name, "value"); + } +} + +TEST(ParquetSchemaTest, NativeSchemaAcceptsRequiredAndOptionalVariantGroups) { + for (const auto repetition : + {tparquet::FieldRepetitionType::REQUIRED, tparquet::FieldRepetitionType::OPTIONAL}) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(repetition); + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + ASSERT_TRUE(status.ok()) << status; + ASSERT_NE(descriptor.get_column(0), nullptr); + } +} + +TEST(ParquetSchemaTest, NestedVariantPropagatesIntoParentLogicalType) { + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(struct_with_variant_schema()).ok()); + descriptor.assign_ids(); + + std::vector> fields; + const auto status = build_parquet_column_schema(descriptor, &fields); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(fields.size(), 1); + const auto* info_type = + assert_cast(remove_nullable(fields[0]->type).get()); + ASSERT_EQ(info_type->get_elements().size(), 2); + EXPECT_EQ(remove_nullable(info_type->get_elements()[1])->get_primitive_type(), TYPE_VARIANT); +} + +TEST(ParquetSchemaTest, NativeSchemaRejectsUnsupportedVariantVersionAndMalformedLayout) { + NativeFieldDescriptor descriptor; + const auto version_status = descriptor.parse_from_thrift(unshredded_variant_schema(2)); + EXPECT_TRUE(version_status.is()) << version_status; + EXPECT_NE(version_status.to_string().find("Variant specification version 2"), + std::string::npos); + + auto missing_metadata = unshredded_variant_schema(); + missing_metadata[2].__set_name("not_metadata"); + const auto layout_status = descriptor.parse_from_thrift(missing_metadata); + EXPECT_TRUE(layout_status.is()) << layout_status; + EXPECT_NE(layout_status.to_string().find("metadata"), std::string::npos); + + auto optional_unshredded_value = unshredded_variant_schema(); + optional_unshredded_value[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + const auto repetition_status = descriptor.parse_from_thrift(optional_unshredded_value); + EXPECT_TRUE(repetition_status.is()) << repetition_status; + EXPECT_NE(repetition_status.to_string().find("required BYTE_ARRAY"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantPreservesUtcTimestampInstant) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(true); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_MICROS(tparquet::MicroSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + const auto* variant = descriptor.get_column(0); + ASSERT_EQ(variant->children.size(), 3); + EXPECT_EQ(remove_nullable(variant->children[2].data_type)->get_primitive_type(), + TYPE_TIMESTAMPTZ); +} + +TEST(ParquetSchemaTest, NativeVariantValidatesEveryShreddedWrapperAndScalar) { + NativeFieldDescriptor descriptor; + const auto unsigned_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(false, true)); + EXPECT_TRUE(unsigned_status.is()) << unsigned_status; + EXPECT_NE(unsigned_status.to_string().find("unsigned"), std::string::npos); + + const auto optional_wrapper_status = + descriptor.parse_from_thrift(shredded_object_variant_schema(true, false)); + EXPECT_TRUE(optional_wrapper_status.is()) << optional_wrapper_status; + EXPECT_NE(optional_wrapper_status.to_string().find("wrapper"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsDuplicateObjectFieldNames) { + auto schema = shredded_object_variant_schema(); + schema[4].__set_num_children(2); + schema.insert(schema.end(), {schema[5], schema[6], schema[7]}); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("duplicate"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedPrimitiveTypePairs) { + std::vector invalid_typed_values; + + tparquet::SchemaElement int96; + int96.__set_type(tparquet::Type::INT96); + invalid_typed_values.push_back(int96); + + tparquet::SchemaElement fixed_binary; + fixed_binary.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + fixed_binary.__set_type_length(16); + invalid_typed_values.push_back(fixed_binary); + + tparquet::SchemaElement json; + json.__set_type(tparquet::Type::BYTE_ARRAY); + json.__set_logicalType(tparquet::LogicalType()); + json.logicalType.__set_JSON(tparquet::JsonType()); + invalid_typed_values.push_back(json); + + tparquet::SchemaElement float16; + float16.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + float16.__set_type_length(2); + float16.__set_logicalType(tparquet::LogicalType()); + float16.logicalType.__set_FLOAT16(tparquet::Float16Type()); + invalid_typed_values.push_back(float16); + + tparquet::SchemaElement mismatched_integer; + mismatched_integer.__set_type(tparquet::Type::INT64); + mismatched_integer.__set_logicalType(tparquet::LogicalType()); + mismatched_integer.logicalType.__set_INTEGER(tparquet::IntType()); + mismatched_integer.logicalType.INTEGER.__set_bitWidth(16); + mismatched_integer.logicalType.INTEGER.__set_isSigned(true); + invalid_typed_values.push_back(mismatched_integer); + + tparquet::SchemaElement mismatched_decimal; + mismatched_decimal.__set_type(tparquet::Type::INT32); + mismatched_decimal.__set_logicalType(tparquet::LogicalType()); + mismatched_decimal.logicalType.__set_DECIMAL(tparquet::DecimalType()); + mismatched_decimal.logicalType.DECIMAL.__set_precision(10); + mismatched_decimal.logicalType.DECIMAL.__set_scale(2); + invalid_typed_values.push_back(mismatched_decimal); + + tparquet::SchemaElement bad_uuid; + bad_uuid.__set_type(tparquet::Type::FIXED_LEN_BYTE_ARRAY); + bad_uuid.__set_type_length(15); + bad_uuid.__set_logicalType(tparquet::LogicalType()); + bad_uuid.logicalType.__set_UUID(tparquet::UUIDType()); + invalid_typed_values.push_back(bad_uuid); + + for (auto& typed_value : invalid_typed_values) { + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift( + shredded_primitive_variant_schema(std::move(typed_value))); + EXPECT_FALSE(status.ok()) << "unsupported Variant typed_value pair was accepted"; + } +} + +TEST(ParquetSchemaTest, NativeVariantRejectsRepeatedOuterGroup) { + auto schema = unshredded_variant_schema(); + schema[1].__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("repeated"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantAcceptsOmittedShreddedWrapperChildren) { + NativeFieldDescriptor descriptor; + + auto value_only_object = shredded_object_variant_schema(); + value_only_object[5].__set_num_children(1); + value_only_object.pop_back(); + ASSERT_TRUE(descriptor.parse_from_thrift(value_only_object).ok()); + + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(true, false)).ok()); + ASSERT_TRUE(descriptor.parse_from_thrift(shredded_array_variant_schema(false, true)).ok()); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsNanosBeforeProjectionChoice) { + auto schema = unshredded_variant_schema(); + schema[1].__set_num_children(3); + schema[3].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + tparquet::SchemaElement typed_value; + typed_value.__set_name("typed_value"); + typed_value.__set_type(tparquet::Type::INT64); + typed_value.__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_TIMESTAMP(tparquet::TimestampType()); + typed_value.logicalType.TIMESTAMP.__set_isAdjustedToUTC(false); + typed_value.logicalType.TIMESTAMP.__set_unit(tparquet::TimeUnit()); + typed_value.logicalType.TIMESTAMP.unit.__set_NANOS(tparquet::NanoSeconds()); + schema.push_back(std::move(typed_value)); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift(schema); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("TIMESTAMP(NANOS)"), std::string::npos); +} + +TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedTimeAnnotations) { + NativeFieldDescriptor descriptor; + const auto adjusted_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(true, false)); + EXPECT_TRUE(adjusted_status.is()) << adjusted_status; + EXPECT_NE(adjusted_status.to_string().find("isAdjustedToUTC"), std::string::npos); + + const auto millis_status = + descriptor.parse_from_thrift(shredded_time_variant_schema(false, true)); + EXPECT_TRUE(millis_status.is()) << millis_status; + EXPECT_NE(millis_status.to_string().find("TIME(MILLIS)"), std::string::npos); +} + TEST(ParquetSchemaTest, NativeMetadataAcceptsRequiredRootWithoutColumns) { tparquet::SchemaElement root; root.__set_name("schema"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 740892367bb04e..1a13e8ecea362e 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -37,10 +37,12 @@ #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_time.h" +#include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" +#include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_column_schema.h" @@ -196,6 +198,107 @@ class MetadataBoundsProbeExpr final : public VExpr { bool _require_false_boolean; const std::string _expr_name = "MetadataBoundsProbeExpr"; }; + +class VariantPathTestExpr final : public VExpr { +public: + VariantPathTestExpr(std::string name, DataTypePtr type, + TExprNodeType::type node_type = TExprNodeType::FUNCTION_CALL) + : VExpr(std::move(type), false), _name(std::move(name)) { + set_node_type(node_type); + } + + const std::string& expr_name() const override { return _name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("VariantPathTestExpr is metadata-only"); + } + +private: + std::string _name; +}; + +VExprContextSPtr variant_path_gt_conjunct(int32_t literal_value, + bool add_narrowing_intermediate_cast = false, + bool decimal_comparison = false) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + DataTypePtr comparison_type = decimal_comparison + ? DataTypePtr(std::make_shared(38, 9)) + : DataTypePtr(std::make_shared()); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + if (add_narrowing_intermediate_cast) { + auto narrowing = std::make_shared( + "CAST", make_nullable(std::make_shared()), TExprNodeType::CAST_EXPR); + narrowing->add_child(element_at); + cast->add_child(narrowing); + } else { + cast->add_child(element_at); + } + auto literal = decimal_comparison + ? VLiteral::create_shared( + comparison_type, + Field::create_field(Decimal128V3( + static_cast<__int128>(literal_value) * 1'000'000'000))) + : VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_float_gt_conjunct(float literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared(comparison_type, + Field::create_field(literal_value)); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + +VExprContextSPtr variant_path_string_gt_conjunct(std::string literal_value) { + auto slot = VSlotRef::create_shared(0, 0, -1, + make_nullable(std::make_shared()), "v"); + auto key = VLiteral::create_shared(std::make_shared(), + Field::create_field("col")); + auto element_at = std::make_shared( + "element_at", make_nullable(std::make_shared())); + element_at->add_child(slot); + element_at->add_child(key); + auto comparison_type = std::make_shared(); + auto cast = std::make_shared("CAST", make_nullable(comparison_type), + TExprNodeType::CAST_EXPR); + cast->add_child(element_at); + auto literal = VLiteral::create_shared( + comparison_type, Field::create_field(std::move(literal_value))); + auto gt = std::make_shared("gt", std::make_shared(), + TExprNodeType::BINARY_PRED); + gt->add_child(cast); + gt->add_child(literal); + return VExprContext::create_shared(std::move(gt)); +} + VExprContextSPtrs bloom_conjuncts(DataTypePtr data_type, std::vector values) { return {VExprContext::create_shared( std::make_shared(0, std::move(data_type), std::move(values)))}; @@ -331,8 +434,9 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru page_indexes.emplace(0, std::move(page_index)); std::vector selected_ranges; std::map skip_plans; + tparquet::RowGroup row_group; EXPECT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, + metadata, row_group, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, nullptr) .ok()); return selected_ranges; @@ -650,8 +754,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_EQ(selected_ranges.size(), 1); @@ -664,8 +768,8 @@ TEST(NativeParquetStatisticsTest, TypeDefinedBoundsRequireSupportedColumnOrder) .ok()); EXPECT_TRUE(selected_row_groups.empty()); ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -737,8 +841,8 @@ TEST(NativeParquetStatisticsTest, ZonemapPruningIgnoresDisabledSessionSwitch) { std::vector selected_ranges; std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 1, &selected_ranges, &skip_plans, - nullptr, nullptr, &state) + metadata, metadata.row_groups[0], page_indexes, schema, request, 1, + &selected_ranges, &skip_plans, nullptr, nullptr, &state) .ok()); EXPECT_TRUE(selected_ranges.empty()); } @@ -780,8 +884,8 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) std::map skip_plans; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( - metadata, page_indexes, schema, request, 10, &selected_ranges, - &skip_plans, nullptr) + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 10, + &selected_ranges, &skip_plans, nullptr) .ok()); // ColumnIndex is optional. An impossible all-null claim must fall back to reading the // ten-row data page instead of proving that no value can satisfy the predicate. @@ -791,5 +895,320 @@ TEST(NativeParquetStatisticsTest, ContradictoryAllNullPageCountsDisablePruning) } } +TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto primitive = [](std::string name, int local_id, int leaf_id) { + auto schema = std::make_unique(); + schema->name = std::move(name); + schema->local_id = local_id; + schema->leaf_column_id = leaf_id; + schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::INT32; + return schema; + }; + auto bytes = [&](std::string name, int local_id, int leaf_id) { + auto schema = primitive(std::move(name), local_id, leaf_id); + schema->type = make_nullable(std::make_shared()); + schema->type_descriptor.doris_type = schema->type; + schema->type_descriptor.physical_type = tparquet::Type::BYTE_ARRAY; + return schema; + }; + + auto variant = std::make_unique(); + variant->name = "v"; + variant->local_id = 0; + variant->kind = format::parquet::ParquetColumnSchemaKind::VARIANT; + variant->type = make_nullable(std::make_shared()); + variant->children.push_back(bytes("metadata", 0, 0)); + variant->children.push_back(bytes("value", 1, 1)); + auto typed_object = std::make_unique(); + typed_object->name = "typed_value"; + typed_object->local_id = 2; + typed_object->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + auto field = std::make_unique(); + field->name = "col"; + field->local_id = 0; + field->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + field->children.push_back(bytes("value", 0, 2)); + field->children.push_back(primitive("typed_value", 1, 3)); + typed_object->children.push_back(std::move(field)); + variant->children.push_back(std::move(typed_object)); + std::vector> schema; + schema.push_back(std::move(variant)); + + auto chunk = [&](tparquet::Type::type type, int64_t num_values, int64_t null_count, + std::optional min_value = std::nullopt, + std::optional max_value = std::nullopt) { + tparquet::Statistics statistics; + statistics.__set_null_count(null_count); + if (min_value.has_value() && max_value.has_value()) { + statistics.__set_min_value(encode_int32(*min_value)); + statistics.__set_max_value(encode_int32(*max_value)); + } + tparquet::ColumnMetaData metadata; + metadata.__set_type(type); + metadata.__set_num_values(num_values); + metadata.__set_statistics(std::move(statistics)); + tparquet::ColumnChunk result; + result.__set_meta_data(std::move(metadata)); + return result; + }; + tparquet::RowGroup row_group; + row_group.__set_num_rows(100); + row_group.__set_columns({chunk(tparquet::Type::BYTE_ARRAY, 100, 0), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::BYTE_ARRAY, 100, 100), + chunk(tparquet::Type::INT32, 100, 0, 1, 200)}); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_row_groups({row_group}); + metadata.__set_column_orders({order, order, order, order}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + request.conjuncts = {variant_path_gt_conjunct(50)}; + + auto footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + std::vector selected_row_groups; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + + auto leaf_projection = format::LocalColumnIndex::partial_local(0); + auto typed_object_projection = format::LocalColumnIndex::partial_local(2); + auto field_projection = format::LocalColumnIndex::partial_local(0); + field_projection.children.push_back(format::LocalColumnIndex::local(1)); + typed_object_projection.children.push_back(std::move(field_projection)); + leaf_projection.children.push_back(std::move(typed_object_projection)); + request.predicate_columns = {std::move(leaf_projection)}; + for (int leaf = 0; leaf < 4; ++leaf) { + footer_only_metadata.row_groups[0].columns[leaf].meta_data.__set_total_compressed_size( + (leaf + 1) * 10); + } + format::parquet::ParquetPruningStats leaf_pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + &leaf_pruning_stats, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + EXPECT_EQ(leaf_pruning_stats.filtered_bytes, 40); + + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + request.conjuncts = {variant_path_gt_conjunct(50)}; + + // Missing typed statistics provide no proof and must retain the row group. + footer_only_metadata.row_groups[0].columns[3].meta_data.__isset.statistics = false; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A populated fallback for the same path invalidates both footer and page pruning. + footer_only_metadata = metadata; + footer_only_metadata.row_groups[0].columns[3].meta_data.statistics.__set_max_value( + encode_int32(2)); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + // A contradictory non-repeated value count cannot prove that every row lacks fallback bytes. + footer_only_metadata.row_groups[0].columns[2].meta_data.__set_num_values(99); + footer_only_metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex typed_pages; + typed_pages.column_index.__set_min_values({encode_int32(1), encode_int32(100)}); + typed_pages.column_index.__set_max_values({encode_int32(2), encode_int32(200)}); + typed_pages.column_index.__set_null_pages({false, false}); + typed_pages.column_index.__set_null_counts({0, 0}); + tparquet::PageLocation first; + first.__set_offset(0); + first.__set_compressed_page_size(10); + first.__set_first_row_index(0); + tparquet::PageLocation second; + second.__set_offset(10); + second.__set_compressed_page_size(10); + second.__set_first_row_index(50); + typed_pages.offset_index.__set_page_locations({first, second}); + std::unordered_map page_indexes; + page_indexes.emplace(3, std::move(typed_pages)); + + std::vector selected_ranges; + std::map skip_plans; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::can_use_parquet_page_index(request, nullptr)); + TQueryOptions query_options; + query_options.__set_enable_expr_zonemap_filter(false); + RuntimeState generic_zonemap_disabled {query_options, TQueryGlobals()}; + EXPECT_TRUE(format::parquet::can_use_parquet_page_index(request, &generic_zonemap_disabled)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, &pruning_stats, nullptr, + &generic_zonemap_disabled) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + EXPECT_EQ(pruning_stats.page_index_read_calls, 1); + EXPECT_EQ(pruning_stats.filtered_page_rows, 50); + + // Direct Variant numeric comparisons coerce integral literals to a wide DECIMAL domain. + request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 50); + EXPECT_EQ(selected_ranges[0].length, 50); + + // Bounds for the raw INT32 typed leaf are not valid for CAST(CAST(v['col'] AS TINYINT) AS INT). + request.conjuncts = {variant_path_gt_conjunct(50, true)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // Raw binary and UUID bounds are physical bytes, while the residual Variant-to-STRING cast + // compares their rendered values. Those domains differ, so neither footer nor page metadata + // may exclude a row solely from the raw byte interval. + auto assert_binary_identity_does_not_prune = [&](bool is_uuid) { + auto* binary_leaf = schema[0]->children[2]->children[0]->children[1].get(); + binary_leaf->type = make_nullable(std::make_shared()); + binary_leaf->type_descriptor = {}; + binary_leaf->type_descriptor.doris_type = binary_leaf->type; + binary_leaf->type_descriptor.physical_type = + is_uuid ? tparquet::Type::FIXED_LEN_BYTE_ARRAY : tparquet::Type::BYTE_ARRAY; + binary_leaf->type_descriptor.fixed_length = is_uuid ? 16 : -1; + binary_leaf->type_descriptor.is_string_like = true; + binary_leaf->type_descriptor.is_uuid = is_uuid; + + auto& binary_chunk = metadata.row_groups[0].columns[3].meta_data; + binary_chunk.__set_type(binary_leaf->type_descriptor.physical_type); + binary_chunk.statistics.__set_min_value("a"); + binary_chunk.statistics.__set_max_value("b"); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_string_gt_conjunct("z")}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex binary_pages; + binary_pages.column_index.__set_min_values({"a"}); + binary_pages.column_index.__set_max_values({"b"}); + binary_pages.column_index.__set_null_pages({false}); + binary_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation binary_location; + binary_location.__set_offset(0); + binary_location.__set_compressed_page_size(10); + binary_location.__set_first_row_index(0); + binary_pages.offset_index.__set_page_locations({binary_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(binary_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + }; + assert_binary_identity_does_not_prune(false); + assert_binary_identity_does_not_prune(true); + + // Parquet floating min/max omits NaN values. Without an explicit no-NaN proof, [0, NaN] + // cannot be represented by max=0 and must not prune a Variant comparison that retains NaN. + auto encode_float = [](float value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto* float_leaf = schema[0]->children[2]->children[0]->children[1].get(); + float_leaf->type = make_nullable(std::make_shared()); + float_leaf->type_descriptor.doris_type = float_leaf->type; + float_leaf->type_descriptor.physical_type = tparquet::Type::FLOAT; + auto& float_chunk = metadata.row_groups[0].columns[3].meta_data; + float_chunk.__set_type(tparquet::Type::FLOAT); + float_chunk.statistics.__set_min_value(encode_float(0.0F)); + float_chunk.statistics.__set_max_value(encode_float(0.0F)); + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(100); + request.conjuncts = {variant_path_float_gt_conjunct(1.0F)}; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, nullptr, + nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + format::parquet::NativeParquetPageIndex float_pages; + float_pages.column_index.__set_min_values({encode_float(0.0F)}); + float_pages.column_index.__set_max_values({encode_float(0.0F)}); + float_pages.column_index.__set_null_pages({false}); + float_pages.column_index.__set_null_counts({0}); + tparquet::PageLocation float_location; + float_location.__set_offset(0); + float_location.__set_compressed_page_size(10); + float_location.__set_first_row_index(0); + float_pages.offset_index.__set_page_locations({float_location}); + page_indexes.clear(); + page_indexes.emplace(3, std::move(float_pages)); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + + // A fallback value in the same row group may have a different Variant type. In that case the + // typed bounds cannot prove anything about the SQL comparison, so all pages must be read. + metadata.row_groups[0].columns[2].meta_data.statistics.__set_null_count(99); + request.conjuncts = {variant_path_gt_conjunct(50)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); +} + } // namespace } // namespace doris diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp new file mode 100644 index 00000000000000..40c18635aad339 --- /dev/null +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -0,0 +1,927 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "format_v2/parquet/reader/variant_column_reader.h" + +#include + +#include +#include +#include + +#include "core/assert_cast.h" +#include "core/column/column_array.h" +#include "core/column/column_nullable.h" +#include "core/column/column_string.h" +#include "core/column/column_struct.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "core/value/variant/variant_parquet_encoding.h" +#include "exprs/function/function_variant_element_v2.h" +#include "format_v2/parquet/parquet_column_schema.h" + +namespace doris::format::parquet { +namespace { + +MutableColumnPtr nullable_strings(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnString::create(); + auto null_map = ColumnUInt8::create(); + for (size_t row = 0; row < values.size(); ++row) { + data->insert_data(values[row].data, values[row].size); + null_map->get_data().push_back(nulls[row]); + } + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +ParquetColumnSchema unshredded_schema() { + ParquetColumnSchema schema; + schema.name = "payload"; + schema.kind = ParquetColumnSchemaKind::VARIANT; + schema.type = make_nullable(std::make_shared()); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary}, Strings {"metadata", "value"})); + + auto metadata = std::make_unique(); + metadata->name = "metadata"; + metadata->kind = ParquetColumnSchemaKind::PRIMITIVE; + metadata->type = binary; + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = binary; + schema.children.push_back(std::move(metadata)); + schema.children.push_back(std::move(value)); + return schema; +} + +ParquetColumnSchema shredded_int64_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::make_shared()); + typed->type_descriptor.integer_bit_width = 64; + schema.children.push_back(std::move(typed)); + const auto binary = make_nullable(std::make_shared()); + schema.variant_physical_type = make_nullable(std::make_shared( + DataTypes {binary, binary, make_nullable(std::make_shared())}, + Strings {"metadata", "value", "typed_value"})); + return schema; +} + +ParquetColumnSchema shredded_object_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + + auto field = std::make_unique(); + field->name = "a"; + field->kind = ParquetColumnSchemaKind::STRUCT; + auto field_typed = std::make_unique(); + field_typed->name = "typed_value"; + field_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + field_typed->type = make_nullable(std::make_shared()); + field_typed->type_descriptor.integer_bit_width = 64; + field->children.push_back(std::move(field_typed)); + typed->children.push_back(std::move(field)); + schema.children.push_back(std::move(typed)); + return schema; +} + +ParquetColumnSchema shredded_binary_object_schema() { + auto schema = shredded_object_schema(); + auto* leaf = schema.children.back()->children[0]->children[0].get(); + leaf->type = make_nullable(std::make_shared()); + return schema; +} + +ParquetColumnSchema shredded_array_schema() { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::LIST; + auto element = std::make_unique(); + element->name = "element"; + element->kind = ParquetColumnSchemaKind::STRUCT; + auto element_typed = std::make_unique(); + element_typed->name = "typed_value"; + element_typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + element_typed->type = make_nullable(std::make_shared()); + element_typed->type_descriptor.integer_bit_width = 64; + element->children.push_back(std::move(element_typed)); + typed->children.push_back(std::move(element)); + schema.children.push_back(std::move(typed)); + return schema; +} + +MutableColumnPtr shredded_int64_physical(const std::vector& values) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + std::vector metadata_rows(values.size(), metadata); + std::vector empty_values(values.size(), {ignored.data(), 0}); + std::vector present(values.size(), 0); + std::vector absent(values.size(), 1); + MutableColumns fields; + fields.push_back(nullable_strings(metadata_rows, present)); + fields.push_back(nullable_strings(empty_values, absent)); + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().resize_fill(values.size(), 0); + return ColumnNullable::create(std::move(structure), std::move(root_nulls)); +} + +MutableColumnPtr projected_shredded_object_physical(const std::vector& values, + const IColumn** decoded_leaf = nullptr) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().resize_fill(values.size(), 0); + MutableColumnPtr leaf = ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + if (decoded_leaf != nullptr) { + *decoded_leaf = leaf.get(); + } + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + +MutableColumnPtr projected_two_field_object_physical(const std::vector& first, + const std::vector& second) { + DORIS_CHECK(first.size() == second.size()); + auto wrapper = [](const std::vector& values) { + auto integers = ColumnInt64::create(); + integers->get_data().assign(values.begin(), values.end()); + auto leaf = + ColumnNullable::create(std::move(integers), ColumnUInt8::create(values.size(), 0)); + MutableColumns fields; + fields.push_back(std::move(leaf)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(values.size(), 0)); + }; + + MutableColumns object_fields; + object_fields.push_back(wrapper(first)); + object_fields.push_back(wrapper(second)); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(first.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(first.size(), 0)); +} + +MutableColumnPtr projected_wide_object_physical(size_t field_count, int64_t value) { + MutableColumns object_fields; + object_fields.reserve(field_count); + for (size_t field = 0; field < field_count; ++field) { + auto integers = ColumnInt64::create(); + integers->insert_value(value + field); + MutableColumns wrapper_fields; + wrapper_fields.push_back( + ColumnNullable::create(std::move(integers), ColumnUInt8::create(1, 0))); + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + } + MutableColumns root_fields; + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return ColumnNullable::create(ColumnStruct::create(std::move(root_fields)), + ColumnUInt8::create(1, 0)); +} + +} // namespace + +TEST(VariantColumnReaderTest, UnshreddedRowsPreserveSqlNullAndVariantNull) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata, metadata, metadata}, {0, 0, 0})); + fields.push_back(nullable_strings( + {{int_seven.data(), int_seven.size()}, {ignored.data(), 0}, {ignored.data(), 0}}, + {0, 1, 1})); + auto physical_struct = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 1, 0}); + auto physical = ColumnNullable::create(std::move(physical_struct), std::move(root_nulls)); + + auto output_type = make_nullable(std::make_shared()); + auto output = output_type->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(output->size(), 3); + + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 1, 0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_TRUE(variants.is_shredded()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + +TEST(VariantColumnReaderTest, RequiredPhysicalGroupAppendsToNullableExternalSlot) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical = ColumnStruct::create(std::move(fields)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, ShreddedIntegerKeepsDeclaredPhysicalWidth) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + auto integers = ColumnInt64::create(); + integers->get_data().push_back(42); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_int64_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_EQ(variants.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { + VariantBatchBuilder first_builder; + auto first_row = first_builder.begin_row(); + auto first_object = first_row.start_object(); + first_object.add_key(StringRef("alpha")); + first_row.add_int(1); + first_object.finish(); + first_row.finish(); + auto first = first_builder.finish_batch(); + + VariantBatchBuilder second_builder; + auto second_row = second_builder.begin_row(); + auto second_object = second_row.start_object(); + second_object.add_key(StringRef("beta")); + second_row.add_int(2); + second_object.finish(); + second_row.finish(); + auto second = second_builder.finish_batch(); + + const VariantRef first_value = first.value_at(0); + const VariantRef second_value = second.value_at(0); + MutableColumns fields; + fields.push_back(nullable_strings({{first_value.metadata.data, first_value.metadata.size}, + {second_value.metadata.data, second_value.metadata.size}}, + {0, 0})); + fields.push_back(nullable_strings({{first_value.value.data, first_value.value.size}, + {second_value.value.data, second_value.value.size}}, + {0, 0})); + auto structure = ColumnStruct::create(std::move(fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().assign({0, 0}); + auto physical = ColumnNullable::create(std::move(structure), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(unshredded_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("alpha"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("beta"), &field)); + EXPECT_EQ(field.get_int(), 2); +} + +TEST(VariantColumnReaderTest, ShreddedObjectFieldMayOmitResidualValueColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 9); + EXPECT_EQ(field.primitive_id(), VariantPrimitiveId::INT64); +} + +TEST(VariantColumnReaderTest, ShreddedTypedPathReusesDecodedLeafColumn) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().push_back(9); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(0); + MutableColumnPtr typed_leaf = + ColumnNullable::create(std::move(integers), std::move(integer_nulls)); + const IColumn* const decoded_typed_leaf = typed_leaf.get(); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(std::move(typed_leaf)); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(0); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(shredded_path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), decoded_typed_leaf); + + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + + const auto& extracted_nullable = assert_cast(*extracted); + const auto& extracted_variant = + assert_cast(extracted_nullable.get_nested_column()); + ASSERT_TRUE(extracted_variant.is_typed()); + EXPECT_EQ(&extracted_variant.typed_column(), decoded_typed_leaf); + const auto& extracted_typed = + assert_cast(extracted_variant.typed_column()); + EXPECT_EQ(assert_cast(extracted_typed.get_nested_column()).get_data()[0], + 9); + EXPECT_TRUE(variants.is_shredded()); +} + +TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializing) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->local_id)); + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(schema.children[2]->children[0]->local_id)); + projection.children.back().children.back().children.push_back(format::LocalColumnIndex::local( + schema.children[2]->children[0]->children[0]->local_id)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + ASSERT_EQ(plan.variant_state_schema.use_count(), 1); + + auto output = make_nullable(std::make_shared())->create_column(); + const IColumn* first_decoded_leaf = nullptr; + ASSERT_TRUE( + materialize_variant_columns( + plan, projected_shredded_object_physical({10, 20}, &first_decoded_leaf), output) + .ok()); + EXPECT_EQ(plan.variant_state_schema.use_count(), 2); + const auto append_status = + materialize_variant_columns(plan, projected_shredded_object_physical({30}), output); + ASSERT_TRUE(append_status.ok()) << append_status; + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + ASSERT_EQ(variants.size(), 3); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(match->column.get(), first_decoded_leaf); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({10, 20, 30})); + + IColumn::Filter filter {1, 0, 1}; + ColumnPtr filtered = output->filter(filter, 2); + EXPECT_EQ(filtered->size(), 2); + EXPECT_EQ(plan.variant_state_schema.use_count(), 3); +} + +TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { + constexpr size_t width = 64; + constexpr size_t batch_count = 16; + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(typed->local_id)); + for (size_t field = 0; field < width; ++field) { + auto wrapper = std::make_unique(); + wrapper->name = "field_" + std::to_string(field); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = static_cast(field); + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + typed->children.push_back(std::move(wrapper)); + + auto wrapper_projection = format::LocalColumnIndex::partial_local(static_cast(field)); + wrapper_projection.children.push_back(format::LocalColumnIndex::local(0)); + projection.children.back().children.push_back(std::move(wrapper_projection)); + } + schema.children.push_back(std::move(typed)); + + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + for (size_t batch = 0; batch < batch_count; ++batch) { + ASSERT_TRUE(materialize_variant_columns( + plan, projected_wide_object_physical(width, batch * width), output) + .ok()); + } + ASSERT_EQ(output->size(), batch_count); + ASSERT_EQ(plan.variant_state_schema.use_count(), 2); + + // Holding derived slices makes schema ownership observable: every state must retain the same + // reader-scoped schema instead of allocating a width-sized clone for each row selection. + std::vector slices; + slices.reserve(batch_count); + for (size_t row = 0; row < batch_count; ++row) { + slices.push_back(output->cut(row, 1)); + } + EXPECT_EQ(plan.variant_state_schema.use_count(), 2 + batch_count); +} + +TEST(VariantColumnReaderTest, RetainedSchemaFollowsDecodedProjectionOrder) { + auto schema = unshredded_schema(); + schema.local_id = 0; + schema.children[0]->local_id = 0; + schema.children[1]->local_id = 1; + + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::STRUCT; + typed->local_id = 2; + auto make_wrapper = [](std::string name, int local_id) { + auto wrapper = std::make_unique(); + wrapper->name = std::move(name); + wrapper->kind = ParquetColumnSchemaKind::STRUCT; + wrapper->local_id = local_id; + auto leaf = std::make_unique(); + leaf->name = "typed_value"; + leaf->kind = ParquetColumnSchemaKind::PRIMITIVE; + leaf->local_id = 0; + leaf->type = make_nullable(std::make_shared()); + leaf->type_descriptor.integer_bit_width = 64; + wrapper->children.push_back(std::move(leaf)); + return wrapper; + }; + typed->children.push_back(make_wrapper("z", 0)); + typed->children.push_back(make_wrapper("a", 1)); + schema.children.push_back(std::move(typed)); + + auto projection = format::LocalColumnIndex::partial_local(schema.local_id); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + for (int local_id : {1, 0}) { + projection.children.back().children.push_back( + format::LocalColumnIndex::partial_local(local_id)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + } + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_two_field_object_physical({11}, {22}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array a_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const std::array z_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("z")}}; + const auto a = variants.find_shredded_typed_value(a_path); + const auto z = variants.find_shredded_typed_value(z_path); + ASSERT_TRUE(a.has_value()); + ASSERT_TRUE(z.has_value()); + EXPECT_EQ(assert_cast( + assert_cast(*a->column).get_nested_column()) + .get_data()[0], + 11); + EXPECT_EQ(assert_cast( + assert_cast(*z->column).get_nested_column()) + .get_data()[0], + 22); +} + +TEST(VariantColumnReaderTest, AmbiguousTypedIdentityRequiresCanonicalMaterialization) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings({StringRef("abc")}, {0})); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(1, 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), ColumnUInt8::create(1, 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto physical = ColumnNullable::create(std::move(root), ColumnUInt8::create(1, 0)); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_binary_object_schema(), *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + EXPECT_FALSE(variants.find_shredded_typed_value(path).has_value()); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_binary(), StringRef("abc")); +} + +TEST(VariantColumnReaderTest, MaterializedCacheParticipatesInMemoryAccounting) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({42, 43}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const size_t physical_bytes = variants.byte_size(); + const size_t physical_allocated = variants.allocated_bytes(); + + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); + EXPECT_GT(variants.byte_size(), physical_bytes); + EXPECT_GT(variants.allocated_bytes(), physical_allocated); +} + +TEST(VariantColumnReaderTest, MaterializedShreddedCopiesDetachBeforeMutation) { + auto first_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({10, 20}), + first_output) + .ok()); + const auto& first = assert_cast( + assert_cast(*first_output).get_nested_column()); + EXPECT_EQ(first.get_value_ref(0).get_int(), 10); + + auto cloned = first.clone_resized(first.size()); + EXPECT_NO_THROW(cloned->pop_back(1)); + ASSERT_EQ(cloned->size(), 1); + EXPECT_EQ(assert_cast(*cloned).get_value_ref(0).get_int(), 10); + + auto second_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({30}), + second_output) + .ok()); + const auto& second = assert_cast( + assert_cast(*second_output).get_nested_column()); + auto appended = ColumnVariantV2::create(); + appended->insert_range_from(first, 0, first.size()); + EXPECT_NO_THROW(appended->insert_range_from(second, 0, second.size())); + ASSERT_EQ(appended->size(), 3); + EXPECT_EQ(appended->get_value_ref(2).get_int(), 30); +} + +TEST(VariantColumnReaderTest, MissingShreddedObjectWrapperMeansAbsentField) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integer = ColumnInt64::create(); + integer->get_data().push_back(0); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().push_back(1); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integer), std::move(integer_nulls))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().push_back(1); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(std::move(wrapper), std::move(wrapper_nulls))); + auto object = ColumnStruct::create(std::move(object_fields)); + auto object_nulls = ColumnUInt8::create(); + object_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(object), std::move(object_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_object_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); +} + +TEST(VariantColumnReaderTest, MaterializesShreddedArrayElements) { + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto integers = ColumnInt64::create(); + integers->get_data().assign({3, 4}); + auto integer_nulls = ColumnUInt8::create(); + integer_nulls->get_data().assign({0, 0}); + MutableColumns wrapper_fields; + wrapper_fields.push_back(ColumnNullable::create(std::move(integers), std::move(integer_nulls))); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto wrapper_nulls = ColumnUInt8::create(); + wrapper_nulls->get_data().assign({0, 0}); + auto elements = ColumnNullable::create(std::move(wrappers), std::move(wrapper_nulls)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto array = ColumnArray::create(std::move(elements), std::move(offsets)); + auto array_nulls = ColumnUInt8::create(); + array_nulls->get_data().push_back(0); + + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata}, {0})); + root_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + root_fields.push_back(ColumnNullable::create(std::move(array), std::move(array_nulls))); + auto root = ColumnStruct::create(std::move(root_fields)); + auto root_nulls = ColumnUInt8::create(); + root_nulls->get_data().push_back(0); + auto physical = ColumnNullable::create(std::move(root), std::move(root_nulls)); + + auto output = make_nullable(std::make_shared())->create_column(); + const auto status = materialize_variant_rows(shredded_array_schema(), *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + const VariantRef value = variants.get_value_ref(0); + ASSERT_EQ(value.num_elements(), 2); + EXPECT_EQ(value.array_at(0).get_int(), 3); + EXPECT_EQ(value.array_at(1).get_int(), 4); +} + +TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back(std::make_unique(unshredded_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + auto child_plan = std::make_unique(); + child_plan->schema = root_schema.children[0].get(); + child_plan->contains_variant = true; + plan.children.push_back(std::move(child_plan)); + + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + const auto& nullable = assert_cast(output_struct.get_column(0)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, AlignsNestedPrimitiveNullabilityAroundVariant) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + MutableColumns physical_fields; + physical_fields.push_back(nullable_strings({StringRef("required")}, {0})); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + physical_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + auto physical = ColumnStruct::create(std::move(physical_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = make_nullable(std::make_shared()); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const auto status = materialize_variant_columns(plan, *physical, output); + ASSERT_TRUE(status.ok()) << status; + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(output_struct.get_column(0).get_data_at(0).to_string(), "required"); + const auto& nullable = assert_cast(output_struct.get_column(1)); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, NestedMaterializationMovesUnaffectedSiblingBuffers) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + + auto label = ColumnString::create(); + label->insert_data("large-sibling", 13); + const IColumn* decoded_label = label.get(); + MutableColumns variant_fields; + variant_fields.push_back(nullable_strings({metadata}, {0})); + variant_fields.push_back(nullable_strings({{int_seven.data(), int_seven.size()}}, {0})); + auto physical_variant = ColumnStruct::create(std::move(variant_fields)); + auto variant_nulls = ColumnUInt8::create(); + variant_nulls->get_data().push_back(0); + MutableColumns root_fields; + root_fields.push_back(std::move(label)); + root_fields.push_back( + ColumnNullable::create(std::move(physical_variant), std::move(variant_nulls))); + ColumnPtr physical = ColumnStruct::create(std::move(root_fields)); + + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + auto label_schema = std::make_unique(); + label_schema->name = "label"; + label_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + label_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(label_schema)); + root_schema.children.push_back(std::make_unique(unshredded_schema())); + + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + for (const auto& child_schema : root_schema.children) { + auto child_plan = std::make_unique(); + child_plan->schema = child_schema.get(); + child_plan->contains_variant = child_schema->kind == ParquetColumnSchemaKind::VARIANT; + plan.children.push_back(std::move(child_plan)); + } + + auto output = std::make_shared( + DataTypes {std::make_shared(), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}) + ->create_column(); + const IColumn* empty_output = output.get(); + ASSERT_TRUE(materialize_variant_columns(plan, std::move(physical), output).ok()); + EXPECT_NE(output.get(), empty_output); + const auto& output_struct = assert_cast(*output); + EXPECT_EQ(&output_struct.get_column(0), decoded_label); +} + +} // namespace doris::format::parquet diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp index 806cc7e729a467..b9fe813c9c68f0 100644 --- a/be/test/format_v2/table/iceberg_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_reader_test.cpp @@ -59,6 +59,7 @@ #include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_varbinary.h" +#include "core/data_type/data_type_variant_v2.h" #include "exec/common/endian.h" #include "exec/scan/access_path_parser.h" #include "exprs/runtime_filter_expr.h" @@ -2627,6 +2628,44 @@ TEST(IcebergV2ReaderTest, IcebergLegacyPlanKeepsAllFieldIdsMappingRule) { TableColumnMappingMode::BY_NAME); } +TEST(IcebergV2ReaderTest, VariantFormatGateUsesPhysicalFileMappings) { + ColumnMapping missing_variant; + missing_variant.table_type = make_nullable(std::make_shared()); + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {missing_variant}) + .ok()); + + ColumnMapping physical_variant = missing_variant; + physical_variant.file_local_id = 0; + const auto orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {physical_variant}); + EXPECT_TRUE(orc_status.is()) << orc_status; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::PARQUET, {physical_variant}) + .ok()); + + ColumnMapping projected_struct; + projected_struct.table_type = make_nullable(std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"})); + projected_struct.file_local_id = 0; + ColumnMapping label; + label.table_type = make_nullable(std::make_shared()); + label.file_local_id = 1; + projected_struct.child_mappings = {label, missing_variant}; + EXPECT_TRUE(doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}) + .ok()); + + projected_struct.child_mappings[1] = physical_variant; + const auto nested_orc_status = + doris::format::iceberg::IcebergTableReader::validate_variant_file_mappings( + FileFormat::ORC, {projected_struct}); + EXPECT_TRUE(nested_orc_status.is()) << nested_orc_status; +} + TEST(IcebergV2ReaderTest, IcebergTableReaderDoesNotPushDownAggregateWithPositionDelete) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_iceberg_aggregate_position_delete_test"; diff --git a/be/test/format_v2/table_reader_request_test.cpp b/be/test/format_v2/table_reader_request_test.cpp index 3845e086cea1b1..a58cc82f45d6cb 100644 --- a/be/test/format_v2/table_reader_request_test.cpp +++ b/be/test/format_v2/table_reader_request_test.cpp @@ -71,6 +71,32 @@ TEST(FileScanRequestBuilderTest, PredicateColumnRemovesDuplicateNonPredicateColu EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(2)); } +TEST(FileScanRequestBuilderTest, DeferredComplexOutputSurvivesLaterPredicateMerge) { + FileScanRequest request; + FileScanRequestBuilder builder(&request); + + auto predicate = LocalColumnIndex::partial_local(5); + predicate.children.push_back(LocalColumnIndex::local(0)); + ASSERT_TRUE(builder.add_predicate_column(std::move(predicate)).ok()); + + auto output = LocalColumnIndex::partial_local(5); + output.children.push_back(LocalColumnIndex::local(0)); + output.children.push_back(LocalColumnIndex::local(1)); + ASSERT_TRUE(builder.add_deferred_non_predicate_column(std::move(output)).ok()); + + auto delete_predicate = LocalColumnIndex::partial_local(5); + delete_predicate.children.push_back(LocalColumnIndex::local(2)); + ASSERT_TRUE(builder.add_predicate_column(std::move(delete_predicate)).ok()); + + ASSERT_EQ(request.predicate_columns.size(), 1); + EXPECT_EQ(request.predicate_columns[0].children.size(), 2); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_EQ(request.non_predicate_columns[0].children.size(), 2); + EXPECT_EQ(request.local_positions.at(LocalColumnId(5)), LocalIndex(0)); + EXPECT_EQ(request.non_predicate_position(LocalColumnId(5)), LocalIndex(1)); + EXPECT_TRUE(request.is_predicate_only(LocalColumnId(5))); +} + // Scenario: TableReader's format-specific customization path delegates to FileScanRequestBuilder // and preserves the same predicate/non-predicate de-duplication rule. TEST(TableReaderRequestTest, AppendPredicateColumnKeepsOtherNonPredicateColumns) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index d306facec300e2..f0718653d09f12 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -471,6 +471,19 @@ public long streamingSplitEstimate(ConnectorSession session, ConnectorTableHandl return fileCount >= threshold ? fileCount : -1; } + @Override + public boolean canServeMetadataOnlyCount(ConnectorSession session, ConnectorTableHandle handle, + Optional filter) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + if (iceHandle.isSystemTable() || filter.isPresent()) { + // Snapshot summaries describe the whole table and cannot prove a filtered row count. + return false; + } + Table table = resolveTable(session, iceHandle); + TableScan scan = buildScan(table, iceHandle, filter, session); + return getCountFromSnapshot(scan, session) >= 0; + } + /** * Lazy streaming split source (FIX-M3), mirroring legacy {@code IcebergScanNode.doStartSplit}: slice files at * a FIXED size ({@code file_split_size} if set, else {@code max_split_size} — NOT the per-table diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java index 9336efdcfdb60a..c8a5f2cd9c2527 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java @@ -93,8 +93,12 @@ public static ConnectorType fromIcebergType(Type icebergType, fieldIds.add(f.fieldId()); } return ConnectorType.structOf(names, types, nullable, comments).withChildrenFieldIds(fieldIds); + case VARIANT: + // Iceberg owns the Parquet Variant physical encoding, so expose an execution-only + // VariantV2 carrier without changing persisted Doris table metadata semantics. + return ConnectorType.of("VARIANT_COMPUTE_V2"); default: - // Any non-primitive iceberg type Doris cannot represent (VARIANT today; future non-primitive + // Any future non-primitive iceberg type Doris cannot represent // typeIds) degrades to UNSUPPORTED: the table still LOADS and only this column is // present-but-unqueryable. This DIVERGES from legacy fe-core (IcebergUtils.icebergTypeToDorisType // threw IllegalArgumentException at schema-load, failing the whole table). Graceful degradation diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java index 2d09fe753a7536..38b57fa6e56dd9 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java @@ -164,6 +164,7 @@ public IcebergWritePlanProvider(Map properties, @Override public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + validateWriteSchema(handle.getColumns(), handle.isWritesDataFiles()); IcebergTableHandle tableHandle = (IcebergTableHandle) handle.getTableHandle(); IcebergConnectorTransaction transaction = currentTransaction(session); @@ -218,7 +219,7 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl case MERGE: { TDataSink dataSink = new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK); dataSink.setIcebergMergeSink(buildMergeSink(table, tableHandle, rewritableDeletes, - handle.isRequireMergeCardinalityCheck(), schemaContext)); + handle.isWritesDataFiles(), handle.isRequireMergeCardinalityCheck(), schemaContext)); return new ConnectorSinkPlan(dataSink); } case REWRITE: { @@ -365,6 +366,27 @@ private static boolean hasMeaningfulTypeParameters(String typeName) { || "TIMESTAMPTZ".equals(typeName); } + static void validateWriteSchema(List columns, boolean writesDataFiles) { + if (!writesDataFiles) { + return; + } + if (columns.stream().anyMatch(column -> containsVariant(column.getType()))) { + // Reject the whole data-file write: validating only selected columns would let an + // unchanged Variant target flow through a writer that cannot preserve its physical identity. + throw new DorisConnectorException( + "Iceberg VARIANT columns are read-only and cannot be written"); + } + } + + private static boolean containsVariant(ConnectorType type) { + String typeName = type.getTypeName(); + if ("VARIANT".equalsIgnoreCase(typeName) + || "VARIANT_COMPUTE_V2".equalsIgnoreCase(typeName)) { + return true; + } + return type.getChildren().stream().anyMatch(IcebergWritePlanProvider::containsVariant); + } + @Override public Optional> getWriteColumns(ConnectorSession session, ConnectorTableHandle tableHandle, Optional branchName) { @@ -756,7 +778,8 @@ private TIcebergDeleteSink buildDeleteSink(Table table, IcebergTableHandle table */ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHandle, Map> rewritableDeletes, - boolean requireMergeCardinalityCheck, IcebergWriteSchemaContext schemaContext) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck, + IcebergWriteSchemaContext schemaContext) { TIcebergMergeSink tSink = new TIcebergMergeSink(); tSink.setDbName(tableHandle.getDbName()); tSink.setTbName(tableHandle.getTableName()); @@ -770,6 +793,7 @@ private TIcebergMergeSink buildMergeSink(Table table, IcebergTableHandle tableHa IcebergWriterHelper.shouldCollectColumnStats(schemaContext, schema)); // #66112: UPDATE and SQL MERGE share this sink, but only SQL MERGE has the one-source-row invariant. tSink.setRequireMergeCardinalityCheck(requireMergeCardinalityCheck); + tSink.setWritesDataFiles(writesDataFiles); PartitionSpec partitionSpec = schemaContext.getPartitionSpec(); if (partitionSpec.isPartitioned()) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 80e0da6816a103..0e5e2b9ed51e74 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -1550,6 +1550,22 @@ public void countPushdownFollowsTheSnapshotPin() { Assertions.assertEquals(10L, pinned.get(0).getPushDownRowCount()); } + @Test + public void metadataOnlyCountCapabilityUsesSnapshotSummary() { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile( + table.spec(), "s3://b/db/t1/f1.parquet", 1000, null, null)).commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider( + Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()); + + Assertions.assertTrue(provider.canServeMetadataOnlyCount( + session, new IcebergTableHandle("db1", "t1"), Optional.empty())); + Assertions.assertFalse(provider.canServeMetadataOnlyCount( + session, IcebergTableHandle.forSystemTable( + "db1", "t1", "snapshots", -1L, null, -1L), Optional.empty())); + } + @Test public void getScanNodePropertiesUnderPinEmitsFullPinnedSchemaDict() throws Exception { // T07 Option A: under a time-travel pin the field-id dict is built from the FULL pinned schema (covering diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java index 97f995ecf60eba..41c7e44f2e2a14 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java @@ -87,9 +87,9 @@ public void flagIndependentPrimitivesMatchLegacy() { @Test public void unknownAndV3TypesDegradeToUnsupportedByDesign() { - // WHY (user decision 2026-07-13, DV-051): iceberg types Doris cannot represent — the v3 primitives - // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — must map to - // UNSUPPORTED WITHOUT throwing, so the table still loads and only the exotic column is + // WHY (user decision 2026-07-13, DV-051): iceberg primitive types Doris cannot represent — + // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN — must map to UNSUPPORTED WITHOUT throwing, + // so the table still loads and only the exotic column is // present-but-unqueryable. This deliberately DIVERGES from legacy fe-core, which threw // IllegalArgumentException("Cannot transform unknown type") at schema-load and failed the whole table. // This test PINS the graceful-degradation choice: MUTATION making either default arm throw -> red, @@ -100,9 +100,7 @@ public void unknownAndV3TypesDegradeToUnsupportedByDesign() { Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeometryType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeographyType.crs84()).getTypeName()); Assertions.assertEquals("UNSUPPORTED", mapOff(Types.UnknownType.get()).getTypeName()); - // VARIANT is NOT a primitive (falls to the nested-switch default); legacy mapped it to UNSUPPORTED - // too, so this stays parity while the primitives above are the intentional divergence. - Assertions.assertEquals("UNSUPPORTED", mapOff(Types.VariantType.get()).getTypeName()); + Assertions.assertEquals("VARIANT_COMPUTE_V2", mapOff(Types.VariantType.get()).getTypeName()); // The mapping flags do not rescue an unrepresentable type (both arms are flag-independent). Assertions.assertEquals("UNSUPPORTED", mapOn(Types.GeometryType.crs84()).getTypeName()); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java index 342c97f91d6caa..25b8e52e22591f 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java @@ -108,6 +108,19 @@ public class IcebergWritePlanProviderTest { private static final Map NON_REST_PROPS = Collections.singletonMap("iceberg.catalog.type", "hadoop"); + @Test + public void rejectsVariantDataWritesButAllowsDeleteOnlyMerge() { + ConnectorColumn nestedVariant = new ConnectorColumn("payload", + ConnectorType.structOf(Collections.singletonList("nested"), + Collections.singletonList(ConnectorType.of("VARIANT"))), + null, true, null); + Assertions.assertThrows(DorisConnectorException.class, + () -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), true)); + Assertions.assertDoesNotThrow(() -> IcebergWritePlanProvider.validateWriteSchema( + Collections.singletonList(nestedVariant), false)); + } + private static InMemoryCatalog freshCatalog() { InMemoryCatalog catalog = new InMemoryCatalog(); catalog.initialize("test", Collections.emptyMap()); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java index 1855fbd75bcee3..75ed6a53df1c2b 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/handle/ConnectorWriteHandle.java @@ -117,6 +117,14 @@ default boolean isRequireMergeCardinalityCheck() { return false; } + /** + * Whether this write can emit data files. A delete-only MERGE returns false so a connector may + * allow position-delete output even when the table has read-only column types. + */ + default boolean isWritesDataFiles() { + return true; + } + /** * The named table branch this write targets ({@code INSERT INTO t@branch(name)}), or * {@link Optional#empty()} when the write goes to the table's default ref. Threaded from the diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java index e9025ae71a5eda..0a901727fc13f8 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ConnectorScanPlanProvider.java @@ -352,6 +352,17 @@ default long streamingSplitEstimate( return -1; } + /** + * Whether this connector can answer the current table-level COUNT(*) without decoding data files. + * The default is false; connectors may use snapshot metadata to prove the stronger condition. + */ + default boolean canServeMetadataOnlyCount( + ConnectorSession session, + ConnectorTableHandle handle, + Optional filter) { + return false; + } + /** * Builds a lazy {@link ConnectorSplitSource} for streaming split generation. Called once, on a * background task, only when {@link #streamingSplitEstimate} returned a non-negative value. The diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java index 62598eec708bef..7b5ae7654a5138 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java @@ -371,6 +371,11 @@ private static Type convertScalarType(String typeName, int precision, int scale) return ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH); case "JSONB": return ScalarType.createType("JSON"); + case "VARIANT_COMPUTE_V2": + // This carrier is execution-only: connector schemas use it for native external + // Variant encodings, while persisted Doris table metadata keeps regular Variant rules. + return new org.apache.doris.catalog.VariantType( + new ArrayList<>(), 0, false, 10000, 0, false, 0L, 64, false, true); case "UNSUPPORTED": return Type.UNSUPPORTED; default: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 51f45809c27fb9..ad4ccc728ee07d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -26,9 +26,14 @@ import org.apache.doris.analysis.TableScanParams; import org.apache.doris.analysis.ToSqlParams; import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.catalog.ArrayType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.MapType; +import org.apache.doris.catalog.StructType; import org.apache.doris.catalog.TableIf; +import org.apache.doris.catalog.Type; +import org.apache.doris.catalog.VariantType; import org.apache.doris.common.UserException; import org.apache.doris.common.profile.RuntimeProfile; import org.apache.doris.common.profile.SummaryProfile; @@ -193,6 +198,64 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, this.currentHandle = tableHandle; } + @Override + protected void doInitialize() throws UserException { + super.doInitialize(); + checkVariantBackendCompatibilityForCurrentScan(backendPolicy.getBackends()); + } + + void checkVariantBackendCompatibilityForCurrentScan(Iterable backends) + throws UserException { + boolean metadataCountProven = false; + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); + if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) { + metadataCountProven = onPluginClassLoader(scanProvider, + () -> scanProvider.canServeMetadataOnlyCount( + connectorSession, currentHandle, Optional.empty())); + } + checkVariantBackendCompatibility( + !metadataCountProven && projectsComputeVariant(desc), backends); + } + + static boolean projectsComputeVariant(TupleDescriptor tuple) { + // Nested-column pruning updates the effective slot type but deliberately keeps the original + // Column metadata; compatibility must follow the payload this scan actually projects. + return tuple.getSlots().stream().anyMatch(slot -> containsComputeVariant(slot.getType())); + } + + private static boolean containsComputeVariant(Type type) { + if (type instanceof VariantType) { + return ((VariantType) type).isComputeV2(); + } + if (type instanceof ArrayType) { + return containsComputeVariant(((ArrayType) type).getItemType()); + } + if (type instanceof MapType) { + MapType map = (MapType) type; + return containsComputeVariant(map.getKeyType()) || containsComputeVariant(map.getValueType()); + } + if (type instanceof StructType) { + return ((StructType) type).getFields().stream() + .anyMatch(field -> containsComputeVariant(field.getType())); + } + return false; + } + + static void checkVariantBackendCompatibility(boolean projectsVariant, Iterable backends) + throws UserException { + if (!projectsVariant) { + return; + } + for (Backend backend : backends) { + if (backend.isSmoothUpgradeSrc()) { + // Old backends cannot distinguish the logical Variant from its physical carrier, + // so scheduling this projection there could corrupt the result shape. + throw new UserException("Iceberg Variant is unavailable while backend " + + backend.getId() + " is a smooth upgrade source"); + } + } + } + // Lazily resolves this node's ConnectorMetadata through the per-statement funnel and caches it, so the // per-method resolvers below share one instance for the statement instead of rebuilding it each time. private ConnectorMetadata metadata() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 099d6a1edb3a2a..5dc082b0635618 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -555,8 +555,8 @@ public PlanFragment visitPhysicalExternalRowLevelDeleteSink( // TIcebergDeleteSink dialect. No output-expr / materialized-name loop is needed: the row id reaches // BE as the __DORIS_ICEBERG_ROWID_COL__ block column (a real hidden column), and viceberg_delete_sink // resolves it by block-name, not by output-expr name. - rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, false, - deleteSink.getBoundWriteMetadataIdentity())); + rootFragment.setSink(buildPluginRowLevelDmlSink(deleteSink, WriteOperation.DELETE, + false, false, deleteSink.getBoundWriteMetadataIdentity())); return rootFragment; } @@ -593,7 +593,8 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( // SQL MERGE INTO carries the cardinality requirement onto the write handle; UPDATE shares this // sink dialect but has no such rule, so it threads false (see PhysicalExternalRowLevelMergeSink). rootFragment.setSink(buildPluginRowLevelDmlSink(mergeSink, WriteOperation.MERGE, - mergeSink.isRequireMergeCardinalityCheck(), mergeSink.getBoundWriteMetadataIdentity())); + mergeSink.isWritesDataFiles(), mergeSink.isRequireMergeCardinalityCheck(), + mergeSink.getBoundWriteMetadataIdentity())); return rootFragment; } @@ -610,7 +611,8 @@ public PlanFragment visitPhysicalExternalRowLevelMergeSink( */ private PluginDrivenTableSink buildPluginRowLevelDmlSink( PhysicalBaseExternalTableSink sink, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck, String boundWriteMetadataIdentity) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck, + String boundWriteMetadataIdentity) { PluginDrivenExternalTable targetTable = (PluginDrivenExternalTable) sink.getTargetTable(); PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) targetTable.getCatalog(); @@ -651,7 +653,7 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( // connector's TIcebergMergeSink.sort_fields, DELETE is unsorted). return new PluginDrivenTableSink(targetTable, writePlanProvider, connSession, providerTableHandle, connectorColumns, connectorColumns, null, writeOperation, - requireMergeCardinalityCheck, boundWriteMetadataIdentity); + writesDataFiles, requireMergeCardinalityCheck, boundWriteMetadataIdentity); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java index 4d84d6ebda60e1..e1994e966d37eb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalExternalRowLevelMergeSinkToPhysicalExternalRowLevelMergeSink.java @@ -40,6 +40,7 @@ public Rule build() { sink.getBoundWriteMetadataIdentity(), sink.getCols(), sink.getOutputExprs(), + sink.isWritesDataFiles(), sink.isRequireMergeCardinalityCheck(), Optional.empty(), sink.getLogicalProperties(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java index 0633637b3377aa..915ef5f7eba26f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java @@ -137,6 +137,15 @@ public Void visitSlotReference(SlotReference slotReference, CollectorContext con path, context.bottomFilter, ColumnAccessPathType.DATA)); return null; } + if (dataType instanceof VariantType) { + // A root Variant consumer must dominate any predicate-only leaf path. Otherwise the + // scanner can legally project a shredded leaf that cannot serve the root expression. + int slotId = slotReference.getExprId().asInt(); + slotToAccessPaths.put(slotId, new CollectAccessPathResult( + ImmutableList.of(slotReference.getName()), + context.bottomFilter, ColumnAccessPathType.DATA)); + return null; + } if (dataType instanceof NestedColumnPrunable) { context.accessPathBuilder.addPrefix(slotReference.getName().toLowerCase()); ImmutableList path = Utils.fastToImmutableList(context.accessPathBuilder.accessPath); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java index 7e7674d20586c7..c9cd9d9046fcd4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathPlanCollector.java @@ -25,6 +25,7 @@ import org.apache.doris.nereids.trees.expressions.Expression; import org.apache.doris.nereids.trees.expressions.NamedExpression; import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; import org.apache.doris.nereids.trees.expressions.functions.Function; import org.apache.doris.nereids.trees.expressions.functions.generator.Explode; import org.apache.doris.nereids.trees.expressions.functions.generator.ExplodeMap; @@ -249,6 +250,12 @@ public Void visitLogicalProject(LogicalProject project, Statemen List outerPath = outerSlotAccessPath.getPath(); List replaceSlotNamePath = new ArrayList<>(); replaceSlotNamePath.add(innerSlot.getName()); + if (outerPath.size() == 1 && innerSlot instanceof SlotReference + && ((SlotReference) innerSlot).hasSubColPath()) { + // A whole access to a derived subcolumn slot is whole only relative to that + // slot; preserve its physical leaf path when propagating to the scan slot. + replaceSlotNamePath.addAll(((SlotReference) innerSlot).getSubPath()); + } replaceSlotNamePath.addAll(outerPath.subList(1, outerPath.size())); allSlotToAccessPaths.put( innerSlot.getExprId().asInt(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java index 6ae6ef2b1abf9d..3faf0d581a5c72 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java @@ -724,6 +724,11 @@ public void setAccessByPath(List path, int accessIndex, ColumnAccessPath // Any other sub-path on a string column means full data is needed. accessAll = true; return; + } else if (type.isVariantType()) { + // Variant object keys stay in the serialized access path. Keeping the terminal type + // here lets BE project a shredded leaf without inventing static schema fields. + accessAll = true; + return; } else if (isRoot) { children.get(path.get(accessIndex).toLowerCase()).setAccessByPath(path, accessIndex + 1, pathType); return; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java index 5e3b601df33958..ca8a3b5ba4c7cd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SlotTypeReplacer.java @@ -64,6 +64,7 @@ import org.apache.doris.nereids.types.MapType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MoreFieldsThread; import com.google.common.collect.ImmutableCollection; @@ -669,6 +670,10 @@ private void replaceAccessPathToFieldId(List originPath, int index, Data break; } } + } else if (type instanceof VariantType) { + // Variant object keys are data, not Iceberg schema field IDs. Replacing them with + // the root ID destroys the physical shredding path before it reaches the scanner. + return; } else { originPath.set(index, String.valueOf(column.getUniqueId())); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java index b3ac202bd4d330..53a31f68729f39 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelMergePlanBuilder.java @@ -417,6 +417,8 @@ LogicalPlan buildMergePlan(ConnectContext ctx, ExternalTable icebergTable) { writeSchema.getWriteMetadataIdentity(), ConnectorWriteSchemaUtils.pinAndGet(ctx, icebergTable), outputExprs, + matchedClauses.stream().anyMatch(clause -> !clause.isDelete()) + || !notMatchedClauses.isEmpty(), true, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java index d0c045df9efe54..240304ea458b74 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExternalRowLevelUpdatePlanBuilder.java @@ -144,6 +144,7 @@ LogicalPlan buildMergePlan(ConnectContext ctx, LogicalPlan logicalQuery, writeSchema.getWriteMetadataIdentity(), writeColumns, outputExprs, + true, false, Optional.empty(), Optional.empty(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java index 768a284c217a86..29268fbe24baa5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalExternalRowLevelMergeSink.java @@ -46,6 +46,8 @@ public class LogicalExternalRowLevelMergeSink extends L private final ExternalDatabase database; private final ExternalTable targetTable; private final String boundWriteMetadataIdentity; + // Delete-only MERGE emits position deletes but never invokes the data-file writer. + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE. MERGE must reject a target row matched by more than one // source row (SQL cardinality rule), which the BE sink can only do when the plan keeps the merge // distribution; UPDATE has no such rule. Read by RequestPropertyDeriver (which otherwise drops the @@ -68,7 +70,7 @@ public LogicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, Optional logicalProperties, CHILD_TYPE child) { - this(database, targetTable, null, cols, outputExprs, requireMergeCardinalityCheck, + this(database, targetTable, null, cols, outputExprs, true, requireMergeCardinalityCheck, groupExpression, logicalProperties, child); } @@ -78,6 +80,7 @@ public LogicalExternalRowLevelMergeSink(ExternalDatabase database, String boundWriteMetadataIdentity, List cols, List outputExprs, + boolean writesDataFiles, boolean requireMergeCardinalityCheck, Optional groupExpression, Optional logicalProperties, @@ -89,6 +92,7 @@ public LogicalExternalRowLevelMergeSink(ExternalDatabase database, this.targetTable = Objects.requireNonNull(targetTable, "targetTable != null in LogicalExternalRowLevelMergeSink"); this.boundWriteMetadataIdentity = boundWriteMetadataIdentity; + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -97,21 +101,21 @@ public Plan withChildAndUpdateOutput(Plan child) { .map(NamedExpression.class::cast) .collect(ImmutableList.toImmutableList()); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, boundWriteMetadataIdentity, cols, output, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); + writesDataFiles, requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child); } @Override public Plan withChildren(List children) { Preconditions.checkArgument(children.size() == 1, "LogicalExternalRowLevelMergeSink only accepts one child"); return new LogicalExternalRowLevelMergeSink<>(database, targetTable, boundWriteMetadataIdentity, - cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), children.get(0)); + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, + Optional.empty(), Optional.empty(), children.get(0)); } public LogicalExternalRowLevelMergeSink withOutputExprs(List outputExprs) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, boundWriteMetadataIdentity, - cols, outputExprs, - requireMergeCardinalityCheck, Optional.empty(), Optional.empty(), child()); + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, + Optional.empty(), Optional.empty(), child()); } public ExternalDatabase getDatabase() { @@ -126,6 +130,10 @@ public String getBoundWriteMetadataIdentity() { return boundWriteMetadataIdentity; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -146,13 +154,14 @@ public boolean equals(Object o) { && Objects.equals(targetTable, that.targetTable) && Objects.equals(boundWriteMetadataIdentity, that.boundWriteMetadataIdentity) && Objects.equals(cols, that.cols) + && writesDataFiles == that.writesDataFiles && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck; } @Override public int hashCode() { return Objects.hash(super.hashCode(), database, targetTable, boundWriteMetadataIdentity, cols, - requireMergeCardinalityCheck); + writesDataFiles, requireMergeCardinalityCheck); } @Override @@ -162,6 +171,7 @@ public String toString() { "database", database.getFullName(), "targetTable", targetTable.getName(), "cols", cols, + "writesDataFiles", writesDataFiles, "requireMergeCardinalityCheck", requireMergeCardinalityCheck); } @@ -173,15 +183,15 @@ public R accept(PlanVisitor visitor, C context) { @Override public Plan withGroupExpression(Optional groupExpression) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, boundWriteMetadataIdentity, - cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, Optional.of(getLogicalProperties()), child()); + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, + groupExpression, Optional.of(getLogicalProperties()), child()); } @Override public Plan withGroupExprLogicalPropChildren(Optional groupExpression, Optional logicalProperties, List children) { return new LogicalExternalRowLevelMergeSink<>(database, targetTable, boundWriteMetadataIdentity, - cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties, children.get(0)); + cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties, children.get(0)); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java index 292cf34fd1d80e..bc528540523161 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java @@ -62,6 +62,7 @@ public class PhysicalExternalRowLevelMergeSink extends PhysicalBaseExternalTableSink { private final String boundWriteMetadataIdentity; + private final boolean writesDataFiles; // True for SQL MERGE INTO, false for UPDATE; see LogicalExternalRowLevelMergeSink. private final boolean requireMergeCardinalityCheck; @@ -76,7 +77,7 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, Optional groupExpression, LogicalProperties logicalProperties, CHILD_TYPE child) { - this(database, targetTable, null, cols, outputExprs, requireMergeCardinalityCheck, + this(database, targetTable, null, cols, outputExprs, true, requireMergeCardinalityCheck, groupExpression, logicalProperties, PhysicalProperties.GATHER, null, child); } @@ -91,6 +92,39 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, LogicalProperties logicalProperties, CHILD_TYPE child) { this(database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, + true, requireMergeCardinalityCheck, groupExpression, logicalProperties, + PhysicalProperties.GATHER, null, child); + } + + /** + * Constructor that records whether the merge writes replacement data files. + */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + CHILD_TYPE child) { + this(database, targetTable, null, cols, outputExprs, writesDataFiles, + requireMergeCardinalityCheck, groupExpression, logicalProperties, + PhysicalProperties.GATHER, null, child); + } + + /** Builds a row-level sink with explicit metadata generation and data-file settings. */ + public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, + ExternalTable targetTable, + String boundWriteMetadataIdentity, + List cols, + List outputExprs, + boolean writesDataFiles, + boolean requireMergeCardinalityCheck, + Optional groupExpression, + LogicalProperties logicalProperties, + CHILD_TYPE child) { + this(database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, writesDataFiles, requireMergeCardinalityCheck, groupExpression, logicalProperties, PhysicalProperties.GATHER, null, child); } @@ -108,16 +142,17 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) { - this(database, targetTable, null, cols, outputExprs, requireMergeCardinalityCheck, + this(database, targetTable, null, cols, outputExprs, true, requireMergeCardinalityCheck, groupExpression, logicalProperties, physicalProperties, statistics, child); } - /** Builds a row-level sink with the write generation captured during logical planning. */ + /** Builds a row-level sink with explicit metadata generation and data-file settings. */ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, ExternalTable targetTable, String boundWriteMetadataIdentity, List cols, List outputExprs, + boolean writesDataFiles, boolean requireMergeCardinalityCheck, Optional groupExpression, LogicalProperties logicalProperties, @@ -127,6 +162,7 @@ public PhysicalExternalRowLevelMergeSink(ExternalDatabase database, super(PlanType.PHYSICAL_EXTERNAL_ROW_LEVEL_MERGE_SINK, database, targetTable, cols, outputExprs, groupExpression, logicalProperties, physicalProperties, statistics, child); this.boundWriteMetadataIdentity = boundWriteMetadataIdentity; + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -134,6 +170,10 @@ public String getBoundWriteMetadataIdentity() { return boundWriteMetadataIdentity; } + public boolean isWritesDataFiles() { + return writesDataFiles; + } + public boolean isRequireMergeCardinalityCheck() { return requireMergeCardinalityCheck; } @@ -142,7 +182,8 @@ public boolean isRequireMergeCardinalityCheck() { public Plan withChildren(List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, - boundWriteMetadataIdentity, cols, outputExprs, requireMergeCardinalityCheck, groupExpression, + boundWriteMetadataIdentity, cols, outputExprs, writesDataFiles, + requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, children.get(0)); } @@ -155,7 +196,8 @@ public R accept(PlanVisitor visitor, C context) { public Plan withGroupExpression(Optional groupExpression) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), child()); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, getLogicalProperties(), child()); } @Override @@ -163,13 +205,15 @@ public Plan withGroupExprLogicalPropChildren(Optional groupExpr Optional logicalProperties, List children) { return new PhysicalExternalRowLevelMergeSink<>( database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, - requireMergeCardinalityCheck, groupExpression, logicalProperties.get(), children.get(0)); + writesDataFiles, requireMergeCardinalityCheck, + groupExpression, logicalProperties.get(), children.get(0)); } @Override public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) { return new PhysicalExternalRowLevelMergeSink<>( - database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, requireMergeCardinalityCheck, + database, targetTable, boundWriteMetadataIdentity, cols, outputExprs, + writesDataFiles, requireMergeCardinalityCheck, groupExpression, getLogicalProperties(), physicalProperties, statistics, child()); } @@ -185,13 +229,15 @@ public boolean equals(Object o) { return false; } PhysicalExternalRowLevelMergeSink that = (PhysicalExternalRowLevelMergeSink) o; - return requireMergeCardinalityCheck == that.requireMergeCardinalityCheck + return writesDataFiles == that.writesDataFiles + && requireMergeCardinalityCheck == that.requireMergeCardinalityCheck && Objects.equals(boundWriteMetadataIdentity, that.boundWriteMetadataIdentity); } @Override public int hashCode() { - return Objects.hash(super.hashCode(), boundWriteMetadataIdentity, requireMergeCardinalityCheck); + return Objects.hash(super.hashCode(), boundWriteMetadataIdentity, + writesDataFiles, requireMergeCardinalityCheck); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java index 79e53cd3f02968..9e6288dcc06b90 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java @@ -75,6 +75,7 @@ public class PluginDrivenTableSink extends BaseExternalTableDataSink { // the INSERT TIcebergTableSink. Threaded onto the write handle so planWrite's buildWriteContext // reads it via ConnectorWriteHandle.getWriteOperation(). private final WriteOperation writeOperation; + private final boolean writesDataFiles; // SQL MERGE INTO only: the statement must reject a target row matched by more than one source row. // Carried from PhysicalExternalRowLevelMergeSink onto the write handle so the connector can stamp the // enforcement flag onto its BE sink; false for UPDATE and for every non-row-level write. @@ -127,7 +128,21 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, ConnectorTableHandle tableHandle, List connectorColumns, TSortInfo writeSortInfo, WriteOperation writeOperation, boolean requireMergeCardinalityCheck) { this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, - connectorColumns, writeSortInfo, writeOperation, requireMergeCardinalityCheck); + connectorColumns, writeSortInfo, writeOperation, true, + requireMergeCardinalityCheck, null); + } + + /** + * Plan-provider mode with explicit data-file and merge-cardinality requirements. + */ + public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, + ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, + ConnectorTableHandle tableHandle, List connectorColumns, + TSortInfo writeSortInfo, WriteOperation writeOperation, boolean writesDataFiles, + boolean requireMergeCardinalityCheck) { + this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, + connectorColumns, writeSortInfo, writeOperation, writesDataFiles, + requireMergeCardinalityCheck, null); } /** @@ -139,7 +154,8 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, List boundTargetColumns, TSortInfo writeSortInfo, WriteOperation writeOperation, boolean requireMergeCardinalityCheck) { this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, - boundTargetColumns, writeSortInfo, writeOperation, requireMergeCardinalityCheck, null); + boundTargetColumns, writeSortInfo, writeOperation, true, + requireMergeCardinalityCheck, null); } /** @@ -151,6 +167,20 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, List boundTargetColumns, TSortInfo writeSortInfo, WriteOperation writeOperation, boolean requireMergeCardinalityCheck, String boundWriteMetadataIdentity) { + this(targetTable, writePlanProvider, connectorSession, tableHandle, connectorColumns, + boundTargetColumns, writeSortInfo, writeOperation, true, + requireMergeCardinalityCheck, boundWriteMetadataIdentity); + } + + /** + * Plan-provider mode with explicit schema, data-file, and metadata-generation settings. + */ + public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, + ConnectorWritePlanProvider writePlanProvider, ConnectorSession connectorSession, + ConnectorTableHandle tableHandle, List connectorColumns, + List boundTargetColumns, TSortInfo writeSortInfo, + WriteOperation writeOperation, boolean writesDataFiles, + boolean requireMergeCardinalityCheck, String boundWriteMetadataIdentity) { super(); this.targetTable = targetTable; this.writePlanProvider = writePlanProvider; @@ -163,6 +193,7 @@ public PluginDrivenTableSink(PluginDrivenExternalTable targetTable, this.writeSortInfo = writeSortInfo; this.boundWriteMetadataIdentity = boundWriteMetadataIdentity; this.writeOperation = writeOperation == null ? WriteOperation.INSERT : writeOperation; + this.writesDataFiles = writesDataFiles; this.requireMergeCardinalityCheck = requireMergeCardinalityCheck; } @@ -196,7 +227,7 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { // EXPLAIN), so the connector derives the detail from the write handle. ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, boundTargetColumns, false, Collections.emptyMap(), null, - null, Optional.empty(), writeOperation, requireMergeCardinalityCheck); + null, Optional.empty(), writeOperation, writesDataFiles, requireMergeCardinalityCheck); writePlanProvider.appendExplainInfo(sb, prefix, connectorSession, handle); return sb.toString(); } @@ -223,7 +254,8 @@ public void bindDataSink(Optional insertCtx) } ConnectorWriteHandle handle = new PluginDrivenWriteHandle( tableHandle, connectorColumns, boundTargetColumns, overwrite, writeContext, writeSortInfo, - boundWriteMetadataIdentity, branchName, writeOperation, requireMergeCardinalityCheck); + boundWriteMetadataIdentity, branchName, writeOperation, writesDataFiles, + requireMergeCardinalityCheck); ConnectorSinkPlan sinkPlan = writePlanProvider.planWrite(connectorSession, handle); this.tDataSink = sinkPlan.getDataSink(); } @@ -246,13 +278,14 @@ private static final class PluginDrivenWriteHandle implements ConnectorWriteHand private final String boundWriteMetadataIdentity; private final Optional branchName; private final WriteOperation writeOperation; + private final boolean writesDataFiles; private final boolean requireMergeCardinalityCheck; private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List columns, List boundTargetColumns, boolean overwrite, Map writeContext, TSortInfo sortInfo, String boundWriteMetadataIdentity, Optional branchName, WriteOperation writeOperation, - boolean requireMergeCardinalityCheck) { + boolean writesDataFiles, boolean requireMergeCardinalityCheck) { this.tableHandle = tableHandle; this.columns = columns; this.boundTargetColumns = boundTargetColumns; @@ -262,9 +295,15 @@ private PluginDrivenWriteHandle(ConnectorTableHandle tableHandle, List PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("backend 7")); + } + + @Test + public void compatibilityCheckIgnoresScansWithoutComputeVariant() throws UserException { + Backend backend = new Backend(7L, "127.0.0.1", 9050); + backend.setSmoothUpgradeSrc(true); + + PluginDrivenScanNode.checkVariantBackendCompatibility( + false, Collections.singletonList(backend)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java index 3c92606e4e3fcc..ca9c59067c625b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java @@ -45,6 +45,9 @@ import org.apache.doris.nereids.types.DataType; import org.apache.doris.nereids.types.NestedColumnPrunable; import org.apache.doris.nereids.types.NullType; +import org.apache.doris.nereids.types.StructField; +import org.apache.doris.nereids.types.StructType; +import org.apache.doris.nereids.types.VariantType; import org.apache.doris.nereids.util.MemoPatternMatchSupported; import org.apache.doris.nereids.util.PlanChecker; import org.apache.doris.planner.OlapScanNode; @@ -1050,6 +1053,22 @@ public void testDataTypeAccessTree() { ); } + @Test + public void testDataTypeAccessTreeKeepsVariantTerminalPath() { + StructType type = new StructType(ImmutableList.of( + new StructField("payload", VariantType.INSTANCE, true, ""))); + SlotReference slot = new SlotReference("info", type); + DataTypeAccessTree tree = DataTypeAccessTree.ofRoot(slot, ColumnAccessPathType.DATA); + + tree.setAccessByPath(ImmutableList.of("info", "payload", "typed_col"), 0, + ColumnAccessPathType.DATA); + + DataType prunedType = tree.pruneDataType().get(); + Assertions.assertInstanceOf(StructType.class, prunedType); + Assertions.assertEquals(VariantType.INSTANCE, + ((StructType) prunedType).getFields().get(0).getDataType()); + } + @Test public void testWithVariant() throws Exception { connectContext.getSessionVariable().enableDecimal256 = true; diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java index 5268569e0b8861..cba617e96ed631 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java @@ -105,6 +105,25 @@ public void testVariantOrPredicatePaths() throws Exception { ); } + @Test + public void testWholeVariantOutputDominatesPredicateLeafProjection() throws Exception { + String rootOutputSql = "select v from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + rootOutputSql, + ImmutableList.of(path("v")), + ImmutableList.of() + ); + assertPredicateAccessPathsEqual(rootOutputSql, ImmutableList.of(path("v", "n"))); + + String predicateOnlySql = "select count(*) from variant_tbl where v['n'] > 1"; + assertAllAccessPathsContain( + predicateOnlySql, + ImmutableList.of(path("v", "n")), + ImmutableList.of(path("v")) + ); + assertPredicateAccessPathsEqual(predicateOnlySql, ImmutableList.of(path("v", "n"))); + } + @Test public void testVariantIfExpressionPaths() throws Exception { assertVariantSubColumnSlots( diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java index 658b5cf2ae2f67..a5c871216bf1d9 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -258,6 +258,20 @@ null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), Assert.assertEquals(WriteOperation.DELETE, provider.seenHandle.getWriteOperation()); } + @Test + public void bindDataSinkThreadsDeleteOnlyMergeToHandle() throws AnalysisException { + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + sink.bindDataSink(Optional.empty()); + + // Delete-only MERGE must bypass data-file validation while retaining cardinality enforcement. + Assert.assertFalse(provider.seenHandle.isWritesDataFiles()); + Assert.assertTrue(provider.seenHandle.isRequireMergeCardinalityCheck()); + } + @Test public void getExplainStringThreadsWriteOperationToHandle() { // WHY: EXPLAIN of a post-flip MERGE/DELETE builds a (degraded) handle for appendExplainInfo; the diff --git a/gensrc/thrift/DataSinks.thrift b/gensrc/thrift/DataSinks.thrift index 9efe29a51eb421..7295da1a6874fd 100644 --- a/gensrc/thrift/DataSinks.thrift +++ b/gensrc/thrift/DataSinks.thrift @@ -539,6 +539,8 @@ struct TIcebergMergeSink { 14: optional bool collect_column_stats; // Unset preserves old-FE UPDATE behavior; execution version gates SQL MERGE validation. 15: optional bool require_merge_cardinality_check; + // Unset preserves old-FE UPDATE behavior, which always writes replacement data rows. + 16: optional bool writes_data_files; // delete side (position delete only) 20: optional TFileContent delete_type diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet new file mode 100644 index 0000000000000000000000000000000000000000..f45415374f2f455c0d31dcb1b98c67005ef0d7b5 GIT binary patch literal 34528 zcmeHw2_RM5_y2Lbp$sX`iBgH4>V*uMqBOkMMAP$NB5LYfT z51FST(?ux>iBN{f{qJ*b6HV{=KL6kM_kQ={wC-MKue~bFcOtQ3X1k6Q3riV_I3?TWD?cd zrdLaVixn>smBVh#x_LoKm8(QdE4v9^XhCt&7Kz&rJdN3P7nIf)NW}Gdn&7wti}Hmf zZWXH-&sGgAsfm<`?Nl*gmI^E`BT7U!lo@kc1eP|nOWYYLGht%WD9S|=qV9_r&r;SX zdFY!E^IXJa<|>Wi619Zek79b3)W%09ja4_+Cd9psF`2Qjv8X^M;a0hkF`H&%Ni8KI z_LY$di$Y^@g>^!7bGPvv`^M6i!Gt^SyL%y)wM99cW>ITmOL1L{@BYb<{5u4 ze)-zs;$3F9tAl&k_1Bg*RG7sL1oz_R-z&N=W_GLeV9)IR_evZOzFJUsui~w->2=z@ z;&NlNXlhdrr_H_6<{q;7aSkD~C zccqWh4m^7e;US0)*llJw@cb~9>AQe=^8&!)@1TIiQNXpCl!Ve$nM-MS{A^ZMJ#A;+ z8f|7-m!%wpqj|iwrmnK5=bqQoyUzRQ=oVR5iSq=bXS{W`1+u4R?bJKs<6omEEPH0< zPQtNj|60pPS!cd-Jz}E&BO{{hS^08;@qmAwLwmB*Oi?`@5A~WONV3Z>?+gF%rFarV z_3c)vKiHv0IkzvCXl1Bgtx-$yvWnF|v9P{siwwnmCy8jGS^rRtLOD+&>Dw#RKiFkW z@z8ojw6U+RJ~T+VaN?D|Ba2+sMow$D&5PPix5_=->t*e!yXb}Oa=8cFcMTq<4LIJE zdt|W7##JihxzP)`I=c#+Q*2soNBmN12x2y8R%tyyR+mz16=&nTu(^$xobrfdY;#tj z`ML2>O5KT`At#n~Z8{fhYjhV6xh!Az+@R97)*^K1^!$uA!X4X3hKGh+Wiy_WdTs0M zTZT@}InkyYHe92>e(22V6VDAF57*jc4mmG*-=?26{K(|w&{?JT&rN8<^Bu|k3KQTO ze{}Bp_x+^H#g7Y75b z>4t0NyDi5iz}0vqsbtpFyo-$i*L1`6YVQ7P6X41$FC^oq2G<-0T+9RIt>F$oXN+)KXXyEOFzvhTp*Qq7!mYGm@hm}An> znt6e>WQtXc;kuX9Y*!g_!cL=OQbE*QUkW*iWMrs7rRI29lM}VNkIAM{^Fjy7$tSuE zl|~h^PjQ;ZZ|3h3KdX>?(aZd66ey!CC)s_@Zig}b!j>0jm;A^o4RC@)AB-k%#-b#UalXq&vstiAHP+x zOX7lku1{$HU46xu@|E^E9*6qL`_sFm@7U)Bwe(YL(qFFY9nC(wzCU5NLzh(8Xs%yo zf0BvAONGayITud$CmtT_lFb^;3mflGb{u0{mOZokGhSdOf341aXlio}7j}kwO}Bfs z@HQ--09S)2B^~!&_xjOH-i5L7n)Zd)f`Ot(6X43I$BCJA!Fe+?QoqrCRY3zF0k;*3TUtUzIT0Ccxjau_>$6mu_YH+b%EcW zo(wiOE-tR3Z04_aQD_q37d?%5t}iI>C=icK@Fej+TTnSH{Q59W>YBTX(LA5PvbIR^ z8@E(Q3#tPv-VnvF2b38hiGk(a?cz66%Sej{0xQRm_^UL{AMT|Xa~Nur{sS8+e`6Qr zQJh)q%Y(hlvftf8`A7Fp#`{)pJf(!g@WR7_@D8{$|6L@9B8-EoZ7#hd z+HFOvwyA6{E>mWK;$?aF&jmYSVC#h$7K3ZNOCPZD!$nLme)3M4u3Qc)lzW%pF=I-% z<6#l_S?=2s4u0758^TvWP@ZnnT-=mo|4q#qs-Rn+lksF&g<+My>idko!ssiEzQPy| z|3@bYoUjc1=&64en6F@&JeW=r`RWS;>rM86EYew=v8VPpgbAFUwOKuPZYb+-%jR8l z-}Gnr+z{UUWk`bi=7Ue?hT+FHY&yAYuKV^)e_Xm4%oDK;zO{X`#>0!D?0U<%1By4P zk^8tCMgk5GU*T_SlRde7eroaNL;V-Sa6A!QXI5_7dv0&=yvq?hH&$*wP_j34_MZ{+ zuH4$R@Alr11vL>!(yh%}U3W$r#d-1s1r z(=viPq;u2$dk;buyHxocdJ{S4MCVpVW^wP0>_pbhd$2GUZ$6I`Cs5^7ME0m+u8Ny3+=>AFDf5wpS?e|z{=RvN3*G& z(#NyZ9TNT@9?Y1khIQ06`{`7VZ6T3qD^{+4Pc4%iiJi_efZ|TWGPbCHTRWQRD zIS)%eK4zRWLjbwI{s2Fyigr)AR0BM$iq%L${76*4SBT$!QC&l`Qc%`z|9I&1j8hwq zZu8#&VASJv=J=2U{LXpO(<{d{|3n(_SaH8ve7KS($OP`2AN?-+Q4IY7);Mv|OO48R zGV#~W8yRs2HI}tg;%~$nkrq-LD+aCOuZMIS%};AA?-`80nbA#JGTK-<&M98=gZzQ9 z=3Bcg?xFbuG*gO2@JlRzV7yasep-Ik#1sPB;F7x0I}4jSgk{t2$Aau%V~aF}rcN=N zw8D%Yi!2t!j-_+#^RF)+NLjAfxpKFCL1O4Y`uy~cUsl`Sk3KYTPd2?%)Wp6ptz{r{ zjzfpglF|H|>jzR-J9Mr(JX(;PIgqhrtYf*-=>0n<2NYB-2wOh`Si}s!>vwVgnDH?g zz#`SFX!3Ryl?{ce^pJ4_B{sR65J!oBPC%bN7L3R#E!P9i#|iVW&c<)z`fZzvz#5(5 zGx07$igVqzEfeHE6ZMg(OZuFN(GyaTFWN3uaTlLjr5UWvrr<*L77pxvmM9*m-n^RABXK-@D<`q@&6Tjn${TEW@R= zI(Eq3t9A~36E1tALs{val+!8v0E;tk8JDZ1XRD|7vg^F%yake8N_>@gVM+I56`fzJ z{VUjsk{((9<^1FAOWGwbjEForJg&RcL%nRCFWK|?SmkUr^0`MbI-)PtE16};?&U^0 ztElxAoYrKI=5C$UqxF?coaSzM{H-er*S3;neZPC9L+h%?!xeK*;tp*W&iRZU zaK9T}?PtD+-beS#N-L!#5vT=ZO^;HFmS3}4siw7POk+Qqx--%;l<{PP0M~+fJgl7f zTwcBSrLNM!t>?9W6XI)Fu|+!Y!Fe4OT|TN@fplo!PVG(H0rks-r2~q0>g;fQy;U`! zaZRLjNN2hB7O8*+5u$WZL%Gf_ivX%(yL8xysP;y-1NDMPlK*{Co$bm88pA|%50oBg znXPvB(v{dF`=SptFQ|Q^Z#d$S6sxQC^1x%}@nf48Hq;BtBn6a{bd)t48f7R+A+KI( zZ&he$5VKASYJR1&+rELielRKQ{h}uuHZN+^J}XzF!)fCpwCMTKLb+NquOV4So73Fy z^S1A@KBpSeZmB`6*Sui&?uXYqguxtg-BFn%Q|ntruD2FOJKm zJlJV$?VXSA_9npDD4I#fVl_m8 z-|3nq?Thx^uZcP(Md7`ii7AMn*-qY9H*6E&ItNC;rE7FpmF04|Pr3ajD7n{d$C_|% zXa9Ub88bI!#iu?_F3SWZwr|@ZdCBMWhWCX%Tq6q2%V*(H_s;^*F=S|pgJH&h1jQ%7?8JQtoUHNL|B%3_9moH z*5g3#AHE5}a7^10wS+*&kqgpel^V4Pp&Ukf+(8YwYLtYKMcsPy(;D&)4JL%m;BVtP zE1$d9%Pg2XxQ*wgeBOZyvrx8!ZS(w6bN9uWg)D4pLy}YTw0g|KSQOjlUa-qm4ebk_ zpWX)FvCGqJ=?k6X(8e7$lDj{XX{N#hN5GRCjwH z@Wfa5jFMB*jf|5hMgY^yp%bfDaIv~DpXG4D<3Q@1caAG>#p^BNllO2O#eXykzPqk8 zlKAayoHHUub3HW5p8Cc|+z=u0Iig2mdhdX-@XA;@{Zx!GhhuVNQVi*D4}`93e~?J$ zD0lzEKp0kG{&678I@o~@gg3-2Qdb@9{B?g?K~kJW208?S!4Qmsj?G<;gzC!kECIJXh1nWM{WcCOg6yPcl z_%xH5begG;1>1)c|IQ|y(mNlepucjasc(L%Mn(VlG4xo|;}|12TBE$H_O;EFcH>N< zX7G0iuqfnTYOIkQD<+06tMF zn4gYj^xAJcwLxHwoQl9QuHhVUWV+1+ z^)Fo4x_!4@qttlsfl9j;wnJ|&`DHxacgL=IVapre^dT*1x%U;lxwbu#u+u znQyM#Iq^*M@kq;@lW%;&-ap-+HL{46>5}(n06`F?`Dgde{PtS#lM6aF?@NBJK`AGs zsfmi{@@Z1c8jQ_SeMureK(K|&@y&7}-Ur*Z$ei1Io@l8%ocNaysf{18w(ze$*e-Ly zu3Y~ln|jqIgyQzQDA7z={ox~Gk0{`>`5L=n7Hh@o46J9#;G&=RD?3xKrY;(nZ>Nd@Zb7*5rQz zkGERAK*r1LyuJgsf7NfoGVa@V63tcpAO0CBbKYR5{+I6e1LQs&FC2D;f$urq1t$#K zcdHHQe>h~A1w61e`BYZp-#TZw=bOmU-S4gJP5EZN3QOO__stk`nERbYk)v4#lxnr^ z*q_>%HAa<=vkIZ5eDpvtFIA)2{KN@E-v+iSWZfTROT#| zqk`N)`4PfWsVh|sepL-BxEm>z!B^ICVZ5#KuN>zfjNK>uCZ=a;fa4tfva2-j-&RY! zCq33Yr4t|bU%hD zm8SHgQ;h@>LwRtd@o-GXov%+dIwodv7#$T_NX?IwNlXJ+&}%P^xB})AXt{e>d0JwPmedrV-cqU-1^Mt?1ij8txO^{c#vf z9II}q=v)0_ZRLoV=`{~9;DM2lX7%yQN+zY*(o)M`^_bpFYkIYK=w9Xf#jmfPRqUR7 z;dt4z(AN<+6<_huhNk{aC7h%UQ71nDh^CP+w#6CS;@{jA{*ZU{uiX~^+npZ|yWxSu zv_xO4W&J88T~}LAJBqgzft{Sm%kCWNM>PyU!A+S)6$kATGV#^%%|==Vi|1hT1o%v9N(n)Cug z#pIgbD8iz@6aL3P(DaI52^#DK2C)C+@8322q89m&%mGaKzX7wU%rQ3D{)aDy09#vz z4Hz~+-H6c-7&c(^1Nwl%@CAl1FnodG3v>%GeBoc>3*ed0e;V2C7y@h=Hn3&b0E7fa zKVaAZ9T^z?fMEkhKR_+O=m!iNVBLTb6aOW?Fr_`Xm4n_OC!^@YpXvV&l5dO)VTGSE zBM^jiF~cp)2-rD7ia0Qe6+$9;Fp3>QToz&!7lb@ricvfelDz_>cp)ToHT(qB<%f`Q zX{<~DLYfpYY8iyw+l)~wAjE$MMu|Yk;2w+;hme~67_|mM$cHdW4nn*Mm^?+0@iHck zJTZa~p&DkAz`wx4NU$-ifq2gVQj9?fs&b79%9;sen4&Be2=O#SS)dJFC{Z*EG|L=i zu|Y_<1`@jlOuYljS_C1K6DVs5Xyi$h1uWHr5=BcP zq|6az2?4u1F`+zQJC{=^RrCw6JD>n}IfSI2#<;+qfzB9rC4>wE3UF6J$U_&53v3*J z2IGoCh?gtIT@4`}fC7?%-3!iQTnPw?aKpG#06jnfF2JJ69pg#^^v+>iSqQo8fpGyo z{TNjw4a3R?jm|!B> zS0cSp0__@}7cmO-gnl{&fUm!Vl>wYHeJ~2Z8gUt;0E*|XU=#rDHJt(&)%s#(0FiV* zi~`mV_s1w;X}1830#@yzQ^0~Xfmj)^*1aH%QUq4`YEK^1`f4zG1JK$v1iKSx-A$)} z)(=CmGN5%z7)AlDL$6{K(Awo1Mggt6=oHYpDjX{VT2roL6wo?20;7P|r*B{s(7J<8 z0j(<{u`;0b-J2K%v<{5IsPAj-bc+dgVnNk*if4jPvtZUfoy-J}flvSxAJ3qd0?nQ7 zGr?Zq2B5lE858URNTB?grrKh7Jrw3AB4S1z{A>x1CM_b<2XWGN3Iv z1fzhm{-GEJbRDNtK-IP|tPE&cauuV1qTp+yP{}|~-*AiqYL3wU)M3NBSdi@KMyxjW#<-w$ z1u+;G#KE{&j0*zcFf-Q3qoU5CdLJ!(JKq%0_zWAR1pY`E!h|sgvR_Fj0+Hn z$;Ik|(0DlyJMpl@-!Gp%)utO$@a zj+f2S&Xk~c7<&%{p|y=sfnVGOgipK=y^G~T3hCr~(RyFX`~_0s2TCAL8~FJ79E2f$ zp!hGW5Fe^7j0eR?wkrFm^G#~gNAX{`%zF`%xm*~;Y~jn1IDTFy9DQ;4fA`8Tfh`CCjDEoA z2e#h|^iEDpAfnQP+v#2qeP$sk{W#c_2xbs<=olumvDq&t}kYx zdz3)Z;l0RYFNlt@f@=jWEX}QSEd@mc=`2wLJ;Al&B7#H%f{~tu;95<4L9|3dM9^4= zgx(L{i3pk#%&ZK|34j4=6GT4BGBCCjJRw4_A^E9>_4g}S>zEtppp~>vK)gFlbWM$;_j>t>t5u}Nd mL`i~#oRo}|uJ{^V9fF+1NF&*bj6~45iveB5L|n73oto~N-8MblV|UFi zn`93j6ubzARgw@4f>9z84_-awB3`@*l9M;{2k22xzN+f_=$`4$%iz2V_WOq(c_{NYNf&=Jb-|)LJz}-2jHZ2`S`AD5G7|~M{U`v*|CMw5VKwDyHDV)GY)GKy(GgP-yoSy=fxkspguu8nao+f1m(6B8TO zthgUVuq8_34Qst&1*C)f5w%+EyJ^WbEdeMb_K6qwyKRyx_oG{L4Qrh+B3`y{le31& z08{KSyDGFwkW)tjCOD9T5lv+z4BwST{C(PNDtFZ?H#xq!U4hg@Bm&-e9RrZ_2x;m z9nwv-d6U76q;(gSGkMAk`7KM6i1vcD1+t9 zyj*S`nn?yfiTd5xW6QNwS!fDboTiv{h>(&l@K}8>x0G|XJ@0MV_JYi&K>Blp*{EnG zu+upjs@4rE$#}XF%mN_W>2>a|4*dYGf1|}3x`bjjB@8gh62YnaW-0d5ogSZ?U>}nEP&-Q6wddlR-6-q980RIOPDbeH z?3Fhp41`^nEAy$()tBKZW|rbMj9=Pzbvts+(MHaeFxih0k(g+U0nO=vmgK5!BOq`+gli%LZH zIi^S1Os}$EhAVgnanu3l07Y`he`avXpJt61i{Y5X2BBk$Fx>pPA@8CeK;2S*Gh{PKph4}a` z4*+g*K-mSv zoGL|}uRZ9a=N^>iAI-<|h;IPepFSqV{z_zPj_CmOzm7>WxTO3mK*)8O zcwni23D8b7p#;lIZvdmXs(cgBUKUg>m0kn1{X!`#h?(AQSTO8}dBdLP|6y;N{0)T9 za`{i-LyUGUIjV?XwF_2jyJKp1ox`Haf!oz8Q#%+f2Qb96s5HquYNL_mrV3FXx!btE4!` z%*mOn%*g@lWKQic#{8Y0f0)ru?pZmeK-A&uvM*^tZF!`UajHwkHA-gk@J;Ea!guv0 z1bi@6T1EJl3PAMWTDhj(=)0$^%qsk-?ihs`d&n*-Ah0m(yELW zZsGliH64RF2uN8_&}CJzlt!=rOP}dT^)xXc#j0W)z-t^yrafzxO3&;G-c(;B2QaP* z16>7wN7dlSWz*rc^&@#C535CtJb?d2zk37)%v3dKx KM=u{9@ctj}>K~&3 literal 0 HcmV?d00001 diff --git a/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro b/regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro new file mode 100644 index 0000000000000000000000000000000000000000..76465f90e3ae1eb12a2e803ae4e88b04cc078ce2 GIT binary patch literal 4758 zcmbVPU1%It6xNuEf3Y!yTCwVN)apYbGn?IXvl_89sZ>%Gw;^gM%iWo~yQA~7&fLwq z4Y3jmieT|4eNZ0+l|o-cB=t=SQl$9MNAX3Zq9A>dA_%1lo;yGH-r3#RpFZyFo%5ah zopZkP&GY3~Zyeu+*F5)eU#}4xpHS8n+Nu+!pm;=gJwusN9Bi}SHg?Pk@oA}IS_B?f zOkx?nQdnK@I8n#61}CCGKJ`qe3cn3khd4{Fj;ZN7X!*rMsA;0x#XjOvY$28)^37uenGPzqDDmTTqrjRJxQ;z)vV2N@o#-V%=82mQ zY}D+()N5aQdr$}C<)$=SdfDq2>y%vg(NgSzJDI|Hr^eJ?DFd#jnE=DJN1eVDUTfyibcOmGRUio$s z?YYg4ToXKp6tH967|`UV=M3}>${9$FD;Fy!pl{fifYhjRz4+uac?hR;RdE`YrGV^2 zQ;R(hx8QFWb{n*gKW`UDFQ?|?o=#{Y(*bNcJ`5=f9LFuYu0^l|4?>Z{5RH2MI)H<>h6mO*j&4jT zFl{zSL$p-#8~kF$hHYa|40aWQ^_rlw=Y3RJ-CrZEl1(UP`+)aN8k38T?zQT{R)zOa z{2nS>Zn-C^`cE#%exZaz~X4GtcZdRSm<+7@p%V%bG2dPg3 zCL9=1_W!9j%a&KoxswpM!x6WhxllNU1{iS@rRBmsIHo@ zJCQr-bViZy@nBJ!9_-Tsgd%+Ahwwpcph6U_6NNjV6r6;5r#d15k>L6A+?Ev7s0mAvPFY0W>j5VL_M% za(9`+PonucycSPh6UFBI3>}{H6Nq!@X9a>NZ%oh705Lt?;;TXixXv)=C3vY|%ja^* z_1FSe4aeG+Eq@@HGOpaxr)j97bG&nfMGF0xKu>6O?3MairEdOAFq-cI8#pt8nz%E! zX#HTjI5dF@%PRV0Pnxk*P}2dk!fZnDeLjdQ6JTOLB@NinVChaQ;k86uRxdDNVbVPu zm&jFgL7TY}LEFlaWTvOf;xHM(f19C7gGoq0@w$x_@boZD7IUPMUAl$&^&8HgJiL7R zx4X}+TsXD)&(}BqIHTZx$3D2D_Q&DOz2E#hIl6ar_Y==u8hdhc z$NYOMSH@4ilK=aYzrOqM$~)D!cHETvFFP?cw*AP!7XYM)3H$&6 delta 7 OcmeysbBkv~6dM2z8v^qH diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out new file mode 100644 index 00000000000000..a7c41169d8dcb3 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -0,0 +1,123 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !variant_evolution_initial_snapshot -- +1 initial 10 v1 + +-- !variant_evolution_initial_tag -- +1 initial 10 v1 + +-- !variant_evolution_initial_time -- +1 initial 10 v1 + +-- !variant_root_projection -- +1 false {"arr":[1,2],"n":10,"name":"alice","nested":{"city":"hz"},"ok":true,"ratio":1.5} +10 false {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 false {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +2 false {"arr":[3,4],"n":20,"name":"bob","nested":{"city":"sh"},"ok":false,"ratio":2.5} +3 false {"n":30,"name":"same","ok":true} +4 false null +5 true \N +6 false 42 +7 false root-string +8 false {"n":30,"name":"same","ok":true} +9 false {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_path_expressions -- +1 ALICE 11 1.5 true 3 hz +10 DAVE 51 5.5 false 15 sz +11 NULL 61 6.5 true 19 null +2 BOB 21 2.5 false 7 sh +9 CAROL 41 4.5 true 11 bj + +-- !variant_filter -- +11 null 60 +3 same 30 +8 same 30 +9 carol 40 + +-- !variant_cross_file_leaf_projection -- +1 10 +10 50 +11 60 +2 20 +3 30 +4 \N +5 \N +6 \N +7 \N +8 30 +9 40 + +-- !variant_implicit_shredded_filter -- +10 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_page_pruning_result -- +1095 3001 4095 + +-- !variant_aggregate -- +false 2 70 4 +true 5 170 4.17 + +-- !variant_join -- +10 fifty dave +2 twenty bob + +-- !variant_null_count_distinct -- +11 10 1 9 + +-- !variant_count_pushdown -- +10 11 + +-- !variant_canonical_group -- +{"n":30,"name":"same","ok":true} 2 + +-- !variant_nested_projection -- +1 first {"deep":{"name":"inside"},"x":11} {"kind":"open","score":101} {"enabled":true,"score":1001} +2 second \N null {"enabled":false,"score":2002} + +-- !variant_nested_filter -- +1 + +-- !variant_nested_expressions -- +1 11 inside open \N true 1002 +2 \N \N \N 202 false 2003 + +-- !variant_evolution_renamed_snapshot -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_renamed_tag -- +1 initial 10 v1 +2 renamed 20 v2 + +-- !variant_evolution_added_reordered -- +1 initial \N v1 +2 renamed \N v2 +3 with-aux 300 v3 + +-- !variant_evolution_dropped -- +1 \N v1 +2 \N v2 +3 300 v3 +4 400 v4 + +-- !variant_evolution_drop_readd -- +1 \N \N \N v1 +2 \N \N \N v2 +3 300 \N \N v3 +4 400 \N \N v4 +5 500 readded 50 v5 + +-- !variant_delete_only_merge -- +0 + +-- !variant_orc_missing_column -- +1 \N + +-- !variant_orc_count_star -- +1 + +-- !variant_mixed_format -- +1 {"format":"parquet"} +2 \N diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy index a5f091298a7a2d..c6d87e32693c45 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy @@ -22,6 +22,7 @@ suite("test_iceberg_varbinary", "p0,external") { logger.info("disable iceberg test.") return } + sql "SET ENABLE_VARIANT_V2=true" String catalog_name_no_mapping = "test_iceberg_no_mapping" String catalog_name_with_mapping = "test_iceberg_with_mapping" @@ -29,13 +30,18 @@ suite("test_iceberg_varbinary", "p0,external") { String rest_port = context.config.otherConfigs.get("iceberg_rest_uri_port") String minio_port = context.config.otherConfigs.get("iceberg_minio_port") String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + // A container-advertised REST URI may differ from the host port-forward address. + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${rest_port}" + } sql """drop catalog if exists ${catalog_name_no_mapping}""" sql """ CREATE CATALOG ${catalog_name_no_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -50,7 +56,7 @@ suite("test_iceberg_varbinary", "p0,external") { CREATE CATALOG ${catalog_name_with_mapping} PROPERTIES ( 'type'='iceberg', 'iceberg.catalog.type'='rest', - 'uri' = 'http://${externalEnvIp}:${rest_port}', + 'uri' = '${restUri}', "s3.access_key" = "admin", "s3.secret_key" = "password", "s3.endpoint" = "http://${externalEnvIp}:${minio_port}", @@ -166,10 +172,8 @@ suite("test_iceberg_varbinary", "p0,external") { qt_select23 """ select id from test_variant_repro; """ - test { - sql """ - select * from test_variant_repro; - """ - exception "UNSUPPORTED" - } + sql """set enable_file_scanner_v2=true""" + qt_select_variant """ + select id, cast(v as string) from test_variant_repro order by id; + """ } diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy new file mode 100644 index 00000000000000..7b3bc3e833d165 --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -0,0 +1,551 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import java.util.regex.Matcher +import java.util.regex.Pattern +import com.amazonaws.auth.AWSStaticCredentialsProvider +import com.amazonaws.auth.BasicAWSCredentials +import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration +import com.amazonaws.services.s3.AmazonS3ClientBuilder +import org.apache.doris.regression.action.ProfileAction + +suite("test_iceberg_variant_read", + "p0,external,iceberg,external_docker,external_docker_iceberg") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (enabled == null || !enabled.equalsIgnoreCase("true")) { + logger.info("disable iceberg test") + return + } + sql "SET ENABLE_VARIANT_V2=true" + + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String restUri = context.config.otherConfigs.get("iceberg_rest_uri") + if (restUri == null) { + restUri = "http://${externalEnvIp}:${restPort}" + } + String catalogName = "test_iceberg_variant_read" + String dbName = "iceberg_variant_read_db" + String fixtureKey = "doris-regression/iceberg-variant/iceberg_variant_shredded.parquet" + File shreddedFixture = new File(context.dataPath, "iceberg_variant_shredded.parquet") + File shreddedTableFixture = new File(context.dataPath, "iceberg_variant_shredded_table") + String shreddedMetadataName = + "00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json" + assertTrue(shreddedFixture.isFile(), "Missing shredded Variant Parquet fixture") + assertTrue(shreddedTableFixture.isDirectory(), "Missing shredded Variant table fixture") + def credentials = new BasicAWSCredentials("admin", "password") + def endpoint = new EndpointConfiguration( + "http://${externalEnvIp}:${minioPort}", "us-east-1") + def minioClient = AmazonS3ClientBuilder.standard() + .withEndpointConfiguration(endpoint) + .withPathStyleAccessEnabled(true) + .withCredentials(new AWSStaticCredentialsProvider(credentials)) + .build() + + def latestSnapshotId = { String tableName -> + List> rows = spark_iceberg """ + SELECT snapshot_id + FROM demo.${dbName}.${tableName}.snapshots + ORDER BY committed_at DESC + LIMIT 1 + """ + assertEquals(1, rows.size()) + return rows[0][0].toString() + } + + spark_iceberg_multi """ + CREATE NAMESPACE IF NOT EXISTS demo.${dbName}; + DROP TABLE IF EXISTS demo.${dbName}.variant_values; + CREATE TABLE demo.${dbName}.variant_values ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false', + 'write.merge.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_values VALUES + (1, parse_json('{"name":"alice","n":10,"ratio":1.5,"ok":true,"arr":[1,2],"nested":{"city":"hz"}}')), + (2, parse_json('{"name":"bob","n":20,"ratio":2.5,"ok":false,"arr":[3,4],"nested":{"city":"sh"}}')), + (3, parse_json('{"name":"same","n":30,"ok":true}')), + (4, parse_json('null')), + (5, NULL), + (6, parse_json('42')), + (7, parse_json('"root-string"')); + ALTER TABLE demo.${dbName}.variant_values SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_values + WITH (`shred-variants`=true, `variant-inference-buffer-size`=100) VALUES + (8, parse_json('{"ok":true,"n":30,"name":"same"}')), + (9, parse_json('{"name":"carol","n":40,"ratio":4.5,"ok":true,"arr":[5,6],"nested":{"city":"bj"},"new_key":"new"}')), + (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), + (11, parse_json('{"name":null,"n":60,"ratio":6.5,"ok":true,"arr":[9,10],"nested":{"city":null}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_page_pruning; + + DROP TABLE IF EXISTS demo.${dbName}.variant_nested; + CREATE TABLE demo.${dbName}.variant_nested ( + id INT, + info STRUCT, + events ARRAY, + attrs MAP + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_nested SELECT + 1, + named_struct('label', 'first', 'payload', parse_json('{"x":11,"deep":{"name":"inside"}}')), + array(parse_json('{"kind":"open","score":101}'), parse_json('2')), + map('primary', parse_json('{"enabled":true,"score":1001}')); + INSERT INTO demo.${dbName}.variant_nested SELECT + 2, + named_struct('label', 'second', 'payload', CAST(NULL AS VARIANT)), + array(parse_json('null'), parse_json('{"kind":"close","score":202}')), + map('primary', parse_json('{"enabled":false,"score":2002}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_signed_selector; + CREATE TABLE demo.${dbName}.variant_signed_selector ( + id INT, + v VARIANT + ) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_signed_selector + VALUES (1, parse_json('{"-1":41}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_evolution; + CREATE TABLE demo.${dbName}.variant_evolution ( + id INT, + payload VARIANT, + note STRING + ) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_evolution + VALUES (1, parse_json('{"stage":"initial","metric":10}'), 'v1'); + + DROP TABLE IF EXISTS demo.${dbName}.variant_write_guard; + CREATE TABLE demo.${dbName}.variant_write_guard (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); + """ + + String writeGuardSourceSnapshot = latestSnapshotId("variant_write_guard") + spark_iceberg """ + ALTER TABLE demo.${dbName}.variant_write_guard ADD COLUMN payload VARIANT + """ + + // Register a stable Iceberg metadata fixture so the page-pruning case always uses a + // standards-compliant shredded Variant file, independent of the Spark writer version. + minioClient.putObject("warehouse", fixtureKey, shreddedFixture) + shreddedTableFixture.eachFile { File fixtureFile -> + minioClient.putObject("warehouse", + "wh/${dbName}/variant_page_pruning/metadata/${fixtureFile.name}", fixtureFile) + } + minioClient.shutdown() + spark_iceberg """ + CALL demo.system.register_table( + table => '${dbName}.variant_page_pruning', + metadata_file => + 's3a://warehouse/wh/${dbName}/variant_page_pruning/metadata/${shreddedMetadataName}') + """ + + sql """drop catalog if exists ${catalogName}""" + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + 'type'='iceberg', + 'iceberg.catalog.type'='rest', + 'uri'='${restUri}', + 's3.access_key'='admin', + 's3.secret_key'='password', + 's3.endpoint'='http://${externalEnvIp}:${minioPort}', + 's3.region'='us-east-1', + 'meta.cache.iceberg.table.ttl-second'='0', + 'meta.cache.iceberg.schema.ttl-second'='0' + ) + """ + + sql """switch ${catalogName}""" + sql """use ${dbName}""" + sql """set enable_file_scanner_v2=true""" + sql """set enable_profile=true""" + sql """set profile_level=2""" + + def profileAction = new ProfileAction(context) + def getProfileByToken = { String token -> + for (int retry = 0; retry < 20; ++retry) { + List profileData = profileAction.getProfileList() + for (final def profileItem in profileData) { + if (profileItem["Sql Statement"].toString().contains(token)) { + return profileAction.getProfile(profileItem["Profile ID"].toString()) + } + } + Thread.sleep(500) + } + throw new IllegalStateException("Missing profile for token: " + token) + } + def counterSum = { String profile, String counterName -> + Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") + Matcher matcher = pattern.matcher(profile) + long sum = 0 + while (matcher.find()) { + sum += Long.parseLong(matcher.group(1).replace(",", "")) + } + return sum + } + + String evolutionInitial = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_initial""" + + order_qt_variant_evolution_initial_snapshot """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionInitial} + ORDER BY id + """ + + order_qt_variant_evolution_initial_tag """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_initial' + ORDER BY id + """ + + List> initialTime = sql """ + SELECT date_format(date_add(committed_at, interval 1 second), '%Y-%m-%d %H:%i:%s') + FROM variant_evolution\$snapshots + WHERE snapshot_id = ${evolutionInitial} + """ + assertEquals(1, initialTime.size()) + order_qt_variant_evolution_initial_time """ + SELECT id, CAST(payload['stage'] AS STRING), CAST(payload['metric'] AS INT), note + FROM variant_evolution FOR TIME AS OF "${initialTime[0][0]}" + ORDER BY id + """ + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN payload TO event; + INSERT INTO demo.${dbName}.variant_evolution + VALUES (2, parse_json('{"stage":"renamed","metric":20}'), 'v2'); + """ + String evolutionRenamed = latestSnapshotId("variant_evolution") + sql """ALTER TABLE variant_evolution CREATE TAG variant_renamed""" + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN aux VARIANT; + ALTER TABLE demo.${dbName}.variant_evolution ALTER COLUMN aux FIRST; + INSERT INTO demo.${dbName}.variant_evolution (id, event, note, aux) + VALUES (3, parse_json('{"stage":"with-aux","metric":30}'), 'v3', + parse_json('{"side":300}')); + """ + String evolutionWithAux = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution RENAME COLUMN aux TO sidecar; + ALTER TABLE demo.${dbName}.variant_evolution DROP COLUMN event; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar) + VALUES (4, 'v4', parse_json('{"side":400}')); + """ + String evolutionDropped = latestSnapshotId("variant_evolution") + + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_evolution ADD COLUMN event VARIANT; + INSERT INTO demo.${dbName}.variant_evolution (id, note, sidecar, event) + VALUES (5, 'v5', parse_json('{"side":500}'), + parse_json('{"stage":"readded","metric":50}')); + + -- Write ORC before evolving the logical schema to Variant. This retains valid ORC files + -- in the snapshots while using Iceberg FileIO instead of Spark native ORC, whose optional + -- S3A implementation may not be installed. + DROP TABLE IF EXISTS demo.${dbName}.variant_orc; + CREATE TABLE demo.${dbName}.variant_orc (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_orc VALUES (1); + ALTER TABLE demo.${dbName}.variant_orc ADD COLUMN v VARIANT; + + DROP TABLE IF EXISTS demo.${dbName}.variant_mixed_format; + CREATE TABLE demo.${dbName}.variant_mixed_format (id INT) USING iceberg + TBLPROPERTIES ('format-version'='3', 'write.format.default'='orc'); + INSERT INTO demo.${dbName}.variant_mixed_format VALUES (2); + ALTER TABLE demo.${dbName}.variant_mixed_format ADD COLUMN v VARIANT; + ALTER TABLE demo.${dbName}.variant_mixed_format SET TBLPROPERTIES + ('write.format.default'='parquet'); + INSERT INTO demo.${dbName}.variant_mixed_format + VALUES (1, parse_json('{"format":"parquet"}')); + """ + String evolutionReadded = latestSnapshotId("variant_evolution") + + // Root projection covers objects, arrays, scalars, Variant null and SQL NULL. + order_qt_variant_root_projection """ + SELECT id, v IS NULL, CAST(v AS STRING) + FROM variant_values + ORDER BY id + """ + + order_qt_variant_path_expressions """ + SELECT id, + UPPER(CAST(v['name'] AS STRING)), + CAST(v['n'] AS INT) + 1, + ROUND(CAST(v['ratio'] AS DOUBLE), 1), + CAST(v['ok'] AS BOOLEAN), + ARRAY_SUM(CAST(v['arr'] AS ARRAY)), + CAST(v['nested']['city'] AS STRING) + FROM variant_values + WHERE id IN (1, 2, 9, 10, 11) + ORDER BY id + """ + + order_qt_variant_filter """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT) + FROM variant_values + WHERE CAST(v['n'] AS INT) >= 20 + AND CAST(v['ok'] AS BOOLEAN) = true + ORDER BY id + """ + + // The first INSERT is unshredded while the second is shredded. Keep both small files on one + // scanner so their complete and leaf-only physical states must be projected before batching. + sql "set parallel_pipeline_task_num=1" + sql "set max_file_scanners_concurrency=1" + order_qt_variant_cross_file_leaf_projection """ + SELECT id, CAST(v['n'] AS INT) + FROM variant_values + ORDER BY id + """ + + // Keep the root Variant as output while the implicit scalar comparison drives the shredded + // typed_value statistics/page-index path. + order_qt_variant_implicit_shredded_filter """ + SELECT id, CAST(v AS STRING) + FROM variant_values + WHERE v['n'] > 35 + ORDER BY id + """ + + // The query projects the root Variant, while the predicate uses typed_value page metadata. + String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() + sql """ + SELECT '${pagePruningToken}', id, CAST(v AS STRING) + FROM variant_page_pruning + WHERE v['n'] > 3000 + ORDER BY id + """ + String pagePruningProfile = getProfileByToken(pagePruningToken).toString() + assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, + "Shredded Variant typed_value did not filter any Parquet page") + // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant + // root is read through the independent deferred-output projection. + assertTrue(counterSum(pagePruningProfile, "VariantLeafProjections") > 0, + "A root Variant output query did not retain its typed predicate leaf projection") + String leafProjectionToken = + "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() + sql """ + SELECT '${leafProjectionToken}', COUNT(*) + FROM variant_page_pruning + WHERE v['n'] > 3000 + """ + String leafProjectionProfile = getProfileByToken(leafProjectionToken).toString() + assertTrue(counterSum(leafProjectionProfile, "VariantLeafProjections") > 0, + "Variant typed predicate did not retain a physical leaf projection") + qt_variant_page_pruning_result """ + SELECT COUNT(*), MIN(id), MAX(id) + FROM variant_page_pruning + WHERE v['n'] > 3000 + """ + + order_qt_variant_aggregate """ + SELECT CAST(v['ok'] AS BOOLEAN), + COUNT(*), + SUM(CAST(v['n'] AS INT)), + ROUND(AVG(CAST(v['ratio'] AS DOUBLE)), 2) + FROM variant_values + WHERE v['name'] IS NOT NULL + GROUP BY CAST(v['ok'] AS BOOLEAN) + ORDER BY 1 + """ + + order_qt_variant_join """ + WITH thresholds AS ( + SELECT 20 AS n, 'twenty' AS label + UNION ALL + SELECT 50 AS n, 'fifty' AS label + ) + SELECT t.id, d.label, CAST(t.v['name'] AS STRING) + FROM variant_values t + JOIN thresholds d ON CAST(t.v['n'] AS INT) = d.n + ORDER BY t.id + """ + + qt_variant_null_count_distinct """ + SELECT COUNT(*), COUNT(v), SUM(v IS NULL), COUNT(DISTINCT v) + FROM variant_values + """ + + qt_variant_count_pushdown """ + SELECT COUNT(v), COUNT(*) + FROM variant_values + """ + + order_qt_variant_canonical_group """ + SELECT CAST(v AS STRING), COUNT(*) + FROM variant_values + GROUP BY v + HAVING COUNT(*) > 1 + ORDER BY 1 + """ + + order_qt_variant_nested_projection """ + SELECT id, + info.label, + CAST(info.payload AS STRING), + CAST(events[1] AS STRING), + CAST(element_at(attrs, 'primary') AS STRING) + FROM variant_nested + ORDER BY id + """ + + // Spark may leave nested Variant values unshredded even when top-level shredding is enabled. + // Keep the external-table regression focused on correctness; mapper/reader unit tests use a + // physical typed_value fixture to verify nested leaf projection. + order_qt_variant_nested_filter """ + SELECT id + FROM variant_nested + WHERE CAST(info.payload['x'] AS INT) > 0 + ORDER BY id + """ + + // Signed integer selectors are array indexes, even when a shredded object has a key with the + // same serialized token. The ambiguous scanner path must retain enough state for both results. + List> signedSelectorRows = sql """ + SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT) + FROM variant_signed_selector + """ + assertEquals(1, signedSelectorRows.size()) + assertEquals(null, signedSelectorRows[0][0]) + assertEquals("41", signedSelectorRows[0][1].toString()) + + order_qt_variant_nested_expressions """ + SELECT id, + CAST(info.payload['x'] AS INT), + CAST(info.payload['deep']['name'] AS STRING), + CAST(events[1]['kind'] AS STRING), + CAST(events[2]['score'] AS INT), + CAST(element_at(attrs, 'primary')['enabled'] AS BOOLEAN), + CAST(element_at(attrs, 'primary')['score'] AS INT) + 1 + FROM variant_nested + ORDER BY id + """ + + order_qt_variant_evolution_renamed_snapshot """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionRenamed} + ORDER BY id + """ + + order_qt_variant_evolution_renamed_tag """ + SELECT id, CAST(event['stage'] AS STRING), CAST(event['metric'] AS INT), note + FROM variant_evolution FOR VERSION AS OF 'variant_renamed' + ORDER BY id + """ + + order_qt_variant_evolution_added_reordered """ + SELECT id, + CAST(event['stage'] AS STRING), + CAST(aux['side'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionWithAux} + ORDER BY id + """ + + order_qt_variant_evolution_dropped """ + SELECT id, CAST(sidecar['side'] AS INT), note + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + ORDER BY id + """ + + test { + sql """ + SELECT event + FROM variant_evolution FOR VERSION AS OF ${evolutionDropped} + """ + exception "event" + } + + order_qt_variant_evolution_drop_readd """ + SELECT id, + CAST(sidecar['side'] AS INT), + CAST(event['stage'] AS STRING), + CAST(event['metric'] AS INT), + note + FROM variant_evolution FOR VERSION AS OF ${evolutionReadded} + ORDER BY id + """ + + test { + sql """SELECT payload FROM variant_evolution""" + exception "payload" + } + + test { + sql """ + INSERT INTO variant_write_guard (id) + SELECT id + FROM variant_write_guard FOR VERSION AS OF ${writeGuardSourceSnapshot} + """ + exception "Iceberg VARIANT columns are read-only and cannot be written" + } + + // A delete-only MERGE emits only position deletes. It must remain available even though + // update/insert actions would route the unchanged Variant through the unsupported data writer. + sql """ + MERGE INTO variant_values t + USING (SELECT 11 AS id) s + ON t.id = s.id + WHEN MATCHED THEN DELETE + """ + qt_variant_delete_only_merge "SELECT COUNT(*) FROM variant_values WHERE id = 11" + + // Files written before the Variant field existed have no physical Variant payload. Schema + // evolution must synthesize NULL instead of rejecting their non-Parquet file format. + order_qt_variant_orc_missing_column """ + SELECT id, CAST(v AS STRING) FROM variant_orc ORDER BY id + """ + qt_variant_orc_count_star "SELECT COUNT(*) FROM variant_orc" + order_qt_variant_mixed_format """ + SELECT id, CAST(v AS STRING) FROM variant_mixed_format ORDER BY id + """ + + sql """set enable_file_scanner_v2=false""" + try { + test { + sql """SELECT CAST(v AS STRING) FROM variant_values ORDER BY id""" + exception "legacy file scanner does not support VARIANT" + } + } finally { + sql """set enable_file_scanner_v2=true""" + } +} diff --git a/regression-test/suites/variant_p0/variant_with_rowstore.groovy b/regression-test/suites/variant_p0/variant_with_rowstore.groovy index db83c8ae1158e4..f7bd05f4922f36 100644 --- a/regression-test/suites/variant_p0/variant_with_rowstore.groovy +++ b/regression-test/suites/variant_p0/variant_with_rowstore.groovy @@ -44,7 +44,9 @@ suite("regression_test_variant_rowstore", "variant_type"){ sql """insert into ${table_name} select * from (select -2, '{"a": 11245, "b" : [123, {"xx" : 1}], "c" : {"c" : 456, "d" : "null", "e" : 7.111}}' as json_str union all select -1, '{"a": 1123}' as json_str union all select *, '{"a" : 1234, "xxxx" : "kaana"}' as json_str from numbers("number" = "4096"))t order by 1 limit 4096 ;""" sql "sync" - qt_sql "select * from ${table_name} order by k limit 10" + // Row-store and column-store Variant readers may emit different insignificant JSON spacing. + // Normalize it so this suite continues to verify that both paths preserve the same values. + qt_sql "select k, replace(cast(v as string), ', ', ',') from ${table_name} order by k limit 10" table_name = "multi_var_rs" @@ -60,7 +62,8 @@ suite("regression_test_variant_rowstore", "variant_type"){ properties("replication_num" = "1", "disable_auto_compaction" = "false", "store_row_column" = "true"); """ sql """insert into ${table_name} select k, cast(v as string), cast(v as string) from var_rowstore""" - qt_sql "select * from ${table_name} order by k limit 10" + qt_sql """select k, replace(cast(v as string), ', ', ','), + replace(cast(v1 as string), ', ', ',') from ${table_name} order by k limit 10""" // Parse url def user = context.config.jdbcUser From 8e3b31681c3354e22a0a2451ead1e22d1afe2e5d Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 12:04:16 +0800 Subject: [PATCH 03/20] [fix](catalog) Safely publish storage adapter snapshots (#66392) ### What problem does this PR solve? Issue Number: None Related PR: #66392 Problem Summary: Master replaced the legacy Hadoop property cache with a shared storage adapter snapshot. Preserve the original fix invariant by publishing an immutable type-keyed map so connector consumers cannot mutate catalog-wide state after publication. Add deterministic coverage for atomic publication and snapshot immutability. ### Release note Prevent connector consumers from modifying shared catalog storage adapter snapshots. ### Check List (For Author) - Test: Unit Test (`CatalogPropertyTest`) - Behavior changed: No. This hardens the existing snapshot contract. - Does this need documentation: No --- .../doris/datasource/CatalogProperty.java | 4 +- .../doris/datasource/CatalogPropertyTest.java | 104 ++++++++++++++++++ 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java index 2fb365e9e4db41..b378ee5f426fa2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java @@ -199,7 +199,9 @@ private StorageBindings initStorageAdapters() { throw new IllegalStateException( "Duplicate storage type: " + a.getType()); }, LinkedHashMap::new)); - local = new StorageBindings(ordered, byType); + // Consumers share the published map without locking, so prevent caller-specific + // mutations from changing the catalog-wide snapshot after publication. + local = new StorageBindings(ordered, Collections.unmodifiableMap(byType)); this.storageBindings = local; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java new file mode 100644 index 00000000000000..6a47944748e4ed --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java @@ -0,0 +1,104 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.datasource.storage.StorageTypeId; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; + +public class CatalogPropertyTest { + + @Test + public void testStorageAdaptersArePublishedAfterInitialization() throws Exception { + CountDownLatch initializationStarted = new CountDownLatch(1); + CountDownLatch allowInitialization = new CountDownLatch(1); + CatalogProperty catalogProperty = new CatalogProperty( + null, Collections.singletonMap("fs.defaultFS", "hdfs://test-ns")); + catalogProperty.setPluginDerivedStorageDefaultsSupplier(() -> { + initializationStarted.countDown(); + awaitInitialization(allowInitialization); + return Collections.emptyMap(); + }); + + ExecutorService executor = Executors.newSingleThreadExecutor(); + AtomicReference> readerResult = new AtomicReference<>(); + Thread concurrentReader = new Thread( + () -> readerResult.set(catalogProperty.getStorageAdaptersMap())); + try { + Future> initializer = + executor.submit(catalogProperty::getStorageAdaptersMap); + Assert.assertTrue(initializationStarted.await(5, TimeUnit.SECONDS)); + + concurrentReader.start(); + Assert.assertTrue(waitUntilBlockedOrTerminated(concurrentReader, 5, TimeUnit.SECONDS)); + Assert.assertEquals("The reader must block until initialization publishes the completed map", + Thread.State.BLOCKED, concurrentReader.getState()); + + allowInitialization.countDown(); + Map initialized = initializer.get(5, TimeUnit.SECONDS); + concurrentReader.join(TimeUnit.SECONDS.toMillis(5)); + Assert.assertFalse(concurrentReader.isAlive()); + Assert.assertSame(initialized, readerResult.get()); + } finally { + allowInitialization.countDown(); + concurrentReader.interrupt(); + executor.shutdownNow(); + } + } + + @Test + public void testStorageAdaptersCacheIsImmutable() { + CatalogProperty catalogProperty = new CatalogProperty( + null, Collections.singletonMap("fs.defaultFS", "hdfs://test-ns")); + catalogProperty.setPluginDerivedStorageDefaultsSupplier(Collections::emptyMap); + + Map storageAdapters = catalogProperty.getStorageAdaptersMap(); + Assert.assertThrows(UnsupportedOperationException.class, storageAdapters::clear); + } + + private static boolean waitUntilBlockedOrTerminated(Thread thread, long timeout, TimeUnit timeUnit) { + long deadline = System.nanoTime() + timeUnit.toNanos(timeout); + while (thread.isAlive() && thread.getState() != Thread.State.BLOCKED + && System.nanoTime() < deadline) { + Thread.yield(); + } + return !thread.isAlive() || thread.getState() == Thread.State.BLOCKED; + } + + private static void awaitInitialization(CountDownLatch allowInitialization) { + try { + if (!allowInitialization.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("Timed out waiting to continue storage adapter initialization"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError("Interrupted while initializing storage adapters", e); + } + } +} From d9f39c979b59d188fef2fc90ab3b64faebc106bc Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 18:44:15 +0800 Subject: [PATCH 04/20] [fix](iceberg) Harden Variant compatibility and validation Issue Number: None Related PR: #66413 Problem Summary: Review follow-ups found that Variant metadata pruning could bypass an earlier error-producing predicate, late runtime-filter refresh could shift deferred Variant output slots, and mixed-version clusters could schedule unsupported Variant scans or delete-only MERGE plans. The connector SPI version also did not reflect its expanded public surface. In addition, debug Boolean validation filtered and copied large nullable complex columns even when they contained no Boolean values, which could exhaust query memory. This change preserves the safe pruning prefix and scan layout, introduces execution-version compatibility gates, bumps and freezes the connector SPI surface, pins metadata-count checks to the selected snapshot, and skips allocation-heavy Boolean filtering when no Boolean subcolumn exists. Iceberg Variant scans and delete-only MERGE now enforce rolling-upgrade compatibility, and debug column validation avoids copying non-Boolean complex payloads. - Test: Unit Test and Regression Test - Focused ASAN BE unit tests for Boolean validation, Variant scan refresh, metadata pruning, page filtering, and MERGE compatibility - FE compatibility and connector SPI surface unit tests - Generated Iceberg Variant regression golden output - FE Checkstyle and BE clang-format - Behavior changed: Yes. Unsafe metadata pruning and mixed-version Variant execution are rejected or conservatively evaluated, and non-Boolean nullable complex columns are validated without copying their payload. - Does this need documentation: No --- be/src/agent/be_exec_version_manager.cpp | 4 +- be/src/agent/be_exec_version_manager.h | 1 + be/src/core/column/column.cpp | 27 ++++++++ be/src/exec/sink/viceberg_merge_sink.cpp | 8 +++ be/src/format_v2/column_mapper.cpp | 12 +++- be/src/format_v2/column_mapper.h | 3 +- be/src/format_v2/file_reader.h | 4 ++ .../format_v2/parquet/parquet_statistics.cpp | 28 ++++---- be/src/format_v2/table_reader.cpp | 57 +++++++--------- be/src/format_v2/table_reader.h | 1 + be/test/core/column/column_self_check.cpp | 65 +++++++++++++++++++ .../exec/sink/viceberg_merge_sink_test.cpp | 22 +++++++ be/test/format_v2/column_mapper_test.cpp | 61 +++++++++++++++++ .../parquet/parquet_statistics_test.cpp | 21 ++++++ be/test/format_v2/table_reader_test.cpp | 42 ++++++++++++ .../java/org/apache/doris/common/Config.java | 2 +- .../spi/ConnectorPluginSurfaceTest.java | 2 + .../resources/connector-plugin-surface.txt | 25 +++++++ .../datasource/scan/PluginDrivenScanNode.java | 19 +++++- .../doris/planner/PluginDrivenTableSink.java | 9 +++ ...PluginDrivenScanNodeCompatibilityTest.java | 41 ++++++++++++ .../planner/PluginDrivenTableSinkTest.java | 21 ++++++ .../iceberg/test_iceberg_variant_read.out | 3 + .../iceberg/test_iceberg_variant_read.groovy | 16 +++-- 24 files changed, 440 insertions(+), 54 deletions(-) diff --git a/be/src/agent/be_exec_version_manager.cpp b/be/src/agent/be_exec_version_manager.cpp index 4c3bdbd9f31445..bbaa0565a9fa3c 100644 --- a/be/src/agent/be_exec_version_manager.cpp +++ b/be/src/agent/be_exec_version_manager.cpp @@ -126,8 +126,10 @@ void BeExecVersionManager::check_function_compatibility(int current_be_exec_vers // a. use new fixed object serialization way. // 11: start from master // a. enforce Iceberg SQL MERGE cardinality only when every executing BE supports it. +// 12: start from master +// a. support Variant columns and delete-only writer omission in Iceberg SQL MERGE. -const int BeExecVersionManager::max_be_exec_version = 11; +const int BeExecVersionManager::max_be_exec_version = 12; const int BeExecVersionManager::min_be_exec_version = 0; std::map> BeExecVersionManager::_function_change_map {}; std::set BeExecVersionManager::_function_restrict_map; diff --git a/be/src/agent/be_exec_version_manager.h b/be/src/agent/be_exec_version_manager.h index c1a40e35a075e1..4582b84c61d744 100644 --- a/be/src/agent/be_exec_version_manager.h +++ b/be/src/agent/be_exec_version_manager.h @@ -27,6 +27,7 @@ namespace doris { constexpr inline int USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10; constexpr inline int SUPPORT_ICEBERG_MERGE_CARDINALITY_VERSION = 11; +constexpr inline int SUPPORT_ICEBERG_VARIANT_VERSION = 12; class BeExecVersionManager { public: diff --git a/be/src/core/column/column.cpp b/be/src/core/column/column.cpp index ce49646b8393bb..ad2900552bc601 100644 --- a/be/src/core/column/column.cpp +++ b/be/src/core/column/column.cpp @@ -28,6 +28,28 @@ namespace doris { +namespace { + +bool contains_boolean_value_column(const IColumn& column) { + if (const auto* nullable = check_and_get_column(column)) { + return contains_boolean_value_column(nullable->get_nested_column()); + } + if (check_and_get_column(column)) { + return true; + } + + bool contains_boolean = false; + IColumn::ColumnCallback callback = [&](const IColumn& subcolumn) { + if (!contains_boolean && contains_boolean_value_column(subcolumn)) { + contains_boolean = true; + } + }; + column.for_each_subcolumn(callback); + return contains_boolean; +} + +} // namespace + std::string IColumn::dump_structure() const { std::stringstream res; res << get_name() << "(size = " << size(); @@ -64,6 +86,11 @@ bool IColumn::column_boolean_check() const { if (const auto* col_nullable = check_and_get_column(*this)) { // for column nullable, we need to skip null values check const auto& nested_col = col_nullable->get_nested_column(); + // Do not materialize complex payloads that cannot contain Boolean values; filtering a + // multi-GB nested column solely for debug validation can exhaust query memory. + if (!contains_boolean_value_column(nested_col)) { + return true; + } const auto& null_map = col_nullable->get_null_map_data(); Filter not_null_filter; not_null_filter.reserve(nested_col.size()); diff --git a/be/src/exec/sink/viceberg_merge_sink.cpp b/be/src/exec/sink/viceberg_merge_sink.cpp index 5008b217228cd2..0b1e77ee1a47be 100644 --- a/be/src/exec/sink/viceberg_merge_sink.cpp +++ b/be/src/exec/sink/viceberg_merge_sink.cpp @@ -71,6 +71,14 @@ Status VIcebergMergeSink::init_properties(ObjectPool* pool, const RowDescriptor& Status VIcebergMergeSink::open(RuntimeState* state, RuntimeProfile* profile) { _state = state; + if (!_writes_data_files && state->be_exec_version() < SUPPORT_ICEBERG_VARIANT_VERSION) { + // The query-wide version keeps delete-only writer omission all-or-nothing; an older BE + // would ignore writes_data_files and parse the unsupported Variant data-writer schema. + return Status::NotSupported( + "Delete-only Iceberg MERGE requires backend execution version {}", + SUPPORT_ICEBERG_VARIANT_VERSION); + } + _written_rows_counter = ADD_COUNTER(profile, "RowsWritten", TUnit::UNIT); _insert_rows_counter = ADD_COUNTER(profile, "InsertRows", TUnit::UNIT); _delete_rows_counter = ADD_COUNTER(profile, "DeleteRows", TUnit::UNIT); diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index b9488b5f78f2ab..1952c6839872ea 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -2259,7 +2259,8 @@ Status TableColumnMapper::create_scan_request( const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state, - const std::map* fixed_local_positions) { + const std::map* fixed_local_positions, + const std::map* fixed_non_predicate_positions) { // FileReader evaluates expressions against a file-local block. This mapper owns the // table-column to file-column conversion, so it also owns the file-local block positions. file_request->predicate_columns.clear(); @@ -2273,7 +2274,13 @@ Status TableColumnMapper::create_scan_request( file_request->local_positions = *fixed_local_positions; } file_request->non_predicate_positions.clear(); + if (fixed_non_predicate_positions != nullptr) { + // Deferred output slots are part of the active reader's immutable block layout, just like + // eager slots; retaining only local_positions can shift a later complex root out of bounds. + file_request->non_predicate_positions = *fixed_non_predicate_positions; + } file_request->conjuncts.clear(); + file_request->metadata_pruning_safe_conjunct_count = 0; file_request->delete_conjuncts.clear(); _filter_entries.clear(); // 1. Build referenced non-predicate columns @@ -2500,6 +2507,9 @@ Status TableColumnMapper::localize_filters(const std::vector& table auto localized_conjunct = VExprContext::create_shared(std::move(localized_root)); RETURN_IF_ERROR(rewrite_context.prepare_created_exprs(localized_conjunct.get())); file_request->conjuncts.push_back(std::move(localized_conjunct)); + if (table_filter.metadata_pruning_safe) { + ++file_request->metadata_pruning_safe_conjunct_count; + } for (const auto global_index : table_filter.global_indices) { const auto* mapping = _find_filter_mapping(global_index); if (mapping != nullptr && mapping->file_local_id.has_value() && diff --git a/be/src/format_v2/column_mapper.h b/be/src/format_v2/column_mapper.h index fe9ac2ebb56c00..2a9870aa6ad1aa 100644 --- a/be/src/format_v2/column_mapper.h +++ b/be/src/format_v2/column_mapper.h @@ -205,7 +205,8 @@ class TableColumnMapper { const std::vector& table_filters, const std::vector& projected_columns, FileScanRequest* file_request, RuntimeState* runtime_state = nullptr, - const std::map* fixed_local_positions = nullptr); + const std::map* fixed_local_positions = nullptr, + const std::map* fixed_non_predicate_positions = nullptr); // Localize table-level filters to the file schema. // Trivial mappings can copy structured predicates directly. Type changes may be localized with diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 0256b8c1eebcb2..ddbabf6329864d 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -83,6 +84,9 @@ struct FileScanRequest { std::map non_predicate_positions; // Row-level filters converted to file-local expressions from table-level predicates. VExprContextSPtrs conjuncts; + // Only this leading subset may participate in footer/page metadata pruning. The boundary is + // inherited from table-conjunct order so an omitted slotless unsafe expression remains a fence. + size_t metadata_pruning_safe_conjunct_count = std::numeric_limits::max(); // Delete predicates converted to file-local expressions. A TRUE result means that the row is // deleted, so readers must invert each result when building their keep filter. VExprContextSPtrs delete_conjuncts; diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 1d10e4cb2c002f..77d6a20c7c825a 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -589,8 +589,15 @@ std::optional extract_variant_shredded_predicate( .op = *op}; } +VExprContextSPtrs metadata_pruning_conjuncts(const format::FileScanRequest& request) { + const size_t safe_count = + std::min(request.metadata_pruning_safe_conjunct_count, request.conjuncts.size()); + return VExprContextSPtrs(request.conjuncts.begin(), request.conjuncts.begin() + safe_count); +} + bool has_variant_shredded_filter(const format::FileScanRequest& request) { - return std::ranges::any_of(request.conjuncts, [](const auto& conjunct) { + const auto conjuncts = metadata_pruning_conjuncts(request); + return std::ranges::any_of(conjuncts, [](const auto& conjunct) { return extract_variant_shredded_predicate(conjunct).has_value(); }); } @@ -778,9 +785,7 @@ bool variant_statistics_exclude(const VariantShreddedPredicate& predicate, bool has_expr_zonemap_filter(const format::FileScanRequest& request, const RuntimeState*) { // FileScannerV2 metadata pruning is a fixed part of its scan pipeline and must not inherit // the legacy scanner's expression ZoneMap session gate. - // TODO: Fence metadata pruning at the first unsafe/error-preserving conjunct so a later - // ZoneMap predicate cannot bypass its row-level evaluation. - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { if (conjunct != nullptr && conjunct->root() != nullptr && conjunct->root()->can_evaluate_zonemap_filter()) { return true; @@ -974,7 +979,8 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, const std::vector>& file_schema, const format::FileScanRequest& request, ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) { - const auto slot_indexes = collect_expr_zonemap_slot_indexes(request.conjuncts); + const auto conjuncts = metadata_pruning_conjuncts(request); + const auto slot_indexes = collect_expr_zonemap_slot_indexes(conjuncts); if (slot_indexes.empty()) { return false; } @@ -1009,7 +1015,7 @@ bool check_native_statistics(const tparquet::FileMetaData& metadata, } add_slot_zonemap(&ctx, slot_index, column_schema->type, std::move(zone_map)); } - const auto result = VExprContext::evaluate_zonemap_filter(request.conjuncts, ctx); + const auto result = VExprContext::evaluate_zonemap_filter(conjuncts, ctx); accumulate_zonemap_stats(ctx, pruning_stats); return result == ZoneMapFilterResult::kNoMatch; } @@ -1018,7 +1024,7 @@ bool check_shredded_variant_statistics( const tparquet::FileMetaData& metadata, const tparquet::RowGroup& row_group, const std::vector>& file_schema, const format::FileScanRequest& request, const cctz::time_zone* timezone) { - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { continue; @@ -1120,7 +1126,7 @@ ParquetRowGroupPruneReason native_dictionary_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_dictionary_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_dictionary_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -1190,7 +1196,7 @@ ParquetRowGroupPruneReason native_bloom_filter_prune_reason( return ParquetRowGroupPruneReason::NONE; } const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - request.conjuncts, expr_zonemap::single_slot_bloom_filter_index); + metadata_pruning_conjuncts(request), expr_zonemap::single_slot_bloom_filter_index); for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { const auto file_column_id = file_column_id_by_block_position(request, slot_index); if (!file_column_id.has_value()) { @@ -1653,7 +1659,7 @@ Status select_row_group_ranges_by_native_page_index( } std::map conjuncts_by_slot; - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct); if (slot_index >= 0) { conjuncts_by_slot[slot_index].push_back(conjunct); @@ -1714,7 +1720,7 @@ Status select_row_group_ranges_by_native_page_index( } } - for (const auto& conjunct : request.conjuncts) { + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { continue; diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 40ffe6cb86eff0..22cc6928762ad0 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -1061,8 +1061,15 @@ Status TableReader::_build_table_filters_from_conjuncts() { if (in_safe_prefix && !_is_safe_to_pre_execute(conjunct)) { in_safe_prefix = false; } + const size_t first_new_filter = _table_filters.size(); RETURN_IF_ERROR( build_table_filters_from_conjunct(conjunct, _runtime_state, &_table_filters)); + for (size_t filter_idx = first_new_filter; filter_idx < _table_filters.size(); + ++filter_idx) { + // Preserve the original conjunct-order fence even when the unsafe expression itself + // had no slot and therefore produced no TableFilter entry. + _table_filters[filter_idx].metadata_pruning_safe = in_safe_prefix; + } if (in_safe_prefix) { _constant_pruning_safe_filter_count = _table_filters.size(); } @@ -1072,47 +1079,29 @@ Status TableReader::_build_table_filters_from_conjuncts() { namespace { -bool same_scan_projection(const LocalColumnIndex& lhs, const LocalColumnIndex& rhs) { - if (lhs.index != rhs.index || lhs.project_all_children != rhs.project_all_children || - lhs.children.size() != rhs.children.size()) { +bool same_scan_projections(const std::vector& lhs, + const std::vector& rhs) { + if (lhs.size() != rhs.size()) { return false; } - for (size_t index = 0; index < lhs.children.size(); ++index) { - if (!same_scan_projection(lhs.children[index], rhs.children[index])) { + for (const auto& lhs_projection : lhs) { + const auto rhs_it = std::ranges::find_if(rhs, [&](const LocalColumnIndex& rhs_projection) { + return rhs_projection.column_id() == lhs_projection.column_id(); + }); + if (rhs_it == rhs.end() || !same_local_column_index(lhs_projection, *rhs_it)) { return false; } } return true; } -const LocalColumnIndex* find_scan_projection(const FileScanRequest& request, - LocalColumnId column_id) { - const auto find_by_id = [column_id](const std::vector& projections) { - return std::ranges::find_if(projections, [column_id](const LocalColumnIndex& projection) { - return projection.column_id() == column_id; - }); - }; - auto it = find_by_id(request.predicate_columns); - if (it != request.predicate_columns.end()) { - return &*it; - } - it = find_by_id(request.non_predicate_columns); - return it == request.non_predicate_columns.end() ? nullptr : &*it; -} - bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest& rhs) { - if (lhs.local_positions != rhs.local_positions) { - return false; - } - for (const auto& [column_id, _] : lhs.local_positions) { - const auto* lhs_projection = find_scan_projection(lhs, column_id); - const auto* rhs_projection = find_scan_projection(rhs, column_id); - if (lhs_projection == nullptr || rhs_projection == nullptr || - !same_scan_projection(*lhs_projection, *rhs_projection)) { - return false; - } - } - return true; + // Deferred complex roots occupy independent output slots. Comparing only eager positions can + // accept a refresh whose second Variant root now aliases or overruns the active block layout. + return lhs.local_positions == rhs.local_positions && + lhs.non_predicate_positions == rhs.non_predicate_positions && + same_scan_projections(lhs.predicate_columns, rhs.predicate_columns) && + same_scan_projections(lhs.non_predicate_columns, rhs.non_predicate_columns); } } // namespace @@ -1141,7 +1130,9 @@ Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { auto refreshed_request = std::make_shared(); RETURN_IF_ERROR(refreshed_mapper->create_scan_request( _table_filters, _projected_columns, refreshed_request.get(), _runtime_state, - _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions)); + _file_scan_request == nullptr ? nullptr : &_file_scan_request->local_positions, + _file_scan_request == nullptr ? nullptr + : &_file_scan_request->non_predicate_positions)); // A refresh does not prove that every future runtime filter has arrived. Keep carrier values // available whenever the split started with pending filters. if (_push_down_agg_type == TPushAggOp::type::COUNT && _push_down_count_columns.has_value() && diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index 0ffaa06a9d15b2..e98f96215e0e0f 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -82,6 +82,7 @@ using DeleteRows = std::vector; struct TableFilter { VExprContextSPtr conjunct; std::vector global_indices; + bool metadata_pruning_safe = true; }; struct ScanTask { diff --git a/be/test/core/column/column_self_check.cpp b/be/test/core/column/column_self_check.cpp index e78b4b6d5fc9b9..eb762334a56499 100644 --- a/be/test/core/column/column_self_check.cpp +++ b/be/test/core/column/column_self_check.cpp @@ -17,6 +17,8 @@ #include +#include "common/config.h" +#include "common/exception.h" #include "core/column/column.h" #include "core/column/column_array.h" #include "core/column/column_const.h" @@ -32,6 +34,29 @@ namespace doris { +namespace { + +class ScopedMemAllocFaultInjection { +public: + ScopedMemAllocFaultInjection() : _old_probability(config::mem_alloc_fault_probability) { + config::mem_alloc_fault_probability = 1.0; + ++enable_thread_catch_bad_alloc; + } + + ~ScopedMemAllocFaultInjection() { + --enable_thread_catch_bad_alloc; + config::mem_alloc_fault_probability = _old_probability; + } + + ScopedMemAllocFaultInjection(const ScopedMemAllocFaultInjection&) = delete; + ScopedMemAllocFaultInjection& operator=(const ScopedMemAllocFaultInjection&) = delete; + +private: + double _old_probability; +}; + +} // namespace + TEST(ColumnSelfCheckTest, const_check_test) { { ColumnPtr col = ColumnHelper::create_column({1, 2, 3}); @@ -173,4 +198,44 @@ TEST(ColumnSelfCheckTest, nullable_complex_without_nulls_does_not_copy_payload) } EXPECT_LT(peak_memory, payload_size); } + +TEST(ColumnSelfCheckTest, non_boolean_complex_payload_does_not_allocate_during_boolean_check) { + auto keys = ColumnString::create(); + keys->insert_data("key", 3); + auto values = ColumnString::create(); + values->insert_data("value", 5); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto nullable_map = ColumnNullable::create(std::move(map), ColumnUInt8::create(1, 0)); + + bool is_valid = false; + { + ScopedMemAllocFaultInjection inject_allocation_failure; + EXPECT_NO_THROW(is_valid = nullable_map->column_boolean_check()); + } + EXPECT_TRUE(is_valid); +} + +TEST(ColumnSelfCheckTest, nested_boolean_check_respects_parent_null_map) { + auto create_nullable_map = [](UInt8 null_row_value, UInt8 non_null_row_value) { + auto keys = ColumnString::create(); + keys->insert_data("first", 5); + keys->insert_data("second", 6); + auto values = ColumnUInt8::create(); + values->insert_value(null_row_value); + values->insert_value(non_null_row_value); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + offsets->insert_value(2); + auto map = ColumnMap::create(std::move(keys), std::move(values), std::move(offsets)); + auto null_map = ColumnUInt8::create(); + null_map->insert_value(1); + null_map->insert_value(0); + return ColumnNullable::create(std::move(map), std::move(null_map)); + }; + + EXPECT_TRUE(create_nullable_map(2, 1)->column_boolean_check()); + EXPECT_FALSE(create_nullable_map(1, 2)->column_boolean_check()); +} } // namespace doris diff --git a/be/test/exec/sink/viceberg_merge_sink_test.cpp b/be/test/exec/sink/viceberg_merge_sink_test.cpp index d1da8fb7bfe1d5..e965fab05e1305 100644 --- a/be/test/exec/sink/viceberg_merge_sink_test.cpp +++ b/be/test/exec/sink/viceberg_merge_sink_test.cpp @@ -473,6 +473,28 @@ TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeSkipsCardinalityState) { EXPECT_TRUE(sink->_matched_row_positions.empty()); } +TEST_F(VIcebergMergeSinkTest, TestRollingUpgradeRejectsDeleteOnlyWriterOmission) { + ObjectPool pool; + MockRuntimeState state; + state.set_be_exec_version(SUPPORT_ICEBERG_VARIANT_VERSION - 1); + + DataTypes types {std::make_shared(), + std::make_shared(DataTypes {std::make_shared(), + std::make_shared()}, + Strings {"file_path", "row_position"}), + std::make_shared(), std::make_shared()}; + MockRowDescriptor row_desc(types, &pool); + auto output_exprs = build_output_exprs(&pool, &state, row_desc); + auto t_sink = build_sink(); + t_sink.iceberg_merge_sink.__set_writes_data_files(false); + auto sink = std::make_shared(t_sink, output_exprs, nullptr, nullptr); + + ASSERT_TRUE(sink->init_properties(&pool, row_desc).ok()); + RuntimeProfile profile("rolling_upgrade_delete_only_iceberg_merge_sink"); + const Status status = sink->open(&state, &profile); + EXPECT_TRUE(status.is()) << status; +} + TEST_F(VIcebergMergeSinkTest, TestErrorCloseRemovesRolledDataFiles) { ObjectPool pool; MockRuntimeState state; diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index 0ef554fc7932f0..0e06473abe06e0 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -4187,6 +4187,67 @@ TEST(ColumnMapperTest, PredicateAccessPathsCreateDeferredVariantRootProjection) EXPECT_TRUE(request.is_predicate_only(LocalColumnId(0))); } +TEST(ColumnMapperTest, RowGroupRefreshPreservesTwoDeferredVariantRootLayouts) { + auto make_file_variant = [](const std::string& name, int32_t field_id, int32_t local_id) { + auto field_wrapper = struct_name_col( + "typed_col", {name_col("value", varbinary(), 0), name_col("typed_value", i64(), 1)}, + 0); + auto typed_value = struct_name_col("typed_value", {std::move(field_wrapper)}, 2); + auto variant = field_id_col(name, field_id, variant_v2(), local_id); + variant.children = {name_col("metadata", varbinary(), 0), name_col("value", varbinary(), 1), + std::move(typed_value)}; + return variant; + }; + auto first = field_id_col("v1", 10, variant_v2()); + first.has_predicate_access_paths = true; + first.predicate_variant_access_paths = {{"typed_col"}}; + auto second = field_id_col("v2", 11, variant_v2()); + second.has_predicate_access_paths = true; + second.predicate_variant_access_paths = {{"typed_col"}}; + const std::vector table_columns {first, second}; + const std::vector file_columns {make_file_variant("v1", 10, 0), + make_file_variant("v2", 11, 1)}; + + std::vector filters; + for (int32_t index = 0; index < 2; ++index) { + auto typed_col = element_at( + table_slot(index, index, table_columns[index].type, table_columns[index].name), + variant_v2(), "typed_col"); + auto predicate = binary_predicate(TExprOpcode::GT, cast_expr(typed_col, i64()), + literal(i64(), Field::create_field(0))); + filters.push_back({.conjunct = VExprContext::create_shared(predicate), + .global_indices = {GlobalIndex(index)}}); + } + + ParquetColumnMapper initial_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(initial_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest initial_request; + ASSERT_TRUE(initial_mapper.create_scan_request(filters, table_columns, &initial_request).ok()); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(0)), LocalIndex(0)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(0)), LocalIndex(1)); + EXPECT_EQ(initial_request.local_positions.at(LocalColumnId(1)), LocalIndex(2)); + EXPECT_EQ(initial_request.non_predicate_position(LocalColumnId(1)), LocalIndex(3)); + + ParquetColumnMapper refreshed_mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ASSERT_TRUE(refreshed_mapper.create_mapping(table_columns, {}, file_columns).ok()); + FileScanRequest refreshed_request; + ASSERT_TRUE(refreshed_mapper + .create_scan_request(filters, table_columns, &refreshed_request, nullptr, + &initial_request.local_positions, + &initial_request.non_predicate_positions) + .ok()); + EXPECT_EQ(refreshed_request.local_positions, initial_request.local_positions); + EXPECT_EQ(refreshed_request.non_predicate_positions, initial_request.non_predicate_positions); + ASSERT_EQ(refreshed_request.predicate_columns.size(), 2); + ASSERT_EQ(refreshed_request.non_predicate_columns.size(), 2); + for (size_t index = 0; index < 2; ++index) { + EXPECT_TRUE(same_local_column_index(refreshed_request.predicate_columns[index], + initial_request.predicate_columns[index])); + EXPECT_TRUE(same_local_column_index(refreshed_request.non_predicate_columns[index], + initial_request.non_predicate_columns[index])); + } +} + TEST(ColumnMapperTest, NestedVariantAccessPathProjectsPhysicalTypedLeaf) { auto table_variant = field_id_col("payload", 2, variant_v2()); table_variant.variant_access_paths = {{"typed_col"}}; diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 1a13e8ecea362e..4c3fdcafa06ffe 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -987,6 +987,17 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) .ok()); EXPECT_TRUE(selected_row_groups.empty()); + // The same predicate can be localized after an earlier unsafe conjunct. Metadata pruning must + // preserve that earlier expression's row-level error instead of skipping the whole row group. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + footer_only_metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr, nullptr, nullptr, nullptr, {}, + format::parquet::ParquetMetadataProbeMode::FOOTER_ONLY) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + auto leaf_projection = format::LocalColumnIndex::partial_local(0); auto typed_object_projection = format::LocalColumnIndex::partial_local(2); auto field_projection = format::LocalColumnIndex::partial_local(0); @@ -1083,6 +1094,16 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) EXPECT_EQ(pruning_stats.page_index_read_calls, 1); EXPECT_EQ(pruning_stats.filtered_page_rows, 50); + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 100, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 100); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + // Direct Variant numeric comparisons coerce integral literals to a wide DECIMAL domain. request.conjuncts = {variant_path_gt_conjunct(50, false, true)}; ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index d959e27b15e907..b4a90f29030c17 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -1588,6 +1588,48 @@ TEST(TableReaderTest, ConstantPruningStopsAtUnsafeSlotlessPredicate) { ASSERT_TRUE(reader.close().ok()); } +TEST(TableReaderTest, MetadataPruningBoundaryKeepsUnsafeSlotlessBarrier) { + std::vector file_schema; + file_schema.push_back(make_file_column(0, "id", std::make_shared())); + std::vector projected_columns; + projected_columns.push_back(make_table_column(0, "id", std::make_shared())); + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + bool predicate_executed = false; + auto fake_state = std::make_shared(); + FakeTableReader reader(file_schema, fake_state); + ASSERT_TRUE( + reader + .init({ + .projected_columns = projected_columns, + .conjuncts = + {prepared_conjunct( + &state, + std::make_shared( + &predicate_executed)), + prepared_conjunct(&state, + table_int32_greater_than_expr(0, 0, 10))}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + + SplitReadOptions split; + split.current_range.__set_path("fake-table-reader-input"); + ASSERT_TRUE(reader.prepare_split(split).ok()); + Block block = build_table_block(projected_columns); + bool eos = false; + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + ASSERT_NE(fake_state->last_request, nullptr); + ASSERT_EQ(fake_state->last_request->conjuncts.size(), 1); + EXPECT_EQ(fake_state->last_request->metadata_pruning_safe_conjunct_count, 0); + ASSERT_TRUE(reader.close().ok()); +} + TEST(TableReaderTest, CanUseInjectedFileReaderForStandaloneUnitTest) { std::vector file_schema; file_schema.push_back(make_file_column(0, "id", std::make_shared())); diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index af7d3a434082e1..392f036af0476a 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -1969,7 +1969,7 @@ public class Config extends ConfigBase { * Max data version of backends serialize block. */ @ConfField(mutable = false) - public static int max_be_exec_version = 11; + public static int max_be_exec_version = 12; /** * Min data version of backends serialize block. diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index 015ba4aaa4cea6..8d45bae956af1d 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -19,6 +19,7 @@ import org.apache.doris.connector.spi.handle.ConnectorColumnHandle; import org.apache.doris.connector.spi.handle.ConnectorWriteHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider; import org.junit.jupiter.api.Assertions; @@ -84,6 +85,7 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException Connector.class, ConnectorColumnHandle.class, ConnectorTableSchema.class, + ConnectorScanPlanProvider.class, ConnectorWriteHandle.class, ConnectorWritePlanProvider.class, org.apache.doris.extension.spi.Plugin.class, diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index 328bde2cfd2008..c61f5760aee8ab 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -64,6 +64,31 @@ org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getTableHandle():org. org.apache.doris.connector.spi.handle.ConnectorWriteHandle#getWriteOperation():org.apache.doris.connector.spi.handle.WriteOperation org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isOverwrite():boolean org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isRequireMergeCardinalityCheck():boolean +org.apache.doris.connector.spi.handle.ConnectorWriteHandle#isWritesDataFiles():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#adjustFileCompressType(org.apache.doris.thrift.TFileCompressType):org.apache.doris.thrift.TFileCompressType +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#canServeMetadataOnlyCount(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#classifyColumn(java.lang.String):org.apache.doris.connector.spi.scan.ConnectorColumnCategory +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#collectScanProfiles(org.apache.doris.connector.spi.ConnectorSession):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getDeleteFiles(org.apache.doris.thrift.TTableFormatFileDesc):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getMustReadColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.Set +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodeProperties(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):java.util.Map +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#getScanNodePropertiesResult(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional):org.apache.doris.connector.spi.scan.ScanNodePropertiesResult +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#ignorePartitionPruneShortCircuit():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#planScanForPartitionBatch(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.scan.ConnectorScanRequest,java.util.List):java.util.List +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#populateScanLevelParams(org.apache.doris.thrift.TFileScanRangeParams,java.util.Map):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#releaseReadTransaction(java.lang.String):void +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#scannedPartitionCount(java.util.List):java.util.OptionalLong +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamingSplitEstimate(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional,boolean):long +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#streamSplits(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List,java.util.Optional,long):org.apache.doris.connector.spi.scan.ConnectorSplitSource +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsBatchScan(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsFileCache():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableIncrementalRead(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableOptions(java.lang.String):boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableTimeTravel():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsTableSample():boolean +org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#usesHiveParquetInt96TimeZone():boolean org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorWriteHandle):void org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getSyntheticWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.List org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):java.util.Optional diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index ad4ccc728ee07d..979826563dc2d4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -34,6 +34,7 @@ import org.apache.doris.catalog.TableIf; import org.apache.doris.catalog.Type; import org.apache.doris.catalog.VariantType; +import org.apache.doris.common.Config; import org.apache.doris.common.UserException; import org.apache.doris.common.profile.RuntimeProfile; import org.apache.doris.common.profile.SummaryProfile; @@ -130,6 +131,7 @@ * */ public class PluginDrivenScanNode extends FileQueryScanNode { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private static final Logger LOG = LogManager.getLogger(PluginDrivenScanNode.class); @@ -201,6 +203,9 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, @Override protected void doInitialize() throws UserException { super.doInitialize(); + // Compatibility must inspect the snapshot-specific handle: latest metadata may answer + // COUNT(*) while an older time-travel snapshot still requires a Variant data scan. + pinMvccSnapshot(); checkVariantBackendCompatibilityForCurrentScan(backendPolicy.getBackends()); } @@ -210,13 +215,17 @@ void checkVariantBackendCompatibilityForCurrentScan(Iterable backends) ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) { metadataCountProven = onPluginClassLoader(scanProvider, - () -> scanProvider.canServeMetadataOnlyCount( - connectorSession, currentHandle, Optional.empty())); + () -> canServeMetadataOnlyCount(scanProvider, connectorSession, currentHandle)); } checkVariantBackendCompatibility( !metadataCountProven && projectsComputeVariant(desc), backends); } + static boolean canServeMetadataOnlyCount(ConnectorScanPlanProvider scanProvider, + ConnectorSession session, ConnectorTableHandle handle) { + return scanProvider.canServeMetadataOnlyCount(session, handle, Optional.empty()); + } + static boolean projectsComputeVariant(TupleDescriptor tuple) { // Nested-column pruning updates the effective slot type but deliberately keeps the original // Column metadata; compatibility must follow the payload this scan actually projects. @@ -246,6 +255,12 @@ static void checkVariantBackendCompatibility(boolean projectsVariant, Iterable */ public class PluginDrivenTableSink extends BaseExternalTableDataSink { + private static final int SUPPORT_ICEBERG_VARIANT_EXEC_VERSION = 12; private final PluginDrivenExternalTable targetTable; // Plan-provider mode (W5): the connector builds its own opaque TDataSink via planWrite(). @@ -243,6 +245,13 @@ public String getExplainString(String prefix, TExplainLevel explainLevel) { @Override public void bindDataSink(Optional insertCtx) throws AnalysisException { + if (writeOperation == WriteOperation.MERGE && !writesDataFiles + && Config.be_exec_version < SUPPORT_ICEBERG_VARIANT_EXEC_VERSION) { + // Older BEs ignore writes_data_files and instantiate the omitted data writer, so reject + // the all-or-nothing query before a Variant schema reaches any rolling-upgrade backend. + throw new AnalysisException("Delete-only Iceberg MERGE with Variant is unavailable " + + "during rolling upgrade"); + } boolean overwrite = false; Map writeContext = Collections.emptyMap(); Optional branchName = Optional.empty(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java index 0c8cbd43cc1b10..e25cd39b7ef699 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCompatibilityTest.java @@ -17,17 +17,24 @@ package org.apache.doris.datasource.scan; +import org.apache.doris.common.Config; import org.apache.doris.common.UserException; +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider; import org.apache.doris.system.Backend; import org.junit.Assert; import org.junit.Test; import java.util.Collections; +import java.util.Optional; /** Tests the mixed-version safety gate for plugin-driven Variant scans. */ public class PluginDrivenScanNodeCompatibilityTest { + private static final int VARIANT_EXEC_VERSION = 12; + @Test public void computeVariantRejectsSmoothUpgradeSourceBackend() { Backend backend = new Backend(7L, "127.0.0.1", 9050); @@ -47,4 +54,38 @@ public void compatibilityCheckIgnoresScansWithoutComputeVariant() throws UserExc PluginDrivenScanNode.checkVariantBackendCompatibility( false, Collections.singletonList(backend)); } + + @Test + public void computeVariantRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = VARIANT_EXEC_VERSION - 1; + Backend backend = new Backend(8L, "127.0.0.1", 9050); + + UserException exception = Assert.assertThrows(UserException.class, + () -> PluginDrivenScanNode.checkVariantBackendCompatibility( + true, Collections.singletonList(backend))); + Assert.assertTrue(exception.getMessage().contains("execution version")); + } finally { + Config.be_exec_version = original; + } + } + + @Test + public void metadataCountCapabilityUsesPinnedHandle() { + ConnectorSession session = org.mockito.Mockito.mock(ConnectorSession.class); + ConnectorTableHandle latest = new ConnectorTableHandle() { }; + ConnectorTableHandle pinned = new ConnectorTableHandle() { }; + ConnectorScanPlanProvider provider = + org.mockito.Mockito.mock(ConnectorScanPlanProvider.class); + org.mockito.Mockito.doAnswer(invocation -> invocation.getArgument(1) == latest) + .when(provider).canServeMetadataOnlyCount(org.mockito.Mockito.same(session), + org.mockito.Mockito.any(ConnectorTableHandle.class), + org.mockito.Mockito.eq(Optional.empty())); + + Assert.assertTrue(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, latest)); + Assert.assertFalse(PluginDrivenScanNode.canServeMetadataOnlyCount( + provider, session, pinned)); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java index a5c871216bf1d9..e4c0262d9d39c8 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkTest.java @@ -18,6 +18,7 @@ package org.apache.doris.planner; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.Config; import org.apache.doris.connector.spi.ConnectorColumn; import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; @@ -272,6 +273,26 @@ null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), Assert.assertTrue(provider.seenHandle.isRequireMergeCardinalityCheck()); } + @Test + public void deleteOnlyMergeRejectsOldQueryWideExecutionVersion() { + int original = Config.be_exec_version; + try { + Config.be_exec_version = 11; + RecordingWritePlanProvider provider = new RecordingWritePlanProvider( + new ConnectorSinkPlan(new TDataSink(TDataSinkType.ICEBERG_MERGE_SINK))); + PluginDrivenTableSink sink = new PluginDrivenTableSink( + null, provider, null, new ConnectorTableHandle() { }, new ArrayList<>(), + null, WriteOperation.MERGE, false, true); + + AnalysisException exception = Assert.assertThrows(AnalysisException.class, + () -> sink.bindDataSink(Optional.empty())); + Assert.assertTrue(exception.getMessage().contains("rolling upgrade")); + Assert.assertNull(provider.seenHandle); + } finally { + Config.be_exec_version = original; + } + } + @Test public void getExplainStringThreadsWriteOperationToHandle() { // WHY: EXPLAIN of a post-flip MERGE/DELETE builds a (degraded) handle for appendExplainInfo; the diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out index a7c41169d8dcb3..0157263d8a83fc 100644 --- a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -79,6 +79,9 @@ true 5 170 4.17 -- !variant_nested_filter -- 1 +-- !variant_signed_selector -- +\N 41 + -- !variant_nested_expressions -- 1 11 inside open \N true 1002 2 \N \N \N 202 false 2003 diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 7b3bc3e833d165..43c6735a6dad74 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -377,6 +377,17 @@ suite("test_iceberg_variant_read", WHERE v['n'] > 3000 """ + // A later Variant metadata predicate must not prune away an earlier error-producing conjunct. + test { + sql """ + SELECT COUNT(*) + FROM variant_page_pruning + WHERE assert_true(id != 1, 'variant_metadata_error_barrier') + AND v['n'] > 5000 + """ + exception "variant_metadata_error_barrier" + } + order_qt_variant_aggregate """ SELECT CAST(v['ok'] AS BOOLEAN), COUNT(*), @@ -440,13 +451,10 @@ suite("test_iceberg_variant_read", // Signed integer selectors are array indexes, even when a shredded object has a key with the // same serialized token. The ambiguous scanner path must retain enough state for both results. - List> signedSelectorRows = sql """ + order_qt_variant_signed_selector """ SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT) FROM variant_signed_selector """ - assertEquals(1, signedSelectorRows.size()) - assertEquals(null, signedSelectorRows[0][0]) - assertEquals("41", signedSelectorRows[0][1].toString()) order_qt_variant_nested_expressions """ SELECT id, From 687d44adc75414d16a9bd9ac45e5400e1534d3af Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 14:33:36 +0800 Subject: [PATCH 05/20] test: expand Iceberg Variant read coverage --- .../parquet/variant_column_reader_test.cpp | 718 ++++++++++++++++++ .../iceberg/test_iceberg_variant_read.out | 59 +- .../iceberg/test_iceberg_variant_read.groovy | 444 ++++++++++- 3 files changed, 1196 insertions(+), 25 deletions(-) diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 40c18635aad339..5bf2d33c10f1af 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -20,20 +20,33 @@ #include #include +#include +#include +#include +#include #include #include +#include "common/exception.h" #include "core/assert_cast.h" #include "core/column/column_array.h" +#include "core/column/column_decimal.h" +#include "core/column/column_map.h" #include "core/column/column_nullable.h" #include "core/column/column_string.h" #include "core/column/column_struct.h" #include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_date_or_datetime_v2.h" +#include "core/data_type/data_type_decimal.h" +#include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_timestamptz.h" #include "core/data_type/data_type_variant_v2.h" +#include "core/value/timestamptz_value.h" #include "core/value/variant/variant_batch_builder.h" #include "core/value/variant/variant_parquet_encoding.h" #include "exprs/function/function_variant_element_v2.h" @@ -90,6 +103,16 @@ ParquetColumnSchema shredded_int64_schema() { return schema; } +ParquetColumnSchema shredded_primitive_schema(DataTypePtr type) { + auto schema = unshredded_schema(); + auto typed = std::make_unique(); + typed->name = "typed_value"; + typed->kind = ParquetColumnSchemaKind::PRIMITIVE; + typed->type = make_nullable(std::move(type)); + schema.children.push_back(std::move(typed)); + return schema; +} + ParquetColumnSchema shredded_object_schema() { auto schema = unshredded_schema(); auto typed = std::make_unique(); @@ -110,6 +133,12 @@ ParquetColumnSchema shredded_object_schema() { return schema; } +ParquetColumnSchema shredded_named_object_schema(std::string field_name) { + auto schema = shredded_object_schema(); + schema.children.back()->children[0]->name = std::move(field_name); + return schema; +} + ParquetColumnSchema shredded_binary_object_schema() { auto schema = shredded_object_schema(); auto* leaf = schema.children.back()->children[0]->children[0].get(); @@ -136,6 +165,17 @@ ParquetColumnSchema shredded_array_schema() { return schema; } +ParquetColumnSchema shredded_mixed_array_schema() { + auto schema = shredded_array_schema(); + auto* element = schema.children.back()->children[0].get(); + auto value = std::make_unique(); + value->name = "value"; + value->kind = ParquetColumnSchemaKind::PRIMITIVE; + value->type = make_nullable(std::make_shared()); + element->children.insert(element->children.begin(), std::move(value)); + return schema; +} + MutableColumnPtr shredded_int64_physical(const std::vector& values) { const std::array ignored {0}; const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); @@ -157,6 +197,20 @@ MutableColumnPtr shredded_int64_physical(const std::vector& values) { return ColumnNullable::create(std::move(structure), std::move(root_nulls)); } +MutableColumnPtr shredded_primitive_physical(MutableColumnPtr typed) { + const size_t rows = typed->size(); + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + MutableColumns fields; + fields.push_back(nullable_strings(std::vector(rows, metadata), + std::vector(rows, 0))); + fields.push_back(nullable_strings(std::vector(rows, {ignored.data(), 0}), + std::vector(rows, 1))); + fields.push_back(std::move(typed)); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), + ColumnUInt8::create(rows, 0)); +} + MutableColumnPtr projected_shredded_object_physical(const std::vector& values, const IColumn** decoded_leaf = nullptr) { auto integers = ColumnInt64::create(); @@ -182,6 +236,37 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls); + +MutableColumnPtr complete_shredded_object_physical(std::string_view residual_key, + int64_t residual_value, int64_t typed_value) { + VariantBatchBuilder builder; + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef(residual_key.data(), residual_key.size())); + row.add_int(residual_value); + object.finish(); + row.finish(); + VariantBatchBuilder batch = builder.finish_batch(); + const VariantRef residual = batch.value_at(0); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({typed_value}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(wrapper_fields)), + ColumnUInt8::create(1, 0))); + MutableColumns root_fields; + root_fields.push_back( + nullable_strings({StringRef(residual.metadata.data, residual.metadata.size)}, {0})); + root_fields.push_back( + nullable_strings({StringRef(residual.value.data, residual.value.size)}, {0})); + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return root_wrapper(std::move(root_fields)); +} + MutableColumnPtr projected_two_field_object_physical(const std::vector& first, const std::vector& second) { DORIS_CHECK(first.size() == second.size()); @@ -226,6 +311,58 @@ MutableColumnPtr projected_wide_object_physical(size_t field_count, int64_t valu ColumnUInt8::create(1, 0)); } +std::string materialization_error(const ParquetColumnSchema& schema, ColumnPtr physical) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows(schema, std::move(physical), output); + if (!status.ok()) { + return status.to_string(); + } + try { + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + (void)variants.get_value_ref(0); + } catch (const Exception& exception) { + return exception.what(); + } + return {}; +} + +MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls) { + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(root_nulls.begin(), root_nulls.end()); + return ColumnNullable::create(ColumnStruct::create(std::move(fields)), std::move(null_map)); +} + +MutableColumnPtr nullable_int64(const std::vector& values, + const std::vector& nulls) { + auto data = ColumnInt64::create(); + data->get_data().assign(values.begin(), values.end()); + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_fixed(std::initializer_list values, + std::initializer_list nulls) { + auto data = ColumnType::create(); + for (const Value& value : values) { + data->insert_value(value); + } + auto null_map = ColumnUInt8::create(); + null_map->get_data().assign(nulls.begin(), nulls.end()); + return ColumnNullable::create(std::move(data), std::move(null_map)); +} + +template +MutableColumnPtr nullable_decimal(uint32_t scale, std::initializer_list values) { + auto data = ColumnType::create(0, scale); + for (const Value& value : values) { + data->insert_value(value); + } + return ColumnNullable::create(std::move(data), ColumnUInt8::create(values.size(), 0)); +} + } // namespace TEST(VariantColumnReaderTest, UnshreddedRowsPreserveSqlNullAndVariantNull) { @@ -304,6 +441,177 @@ TEST(VariantColumnReaderTest, ShreddedIntegerKeepsDeclaredPhysicalWidth) { EXPECT_EQ(variants.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); } +TEST(VariantColumnReaderTest, ReconstructsShreddedPrimitiveTypeMatrix) { + auto decode = [](ParquetColumnSchema schema, MutableColumnPtr typed, + const std::function& verify) { + auto output = make_nullable(std::make_shared())->create_column(); + const Status status = materialize_variant_rows( + schema, shredded_primitive_physical(std::move(typed)), output); + ASSERT_TRUE(status.ok()) << status; + verify(assert_cast( + assert_cast(*output).get_nested_column())); + }; + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({0, 1}, {0, 0}), [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).primitive_id(), VariantPrimitiveId::FALSE_VALUE); + EXPECT_EQ(values.get_value_ref(1).primitive_id(), VariantPrimitiveId::TRUE_VALUE); + }); + + auto verify_integer = [&](DataTypePtr type, MutableColumnPtr typed, int width, int64_t first, + int64_t second) { + auto schema = shredded_primitive_schema(std::move(type)); + schema.children.back()->type_descriptor.integer_bit_width = width; + decode(std::move(schema), std::move(typed), [&](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_int(), first); + EXPECT_EQ(values.get_value_ref(1).get_int(), second); + }); + }; + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 8, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 16, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 32, std::numeric_limits::min(), std::numeric_limits::max()); + verify_integer( + std::make_shared(), + nullable_fixed( + {std::numeric_limits::min(), std::numeric_limits::max()}, {0, 0}), + 64, std::numeric_limits::min(), std::numeric_limits::max()); + + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({std::numeric_limits::quiet_NaN(), + std::numeric_limits::infinity()}, + {0, 0}), + [](const auto& values) { + EXPECT_TRUE(std::isnan(values.get_value_ref(0).get_float())); + EXPECT_TRUE(std::isinf(values.get_value_ref(1).get_float())); + }); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed({-std::numeric_limits::infinity(), 1.25}, + {0, 0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_double(), + -std::numeric_limits::infinity()); + EXPECT_EQ(values.get_value_ref(1).get_double(), 1.25); + }); + + { + auto schema = shredded_primitive_schema(std::make_shared(9, 2)); + schema.children.back()->type_descriptor.decimal_precision = 9; + schema.children.back()->type_descriptor.decimal_scale = 2; + decode(std::move(schema), + nullable_decimal(2, {Decimal32 {12345}, Decimal32 {-1}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), (VariantDecimal {12345, 2, 4})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), (VariantDecimal {-1, 2, 4})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(18, 3)); + schema.children.back()->type_descriptor.decimal_precision = 18; + schema.children.back()->type_descriptor.decimal_scale = 3; + decode(std::move(schema), + nullable_decimal( + 3, {Decimal64 {123456789}, Decimal64 {-123456789}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {123456789, 3, 8})); + EXPECT_EQ(values.get_value_ref(1).get_decimal(), + (VariantDecimal {-123456789, 3, 8})); + }); + } + { + auto schema = shredded_primitive_schema(std::make_shared(38, 4)); + schema.children.back()->type_descriptor.decimal_precision = 38; + schema.children.back()->type_descriptor.decimal_scale = 4; + decode(std::move(schema), + nullable_decimal( + 4, {Decimal128V3 {static_cast(1234567890123456789LL)}}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_decimal(), + (VariantDecimal {1234567890123456789LL, 4, 16})); + }); + } + + const auto date = DateV2Value::create_from_olap_date( + (static_cast(1970) << 9) | (static_cast(1) << 5) | 2); + decode(shredded_primitive_schema(std::make_shared()), + nullable_fixed>({date}, {0}), + [](const auto& values) { EXPECT_EQ(values.get_value_ref(0).get_date(), 1); }); + + auto datetime = DateV2Value::create_from_olap_datetime(19700101000001ULL); + datetime.set_microsecond(234567); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = false; + decode(std::move(schema), + nullable_fixed>({datetime}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_ntz_micros(), 1234567); + }); + } + TimestampTzValue timestamp; + timestamp.unchecked_set_time(1970, 1, 1, 0, 0, 2, 345678); + { + auto schema = shredded_primitive_schema(std::make_shared(6)); + schema.children.back()->type_descriptor.time_unit = ParquetTimeUnit::MICROS; + schema.children.back()->type_descriptor.timestamp_is_adjusted_to_utc = true; + decode(std::move(schema), + nullable_fixed({timestamp}, {0}), + [](const auto& values) { + EXPECT_EQ(values.get_value_ref(0).get_timestamp_micros(), 2345678); + }); + } + + auto verify_bytes = [&](bool string_annotation, bool uuid) { + const std::array bytes {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}; + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_string_annotation = string_annotation; + schema.children.back()->type_descriptor.is_uuid = uuid; + auto strings = ColumnString::create(); + if (uuid) { + strings->insert_data(reinterpret_cast(bytes.data()), bytes.size()); + } else { + strings->insert_data("bytes", 5); + } + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + decode(std::move(schema), std::move(typed), [&](const auto& values) { + if (uuid) { + EXPECT_EQ(values.get_value_ref(0).get_uuid(), bytes); + } else if (string_annotation) { + EXPECT_EQ(values.get_value_ref(0).get_string(), StringRef("bytes")); + } else { + EXPECT_EQ(values.get_value_ref(0).get_binary(), StringRef("bytes")); + } + }); + }; + verify_bytes(false, false); + verify_bytes(true, false); + verify_bytes(false, true); +} + +TEST(VariantColumnReaderTest, RejectsInvalidShreddedUuidWidth) { + auto schema = shredded_primitive_schema(std::make_shared()); + schema.children.back()->type_descriptor.is_uuid = true; + auto strings = ColumnString::create(); + strings->insert_data("short", 5); + auto typed = ColumnNullable::create(std::move(strings), ColumnUInt8::create(1, 0)); + const std::string error = + materialization_error(schema, shredded_primitive_physical(std::move(typed))); + EXPECT_NE(error.find("UUID has 5 bytes instead of 16"), std::string::npos) << error; +} + TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { VariantBatchBuilder first_builder; auto first_row = first_builder.begin_row(); @@ -349,6 +657,31 @@ TEST(VariantColumnReaderTest, DifferentMetadataDictionariesRemainIndependent) { EXPECT_EQ(field.get_int(), 2); } +TEST(VariantColumnReaderTest, AppendsCompleteShreddedStatesWithDifferentSchemasAndMetadata) { + auto output = make_nullable(std::make_shared())->create_column(); + auto first_schema = shredded_named_object_schema("a"); + ASSERT_TRUE(materialize_variant_rows(first_schema, + complete_shredded_object_physical("left", 1, 11), output) + .ok()); + auto second_schema = shredded_named_object_schema("b"); + ASSERT_TRUE(materialize_variant_rows(second_schema, + complete_shredded_object_physical("right", 2, 22), output) + .ok()); + + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_EQ(variants.size(), 2); + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("left"), &field)); + EXPECT_EQ(field.get_int(), 1); + ASSERT_TRUE(variants.get_value_ref(0).object_find(StringRef("a"), &field)); + EXPECT_EQ(field.get_int(), 11); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("right"), &field)); + EXPECT_EQ(field.get_int(), 2); + ASSERT_TRUE(variants.get_value_ref(1).object_find(StringRef("b"), &field)); + EXPECT_EQ(field.get_int(), 22); +} + TEST(VariantColumnReaderTest, ShreddedObjectFieldMayOmitResidualValueColumn) { const std::array ignored {0}; const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); @@ -776,6 +1109,273 @@ TEST(VariantColumnReaderTest, MaterializesShreddedArrayElements) { EXPECT_EQ(value.array_at(1).get_int(), 4); } +TEST(VariantColumnReaderTest, RejectsCorruptShreddedWrappersWithoutCrashing) { + const std::array int_seven { + static_cast(static_cast(VariantPrimitiveId::INT8) + << VARIANT_VALUE_HEADER_SHIFT), + 7}; + const std::array invalid_value {static_cast(0xff)}; + const std::array ignored {0}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + const StringRef residual_int(int_seven.data(), int_seven.size()); + auto expect_error = [](const std::string& error, std::string_view expected) { + EXPECT_NE(error.find(expected), std::string::npos) << error; + }; + std::string_view current_case; + + try { + { + current_case = "null metadata"; + SCOPED_TRACE("null metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {1})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "null metadata"); + } + { + current_case = "wrapper without carriers"; + SCOPED_TRACE("wrapper without carriers"); + auto schema = unshredded_schema(); + schema.children.pop_back(); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + expect_error(materialization_error(schema, root_wrapper(std::move(fields))), + "neither value nor typed_value"); + } + { + current_case = "scalar with residual"; + SCOPED_TRACE("scalar with residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "scalar typed_value cannot have residual"); + } + { + current_case = "object with scalar residual"; + SCOPED_TRACE("object with scalar residual"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({9}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(wrapper_fields)), ColumnUInt8::create(1, 0))); + fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "non-object residual"); + } + { + current_case = "object field count mismatch"; + SCOPED_TRACE("object field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + MutableColumns unexpected_object_fields; + unexpected_object_fields.push_back(nullable_int64({1}, {0})); + unexpected_object_fields.push_back(nullable_int64({2}, {0})); + fields.push_back(ColumnNullable::create( + ColumnStruct::create(std::move(unexpected_object_fields)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_object_schema(), + root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "array with residual"; + SCOPED_TRACE("array with residual"); + MutableColumns empty_wrapper_fields; + empty_wrapper_fields.push_back(nullable_int64({}, {})); + auto empty_elements = ColumnNullable::create( + ColumnStruct::create(std::move(empty_wrapper_fields)), ColumnUInt8::create()); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(0); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(empty_elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array typed_value cannot have residual"); + } + { + current_case = "null array element wrapper"; + SCOPED_TRACE("null array element wrapper"); + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({0}, {1})); + auto wrappers = ColumnStruct::create(std::move(wrapper_fields)); + auto elements = ColumnNullable::create(std::move(wrappers), ColumnUInt8::create(1, 1)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error( + materialization_error(shredded_array_schema(), root_wrapper(std::move(fields))), + "array element wrapper is null"); + } + { + current_case = "missing array element"; + SCOPED_TRACE("missing array element"); + MutableColumns element_fields; + element_fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + element_fields.push_back(nullable_int64({0}, {1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(1, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{ignored.data(), 0}}, {1})); + fields.push_back(ColumnNullable::create( + ColumnArray::create(std::move(elements), std::move(offsets)), + ColumnUInt8::create(1, 0))); + expect_error(materialization_error(shredded_mixed_array_schema(), + root_wrapper(std::move(fields))), + "array element is missing"); + } + { + current_case = "root field count mismatch"; + SCOPED_TRACE("root field count mismatch"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + fields.push_back(nullable_int64({8}, {0})); + fields.push_back(nullable_int64({9}, {0})); + expect_error( + materialization_error(shredded_int64_schema(), root_wrapper(std::move(fields))), + "physical field count mismatch"); + } + { + current_case = "invalid metadata"; + SCOPED_TRACE("invalid metadata"); + MutableColumns fields; + fields.push_back(nullable_strings({StringRef("bad")}, {0})); + fields.push_back(nullable_strings({residual_int}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "metadata"); + } + { + current_case = "invalid residual value"; + SCOPED_TRACE("invalid residual value"); + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + expect_error( + materialization_error(unshredded_schema(), root_wrapper(std::move(fields))), + "Variant"); + } + } catch (const std::exception& error) { + FAIL() << "Unexpected exception in " << current_case << ": " << error.what(); + } +} + +TEST(VariantColumnReaderTest, ImmediateCorruptionLeavesDestinationUnchanged) { + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({7}), output) + .ok()); + MutableColumns invalid_fields; + invalid_fields.push_back(nullable_strings( + {{VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()}}, {0})); + const Status status = materialize_variant_rows(shredded_int64_schema(), + root_wrapper(std::move(invalid_fields)), output); + EXPECT_FALSE(status.ok()); + ASSERT_EQ(output->size(), 1); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); +} + +TEST(VariantColumnReaderTest, MaterializesMixedRootArraysAndNullKinds) { + VariantBatchBuilder residual_builder; + { + auto row = residual_builder.begin_row(); + row.add_null(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef("x")); + row.add_int(2); + object.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + auto array = row.start_array(); + row.add_int(3); + row.add_int(4); + array.finish(); + row.finish(); + } + { + auto row = residual_builder.begin_row(); + row.add_string(StringRef("tail")); + row.finish(); + } + VariantBatchBuilder residuals = residual_builder.finish_batch(); + const VariantRef first = residuals.value_at(0); + std::vector residual_values; + for (size_t row = 0; row < residuals.num_rows(); ++row) { + residual_values.push_back(residuals.value_at(row).value); + } + residual_values.insert(residual_values.begin() + 1, StringRef {}); + + MutableColumns element_fields; + element_fields.push_back(nullable_strings(residual_values, {0, 1, 0, 0, 0})); + element_fields.push_back(nullable_int64({0, 1, 0, 0, 0}, {1, 0, 1, 1, 1})); + auto elements = ColumnNullable::create(ColumnStruct::create(std::move(element_fields)), + ColumnUInt8::create(5, 0)); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->get_data().assign({0, 5, 5, 5}); + auto arrays = ColumnArray::create(std::move(elements), std::move(offsets)); + + const StringRef metadata(first.metadata.data, first.metadata.size); + const std::array ignored {0}; + MutableColumns root_fields; + root_fields.push_back(nullable_strings({metadata, metadata, metadata, metadata}, {0, 0, 0, 0})); + root_fields.push_back(nullable_strings( + {{ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}, {ignored.data(), 0}}, + {1, 1, 1, 1})); + auto typed_nulls = ColumnUInt8::create(4, 0); + typed_nulls->get_data()[2] = 1; + typed_nulls->get_data()[3] = 1; + root_fields.push_back(ColumnNullable::create(std::move(arrays), std::move(typed_nulls))); + auto physical = root_wrapper(std::move(root_fields), {0, 0, 0, 1}); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_mixed_array_schema(), *physical, output).ok()); + const auto& nullable = assert_cast(*output); + EXPECT_EQ(nullable.get_null_map_data(), (NullMap {0, 0, 0, 1})); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).num_elements(), 0); + const VariantRef mixed = variants.get_value_ref(1); + ASSERT_EQ(mixed.num_elements(), 5); + EXPECT_TRUE(mixed.array_at(0).is_null()); + EXPECT_EQ(mixed.array_at(1).get_int(), 1); + VariantRef object_field; + ASSERT_TRUE(mixed.array_at(2).object_find(StringRef("x"), &object_field)); + EXPECT_EQ(object_field.get_int(), 2); + EXPECT_EQ(mixed.array_at(3).array_at(1).get_int(), 4); + EXPECT_EQ(mixed.array_at(4).get_string(), StringRef("tail")); + EXPECT_TRUE(variants.get_value_ref(2).is_null()); +} + TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { const std::array int_seven { static_cast(static_cast(VariantPrimitiveId::INT8) @@ -817,6 +1417,124 @@ TEST(VariantColumnReaderTest, MaterializesVariantNestedInStruct) { EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); } +TEST(VariantColumnReaderTest, MaterializesPhysicallyShreddedVariantInStructArrayAndMap) { + auto make_plan_child = [](const ParquetColumnSchema* schema) { + auto child = std::make_unique(); + child->schema = schema; + child->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + return child; + }; + + { + ParquetColumnSchema root_schema; + root_schema.name = "root"; + root_schema.kind = ParquetColumnSchemaKind::STRUCT; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + MutableColumns physical_fields; + physical_fields.push_back(shredded_int64_physical({11})); + auto physical = ColumnStruct::create(std::move(physical_fields)); + auto output = std::make_shared( + DataTypes {make_nullable(std::make_shared())}, + Strings {"v"}) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_column(0)) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 11); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "items"; + root_schema.kind = ParquetColumnSchemaKind::LIST; + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnArray::create(shredded_int64_physical({12, 13}), std::move(offsets)); + auto output = std::make_shared( + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_data()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 12); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 13); + } + + { + ParquetColumnSchema root_schema; + root_schema.name = "entries"; + root_schema.kind = ParquetColumnSchemaKind::MAP; + auto key_schema = std::make_unique(); + key_schema->name = "key"; + key_schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + key_schema->type = std::make_shared(); + root_schema.children.push_back(std::move(key_schema)); + root_schema.children.push_back( + std::make_unique(shredded_int64_schema())); + VariantMaterializationNode plan; + plan.schema = &root_schema; + plan.contains_variant = true; + plan.children.push_back(make_plan_child(root_schema.children[0].get())); + plan.children.push_back(make_plan_child(root_schema.children[1].get())); + auto keys = ColumnString::create(); + keys->insert_data("a", 1); + keys->insert_data("b", 1); + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(2); + auto physical = ColumnMap::create(std::move(keys), shredded_int64_physical({14, 15}), + std::move(offsets)); + auto output = + std::make_shared(std::make_shared(), + make_nullable(std::make_shared())) + ->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, *physical, output).ok()); + const auto& variants = assert_cast( + assert_cast( + assert_cast(*output).get_values()) + .get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 14); + EXPECT_EQ(variants.get_value_ref(1).get_int(), 15); + } +} + +TEST(VariantColumnReaderTest, ProjectedShreddedStateRejectsRootMaterialization) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({17}), output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + EXPECT_THROW((void)variants.get_value_ref(0), Exception); +} + TEST(VariantColumnReaderTest, AlignsNestedPrimitiveNullabilityAroundVariant) { const std::array int_seven { static_cast(static_cast(VariantPrimitiveId::INT8) diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out index 0157263d8a83fc..6abb509f237f85 100644 --- a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -21,6 +21,13 @@ 8 false {"n":30,"name":"same","ok":true} 9 false {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} +-- !variant_root_array_projection -- +1 false [] \N \N \N \N +2 false [null,1,{"x":2},[3,4],"tail"] null 1 2 4 +3 false [{"nested":[null,{"y":5}]}] {"nested":[null,{"y":5}]} \N \N \N +4 false null \N \N \N \N +5 true \N \N \N \N \N + -- !variant_path_expressions -- 1 ALICE 11 1.5 true 3 hz 10 DAVE 51 5.5 false 15 sz @@ -47,13 +54,52 @@ 8 30 9 40 --- !variant_implicit_shredded_filter -- +-- !variant_multi_file_serial -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_multi_file_parallel -- +2 20 \N 2 \N {"b":2,"shared":20,"z":200} +3 30 3 \N \N {"a":3,"shared":30,"z":300} +4 40 \N \N \N {"c":4,"shared":40} +5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} + +-- !variant_type_matrix -- +true -128 -32768 2147483647 -9223372036854775808 true true -1234567890.1234 1970-01-02 1970-01-01T00:00:01.234567 "YmluYXJ5" false + +-- !variant_multi_row_group_result -- +192 8000 8191 1554336 + +-- !variant_deletion_vector_current -- +2048 0 4094 4192256 + +-- !variant_deletion_vector_before_delete -- +4096 0 4095 8386560 + +-- !variant_equality_delete_current -- +1 10 keep-one {"label":"keep-one","n":10} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_equality_delete_before_delete -- +1 10 keep-one {"label":"keep-one","n":10} +2 20 delete {"label":"delete","n":20} +3 30 keep-three {"label":"keep-three","n":30} + +-- !variant_implicit_filter -- 10 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} 11 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} 9 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} +-- !variant_shredded_only_time_travel -- +1095 3001 4095 3885060 + +-- !variant_mixed_before_delete -- +1096 3001 5000 3890060 + -- !variant_page_pruning_result -- -1095 3001 4095 +1094 3001 4094 -- !variant_aggregate -- false 2 70 4 @@ -115,6 +161,15 @@ true 5 170 4.17 -- !variant_delete_only_merge -- 0 +-- !variant_position_delete_alignment -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + +-- !variant_before_position_delete -- +10 dave 50 {"arr":[7,8],"n":50,"name":"dave","nested":{"city":"sz"},"ok":false,"ratio":5.5} +11 null 60 {"arr":[9,10],"n":60,"name":null,"nested":{"city":null},"ok":true,"ratio":6.5} +9 carol 40 {"arr":[5,6],"n":40,"name":"carol","nested":{"city":"bj"},"new_key":"new","ok":true,"ratio":4.5} + -- !variant_orc_missing_column -- 1 \N diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 43c6735a6dad74..65857764eee781 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -56,6 +56,39 @@ suite("test_iceberg_variant_read", .withPathStyleAccessEnabled(true) .withCredentials(new AWSStaticCredentialsProvider(credentials)) .build() + def executeCommand = { String command, int timeoutSeconds = 300 -> + StringBuilder stdout = new StringBuilder() + StringBuilder stderr = new StringBuilder() + def process = new ProcessBuilder("/bin/bash", "-c", command).start() + process.consumeProcessOutput(stdout, stderr) + process.waitForOrKill(timeoutSeconds * 1000) + assertEquals(0, process.exitValue(), + "Command failed\nstdout:\n${stdout}\nstderr:\n${stderr}") + return stdout.toString() + } + String dockerCommand = context.config.otherConfigs.get("externalDockerCommand") ?: "docker" + String sparkContainer = context.config.otherConfigs.get("icebergSparkContainer") + if (sparkContainer == null || sparkContainer.isEmpty()) { + String containers = executeCommand( + "${dockerCommand} ps --format '{{.ID}}\t{{.Names}}'", 30) + def matches = [] + containers.readLines().each { String line -> + String containerId = line.split(/\t/, 2)[0] + String probe = "${dockerCommand} exec ${containerId} bash -lc " + + "'test -f /mnt/SUCCESS && command -v spark-sql >/dev/null'" + try { + executeCommand(probe, 30) + matches.add(containerId) + } catch (Throwable ignored) { + // Only the Spark service contains the Iceberg writer dependencies. + } + } + assertEquals(1, matches.size(), "Expected exactly one usable Spark Iceberg container") + sparkContainer = matches[0] + } + def runInSparkContainer = { String command -> + executeCommand("${dockerCommand} exec ${sparkContainer} bash -lc '${command}'", 300) + } def latestSnapshotId = { String tableName -> List> rows = spark_iceberg """ @@ -100,6 +133,115 @@ suite("test_iceberg_variant_read", (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), (11, parse_json('{"name":null,"n":60,"ratio":6.5,"ok":true,"arr":[9,10],"nested":{"city":null}}')); + DROP TABLE IF EXISTS demo.${dbName}.variant_root_arrays; + CREATE TABLE demo.${dbName}.variant_root_arrays (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_root_arrays VALUES + (1, parse_json('[]')), + (2, parse_json('[null,1,{"x":2},[3,4],"tail"]')), + (3, parse_json('[{"nested":[null,{"y":5}]}]')), + (4, parse_json('null')), + (5, NULL); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_file; + CREATE TABLE demo.${dbName}.variant_multi_file (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='false' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (1, parse_json('{"a":1,"shared":10}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES ( + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='1' + ); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (2, parse_json('{"b":2,"shared":20,"z":200}')); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (3, parse_json('{"z":300,"shared":30,"a":3}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='false'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (4, parse_json('{"c":4,"shared":40}')); + ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES + ('write.parquet.shred-variants'='true'); + INSERT INTO demo.${dbName}.variant_multi_file + VALUES (5, parse_json('{"shared":50,"b":5,"new_field":{"k":500}}')); + + DROP TABLE IF EXISTS demo.${dbName}.variant_type_matrix; + CREATE TABLE demo.${dbName}.variant_type_matrix (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_type_matrix SELECT 1, to_variant_object(named_struct( + 'bool_value', true, + 'tiny_value', CAST(-128 AS TINYINT), + 'small_value', CAST(-32768 AS SMALLINT), + 'int_value', CAST(2147483647 AS INT), + 'big_value', CAST('-9223372036854775808' AS BIGINT), + 'float_value', CAST('NaN' AS FLOAT), + 'double_value', CAST('Infinity' AS DOUBLE), + 'decimal_value', CAST('-1234567890.1234' AS DECIMAL(20, 4)), + 'date_value', CAST('1970-01-02' AS DATE), + 'timestamp_value', TIMESTAMP'1970-01-01 00:00:01.234567', + 'binary_value', CAST('binary' AS BINARY), + 'null_value', CAST(NULL AS INT) + )); + + DROP TABLE IF EXISTS demo.${dbName}.variant_multi_row_group; + CREATE TABLE demo.${dbName}.variant_multi_row_group (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.parquet.row-group-size-bytes'='4096' + ); + SET spark.sql.shuffle.partitions=1; + INSERT INTO demo.${dbName}.variant_multi_row_group + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat( + '{"n":', id, ',"padding":"', repeat('x', 256), '"}')) + FROM range(0, 8192); + + DROP TABLE IF EXISTS demo.${dbName}.variant_deletion_vector; + CREATE TABLE demo.${dbName}.variant_deletion_vector (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100', + 'write.delete.mode'='merge-on-read', + 'read.parquet.vectorization.enabled'='false', + 'write.parquet.row-group-size-bytes'='4096' + ); + INSERT INTO demo.${dbName}.variant_deletion_vector + SELECT /*+ COALESCE(1) */ CAST(id AS INT), parse_json(concat('{"n":', id, ',"keep":', + IF(id % 2 = 0, 'true', 'false'), '}')) + FROM range(0, 4096); + RESET spark.sql.shuffle.partitions; + + DROP TABLE IF EXISTS demo.${dbName}.variant_equality_delete; + CREATE TABLE demo.${dbName}.variant_equality_delete (id INT, v VARIANT) USING iceberg + TBLPROPERTIES ( + 'format-version'='3', + 'write.format.default'='parquet', + 'write.parquet.shred-variants'='true', + 'write.parquet.variant-inference-buffer-size'='100' + ); + INSERT INTO demo.${dbName}.variant_equality_delete VALUES + (1, parse_json('{"n":10,"label":"keep-one"}')), + (2, parse_json('{"n":20,"label":"delete"}')), + (3, parse_json('{"n":30,"label":"keep-three"}')); + DROP TABLE IF EXISTS demo.${dbName}.variant_page_pruning; DROP TABLE IF EXISTS demo.${dbName}.variant_nested; @@ -156,10 +298,83 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); """ + List> multiRowGroupFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_row_group.files WHERE content = 0 + """ + assertEquals(1, multiRowGroupFiles.size()) + assertEquals("1", multiRowGroupFiles[0][0].toString(), + "The multi-row-group fixture must contain exactly one data file") + + String equalityDeleteBaseSnapshot = latestSnapshotId("variant_equality_delete") + String equalityDeleteJava = ''' +import java.util.HashMap; +import java.util.Map; +import org.apache.iceberg.CatalogUtil; +import org.apache.iceberg.DeleteFile; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Catalog; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.data.GenericRecord; +import org.apache.iceberg.data.Record; +import org.apache.iceberg.data.parquet.GenericParquetWriter; +import org.apache.iceberg.deletes.EqualityDeleteWriter; +import org.apache.iceberg.io.OutputFile; +import org.apache.iceberg.parquet.Parquet; + +public class AppendVariantEqualityDelete { + public static void main(String[] args) throws Exception { + Map props = new HashMap<>(); + props.put("type", "rest"); + props.put("uri", "http://rest:8181"); + props.put("warehouse", "s3://warehouse/wh/"); + props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO"); + props.put("s3.endpoint", "http://minio:9000"); + props.put("s3.path-style-access", "true"); + props.put("s3.region", "us-east-1"); + Catalog catalog = CatalogUtil.buildIcebergCatalog("demo", props, null); + Table table = catalog.loadTable(TableIdentifier.of(args[0], args[1])); + Schema equalitySchema = table.schema().select("id"); + int fieldId = table.schema().findField("id").fieldId(); + OutputFile output = table.io().newOutputFile( + table.location() + "/data/variant-equality-delete-" + + System.currentTimeMillis() + ".parquet"); + EqualityDeleteWriter writer = Parquet.writeDeletes(output) + .forTable(table) + .rowSchema(equalitySchema) + .withSpec(PartitionSpec.unpartitioned()) + .createWriterFunc(GenericParquetWriter::create) + .equalityFieldIds(fieldId) + .overwrite() + .buildEqualityWriter(); + GenericRecord record = GenericRecord.create(equalitySchema); + record.setField("id", Integer.valueOf(args[2])); + writer.write(record); + writer.close(); + DeleteFile deleteFile = writer.toDeleteFile(); + table.newRowDelta().addDeletes(deleteFile).commit(); + } +} +''' + String encodedEqualityDeleteJava = + equalityDeleteJava.getBytes("UTF-8").encodeBase64().toString() + runInSparkContainer( + "echo ${encodedEqualityDeleteJava} | base64 -d " + + ">/tmp/AppendVariantEqualityDelete.java && " + + "javac -cp \"/opt/spark/jars/*\" " + + "/tmp/AppendVariantEqualityDelete.java && " + + "java -cp \"/tmp:/opt/spark/jars/*\" AppendVariantEqualityDelete " + + "${dbName} variant_equality_delete 2") + String writeGuardSourceSnapshot = latestSnapshotId("variant_write_guard") + String deletionVectorBaseSnapshot = latestSnapshotId("variant_deletion_vector") spark_iceberg """ ALTER TABLE demo.${dbName}.variant_write_guard ADD COLUMN payload VARIANT """ + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_deletion_vector WHERE id % 2 = 1 + """ // Register a stable Iceberg metadata fixture so the page-pruning case always uses a // standards-compliant shredded Variant file, independent of the Spark writer version. @@ -175,6 +390,21 @@ suite("test_iceberg_variant_read", metadata_file => 's3a://warehouse/wh/${dbName}/variant_page_pruning/metadata/${shreddedMetadataName}') """ + String shreddedOnlySnapshot = latestSnapshotId("variant_page_pruning") + spark_iceberg_multi """ + ALTER TABLE demo.${dbName}.variant_page_pruning SET TBLPROPERTIES ( + 'read.parquet.vectorization.enabled'='false', + 'write.delete.mode'='merge-on-read' + ); + INSERT INTO demo.${dbName}.variant_page_pruning VALUES + (5000, parse_json('{"n":5000,"padding":"mixed-unshredded"}')); + """ + String mixedBeforeDeleteSnapshot = latestSnapshotId("variant_page_pruning") + // One deletion vector targets the shredded fixture and another targets the appended + // unshredded file, forcing both physical states through the same scan and delete alignment. + spark_iceberg """ + DELETE FROM demo.${dbName}.variant_page_pruning WHERE id IN (4095, 5000) + """ sql """drop catalog if exists ${catalogName}""" sql """ @@ -198,18 +428,6 @@ suite("test_iceberg_variant_read", sql """set profile_level=2""" def profileAction = new ProfileAction(context) - def getProfileByToken = { String token -> - for (int retry = 0; retry < 20; ++retry) { - List profileData = profileAction.getProfileList() - for (final def profileItem in profileData) { - if (profileItem["Sql Statement"].toString().contains(token)) { - return profileAction.getProfile(profileItem["Profile ID"].toString()) - } - } - Thread.sleep(500) - } - throw new IllegalStateException("Missing profile for token: " + token) - } def counterSum = { String profile, String counterName -> Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") Matcher matcher = pattern.matcher(profile) @@ -219,6 +437,25 @@ suite("test_iceberg_variant_read", } return sum } + def getProfileByToken = { String token, List positiveCounters = [] -> + String lastProfile = "" + for (int retry = 0; retry < 20; ++retry) { + List profileData = profileAction.getProfileList() + for (final def profileItem in profileData) { + if (profileItem["Sql Statement"].toString().contains(token)) { + lastProfile = profileAction.getProfile( + profileItem["Profile ID"].toString()).toString() + if (positiveCounters.every { counterSum(lastProfile, it) > 0 }) { + return lastProfile + } + } + } + Thread.sleep(500) + } + throw new IllegalStateException( + "Profile did not expose positive counters ${positiveCounters} for token ${token}: " + + lastProfile) + } String evolutionInitial = latestSnapshotId("variant_evolution") sql """ALTER TABLE variant_evolution CREATE TAG variant_initial""" @@ -306,6 +543,18 @@ suite("test_iceberg_variant_read", ORDER BY id """ + order_qt_variant_root_array_projection """ + SELECT id, + v IS NULL, + CAST(v AS STRING), + CAST(v[1] AS STRING), + CAST(v[2] AS INT), + CAST(v[3]['x'] AS INT), + CAST(v[4][2] AS INT) + FROM variant_root_arrays + ORDER BY id + """ + order_qt_variant_path_expressions """ SELECT id, UPPER(CAST(v['name'] AS STRING)), @@ -327,8 +576,8 @@ suite("test_iceberg_variant_read", ORDER BY id """ - // The first INSERT is unshredded while the second is shredded. Keep both small files on one - // scanner so their complete and leaf-only physical states must be projected before batching. + // Keep the independent Spark writes on one scanner to exercise metadata dictionaries and + // complete Variant state transitions across file boundaries before batching. sql "set parallel_pipeline_task_num=1" sql "set max_file_scanners_concurrency=1" order_qt_variant_cross_file_leaf_projection """ @@ -337,44 +586,167 @@ suite("test_iceberg_variant_read", ORDER BY id """ - // Keep the root Variant as output while the implicit scalar comparison drives the shredded - // typed_value statistics/page-index path. - order_qt_variant_implicit_shredded_filter """ + order_qt_variant_multi_file_serial """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + sql "set parallel_pipeline_task_num=4" + sql "set max_file_scanners_concurrency=8" + order_qt_variant_multi_file_parallel """ + SELECT id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + + order_qt_variant_type_matrix """ + SELECT CAST(v['bool_value'] AS BOOLEAN), + CAST(v['tiny_value'] AS TINYINT), + CAST(v['small_value'] AS SMALLINT), + CAST(v['int_value'] AS INT), + CAST(v['big_value'] AS BIGINT), + ISNAN(CAST(v['float_value'] AS FLOAT)), + ISINF(CAST(v['double_value'] AS DOUBLE)), + CAST(v['decimal_value'] AS DECIMAL(20, 4)), + CAST(v['date_value'] AS DATE), + CAST(v['timestamp_value'] AS DATETIMEV2(6)), + CAST(v['binary_value'] AS STRING), + v['null_value'] IS NULL + FROM variant_type_matrix + """ + + String multiRowGroupColdToken = + "iceberg_variant_multi_row_group_cold_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupColdToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupColdProfile = getProfileByToken(multiRowGroupColdToken, + ["RowGroupsTotalNum", "VariantDirectLeafPathMisses", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + assertTrue(counterSum(multiRowGroupColdProfile, "RowGroupsTotalNum") > 1, + "The generated Variant file did not contain multiple Parquet row groups") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantDirectLeafPathMisses") > 0, + "The unshredded scan did not record its direct-leaf fallback") + assertTrue(counterSum(multiRowGroupColdProfile, "VariantReconstructedRows") > 0, + "The unshredded scan did not reconstruct Variant rows") + assertTrue(counterSum(multiRowGroupColdProfile, "FilteredRowsByLazyRead") > 0, + "The unshredded Variant predicate did not defer non-predicate columns") + String multiRowGroupWarmToken = + "iceberg_variant_multi_row_group_warm_" + UUID.randomUUID().toString() + sql """ + SELECT '${multiRowGroupWarmToken}', COUNT(*), MIN(id), MAX(id) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + String multiRowGroupWarmProfile = getProfileByToken(multiRowGroupWarmToken, + ["VariantDirectLeafPathMisses"]).toString() + assertTrue(counterSum(multiRowGroupWarmProfile, "VariantDirectLeafPathMisses") > 0, + "The warm unshredded scan did not preserve its direct-leaf fallback") + qt_variant_multi_row_group_result """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_multi_row_group + WHERE CAST(v['n'] AS INT) >= 8000 + """ + + qt_variant_deletion_vector_current """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector + WHERE v['keep'] = true + """ + qt_variant_deletion_vector_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_deletion_vector FOR VERSION AS OF ${deletionVectorBaseSnapshot} + WHERE v['n'] >= 0 + """ + order_qt_variant_equality_delete_current """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete + WHERE v['n'] >= 0 + ORDER BY id + """ + order_qt_variant_equality_delete_before_delete """ + SELECT id, CAST(v['n'] AS INT), CAST(v['label'] AS STRING), CAST(v AS STRING) + FROM variant_equality_delete FOR VERSION AS OF ${equalityDeleteBaseSnapshot} + WHERE v['n'] >= 0 + ORDER BY id + """ + + // Keep the root Variant as output while the scalar comparison exercises the fallback path for + // the unshredded Spark files. + order_qt_variant_implicit_filter """ SELECT id, CAST(v AS STRING) FROM variant_values WHERE v['n'] > 35 ORDER BY id """ - // The query projects the root Variant, while the predicate uses typed_value page metadata. + qt_variant_shredded_only_time_travel """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + qt_variant_mixed_before_delete """ + SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + + // The query projects the complete Variant while its predicate reads the shredded typed leaf. + // The appended unshredded file must fall back independently in the same scan. String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() sql """ SELECT '${pagePruningToken}', id, CAST(v AS STRING) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 ORDER BY id """ - String pagePruningProfile = getProfileByToken(pagePruningToken).toString() + String pagePruningProfile = getProfileByToken(pagePruningToken, + ["FilteredRowsByPage", "VariantLeafProjections", "VariantDirectLeafPathMisses", + "VariantDirectLeafRows", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, "Shredded Variant typed_value did not filter any Parquet page") // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant // root is read through the independent deferred-output projection. assertTrue(counterSum(pagePruningProfile, "VariantLeafProjections") > 0, "A root Variant output query did not retain its typed predicate leaf projection") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafPathMisses") > 0, + "The mixed scan did not fall back for its unshredded Variant file") + assertTrue(counterSum(pagePruningProfile, "VariantDirectLeafRows") > 0, + "The mixed scan did not evaluate rows from the shredded typed leaf") + assertTrue(counterSum(pagePruningProfile, "VariantReconstructedRows") > 0, + "The mixed scan did not reconstruct complete Variant output") + assertTrue(counterSum(pagePruningProfile, "FilteredRowsByLazyRead") > 0, + "The mixed Variant scan did not delay output materialization") String leafProjectionToken = "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() sql """ SELECT '${leafProjectionToken}', COUNT(*) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 """ - String leafProjectionProfile = getProfileByToken(leafProjectionToken).toString() + String leafProjectionProfile = getProfileByToken(leafProjectionToken, + ["VariantLeafProjections"]).toString() assertTrue(counterSum(leafProjectionProfile, "VariantLeafProjections") > 0, "Variant typed predicate did not retain a physical leaf projection") qt_variant_page_pruning_result """ SELECT COUNT(*), MIN(id), MAX(id) FROM variant_page_pruning - WHERE v['n'] > 3000 + WHERE CAST(v['n'] AS INT) > 3000 """ // A later Variant metadata predicate must not prune away an earlier error-producing conjunct. @@ -529,6 +901,7 @@ suite("test_iceberg_variant_read", // A delete-only MERGE emits only position deletes. It must remain available even though // update/insert actions would route the unchanged Variant through the unsupported data writer. + String beforePositionDeleteSnapshot = latestSnapshotId("variant_values") sql """ MERGE INTO variant_values t USING (SELECT 11 AS id) s @@ -536,6 +909,31 @@ suite("test_iceberg_variant_read", WHEN MATCHED THEN DELETE """ qt_variant_delete_only_merge "SELECT COUNT(*) FROM variant_values WHERE id = 11" + order_qt_variant_position_delete_alignment """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values + WHERE v['n'] >= 40 + ORDER BY id + """ + order_qt_variant_before_position_delete """ + SELECT id, CAST(v['name'] AS STRING), CAST(v['n'] AS INT), CAST(v AS STRING) + FROM variant_values FOR VERSION AS OF ${beforePositionDeleteSnapshot} + WHERE v['n'] >= 40 + ORDER BY id + """ + String positionDeleteToken = + "iceberg_variant_position_delete_" + UUID.randomUUID().toString() + sql """ + SELECT '${positionDeleteToken}', COUNT(*) + FROM variant_values + WHERE v['n'] >= 40 + """ + String positionDeleteProfile = getProfileByToken(positionDeleteToken, + ["VariantDirectLeafPathMisses", "VariantReconstructedRows"]).toString() + assertTrue(counterSum(positionDeleteProfile, "VariantDirectLeafPathMisses") > 0, + "Position-delete filtering did not preserve the unshredded Variant fallback") + assertTrue(counterSum(positionDeleteProfile, "VariantReconstructedRows") > 0, + "Position-delete filtering did not reconstruct its Variant rows") // Files written before the Variant field existed have no physical Variant payload. Schema // evolution must synthesize NULL instead of rejecting their non-Parquet file format. From f3b7efb7cb0ed04eb0d11adf3197eb382d879e1d Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 16:47:16 +0800 Subject: [PATCH 06/20] fix: preserve nested Variant append atomicity --- .../parquet/reader/variant_column_reader.cpp | 101 +++++++--- .../parquet/variant_column_reader_test.cpp | 176 ++++++++++++++++++ .../iceberg/test_iceberg_variant_read.groovy | 170 +++++++++++++---- 3 files changed, 390 insertions(+), 57 deletions(-) diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index 4827f951c94fe5..b51358a4f0649c 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -26,6 +26,7 @@ #include #include #include +#include #include "common/exception.h" #include "core/assert_cast.h" @@ -852,16 +853,30 @@ ColumnPtr transform_node(const VariantMaterializationNode& plan, ColumnPtr physi void append_compatible_column(IColumn& output, const IColumn& converted) { if (auto* output_nullable = check_and_get_column(output)) { - if (const auto* converted_nullable = check_and_get_column(converted)) { - append_compatible_column(output_nullable->get_nested_column(), - converted_nullable->get_nested_column()); - output_nullable->get_null_map_column().insert_range_from( - converted_nullable->get_null_map_column(), 0, converted.size()); - } else { - append_compatible_column(output_nullable->get_nested_column(), converted); - // External slots and nested Iceberg fields may remain nullable even when one file's - // physical node is required. Preserve that destination invariant with non-null bits. - output_nullable->push_false_to_nullmap(converted.size()); + auto& nested = output_nullable->get_nested_column(); + auto& null_map = output_nullable->get_null_map_column(); + const size_t nested_size = nested.size(); + const size_t null_map_size = null_map.size(); + try { + if (const auto* converted_nullable = check_and_get_column(converted)) { + append_compatible_column(nested, converted_nullable->get_nested_column()); + null_map.insert_range_from(converted_nullable->get_null_map_column(), 0, + converted.size()); + } else { + append_compatible_column(nested, converted); + // External slots and nested Iceberg fields may remain nullable even when one + // file's physical node is required. Preserve that destination invariant with + // non-null bits. + output_nullable->push_false_to_nullmap(converted.size()); + } + } catch (...) { + if (nested.size() > nested_size) { + nested.pop_back(nested.size() - nested_size); + } + if (null_map.size() > null_map_size) { + null_map.pop_back(null_map.size() - null_map_size); + } + throw; } return; } @@ -885,8 +900,25 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible STRUCT"); } + std::vector original_sizes(output_struct->tuple_size()); for (size_t i = 0; i < output_struct->tuple_size(); ++i) { - append_compatible_column(output_struct->get_column(i), converted_struct->get_column(i)); + original_sizes[i] = output_struct->get_column(i).size(); + } + try { + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + append_compatible_column(output_struct->get_column(i), + converted_struct->get_column(i)); + } + } catch (...) { + // Variant corruption can surface only during lazy fallback after earlier siblings + // were appended. Roll every child back to preserve the failed-append invariant. + for (size_t i = 0; i < output_struct->tuple_size(); ++i) { + auto& child = output_struct->get_column(i); + if (child.size() > original_sizes[i]) { + child.pop_back(child.size() - original_sizes[i]); + } + } + throw; } return; } @@ -897,12 +929,22 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible ARRAY"); } - const size_t element_base = output_array->get_data().size(); - append_compatible_column(output_array->get_data(), converted_array->get_data()); + auto& output_data = output_array->get_data(); auto& output_offsets = output_array->get_offsets(); - output_offsets.reserve(output_offsets.size() + converted_array->size()); - for (const auto offset : converted_array->get_offsets()) { - output_offsets.push_back(element_base + offset); + const size_t element_base = output_data.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_data, converted_array->get_data()); + output_offsets.reserve(output_offsets.size() + converted_array->size()); + for (const auto offset : converted_array->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_data.size() > element_base) { + output_data.pop_back(output_data.size() - element_base); + } + output_offsets.resize(offsets_size); + throw; } return; } @@ -913,13 +955,28 @@ void append_compatible_column(IColumn& output, const IColumn& converted) { throw Exception(ErrorCode::CORRUPTION, "Parquet Variant materialization produced an incompatible MAP"); } - const size_t element_base = output_map->get_keys().size(); - append_compatible_column(output_map->get_keys(), converted_map->get_keys()); - append_compatible_column(output_map->get_values(), converted_map->get_values()); + auto& output_keys = output_map->get_keys(); + auto& output_values = output_map->get_values(); auto& output_offsets = output_map->get_offsets(); - output_offsets.reserve(output_offsets.size() + converted_map->size()); - for (const auto offset : converted_map->get_offsets()) { - output_offsets.push_back(element_base + offset); + const size_t element_base = output_keys.size(); + const size_t values_size = output_values.size(); + const size_t offsets_size = output_offsets.size(); + try { + append_compatible_column(output_keys, converted_map->get_keys()); + append_compatible_column(output_values, converted_map->get_values()); + output_offsets.reserve(output_offsets.size() + converted_map->size()); + for (const auto offset : converted_map->get_offsets()) { + output_offsets.push_back(element_base + offset); + } + } catch (...) { + if (output_keys.size() > element_base) { + output_keys.pop_back(output_keys.size() - element_base); + } + if (output_values.size() > values_size) { + output_values.pop_back(output_values.size() - values_size); + } + output_offsets.resize(offsets_size); + throw; } return; } diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 5bf2d33c10f1af..24ace262faba37 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -1300,6 +1300,182 @@ TEST(VariantColumnReaderTest, ImmediateCorruptionLeavesDestinationUnchanged) { EXPECT_EQ(variants.get_value_ref(0).get_int(), 7); } +TEST(VariantColumnReaderTest, LazyNestedCorruptionLeavesDestinationUnchanged) { + const std::array invalid_value {static_cast(0xff)}; + const StringRef metadata(VARIANT_EMPTY_METADATA.data(), VARIANT_EMPTY_METADATA.size()); + auto corrupt_variant = [&]() { + MutableColumns fields; + fields.push_back(nullable_strings({metadata}, {0})); + fields.push_back(nullable_strings({{invalid_value.data(), invalid_value.size()}}, {0})); + return root_wrapper(std::move(fields)); + }; + auto label_schema = []() { + auto schema = std::make_unique(); + schema->name = "label"; + schema->kind = ParquetColumnSchemaKind::PRIMITIVE; + schema->type = make_nullable(std::make_shared()); + return schema; + }; + auto make_plan = [](const ParquetColumnSchema& root) { + auto build = [&](auto&& self, const ParquetColumnSchema* schema) + -> std::unique_ptr { + auto node = std::make_unique(); + node->schema = schema; + node->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + for (const auto& child_schema : schema->children) { + auto child = self(self, child_schema.get()); + node->contains_variant = node->contains_variant || child->contains_variant; + node->children.push_back(std::move(child)); + } + return node; + }; + return build(build, &root); + }; + auto make_struct_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "row"; + root.kind = ParquetColumnSchemaKind::STRUCT; + root.children.push_back(label_schema()); + root.children.push_back(std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_struct_physical = [&](std::string_view label, MutableColumnPtr variant) { + MutableColumns fields; + fields.push_back(nullable_strings({StringRef(label.data(), label.size())}, {0})); + fields.push_back(std::move(variant)); + return ColumnStruct::create(std::move(fields)); + }; + const auto element_type = std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"label", "payload"}); + + { + auto output = element_type->create_column(); + auto valid_schema = make_struct_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, + *make_struct_physical("before", shredded_int64_physical({7})), output) + .ok()); + + auto corrupt_schema = make_struct_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_struct_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& structure = assert_cast(*output); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = std::make_shared(element_type)->create_column(); + auto valid_element_schema = make_struct_schema(shredded_int64_schema()); + ParquetColumnSchema valid_schema; + valid_schema.name = "rows"; + valid_schema.kind = ParquetColumnSchemaKind::LIST; + valid_schema.children.push_back( + std::make_unique(std::move(valid_element_schema))); + auto valid_plan = make_plan(valid_schema); + auto valid_offsets = ColumnArray::ColumnOffsets::create(); + valid_offsets->insert_value(1); + auto valid_physical = + ColumnArray::create(make_struct_physical("before", shredded_int64_physical({7})), + std::move(valid_offsets)); + ASSERT_TRUE(materialize_variant_columns(*valid_plan, *valid_physical, output).ok()); + + auto corrupt_element_schema = make_struct_schema(unshredded_schema()); + ParquetColumnSchema corrupt_schema; + corrupt_schema.name = "rows"; + corrupt_schema.kind = ParquetColumnSchemaKind::LIST; + corrupt_schema.children.push_back( + std::make_unique(std::move(corrupt_element_schema))); + auto corrupt_plan = make_plan(corrupt_schema); + auto corrupt_offsets = ColumnArray::ColumnOffsets::create(); + corrupt_offsets->insert_value(1); + auto corrupt_physical = ColumnArray::create( + make_struct_physical("after", corrupt_variant()), std::move(corrupt_offsets)); + const Status status = materialize_variant_columns(*corrupt_plan, *corrupt_physical, output); + EXPECT_FALSE(status.ok()); + + const auto& array = assert_cast(*output); + EXPECT_EQ(array.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& element = assert_cast(array.get_data()); + EXPECT_EQ(element.get_null_map_data(), (NullMap {0})); + const auto& structure = assert_cast(element.get_nested_column()); + const auto& label = assert_cast(structure.get_column(0)); + EXPECT_EQ(label.size(), 1); + EXPECT_EQ(label.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(label.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& payload = assert_cast(structure.get_column(1)); + EXPECT_EQ(payload.size(), 1); + EXPECT_EQ(payload.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(payload.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } + + { + auto output = + std::make_shared(make_nullable(std::make_shared()), + make_nullable(std::make_shared())) + ->create_column(); + auto make_map_schema = [&](ParquetColumnSchema variant_schema) { + ParquetColumnSchema root; + root.name = "entries"; + root.kind = ParquetColumnSchemaKind::MAP; + root.children.push_back(label_schema()); + root.children.push_back( + std::make_unique(std::move(variant_schema))); + return root; + }; + auto make_map_physical = [&](std::string_view key, MutableColumnPtr variant) { + auto offsets = ColumnArray::ColumnOffsets::create(); + offsets->insert_value(1); + return ColumnMap::create(nullable_strings({StringRef(key.data(), key.size())}, {0}), + std::move(variant), std::move(offsets)); + }; + + auto valid_schema = make_map_schema(shredded_int64_schema()); + auto valid_plan = make_plan(valid_schema); + ASSERT_TRUE(materialize_variant_columns( + *valid_plan, *make_map_physical("before", shredded_int64_physical({7})), + output) + .ok()); + auto corrupt_schema = make_map_schema(unshredded_schema()); + auto corrupt_plan = make_plan(corrupt_schema); + const Status status = materialize_variant_columns( + *corrupt_plan, *make_map_physical("after", corrupt_variant()), output); + EXPECT_FALSE(status.ok()); + + const auto& map = assert_cast(*output); + EXPECT_EQ(map.get_offsets(), (ColumnArray::Offsets64 {1})); + const auto& keys = assert_cast(map.get_keys()); + EXPECT_EQ(keys.size(), 1); + EXPECT_EQ(keys.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(keys.get_nested_column().get_data_at(0).to_string(), "before"); + const auto& values = assert_cast(map.get_values()); + EXPECT_EQ(values.size(), 1); + EXPECT_EQ(values.get_null_map_data(), (NullMap {0})); + EXPECT_EQ(assert_cast(values.get_nested_column()) + .get_value_ref(0) + .get_int(), + 7); + } +} + TEST(VariantColumnReaderTest, MaterializesMixedRootArraysAndNullKinds) { VariantBatchBuilder residual_builder; { diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index 65857764eee781..ffc99236a1976f 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -123,11 +123,11 @@ suite("test_iceberg_variant_read", (6, parse_json('42')), (7, parse_json('"root-string"')); ALTER TABLE demo.${dbName}.variant_values SET TBLPROPERTIES ( - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_values - WITH (`shred-variants`=true, `variant-inference-buffer-size`=100) VALUES + VALUES (8, parse_json('{"ok":true,"n":30,"name":"same"}')), (9, parse_json('{"name":"carol","n":40,"ratio":4.5,"ok":true,"arr":[5,6],"nested":{"city":"bj"},"new_key":"new"}')), (10, parse_json('{"name":"dave","n":50,"ratio":5.5,"ok":false,"arr":[7,8],"nested":{"city":"sz"}}')), @@ -138,7 +138,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_root_arrays VALUES @@ -158,7 +158,7 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_multi_file VALUES (1, parse_json('{"a":1,"shared":10}')); ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES ( - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='1' ); INSERT INTO demo.${dbName}.variant_multi_file @@ -170,7 +170,7 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_multi_file VALUES (4, parse_json('{"c":4,"shared":40}')); ALTER TABLE demo.${dbName}.variant_multi_file SET TBLPROPERTIES - ('write.parquet.shred-variants'='true'); + ('write.parquet.shred-variants'='false'); INSERT INTO demo.${dbName}.variant_multi_file VALUES (5, parse_json('{"shared":50,"b":5,"new_field":{"k":500}}')); @@ -179,7 +179,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_type_matrix SELECT 1, to_variant_object(named_struct( @@ -202,7 +202,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100', 'write.parquet.row-group-size-bytes'='4096' ); @@ -217,7 +217,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100', 'write.delete.mode'='merge-on-read', 'read.parquet.vectorization.enabled'='false', @@ -234,7 +234,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_equality_delete VALUES @@ -254,7 +254,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_nested SELECT @@ -276,7 +276,7 @@ suite("test_iceberg_variant_read", TBLPROPERTIES ( 'format-version'='3', 'write.format.default'='parquet', - 'write.parquet.shred-variants'='true', + 'write.parquet.shred-variants'='false', 'write.parquet.variant-inference-buffer-size'='100' ); INSERT INTO demo.${dbName}.variant_signed_selector @@ -298,6 +298,13 @@ suite("test_iceberg_variant_read", INSERT INTO demo.${dbName}.variant_write_guard VALUES (1); """ + List> multiFileDataFiles = spark_iceberg """ + SELECT COUNT(*) FROM demo.${dbName}.variant_multi_file.files WHERE content = 0 + """ + assertEquals(1, multiFileDataFiles.size()) + assertTrue(Long.parseLong(multiFileDataFiles[0][0].toString()) > 1, + "The parallel Variant fixture must contain multiple data files") + List> multiRowGroupFiles = spark_iceberg """ SELECT COUNT(*) FROM demo.${dbName}.variant_multi_row_group.files WHERE content = 0 """ @@ -375,6 +382,21 @@ public class AppendVariantEqualityDelete { spark_iceberg """ DELETE FROM demo.${dbName}.variant_deletion_vector WHERE id % 2 = 1 """ + List> deletionVectorFiles = spark_iceberg """ + SELECT file_format, content_offset, content_size_in_bytes + FROM demo.${dbName}.variant_deletion_vector.files + WHERE content = 1 + """ + assertFalse(deletionVectorFiles.isEmpty(), + "The Variant deletion fixture must expose a live delete file") + deletionVectorFiles.each { List deleteFile -> + assertEquals("PUFFIN", deleteFile[0].toString().toUpperCase(), + "The format-v3 Variant fixture must use PUFFIN deletion vectors") + assertTrue(Long.parseLong(deleteFile[1].toString()) >= 0, + "A PUFFIN deletion vector must expose its content offset") + assertTrue(Long.parseLong(deleteFile[2].toString()) > 0, + "A PUFFIN deletion vector must expose its content size") + } // Register a stable Iceberg metadata fixture so the page-pruning case always uses a // standards-compliant shredded Variant file, independent of the Spark writer version. @@ -428,33 +450,60 @@ public class AppendVariantEqualityDelete { sql """set profile_level=2""" def profileAction = new ProfileAction(context) + def mergedProfile = { String profile -> + if (!profile.contains("MergedProfile:")) { + return profile + } + String merged = profile.substring(profile.indexOf("MergedProfile:")) + int end = merged.length() + ["DetailProfile(", "Execution Profile:", "Appendix:"].each { String sectionName -> + int sectionIndex = merged.indexOf(sectionName) + if (sectionIndex > 0) { + end = Math.min(end, sectionIndex) + } + } + return merged.substring(0, end) + } def counterSum = { String profile, String counterName -> - Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)") - Matcher matcher = pattern.matcher(profile) + Pattern pattern = Pattern.compile("(?m)^\\s*(?:-\\s*)?" + + Pattern.quote(counterName) + ":\\s+([^\\n]+)") + Matcher matcher = pattern.matcher(mergedProfile(profile)) long sum = 0 while (matcher.find()) { - sum += Long.parseLong(matcher.group(1).replace(",", "")) + String valueText = matcher.group(1) + // Merged counters may be human-readable; the parenthesized value is the exact sum. + Matcher exact = Pattern.compile("\\(([0-9,]+)\\)").matcher(valueText) + Matcher number = Pattern.compile("([0-9,]+)").matcher(valueText) + if (exact.find()) { + sum += Long.parseLong(exact.group(1).replace(",", "")) + } else if (number.find()) { + sum += Long.parseLong(number.group(1).replace(",", "")) + } } return sum } + def profileInfoValues = { String profile, String infoName -> + Pattern pattern = Pattern.compile( + Pattern.quote(infoName) + ":\\s*\\[([^\\]]*)\\]") + Matcher matcher = pattern.matcher(profile) + if (!matcher.find()) { + return [] + } + return matcher.group(1).split(",").collect { String value -> value.trim() } + .findAll { String value -> !value.isEmpty() } + .collect { String value -> Long.parseLong(value.replace(",", "")) } + } def getProfileByToken = { String token, List positiveCounters = [] -> - String lastProfile = "" - for (int retry = 0; retry < 20; ++retry) { - List profileData = profileAction.getProfileList() - for (final def profileItem in profileData) { - if (profileItem["Sql Statement"].toString().contains(token)) { - lastProfile = profileAction.getProfile( - profileItem["Profile ID"].toString()).toString() - if (positiveCounters.every { counterSum(lastProfile, it) > 0 }) { - return lastProfile - } - } - } - Thread.sleep(500) + String lastProfile = profileAction.getProfileBySql(token, positiveCounters) + if (positiveCounters.every { String counter -> counterSum(lastProfile, counter) > 0 }) { + return lastProfile } - throw new IllegalStateException( - "Profile did not expose positive counters ${positiveCounters} for token ${token}: " + - lastProfile) + return profileAction.waitProfile({ + lastProfile = profileAction.getProfileBySql(token, positiveCounters) + return positiveCounters.every { + String counter -> counterSum(lastProfile, counter) > 0 + } ? lastProfile : "" + }, [], "Completed profile with positive counters ${positiveCounters} for ${token}") } String evolutionInitial = latestSnapshotId("variant_evolution") @@ -599,6 +648,7 @@ public class AppendVariantEqualityDelete { """ sql "set parallel_pipeline_task_num=4" sql "set max_file_scanners_concurrency=8" + sql "set min_file_scanners_concurrency=4" order_qt_variant_multi_file_parallel """ SELECT id, CAST(v['shared'] AS INT), @@ -610,6 +660,36 @@ public class AppendVariantEqualityDelete { WHERE v['shared'] >= 20 ORDER BY id """ + String parallelScanToken = + "iceberg_variant_parallel_scan_" + UUID.randomUUID().toString() + List> parallelScanRows = sql """ + SELECT '${parallelScanToken}', id, + CAST(v['shared'] AS INT), + CAST(v['a'] AS INT), + CAST(v['b'] AS INT), + CAST(v['new_field']['k'] AS INT), + CAST(v AS STRING) + FROM variant_multi_file + WHERE v['shared'] >= 20 + ORDER BY id + """ + assertEquals(4, parallelScanRows.size(), + "The parallel Variant query must read rows from multiple data files") + String parallelScanProfile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + if (profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } <= 1) { + parallelScanProfile = profileAction.waitProfile({ + String profile = profileAction.getProfileBySql( + parallelScanToken, ["PerScannerRowsRead"]) + return profileInfoValues(profile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1 ? profile : "" + }, [], "Completed parallel Variant profile with multiple non-empty scanners") + } + assertTrue(profileInfoValues(parallelScanProfile, "PerScannerRowsRead") + .count { long rows -> rows > 0 } > 1, + "The parallel Variant query did not use multiple non-empty scanners") + sql "set min_file_scanners_concurrency=1" order_qt_variant_type_matrix """ SELECT CAST(v['bool_value'] AS BOOLEAN), @@ -665,7 +745,7 @@ public class AppendVariantEqualityDelete { qt_variant_deletion_vector_current """ SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) FROM variant_deletion_vector - WHERE v['keep'] = true + WHERE v['n'] >= 0 """ qt_variant_deletion_vector_before_delete """ SELECT COUNT(*), MIN(id), MAX(id), SUM(CAST(v['n'] AS BIGINT)) @@ -705,6 +785,29 @@ public class AppendVariantEqualityDelete { WHERE CAST(v['n'] AS INT) > 3000 """ + // The complete Variant is the only scanned output column outside the predicate. A positive + // lazy-read count therefore proves Variant output deferral rather than deferral of an id + // sibling, while the row relationship proves reconstruction happens after filtering. + String lazyVariantToken = + "iceberg_variant_lazy_materialization_" + UUID.randomUUID().toString() + List> lazyVariantRows = sql """ + SELECT '${lazyVariantToken}', CAST(v AS STRING) + FROM variant_page_pruning FOR VERSION AS OF ${shreddedOnlySnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + """ + String lazyVariantProfile = getProfileByToken(lazyVariantToken, + ["VariantDirectLeafRows", "VariantReconstructedRows", + "FilteredRowsByLazyRead"]).toString() + long reconstructedVariantRows = + counterSum(lazyVariantProfile, "VariantReconstructedRows") + assertEquals((long) lazyVariantRows.size(), reconstructedVariantRows, + "Complete Variant reconstruction must be limited to selected output rows") + assertTrue(counterSum(lazyVariantProfile, "VariantDirectLeafRows") > + reconstructedVariantRows, + "Variant output was not deferred until after its shredded-leaf predicate") + assertTrue(counterSum(lazyVariantProfile, "FilteredRowsByLazyRead") > 0, + "The shredded predicate did not defer complete Variant output") + // The query projects the complete Variant while its predicate reads the shredded typed leaf. // The appended unshredded file must fall back independently in the same scan. String pagePruningToken = "iceberg_variant_page_pruning_" + UUID.randomUUID().toString() @@ -716,8 +819,7 @@ public class AppendVariantEqualityDelete { """ String pagePruningProfile = getProfileByToken(pagePruningToken, ["FilteredRowsByPage", "VariantLeafProjections", "VariantDirectLeafPathMisses", - "VariantDirectLeafRows", "VariantReconstructedRows", - "FilteredRowsByLazyRead"]).toString() + "VariantDirectLeafRows", "VariantReconstructedRows"]).toString() assertTrue(counterSum(pagePruningProfile, "FilteredRowsByPage") > 0, "Shredded Variant typed_value did not filter any Parquet page") // The predicate_access_paths contract keeps the typed leaf eager while the complete Variant @@ -730,8 +832,6 @@ public class AppendVariantEqualityDelete { "The mixed scan did not evaluate rows from the shredded typed leaf") assertTrue(counterSum(pagePruningProfile, "VariantReconstructedRows") > 0, "The mixed scan did not reconstruct complete Variant output") - assertTrue(counterSum(pagePruningProfile, "FilteredRowsByLazyRead") > 0, - "The mixed Variant scan did not delay output materialization") String leafProjectionToken = "iceberg_variant_leaf_projection_" + UUID.randomUUID().toString() sql """ From 07dc32521f2afda29c244400512f9d32846fc88b Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 16:13:32 +0800 Subject: [PATCH 07/20] [fix](parquet) Isolate Variant planning from ordinary scans --- .../parquet/parquet_column_schema.cpp | 14 +++-------- .../format_v2/parquet/parquet_column_schema.h | 4 +++ .../parquet/parquet_file_context.cpp | 2 ++ .../format_v2/parquet/parquet_file_context.h | 3 +++ be/src/format_v2/parquet/parquet_reader.cpp | 25 +++++++++++++------ .../format_v2/parquet/parquet_statistics.cpp | 10 ++++++-- .../parquet/reader/native_column_reader.cpp | 20 +++++++++------ .../parquet/reader/variant_column_reader.cpp | 1 + .../format_v2/parquet/parquet_schema_test.cpp | 8 ++++++ .../parquet/parquet_statistics_test.cpp | 1 + .../parquet/variant_column_reader_test.cpp | 1 + 11 files changed, 62 insertions(+), 27 deletions(-) diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 623f1ece3dd409..71416e17dc9209 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -178,16 +178,6 @@ void propagate_native_max_levels(ParquetColumnSchema* schema) { } } -bool contains_variant_node(const ParquetColumnSchema& schema) { - if (schema.kind == ParquetColumnSchemaKind::VARIANT) { - return true; - } - return std::ranges::any_of(schema.children, [](const auto& child) { - DORIS_CHECK(child != nullptr); - return contains_variant_node(*child); - }); -} - std::unique_ptr build_native_node_schema(const NativeFieldSchema& field, int32_t local_id) { auto result = std::make_unique(); @@ -222,6 +212,7 @@ std::unique_ptr build_native_node_schema(const NativeFieldS } if (field.variant_physical_type != nullptr) { result->kind = ParquetColumnSchemaKind::VARIANT; + result->contains_variant = true; } else if (primitive_type == TYPE_ARRAY) { result->kind = ParquetColumnSchemaKind::LIST; } else if (primitive_type == TYPE_MAP) { @@ -233,10 +224,11 @@ std::unique_ptr build_native_node_schema(const NativeFieldS for (size_t child_idx = 0; child_idx < field.children.size(); ++child_idx) { result->children.push_back( build_native_node_schema(field.children[child_idx], cast_set(child_idx))); + result->contains_variant |= result->children.back()->contains_variant; } // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild // every enclosing complex type so file-block columns keep the same logical shape as readers. - if (result->kind != ParquetColumnSchemaKind::VARIANT && contains_variant_node(*result)) { + if (result->kind != ParquetColumnSchemaKind::VARIANT && result->contains_variant) { DataTypePtr logical_type; if (result->kind == ParquetColumnSchemaKind::LIST) { DORIS_CHECK(result->children.size() == 1); diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index ad92af269770f5..11b5d2f15a35b1 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -59,6 +59,10 @@ struct ParquetColumnSchema { ParquetColumnSchemaKind kind = ParquetColumnSchemaKind::PRIMITIVE; + // Cached during schema construction so readers created per row group do not repeatedly walk + // ordinary nested schemas to discover whether Variant-specific planning is needed. + bool contains_variant = false; + // ======== Dremel Levels ======== int16_t max_definition_level = 0; diff --git a/be/src/format_v2/parquet/parquet_file_context.cpp b/be/src/format_v2/parquet/parquet_file_context.cpp index c178e491b8ac21..8ba8cf94662f9a 100644 --- a/be/src/format_v2/parquet/parquet_file_context.cpp +++ b/be/src/format_v2/parquet/parquet_file_context.cpp @@ -273,6 +273,7 @@ Status ParquetFileContext::open(io::FileReaderSPtr input_file_reader, io::IOCont bool enable_page_cache, const io::FileDescription& file_description, bool enable_mapping_timestamp_tz, bool enable_mapping_varbinary) { DORIS_CHECK(input_file_reader != nullptr); + contains_variant = false; if (detail::should_stage_small_http_file(input_file_reader->path().native(), input_file_reader->size(), config::in_memory_file_size)) { @@ -602,6 +603,7 @@ Status ParquetFileContext::close() { native_io_ctx = nullptr; native_page_cache_enabled = false; native_page_cache_file_key.clear(); + contains_variant = false; return Status::OK(); } diff --git a/be/src/format_v2/parquet/parquet_file_context.h b/be/src/format_v2/parquet/parquet_file_context.h index 0cd413e10557ac..38e78438e39b29 100644 --- a/be/src/format_v2/parquet/parquet_file_context.h +++ b/be/src/format_v2/parquet/parquet_file_context.h @@ -139,6 +139,9 @@ struct ParquetFileContext { int64_t native_footer_cache_hits = 0; bool native_page_cache_enabled = false; std::string native_page_cache_file_key; + // Set once after the logical file schema is built. Per-request planning uses this guard so + // ordinary files never enter Variant projection or shredded-statistics paths. + bool contains_variant = false; Status open(io::FileReaderSPtr input_file_reader, io::IOContext* io_ctx, bool enable_page_cache, const io::FileDescription& file_description, diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index 47a907d0a638ce..dcb46d9b5ba643 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -165,6 +165,9 @@ size_t finalize_variant_leaf_projections( if (local_id < 0 || local_id >= static_cast(file_schema.size())) { continue; } + if (!file_schema[local_id]->contains_variant) { + continue; + } retained += detail::finalize_variant_leaf_projection(metadata.to_thrift(), *file_schema[local_id], &projection); } @@ -557,6 +560,11 @@ Status ParquetReader::init(RuntimeState* state) { SCOPED_TIMER(_parquet_profile.parse_meta_time); RETURN_IF_ERROR(build_parquet_column_schema(_state->file_context.native_metadata->schema(), &_state->file_schema)); + _state->file_context.contains_variant = + std::ranges::any_of(_state->file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); if (_enable_mapping_timestamp_tz) { for (auto& column_schema : _state->file_schema) { apply_timestamp_tz_mapping(column_schema.get()); @@ -608,13 +616,16 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); - const size_t retained_variant_leaf_projections = - finalize_variant_leaf_projections(*_state->file_context.native_metadata, - _state->file_schema, - &request_snapshot->predicate_columns) + - finalize_variant_leaf_projections(*_state->file_context.native_metadata, - _state->file_schema, - &request_snapshot->non_predicate_columns); + size_t retained_variant_leaf_projections = 0; + if (_state->file_context.contains_variant) { + retained_variant_leaf_projections = + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->predicate_columns) + + finalize_variant_leaf_projections(*_state->file_context.native_metadata, + _state->file_schema, + &request_snapshot->non_predicate_columns); + } if (_parquet_profile.variant_leaf_projections != nullptr) { COUNTER_UPDATE(_parquet_profile.variant_leaf_projections, retained_variant_leaf_projections); diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 77d6a20c7c825a..5ba015373a404f 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -1288,6 +1288,12 @@ Status select_row_groups_by_metadata( if (pruning_stats != nullptr) { pruning_stats->total_row_groups = cast_set(candidate_size); } + const bool contains_variant = + file_context != nullptr ? file_context->contains_variant + : std::ranges::any_of(file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); selected_row_groups->reserve(candidate_size); for (size_t candidate_idx = 0; candidate_idx < candidate_size; ++candidate_idx) { const int row_group_idx = candidate_row_groups == nullptr @@ -1314,8 +1320,8 @@ Status select_row_groups_by_metadata( has_expr_zonemap_filter(request, runtime_state) && (check_native_statistics(metadata, row_group, file_schema, request, pruning_stats, timezone) || - check_shredded_variant_statistics(metadata, row_group, file_schema, request, - timezone))) { + (contains_variant && check_shredded_variant_statistics( + metadata, row_group, file_schema, request, timezone)))) { prune_reason = ParquetRowGroupPruneReason::STATISTICS; } if (probe_mode != ParquetMetadataProbeMode::FOOTER_ONLY && diff --git a/be/src/format_v2/parquet/reader/native_column_reader.cpp b/be/src/format_v2/parquet/reader/native_column_reader.cpp index 7352cc2d58b2bf..68a8905226d432 100644 --- a/be/src/format_v2/parquet/reader/native_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/native_column_reader.cpp @@ -316,8 +316,14 @@ Status NativeColumnReader::create( } auto logical_type = projected_type(column_schema, projection, false); - auto native_type = projected_type(column_schema, projection, true); - auto variant_plan = build_variant_plan(column_schema, projection); + auto native_type = logical_type; + std::unique_ptr variant_plan; + if (column_schema.contains_variant) { + // Native readers are instantiated per projected column and row group. Keep Variant tree + // construction out of ordinary scans instead of charging that setup cost at every split. + native_type = projected_type(column_schema, projection, true); + variant_plan = build_variant_plan(column_schema, projection); + } auto native_reader = std::unique_ptr(new NativeColumnReader( column_schema, std::move(logical_type), native_type, std::move(variant_plan), profile)); // Footer metadata is cached and shared across scans. Keep per-request timestamp semantics on a @@ -394,7 +400,7 @@ Status NativeColumnReader::init( int96_timezone_override)); DORIS_CHECK(_native_reader != nullptr); _skip_column = _native_type->create_column(); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { _variant_physical_column = _native_type->create_column(); } return Status::OK(); @@ -417,7 +423,7 @@ Status NativeColumnReader::read_with_filter(int64_t rows, const uint8_t* filter_ RETURN_IF_ERROR(filter.init(filter_data, static_cast(rows), filter_all)); _native_reader->reset_filter_map_index(); const bool materialize_variant = - !dictionary_ids && _variant_plan->contains_variant && output_type->equals(*_type); + !dictionary_ids && _variant_plan != nullptr && output_type->equals(*_type); if (materialize_variant) { _variant_physical_column->clear(); } @@ -745,7 +751,7 @@ Status NativeColumnReader::select_with_dictionary_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(survivor_count != nullptr); DORIS_CHECK(used_filter != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { row_filter->clear(); *used_filter = false; return Status::OK(); @@ -901,7 +907,7 @@ Status NativeColumnReader::select_with_fixed_width_filter( DORIS_CHECK(row_filter != nullptr); DORIS_CHECK(used_filter != nullptr); DORIS_CHECK(execution_kind != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { // Direct fixed-width evaluation cannot preserve a Variant physical subtree's row shape. row_filter->clear(); *used_filter = false; @@ -1038,7 +1044,7 @@ bool NativeColumnReader::crossed_page_since_last_batch() { Result NativeColumnReader::dictionary_values() { DORIS_CHECK(_native_reader != nullptr); - if (_variant_plan->contains_variant) { + if (_variant_plan != nullptr) { return ResultError( Status::NotSupported("Parquet Variant columns do not expose dictionary values")); } diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index b51358a4f0649c..da5ecbb8310a20 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -470,6 +470,7 @@ std::unique_ptr clone_schema( result->leaf_column_id = source.leaf_column_id; result->type_descriptor = source.type_descriptor; result->kind = source.kind; + result->contains_variant = source.contains_variant; result->max_definition_level = source.max_definition_level; result->max_repetition_level = source.max_repetition_level; result->nullable_definition_level = source.nullable_definition_level; diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index 9ecc1edf6fc582..acd5e8600789ec 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -217,6 +217,7 @@ TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { ASSERT_TRUE(status.ok()) << status; ASSERT_EQ(fields.size(), 1); EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_TRUE(fields[0]->contains_variant); EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); EXPECT_NE(typeid_cast(remove_nullable(fields[0]->type).get()), nullptr); @@ -247,6 +248,10 @@ TEST(ParquetSchemaTest, NestedVariantPropagatesIntoParentLogicalType) { const auto status = build_parquet_column_schema(descriptor, &fields); ASSERT_TRUE(status.ok()) << status; ASSERT_EQ(fields.size(), 1); + EXPECT_TRUE(fields[0]->contains_variant); + ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_TRUE(fields[0]->children[1]->contains_variant); const auto* info_type = assert_cast(remove_nullable(fields[0]->type).get()); ASSERT_EQ(info_type->get_elements().size(), 2); @@ -481,9 +486,12 @@ TEST(ParquetSchemaTest, NativeMetadataTreePreservesNestedFieldNamesAndIds) { std::vector> fields; ASSERT_TRUE(build_parquet_column_schema(native_schema, &fields).ok()); ASSERT_EQ(fields.size(), 1); + EXPECT_FALSE(fields[0]->contains_variant); EXPECT_EQ(fields[0]->name, "protocol"); EXPECT_EQ(fields[0]->parquet_field_id, 10); ASSERT_EQ(fields[0]->children.size(), 2); + EXPECT_FALSE(fields[0]->children[0]->contains_variant); + EXPECT_FALSE(fields[0]->children[1]->contains_variant); EXPECT_EQ(fields[0]->children[0]->name, "minReaderVersion"); EXPECT_EQ(fields[0]->children[0]->leaf_column_id, 0); EXPECT_EQ(fields[0]->children[1]->name, "minWriterVersion"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 4c3fdcafa06ffe..533b526d613d9f 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -924,6 +924,7 @@ TEST(NativeParquetStatisticsTest, ShreddedVariantTypedValueDrivesPageFiltering) variant->name = "v"; variant->local_id = 0; variant->kind = format::parquet::ParquetColumnSchemaKind::VARIANT; + variant->contains_variant = true; variant->type = make_nullable(std::make_shared()); variant->children.push_back(bytes("metadata", 0, 0)); variant->children.push_back(bytes("value", 1, 1)); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 24ace262faba37..51a50319285c5b 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -70,6 +70,7 @@ ParquetColumnSchema unshredded_schema() { ParquetColumnSchema schema; schema.name = "payload"; schema.kind = ParquetColumnSchemaKind::VARIANT; + schema.contains_variant = true; schema.type = make_nullable(std::make_shared()); const auto binary = make_nullable(std::make_shared()); schema.variant_physical_type = make_nullable(std::make_shared( From b79bb0480dc30ba77489be28cc32b920e53212dc Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 20:32:30 +0800 Subject: [PATCH 08/20] [fix](variant) Preserve projected shredded states across exchange --- .../column/variant_v2/column_variant_v2.cpp | 293 +++++++++++++++++- .../column/variant_v2/column_variant_v2.h | 5 + .../function/function_variant_element_v2.cpp | 7 +- .../parquet/reader/variant_column_reader.cpp | 41 ++- .../parquet/variant_column_reader_test.cpp | 141 +++++++++ 5 files changed, 480 insertions(+), 7 deletions(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 0d5a7fb74bb6df..e0ba9bd048cb97 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -22,9 +22,11 @@ #include #include #include +#include #include #include #include +#include #include "common/check.h" #include "common/exception.h" @@ -313,6 +315,266 @@ ValidatedTypedInput validate_typed_input(ColumnPtr column, DataTypePtr scalar_ty "ColumnVariantV2::{} is intentionally unsupported for Variant values", method); } +class CompositeVariantShreddedState final : public VariantShreddedState { +public: + explicit CompositeVariantShreddedState( + std::vector> segments) + : _segments(std::move(segments)) { + DORIS_CHECK(std::ranges::all_of(_segments, [](const auto& segment) { + return segment != nullptr; + })) << "composite Variant shredded segments must not be null"; + } + + size_t size() const override { + size_t rows = 0; + for (const auto& segment : _segments) { + DORIS_CHECK_LE(segment->size(), std::numeric_limits::max() - rows) + << "composite Variant shredded row count overflows size_t"; + rows += segment->size(); + } + return rows; + } + + size_t byte_size() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->byte_size(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->byte_size() : 0); + } + + size_t allocated_bytes() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->allocated_bytes(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->allocated_bytes() : 0); + } + + void sanity_check() const override { + for (const auto& segment : _segments) { + segment->sanity_check(); + } + } + + void for_each_subcolumn(const IColumn::ImutableColumnCallback& callback) const override { + for (const auto& segment : _segments) { + segment->for_each_subcolumn(callback); + } + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t /*result_size_hint*/) const override { + DORIS_CHECK_EQ(filter.size(), size()) + << "composite Variant shredded filter size does not match row count"; + std::vector> selected; + selected.reserve(_segments.size()); + size_t offset = 0; + for (const auto& segment : _segments) { + IColumn::Filter segment_filter; + segment_filter.insert(filter.begin() + offset, + filter.begin() + offset + segment->size()); + auto filtered = segment->filter(segment_filter, -1); + if (filtered->size() != 0) { + selected.push_back(std::move(filtered)); + } + offset += segment->size(); + } + return pack(std::move(selected)); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + DORIS_CHECK_LE(start, size()) << "composite Variant range starts past source size"; + DORIS_CHECK_LE(length, size() - start) << "composite Variant range exceeds source size"; + std::vector> selected; + if (length == 0) { + return pack(std::move(selected)); + } + const size_t end = start + length; + size_t offset = 0; + for (const auto& segment : _segments) { + const size_t segment_end = offset + segment->size(); + const size_t overlap_begin = std::max(start, offset); + const size_t overlap_end = std::min(end, segment_end); + if (overlap_begin < overlap_end) { + selected.push_back( + segment->select_range(overlap_begin - offset, overlap_end - overlap_begin)); + } + offset = segment_end; + if (offset >= end) { + break; + } + } + return pack(std::move(selected)); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + if (indices_begin == indices_end) { + return pack({}); + } + DORIS_CHECK(indices_begin != nullptr && indices_end != nullptr && + indices_begin < indices_end) + << "composite Variant indices are invalid"; + + std::vector segment_ends; + segment_ends.reserve(_segments.size()); + size_t rows = 0; + for (const auto& segment : _segments) { + rows += segment->size(); + segment_ends.push_back(rows); + } + + std::vector> selected; + const uint32_t* cursor = indices_begin; + while (cursor != indices_end) { + DORIS_CHECK_LT(*cursor, rows) << "composite Variant source index is out of range"; + const size_t segment_index = + std::upper_bound(segment_ends.begin(), segment_ends.end(), *cursor) - + segment_ends.begin(); + const size_t segment_begin = segment_index == 0 ? 0 : segment_ends[segment_index - 1]; + DorisVector local_indices; + while (cursor != indices_end && *cursor >= segment_begin && + *cursor < segment_ends[segment_index]) { + local_indices.push_back(static_cast(*cursor - segment_begin)); + ++cursor; + } + selected.push_back(_segments[segment_index]->select_indices( + local_indices.data(), local_indices.data() + local_indices.size())); + } + return pack(std::move(selected)); + } + + bool can_materialize() const override { + return std::ranges::all_of(_segments, + [](const auto& segment) { return segment->can_materialize(); }); + } + + bool try_append(const VariantShreddedState& source) override { + if (const auto* composite = dynamic_cast(&source)) { + for (const auto& segment : composite->_segments) { + append(segment); + } + } else { + append(source.select_range(0, source.size())); + } + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + return true; + } + + std::optional find_typed_value( + std::span path) const override { + if (_segments.empty()) { + return std::nullopt; + } + std::vector matches; + matches.reserve(_segments.size()); + for (const auto& segment : _segments) { + auto match = segment->find_typed_value(path); + if (!match.has_value()) { + return std::nullopt; + } + matches.push_back(std::move(*match)); + } + + const bool homogeneous = matches.front().column && matches.front().type && + std::ranges::all_of(matches, [&](const auto& match) { + return match.column && match.type && !match.normalized && + exact_typed_identity(matches.front().type, match.type); + }); + if (homogeneous) { + MutableColumnPtr combined = matches.front().column->clone_empty(); + for (const auto& match : matches) { + combined->insert_range_from(*match.column, 0, match.column->size()); + } + return VariantShreddedTypedValue {.column = std::move(combined), + .type = matches.front().type, + .normalized = nullptr}; + } + + auto values = ColumnVariantV2::create(); + auto nulls = ColumnUInt8::create(); + nulls->reserve(size()); + for (const auto& match : matches) { + if (match.normalized) { + const auto& nullable = assert_cast(*match.normalized); + const auto& variants = + assert_cast(nullable.get_nested_column()); + values->insert_range_from(variants, 0, variants.size()); + nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); + continue; + } + auto typed = ColumnVariantV2::create_typed(match.column, match.type); + values->insert_range_from(*typed, 0, typed->size()); + const auto& nullable = assert_cast(*match.column); + nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); + } + return VariantShreddedTypedValue { + .column = nullptr, + .type = nullptr, + .normalized = ColumnNullable::create(std::move(values), std::move(nulls))}; + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_materialized) { + auto materialized = ColumnVariantV2::create(); + for (const auto& segment : _segments) { + const ColumnVariantV2& source = segment->materialized_column(); + materialized->insert_range_from(source, 0, source.size()); + } + _materialized = std::move(materialized); + } + return *_materialized; + } + +private: + static std::shared_ptr pack( + std::vector> segments) { + if (segments.size() == 1) { + return std::move(segments.front()); + } + return std::make_shared(std::move(segments)); + } + + void append(std::shared_ptr source) { + if (source->size() == 0) { + return; + } + if (const auto* composite = + dynamic_cast(source.get())) { + _segments.insert(_segments.end(), composite->_segments.begin(), + composite->_segments.end()); + return; + } + if (!_segments.empty()) { + auto& tail = _segments.back(); + if (tail.use_count() != 1) { + tail = tail->select_range(0, tail->size()); + } + if (tail->try_append(*source)) { + return; + } + } + _segments.push_back(std::move(source)); + } + + std::vector> _segments; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; +}; + +std::shared_ptr combine_shredded_states( + std::shared_ptr left, std::shared_ptr right) { + auto combined = std::make_shared( + std::vector> {std::move(left)}); + combined->try_append(*right); + return combined; +} + } // namespace #ifdef BE_TEST @@ -818,6 +1080,14 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (!_shredded->can_materialize() && !selected_source->can_materialize()) { + // Different files may shred the same projected path with different physical + // identities. Keep both incomplete states ordered because neither can reconstruct the + // root value. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } } if (_shredded) { ensure_encoded(); @@ -911,9 +1181,6 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } - if (_shredded) { - ensure_encoded(); - } if (!_typed && empty() && _metadatas->empty() && source._shredded) { // Gather into the native shredded representation for the same reason as range selection: // row selection does not require, and may not have, a complete logical Variant value. @@ -921,6 +1188,26 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (_shredded && source._shredded) { + auto selected_source = source._shredded->select_indices(indices_begin, indices_end); + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + if (!_shredded->can_materialize() && !selected_source->can_materialize()) { + // Exchange channels can gather projected rows from files whose shredded leaf types + // differ. Preserve the segments instead of encoding an incomplete logical Variant. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } + } + if (_shredded) { + ensure_encoded(); + } if (source._shredded) { insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); return; diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index abe4df4b8223d4..e5be6e67e88b44 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -53,6 +53,9 @@ struct VariantShreddedTypedValue { // retain the decoded leaf without copying it or depending on scanner lifetime. ColumnPtr column; DataTypePtr type; + // Physical identities such as binary annotations cannot use the typed scalar state. In that + // case the format reader may return an exact Nullable leaf instead. + ColumnPtr normalized; }; // Format readers keep their native shredded representation behind this interface. Core Variant @@ -76,6 +79,8 @@ class VariantShreddedState { size_t length) const = 0; virtual std::shared_ptr select_indices( const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // False means the state contains only projected leaves and cannot reconstruct root values. + virtual bool can_materialize() const = 0; // Appends another state only when both format-owned physical layouts have identical semantics. // An incompatible source must leave this state unchanged and return false. virtual bool try_append(const VariantShreddedState& source) = 0; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 90863fe86c5c72..2c506c05377e65 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -152,7 +152,8 @@ std::optional extract_shredded_typed_variant_element( if (!match.has_value()) { return std::nullopt; } - const auto& leaf = assert_cast(*match->column); + const ColumnPtr& matched_column = match->normalized ? match->normalized : match->column; + const auto& leaf = assert_cast(*matched_column); auto nulls = leaf.get_null_map_column().clone_resized(source.size()); auto& null_data = assert_cast(*nulls).get_data(); for (size_t row = 0; row < source.size(); ++row) { @@ -160,6 +161,10 @@ std::optional extract_shredded_typed_variant_element( static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); } + if (match->normalized) { + return ColumnNullable::create(leaf.get_nested_column_ptr(), std::move(nulls)); + } + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null // map is produced here, so predicates and casts can consume the leaf without reconstructing // canonical Variant rows. diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index da5ecbb8310a20..09ec88841e459e 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -553,6 +553,25 @@ bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { } } +ColumnPtr normalize_projected_primitive_leaf(const ParquetColumnSchema& schema, + const ColumnPtr& typed) { + const auto& nullable = assert_cast(*typed); + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = nullable.size()}); + for (size_t row = 0; row < nullable.size(); ++row) { + auto output_row = builder.begin_row(); + if (nullable.get_null_map_data()[row] != 0) { + output_row.add_null(); + } else { + append_typed_scalar(schema, nullable.get_nested_column(), row, output_row); + } + output_row.finish(); + } + auto values = ColumnVariantV2::create(); + values->insert_encoded_batch(builder.finish_batch()); + auto nulls = nullable.get_null_map_column().clone_resized(nullable.size()); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { return (!left && !right) || (left && right && left->equals(*right)); } @@ -648,6 +667,8 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _complete, _profile); } + bool can_materialize() const override { return _complete; } + bool try_append(const VariantShreddedState& source) override { const auto* parquet_source = dynamic_cast(&source); if (parquet_source == nullptr || _complete != parquet_source->_complete || @@ -705,15 +726,29 @@ class ParquetVariantShreddedState final : public VariantShreddedState { } if (position + 1 == path.size()) { if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || - check_and_get_column(*typed) == nullptr || - !supports_direct_typed_variant_state(*typed_schema)) { + check_and_get_column(*typed) == nullptr) { update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); return std::nullopt; } + if (!supports_direct_typed_variant_state(*typed_schema)) { + if (_complete) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + // A partial projection cannot reconstruct its root. Normalize only the exact + // requested leaf so Parquet annotations survive heterogeneous file schemas. + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue { + .column = nullptr, + .type = nullptr, + .normalized = normalize_projected_primitive_leaf(*typed_schema, typed)}; + } update_counter(_profile.variant_direct_leaf_rows, static_cast(typed->size())); return VariantShreddedTypedValue {.column = std::move(typed), - .type = remove_nullable(typed_schema->type)}; + .type = remove_nullable(typed_schema->type), + .normalized = nullptr}; } if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { return path_miss(); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 51a50319285c5b..8dd080f6387253 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -237,6 +237,27 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } +MutableColumnPtr projected_shredded_binary_object_physical( + const std::vector& values) { + std::vector refs; + refs.reserve(values.size()); + for (const auto value : values) { + refs.emplace_back(value.data(), value.size()); + } + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings(refs, std::vector(values.size(), 0))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); MutableColumnPtr nullable_int64(const std::vector& values, const std::vector& nulls); @@ -840,6 +861,126 @@ TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializin EXPECT_EQ(plan.variant_state_schema.use_count(), 3); } +TEST(VariantColumnReaderTest, GathersConsecutiveProjectedShreddedBatches) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto first = make_nullable(std::make_shared())->create_column(); + auto second = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), first) + .ok()); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({30}), second) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array first_indices {1, 0}; + const std::array second_indices {0}; + gathered->insert_indices_from(*first, first_indices.begin(), first_indices.end()); + gathered->insert_indices_from(*second, second_indices.begin(), second_indices.end()); + + const auto& variants = assert_cast( + assert_cast(*gathered).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({20, 10, 30})); +} + +TEST(VariantColumnReaderTest, GathersProjectedShreddedBatchesWithDifferentLeafTypes) { + auto integer_schema = shredded_object_schema(); + auto string_schema = shredded_binary_object_schema(); + for (auto* schema : {&integer_schema, &string_schema}) { + schema->local_id = 0; + schema->children[2]->local_id = 2; + schema->children[2]->children[0]->local_id = 0; + schema->children[2]->children[0]->children[0]->local_id = 0; + } + auto make_plan = [](const ParquetColumnSchema& schema) { + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + return plan; + }; + auto integer_plan = make_plan(integer_schema); + auto string_plan = make_plan(string_schema); + auto integers = make_nullable(std::make_shared())->create_column(); + auto strings = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(integer_plan, projected_shredded_object_physical({7}), + integers) + .ok()); + ASSERT_TRUE(materialize_variant_columns( + string_plan, projected_shredded_binary_object_physical({"seven"}), strings) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array selected {0}; + gathered->insert_indices_from(*integers, selected.begin(), selected.end()); + gathered->insert_indices_from(*strings, selected.begin(), selected.end()); + + const auto& nullable = assert_cast(*gathered); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + const auto& extracted_variants = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(extracted_variants.get_value_ref(0).get_int(), 7); + EXPECT_EQ(extracted_variants.get_value_ref(1).get_binary(), StringRef("seven")); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + auto reordered = ColumnVariantV2::create(); + const std::array reversed {1, 0}; + reordered->insert_indices_from(variants, reversed.begin(), reversed.end()); + const auto reordered_match = reordered->find_shredded_typed_value(shredded_path); + ASSERT_TRUE(reordered_match.has_value()); + ASSERT_TRUE(reordered_match->normalized); + const auto& reordered_values = assert_cast( + assert_cast(*reordered_match->normalized).get_nested_column()); + EXPECT_EQ(reordered_values.get_value_ref(0).get_binary(), StringRef("seven")); + EXPECT_EQ(reordered_values.get_value_ref(1).get_int(), 7); + + IColumn::Filter keep_integer {1, 0}; + const auto filtered = variants.filter(keep_integer, 1); + const auto filtered_match = + assert_cast(*filtered).find_shredded_typed_value(shredded_path); + ASSERT_TRUE(filtered_match.has_value()); + ASSERT_TRUE(filtered_match->column); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + 7); +} + TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { constexpr size_t width = 64; constexpr size_t batch_count = 16; From b2e1ff2b9fe4e0cd55d4cd43004dff6b5e057fe3 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Tue, 4 Aug 2026 22:09:34 +0800 Subject: [PATCH 09/20] [fix](variant) Adapt shredded callback to master MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What problem does this PR solve? Issue Number: None Related PR: #66446 Problem Summary: Master exposes immutable subcolumn traversal through IColumn::ColumnCallback, while branch-4.1 still uses ImutableColumnCallback. Adapt the forward-port implementation to master’s callback contract so the composite shredded state overrides the interface and compiles. ### Release note None ### Check List (For Author) - Test: Unit Test - Behavior changed: No - Does this need documentation: No --- be/src/core/column/variant_v2/column_variant_v2.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index e0ba9bd048cb97..cb16694dd7f7cc 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -359,7 +359,8 @@ class CompositeVariantShreddedState final : public VariantShreddedState { } } - void for_each_subcolumn(const IColumn::ImutableColumnCallback& callback) const override { + // Preserve master's immutable traversal contract while forwarding into each segment. + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { for (const auto& segment : _segments) { segment->for_each_subcolumn(callback); } From e29e1dadd3e63e4661fbcd012cbf9768d7521b44 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Wed, 5 Aug 2026 20:40:48 +0800 Subject: [PATCH 10/20] [fix](variant) Revert projected shredded state preservation ### What problem does this PR solve? Issue Number: None Related PR: #66446 Problem Summary: Remove the forward port of projected shredded Variant state preservation and its master-only callback adaptation so the branch returns exactly to the state before that pull request. ### Release note None ### Check List (For Author) - Test: No need to test (the resulting tree exactly matches the pre-#66446 state) - Behavior changed: Yes (removes the #66446 forward port) - Does this need documentation: No --- .../column/variant_v2/column_variant_v2.cpp | 294 +----------------- .../column/variant_v2/column_variant_v2.h | 5 - .../function/function_variant_element_v2.cpp | 7 +- .../parquet/reader/variant_column_reader.cpp | 41 +-- .../parquet/variant_column_reader_test.cpp | 141 --------- 5 files changed, 7 insertions(+), 481 deletions(-) diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index cb16694dd7f7cc..0d5a7fb74bb6df 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -22,11 +22,9 @@ #include #include #include -#include #include #include #include -#include #include "common/check.h" #include "common/exception.h" @@ -315,267 +313,6 @@ ValidatedTypedInput validate_typed_input(ColumnPtr column, DataTypePtr scalar_ty "ColumnVariantV2::{} is intentionally unsupported for Variant values", method); } -class CompositeVariantShreddedState final : public VariantShreddedState { -public: - explicit CompositeVariantShreddedState( - std::vector> segments) - : _segments(std::move(segments)) { - DORIS_CHECK(std::ranges::all_of(_segments, [](const auto& segment) { - return segment != nullptr; - })) << "composite Variant shredded segments must not be null"; - } - - size_t size() const override { - size_t rows = 0; - for (const auto& segment : _segments) { - DORIS_CHECK_LE(segment->size(), std::numeric_limits::max() - rows) - << "composite Variant shredded row count overflows size_t"; - rows += segment->size(); - } - return rows; - } - - size_t byte_size() const override { - size_t bytes = 0; - for (const auto& segment : _segments) { - bytes += segment->byte_size(); - } - std::lock_guard lock(_materialization_lock); - return bytes + (_materialized ? _materialized->byte_size() : 0); - } - - size_t allocated_bytes() const override { - size_t bytes = 0; - for (const auto& segment : _segments) { - bytes += segment->allocated_bytes(); - } - std::lock_guard lock(_materialization_lock); - return bytes + (_materialized ? _materialized->allocated_bytes() : 0); - } - - void sanity_check() const override { - for (const auto& segment : _segments) { - segment->sanity_check(); - } - } - - // Preserve master's immutable traversal contract while forwarding into each segment. - void for_each_subcolumn(IColumn::ColumnCallback callback) const override { - for (const auto& segment : _segments) { - segment->for_each_subcolumn(callback); - } - } - - std::shared_ptr filter(const IColumn::Filter& filter, - ssize_t /*result_size_hint*/) const override { - DORIS_CHECK_EQ(filter.size(), size()) - << "composite Variant shredded filter size does not match row count"; - std::vector> selected; - selected.reserve(_segments.size()); - size_t offset = 0; - for (const auto& segment : _segments) { - IColumn::Filter segment_filter; - segment_filter.insert(filter.begin() + offset, - filter.begin() + offset + segment->size()); - auto filtered = segment->filter(segment_filter, -1); - if (filtered->size() != 0) { - selected.push_back(std::move(filtered)); - } - offset += segment->size(); - } - return pack(std::move(selected)); - } - - std::shared_ptr select_range(size_t start, size_t length) const override { - DORIS_CHECK_LE(start, size()) << "composite Variant range starts past source size"; - DORIS_CHECK_LE(length, size() - start) << "composite Variant range exceeds source size"; - std::vector> selected; - if (length == 0) { - return pack(std::move(selected)); - } - const size_t end = start + length; - size_t offset = 0; - for (const auto& segment : _segments) { - const size_t segment_end = offset + segment->size(); - const size_t overlap_begin = std::max(start, offset); - const size_t overlap_end = std::min(end, segment_end); - if (overlap_begin < overlap_end) { - selected.push_back( - segment->select_range(overlap_begin - offset, overlap_end - overlap_begin)); - } - offset = segment_end; - if (offset >= end) { - break; - } - } - return pack(std::move(selected)); - } - - std::shared_ptr select_indices( - const uint32_t* indices_begin, const uint32_t* indices_end) const override { - if (indices_begin == indices_end) { - return pack({}); - } - DORIS_CHECK(indices_begin != nullptr && indices_end != nullptr && - indices_begin < indices_end) - << "composite Variant indices are invalid"; - - std::vector segment_ends; - segment_ends.reserve(_segments.size()); - size_t rows = 0; - for (const auto& segment : _segments) { - rows += segment->size(); - segment_ends.push_back(rows); - } - - std::vector> selected; - const uint32_t* cursor = indices_begin; - while (cursor != indices_end) { - DORIS_CHECK_LT(*cursor, rows) << "composite Variant source index is out of range"; - const size_t segment_index = - std::upper_bound(segment_ends.begin(), segment_ends.end(), *cursor) - - segment_ends.begin(); - const size_t segment_begin = segment_index == 0 ? 0 : segment_ends[segment_index - 1]; - DorisVector local_indices; - while (cursor != indices_end && *cursor >= segment_begin && - *cursor < segment_ends[segment_index]) { - local_indices.push_back(static_cast(*cursor - segment_begin)); - ++cursor; - } - selected.push_back(_segments[segment_index]->select_indices( - local_indices.data(), local_indices.data() + local_indices.size())); - } - return pack(std::move(selected)); - } - - bool can_materialize() const override { - return std::ranges::all_of(_segments, - [](const auto& segment) { return segment->can_materialize(); }); - } - - bool try_append(const VariantShreddedState& source) override { - if (const auto* composite = dynamic_cast(&source)) { - for (const auto& segment : composite->_segments) { - append(segment); - } - } else { - append(source.select_range(0, source.size())); - } - std::lock_guard lock(_materialization_lock); - _materialized.reset(); - return true; - } - - std::optional find_typed_value( - std::span path) const override { - if (_segments.empty()) { - return std::nullopt; - } - std::vector matches; - matches.reserve(_segments.size()); - for (const auto& segment : _segments) { - auto match = segment->find_typed_value(path); - if (!match.has_value()) { - return std::nullopt; - } - matches.push_back(std::move(*match)); - } - - const bool homogeneous = matches.front().column && matches.front().type && - std::ranges::all_of(matches, [&](const auto& match) { - return match.column && match.type && !match.normalized && - exact_typed_identity(matches.front().type, match.type); - }); - if (homogeneous) { - MutableColumnPtr combined = matches.front().column->clone_empty(); - for (const auto& match : matches) { - combined->insert_range_from(*match.column, 0, match.column->size()); - } - return VariantShreddedTypedValue {.column = std::move(combined), - .type = matches.front().type, - .normalized = nullptr}; - } - - auto values = ColumnVariantV2::create(); - auto nulls = ColumnUInt8::create(); - nulls->reserve(size()); - for (const auto& match : matches) { - if (match.normalized) { - const auto& nullable = assert_cast(*match.normalized); - const auto& variants = - assert_cast(nullable.get_nested_column()); - values->insert_range_from(variants, 0, variants.size()); - nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); - continue; - } - auto typed = ColumnVariantV2::create_typed(match.column, match.type); - values->insert_range_from(*typed, 0, typed->size()); - const auto& nullable = assert_cast(*match.column); - nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); - } - return VariantShreddedTypedValue { - .column = nullptr, - .type = nullptr, - .normalized = ColumnNullable::create(std::move(values), std::move(nulls))}; - } - - const ColumnVariantV2& materialized_column() const override { - std::lock_guard lock(_materialization_lock); - if (!_materialized) { - auto materialized = ColumnVariantV2::create(); - for (const auto& segment : _segments) { - const ColumnVariantV2& source = segment->materialized_column(); - materialized->insert_range_from(source, 0, source.size()); - } - _materialized = std::move(materialized); - } - return *_materialized; - } - -private: - static std::shared_ptr pack( - std::vector> segments) { - if (segments.size() == 1) { - return std::move(segments.front()); - } - return std::make_shared(std::move(segments)); - } - - void append(std::shared_ptr source) { - if (source->size() == 0) { - return; - } - if (const auto* composite = - dynamic_cast(source.get())) { - _segments.insert(_segments.end(), composite->_segments.begin(), - composite->_segments.end()); - return; - } - if (!_segments.empty()) { - auto& tail = _segments.back(); - if (tail.use_count() != 1) { - tail = tail->select_range(0, tail->size()); - } - if (tail->try_append(*source)) { - return; - } - } - _segments.push_back(std::move(source)); - } - - std::vector> _segments; - mutable std::mutex _materialization_lock; - mutable ColumnVariantV2::MutablePtr _materialized; -}; - -std::shared_ptr combine_shredded_states( - std::shared_ptr left, std::shared_ptr right) { - auto combined = std::make_shared( - std::vector> {std::move(left)}); - combined->try_append(*right); - return combined; -} - } // namespace #ifdef BE_TEST @@ -1081,14 +818,6 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) _check_invariants(); return; } - if (!_shredded->can_materialize() && !selected_source->can_materialize()) { - // Different files may shred the same projected path with different physical - // identities. Keep both incomplete states ordered because neither can reconstruct the - // root value. - _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); - _check_invariants(); - return; - } } if (_shredded) { ensure_encoded(); @@ -1182,6 +911,9 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } + if (_shredded) { + ensure_encoded(); + } if (!_typed && empty() && _metadatas->empty() && source._shredded) { // Gather into the native shredded representation for the same reason as range selection: // row selection does not require, and may not have, a complete logical Variant value. @@ -1189,26 +921,6 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) _check_invariants(); return; } - if (_shredded && source._shredded) { - auto selected_source = source._shredded->select_indices(indices_begin, indices_end); - if (_shredded.use_count() != 1) { - _shredded = _shredded->select_range(0, size()); - } - if (_shredded->try_append(*selected_source)) { - _check_invariants(); - return; - } - if (!_shredded->can_materialize() && !selected_source->can_materialize()) { - // Exchange channels can gather projected rows from files whose shredded leaf types - // differ. Preserve the segments instead of encoding an incomplete logical Variant. - _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); - _check_invariants(); - return; - } - } - if (_shredded) { - ensure_encoded(); - } if (source._shredded) { insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); return; diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index e5be6e67e88b44..abe4df4b8223d4 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -53,9 +53,6 @@ struct VariantShreddedTypedValue { // retain the decoded leaf without copying it or depending on scanner lifetime. ColumnPtr column; DataTypePtr type; - // Physical identities such as binary annotations cannot use the typed scalar state. In that - // case the format reader may return an exact Nullable leaf instead. - ColumnPtr normalized; }; // Format readers keep their native shredded representation behind this interface. Core Variant @@ -79,8 +76,6 @@ class VariantShreddedState { size_t length) const = 0; virtual std::shared_ptr select_indices( const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; - // False means the state contains only projected leaves and cannot reconstruct root values. - virtual bool can_materialize() const = 0; // Appends another state only when both format-owned physical layouts have identical semantics. // An incompatible source must leave this state unchanged and return false. virtual bool try_append(const VariantShreddedState& source) = 0; diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 2c506c05377e65..90863fe86c5c72 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -152,8 +152,7 @@ std::optional extract_shredded_typed_variant_element( if (!match.has_value()) { return std::nullopt; } - const ColumnPtr& matched_column = match->normalized ? match->normalized : match->column; - const auto& leaf = assert_cast(*matched_column); + const auto& leaf = assert_cast(*match->column); auto nulls = leaf.get_null_map_column().clone_resized(source.size()); auto& null_data = assert_cast(*nulls).get_data(); for (size_t row = 0; row < source.size(); ++row) { @@ -161,10 +160,6 @@ std::optional extract_shredded_typed_variant_element( static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); } - if (match->normalized) { - return ColumnNullable::create(leaf.get_nested_column_ptr(), std::move(nulls)); - } - // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null // map is produced here, so predicates and casts can consume the leaf without reconstructing // canonical Variant rows. diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index 09ec88841e459e..da5ecbb8310a20 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -553,25 +553,6 @@ bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { } } -ColumnPtr normalize_projected_primitive_leaf(const ParquetColumnSchema& schema, - const ColumnPtr& typed) { - const auto& nullable = assert_cast(*typed); - VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = nullable.size()}); - for (size_t row = 0; row < nullable.size(); ++row) { - auto output_row = builder.begin_row(); - if (nullable.get_null_map_data()[row] != 0) { - output_row.add_null(); - } else { - append_typed_scalar(schema, nullable.get_nested_column(), row, output_row); - } - output_row.finish(); - } - auto values = ColumnVariantV2::create(); - values->insert_encoded_batch(builder.finish_batch()); - auto nulls = nullable.get_null_map_column().clone_resized(nullable.size()); - return ColumnNullable::create(std::move(values), std::move(nulls)); -} - bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { return (!left && !right) || (left && right && left->equals(*right)); } @@ -667,8 +648,6 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _complete, _profile); } - bool can_materialize() const override { return _complete; } - bool try_append(const VariantShreddedState& source) override { const auto* parquet_source = dynamic_cast(&source); if (parquet_source == nullptr || _complete != parquet_source->_complete || @@ -726,29 +705,15 @@ class ParquetVariantShreddedState final : public VariantShreddedState { } if (position + 1 == path.size()) { if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || - check_and_get_column(*typed) == nullptr) { + check_and_get_column(*typed) == nullptr || + !supports_direct_typed_variant_state(*typed_schema)) { update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); return std::nullopt; } - if (!supports_direct_typed_variant_state(*typed_schema)) { - if (_complete) { - update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); - return std::nullopt; - } - // A partial projection cannot reconstruct its root. Normalize only the exact - // requested leaf so Parquet annotations survive heterogeneous file schemas. - update_counter(_profile.variant_direct_leaf_rows, - static_cast(typed->size())); - return VariantShreddedTypedValue { - .column = nullptr, - .type = nullptr, - .normalized = normalize_projected_primitive_leaf(*typed_schema, typed)}; - } update_counter(_profile.variant_direct_leaf_rows, static_cast(typed->size())); return VariantShreddedTypedValue {.column = std::move(typed), - .type = remove_nullable(typed_schema->type), - .normalized = nullptr}; + .type = remove_nullable(typed_schema->type)}; } if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { return path_miss(); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 8dd080f6387253..51a50319285c5b 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -237,27 +237,6 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } -MutableColumnPtr projected_shredded_binary_object_physical( - const std::vector& values) { - std::vector refs; - refs.reserve(values.size()); - for (const auto value : values) { - refs.emplace_back(value.data(), value.size()); - } - MutableColumns wrapper_fields; - wrapper_fields.push_back(nullable_strings(refs, std::vector(values.size(), 0))); - auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); - MutableColumns object_fields; - object_fields.push_back( - ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); - auto object = ColumnStruct::create(std::move(object_fields)); - MutableColumns root_fields; - root_fields.push_back( - ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); - auto root = ColumnStruct::create(std::move(root_fields)); - return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); -} - MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); MutableColumnPtr nullable_int64(const std::vector& values, const std::vector& nulls); @@ -861,126 +840,6 @@ TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializin EXPECT_EQ(plan.variant_state_schema.use_count(), 3); } -TEST(VariantColumnReaderTest, GathersConsecutiveProjectedShreddedBatches) { - auto schema = shredded_object_schema(); - schema.local_id = 0; - schema.children[2]->local_id = 2; - schema.children[2]->children[0]->local_id = 0; - schema.children[2]->children[0]->children[0]->local_id = 0; - auto projection = format::LocalColumnIndex::partial_local(0); - projection.children.push_back(format::LocalColumnIndex::partial_local(2)); - projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); - projection.children.back().children.back().children.push_back( - format::LocalColumnIndex::local(0)); - VariantMaterializationNode plan; - plan.schema = &schema; - plan.contains_variant = true; - plan.variant_projection = std::move(projection); - plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); - - auto first = make_nullable(std::make_shared())->create_column(); - auto second = make_nullable(std::make_shared())->create_column(); - ASSERT_TRUE( - materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), first) - .ok()); - ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({30}), second) - .ok()); - - auto gathered = make_nullable(std::make_shared())->create_column(); - const std::array first_indices {1, 0}; - const std::array second_indices {0}; - gathered->insert_indices_from(*first, first_indices.begin(), first_indices.end()); - gathered->insert_indices_from(*second, second_indices.begin(), second_indices.end()); - - const auto& variants = assert_cast( - assert_cast(*gathered).get_nested_column()); - const std::array path {VariantShreddedPathSegment { - .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; - const auto match = variants.find_shredded_typed_value(path); - ASSERT_TRUE(match.has_value()); - const auto& values = assert_cast( - assert_cast(*match->column).get_nested_column()); - EXPECT_EQ(values.get_data(), ColumnInt64::Container({20, 10, 30})); -} - -TEST(VariantColumnReaderTest, GathersProjectedShreddedBatchesWithDifferentLeafTypes) { - auto integer_schema = shredded_object_schema(); - auto string_schema = shredded_binary_object_schema(); - for (auto* schema : {&integer_schema, &string_schema}) { - schema->local_id = 0; - schema->children[2]->local_id = 2; - schema->children[2]->children[0]->local_id = 0; - schema->children[2]->children[0]->children[0]->local_id = 0; - } - auto make_plan = [](const ParquetColumnSchema& schema) { - auto projection = format::LocalColumnIndex::partial_local(0); - projection.children.push_back(format::LocalColumnIndex::partial_local(2)); - projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); - projection.children.back().children.back().children.push_back( - format::LocalColumnIndex::local(0)); - VariantMaterializationNode plan; - plan.schema = &schema; - plan.contains_variant = true; - plan.variant_projection = std::move(projection); - plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); - return plan; - }; - auto integer_plan = make_plan(integer_schema); - auto string_plan = make_plan(string_schema); - auto integers = make_nullable(std::make_shared())->create_column(); - auto strings = make_nullable(std::make_shared())->create_column(); - ASSERT_TRUE(materialize_variant_columns(integer_plan, projected_shredded_object_physical({7}), - integers) - .ok()); - ASSERT_TRUE(materialize_variant_columns( - string_plan, projected_shredded_binary_object_physical({"seven"}), strings) - .ok()); - - auto gathered = make_nullable(std::make_shared())->create_column(); - const std::array selected {0}; - gathered->insert_indices_from(*integers, selected.begin(), selected.end()); - gathered->insert_indices_from(*strings, selected.begin(), selected.end()); - - const auto& nullable = assert_cast(*gathered); - const auto& variants = assert_cast(nullable.get_nested_column()); - const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; - std::unique_ptr path; - ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); - ColumnPtr extracted; - ASSERT_TRUE( - extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) - .ok()); - const auto& extracted_variants = assert_cast( - assert_cast(*extracted).get_nested_column()); - EXPECT_EQ(extracted_variants.get_value_ref(0).get_int(), 7); - EXPECT_EQ(extracted_variants.get_value_ref(1).get_binary(), StringRef("seven")); - - const std::array shredded_path {VariantShreddedPathSegment { - .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; - auto reordered = ColumnVariantV2::create(); - const std::array reversed {1, 0}; - reordered->insert_indices_from(variants, reversed.begin(), reversed.end()); - const auto reordered_match = reordered->find_shredded_typed_value(shredded_path); - ASSERT_TRUE(reordered_match.has_value()); - ASSERT_TRUE(reordered_match->normalized); - const auto& reordered_values = assert_cast( - assert_cast(*reordered_match->normalized).get_nested_column()); - EXPECT_EQ(reordered_values.get_value_ref(0).get_binary(), StringRef("seven")); - EXPECT_EQ(reordered_values.get_value_ref(1).get_int(), 7); - - IColumn::Filter keep_integer {1, 0}; - const auto filtered = variants.filter(keep_integer, 1); - const auto filtered_match = - assert_cast(*filtered).find_shredded_typed_value(shredded_path); - ASSERT_TRUE(filtered_match.has_value()); - ASSERT_TRUE(filtered_match->column); - EXPECT_EQ( - assert_cast( - assert_cast(*filtered_match->column).get_nested_column()) - .get_data()[0], - 7); -} - TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { constexpr size_t width = 64; constexpr size_t batch_count = 16; From 9e0b163070029e7c3b781ffb20b4391243aed28d Mon Sep 17 00:00:00 2001 From: Gabriel Date: Wed, 5 Aug 2026 20:42:08 +0800 Subject: [PATCH 11/20] [improvement](parquet) Use predicate tree for compound page pruning ### What problem does this PR solve? Issue Number: None Related PR: #66412 Problem Summary: Forward-port compound Parquet Page Index pruning to master. AND nodes intersect conservative candidate ranges, OR nodes union them, and unavailable OR branches retain the complete range. The master adaptation also keeps compound predicates behind the existing metadata-pruning safety fence. ### Release note Support compound Parquet Page Index pruning in File Scanner V2. ### Check List (For Author) - Test: Unit Test (2 focused BE ASAN tests passed) - Behavior changed: Yes (enables conservative compound Page Index pruning) - Does this need documentation: Yes (included in this commit) --- be/benchmark/parquet/AGENTS.md | 8 +- be/benchmark/parquet/README.md | 12 + .../parquet/benchmark_parquet_reader.hpp | 169 ++++++++++++++ .../format_v2/parquet/parquet_statistics.cpp | 209 +++++++++++++++++- .../format_v2/parquet/parquet_scan_test.cpp | 57 +++++ .../parquet/parquet_statistics_test.cpp | 171 ++++++++++++++ docs/file-scanner-v2-parquet-scan-design.md | 17 +- 7 files changed, 628 insertions(+), 15 deletions(-) diff --git a/be/benchmark/parquet/AGENTS.md b/be/benchmark/parquet/AGENTS.md index 4d8c0610f17b5d..da51348852002e 100644 --- a/be/benchmark/parquet/AGENTS.md +++ b/be/benchmark/parquet/AGENTS.md @@ -57,7 +57,7 @@ be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^ParquetSelection/' # currently 25 be/output/lib/benchmark_test --benchmark_list_tests \ - | grep -c '^ParquetReader/' # currently 167 + | grep -c '^ParquetReader/' # currently 169 be/output/lib/benchmark_test --benchmark_list_tests \ | grep -c '^FileScannerExpr/' # currently 8 @@ -189,6 +189,10 @@ Except for the axis being varied, reader cases inherit the baseline: nullable IN alternating 10% nulls, 10% selectivity, 32 columns, predicate at column zero, and predicate plus payload projection. +Two dedicated multi-column OR cases scan the same Page Index fixture and change only the Doris Page +Index switch. They retain the complete residual expression and validate the same selected row +count before reporting throughput. + ## How decoder data is generated Decoder pages are constructed in memory before the timed loop. There is no Parquet file, Python @@ -346,7 +350,7 @@ be simulated by silently changing the local reader benchmark. ## Current validation record -The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 167 reader, and +The current expected registration counts are 228 decoder, 92 kernel, 25 selection, 169 reader, and 8 expression-lifecycle cases. A smoke run is an execution record only, not a reviewed performance baseline, because repetitions, host isolation, warmups, cache control, `perf` data, variance, and before/after comparison are not collected. diff --git a/be/benchmark/parquet/README.md b/be/benchmark/parquet/README.md index 891fd02d8375ed..b4761855f5ceae 100644 --- a/be/benchmark/parquet/README.md +++ b/be/benchmark/parquet/README.md @@ -137,6 +137,18 @@ be/output/lib/benchmark_test \ --benchmark_min_time=1s ``` +The multi-column OR pair scans the same ColumnIndex/OffsetIndex fixture and changes only the Doris +Page Index switch. Both variants retain the full residual expression, so the comparison measures +metadata pruning without changing result semantics: + +```shell +be/output/lib/benchmark_test \ + --benchmark_filter='^ParquetReader/multi_column_or/page_index_(off|on)$' \ + --benchmark_min_time=1s \ + --benchmark_repetitions=10 \ + --benchmark_report_aggregates_only=true +``` + Every result reports throughput plus `raw_rows`, `selected_rows`, `fixture_bytes`, `ns/raw_row`, and (when at least one row survives) `ns/selected_row`. Keep CPU frequency, build type, compiler, machine placement, and benchmark filters fixed when comparing two commits. diff --git a/be/benchmark/parquet/benchmark_parquet_reader.hpp b/be/benchmark/parquet/benchmark_parquet_reader.hpp index b456244ba0e3ba..4763ffe4c9867e 100644 --- a/be/benchmark/parquet/benchmark_parquet_reader.hpp +++ b/be/benchmark/parquet/benchmark_parquet_reader.hpp @@ -34,6 +34,7 @@ #include #include +#include "common/config.h" #include "core/assert_cast.h" #include "core/block/block.h" #include "core/column/column_nullable.h" @@ -61,6 +62,19 @@ namespace reader_detail { constexpr size_t READER_ROWS = 1UL << 14; constexpr size_t READER_ROW_GROUP_ROWS = 1UL << 12; +constexpr size_t MULTI_COLUMN_OR_ROWS = 1UL << 20; +constexpr size_t MULTI_COLUMN_OR_ROW_GROUP_ROWS = 1UL << 18; + +class ScopedPageIndexConfig { +public: + explicit ScopedPageIndexConfig(bool enabled) : _previous(config::enable_parquet_page_index) { + config::enable_parquet_page_index = enabled; + } + ~ScopedPageIndexConfig() { config::enable_parquet_page_index = _previous; } + +private: + bool _previous; +}; inline void throw_if_error(const Status& status) { if (!status.ok()) { @@ -256,6 +270,8 @@ class Int32LessThanExpr final : public VExpr { _column_id(column_id), _upper_bound(upper_bound) {} + bool is_constant() const override { return false; } + Status execute_column_impl(VExprContext*, const Block* block, const Selector* selector, size_t count, ColumnPtr& result_column) const override { DORIS_CHECK(block != nullptr); @@ -612,6 +628,151 @@ inline void run_reader(benchmark::State& state, ReaderScenario scenario) { } } +inline std::filesystem::path ensure_multi_column_or_fixture() { + static std::mutex fixture_mutex; + const auto directory = + std::filesystem::temp_directory_path() / "doris_parquet_reader_benchmark"; + const auto path = directory / "v2_multi_column_or_page_index_v2.parquet"; + std::lock_guard guard(fixture_mutex); + if (std::filesystem::exists(path)) { + return path; + } + + arrow::Int32Builder ascending_builder; + arrow::Int32Builder descending_builder; + arrow::Int32Builder payload_builder; + PARQUET_THROW_NOT_OK(ascending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(descending_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + PARQUET_THROW_NOT_OK(payload_builder.Reserve(MULTI_COLUMN_OR_ROWS)); + for (size_t row = 0; row < MULTI_COLUMN_OR_ROWS; ++row) { + const int32_t row_in_group = static_cast(row % MULTI_COLUMN_OR_ROW_GROUP_ROWS); + PARQUET_THROW_NOT_OK(ascending_builder.Append(row_in_group)); + PARQUET_THROW_NOT_OK(descending_builder.Append( + static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS - 1) - row_in_group)); + PARQUET_THROW_NOT_OK(payload_builder.Append(static_cast(row))); + } + auto table = arrow::Table::Make( + arrow::schema({arrow::field("ascending", arrow::int32(), true), + arrow::field("descending", arrow::int32(), true), + arrow::field("payload", arrow::int32(), true)}), + {ascending_builder.Finish().ValueOrDie(), descending_builder.Finish().ValueOrDie(), + payload_builder.Finish().ValueOrDie()}); + + std::filesystem::create_directories(directory); + const auto temporary_path = path.string() + ".tmp"; + std::filesystem::remove(temporary_path); + const auto output_result = arrow::io::FileOutputStream::Open(temporary_path); + if (!output_result.ok()) { + throw std::runtime_error(output_result.status().ToString()); + } + const auto output = *output_result; + ::parquet::WriterProperties::Builder properties; + properties.version(::parquet::ParquetVersion::PARQUET_2_6); + properties.data_page_version(::parquet::ParquetDataPageVersion::V2); + properties.compression(::parquet::Compression::UNCOMPRESSED); + properties.disable_dictionary(); + properties.encoding(::parquet::Encoding::PLAIN); + properties.enable_write_page_index(); + properties.write_batch_size(8192); + properties.data_pagesize(64 * 1024); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), output, + MULTI_COLUMN_OR_ROW_GROUP_ROWS, + properties.build())); + PARQUET_THROW_NOT_OK(output->Close()); + std::filesystem::rename(temporary_path, path); + return path; +} + +inline std::unique_ptr open_multi_column_or_reader( + const std::filesystem::path& path) { + auto session = std::make_unique(); + auto properties = std::make_shared(); + properties->system_type = TFileType::FILE_LOCAL; + auto description = std::make_unique(); + description->path = path.string(); + description->file_size = static_cast(std::filesystem::file_size(path)); + description->range_start_offset = 0; + description->range_size = -1; + session->reader = std::make_unique(properties, description, + nullptr, nullptr); + throw_if_error(session->reader->init(&session->runtime_state)); + throw_if_error(session->reader->get_schema(&session->schema)); + + session->request = std::make_shared(); + format::FileScanRequestBuilder request_builder(session->request.get()); + std::array predicate_positions {}; + for (int column = 0; column < 2; ++column) { + const auto column_id = format::LocalColumnId(column); + throw_if_error(request_builder.add_predicate_column(column_id)); + session->request->predicate_only_columns.push_back(column_id); + predicate_positions[column] = + static_cast(session->request->local_positions.at(column_id).value()); + } + throw_if_error(request_builder.add_non_predicate_column(format::LocalColumnId(2))); + + TExprNode node; + node.__set_node_type(TExprNodeType::COMPOUND_PRED); + node.__set_opcode(TExprOpcode::COMPOUND_OR); + node.__set_type(std::make_shared()->to_thrift()); + node.__set_num_children(2); + node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(node); + constexpr int32_t UPPER_BOUND = static_cast(MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + compound->add_child(std::make_shared(predicate_positions[0], UPPER_BOUND)); + compound->add_child(std::make_shared(predicate_positions[1], UPPER_BOUND)); + auto context = VExprContext::create_shared(std::move(compound)); + throw_if_error(context->prepare(&session->runtime_state, RowDescriptor())); + throw_if_error(context->open(&session->runtime_state)); + session->request->conjuncts.push_back(context); + session->opened_conjuncts.push_back(std::move(context)); + throw_if_error(session->reader->open(session->request)); + return session; +} + +inline void run_multi_column_or_reader(benchmark::State& state, bool enable_page_index) { + try { + const auto fixture = ensure_multi_column_or_fixture(); + ScopedPageIndexConfig page_index_config(enable_page_index); + size_t selected_rows = 0; + for (auto _ : state) { + state.PauseTiming(); + auto session = open_multi_column_or_reader(fixture); + state.ResumeTiming(); + const ReaderScenario scenario {.operation = ReaderOperation::PREDICATE_SCAN, + .encoding = Encoding::PLAIN, + .null_percent = 0, + .null_pattern = Pattern::CLUSTERED, + .selectivity_percent = 20, + .projection = Projection::PREDICATE_ONLY, + .schema_width = 3, + .predicate_position = 0}; + selected_rows = scan_reader(session.get(), scenario); + state.PauseTiming(); + throw_if_error(session->reader->close()); + state.ResumeTiming(); + benchmark::ClobberMemory(); + } + constexpr size_t ROW_GROUPS = MULTI_COLUMN_OR_ROWS / MULTI_COLUMN_OR_ROW_GROUP_ROWS; + constexpr size_t EXPECTED_ROWS = ROW_GROUPS * 2 * (MULTI_COLUMN_OR_ROW_GROUP_ROWS / 10); + if (selected_rows != EXPECTED_ROWS) { + state.SkipWithError("multi-column OR benchmark returned unexpected rows"); + return; + } + state.SetItemsProcessed(static_cast(state.iterations() * selected_rows)); + state.counters["raw_rows"] = static_cast(MULTI_COLUMN_OR_ROWS); + state.counters["selected_rows"] = static_cast(selected_rows); + state.counters["fixture_bytes"] = static_cast(std::filesystem::file_size(fixture)); + state.counters["ns/raw_row"] = benchmark::Counter( + static_cast(MULTI_COLUMN_OR_ROWS), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + state.counters["ns/selected_row"] = benchmark::Counter( + static_cast(selected_rows), + benchmark::Counter::kIsIterationInvariantRate | benchmark::Counter::kInvert); + } catch (const std::exception& error) { + state.SkipWithError(error.what()); + } +} + inline bool register_reader_benchmarks() { for (const auto& scenario : reader_scenarios()) { std::string name = "ParquetReader/" + reader_scenario_name(scenario); @@ -619,6 +780,14 @@ inline bool register_reader_benchmarks() { run_reader(state, scenario); })->Unit(benchmark::kNanosecond); } + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_off", + [](benchmark::State& state) { run_multi_column_or_reader(state, false); }) + ->Unit(benchmark::kNanosecond); + benchmark::RegisterBenchmark( + "ParquetReader/multi_column_or/page_index_on", + [](benchmark::State& state) { run_multi_column_or_reader(state, true); }) + ->Unit(benchmark::kNanosecond); return true; } diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 5ba015373a404f..30a8cc6da92600 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -1381,6 +1381,38 @@ std::vector intersect_ranges(const std::vector& left, return result; } +std::vector union_ranges(const std::vector& left, + const std::vector& right) { + std::vector result; + result.reserve(left.size() + right.size()); + auto append = [&](const RowRange& range) { + if (range.length == 0) { + return; + } + if (!result.empty()) { + auto& previous = result.back(); + const int64_t previous_end = previous.start + previous.length; + if (range.start <= previous_end) { + previous.length = + std::max(previous_end, range.start + range.length) - previous.start; + return; + } + } + result.push_back(range); + }; + size_t left_idx = 0; + size_t right_idx = 0; + while (left_idx < left.size() || right_idx < right.size()) { + if (right_idx == right.size() || + (left_idx < left.size() && left[left_idx].start <= right[right_idx].start)) { + append(left[left_idx++]); + } else { + append(right[right_idx++]); + } + } + return result; +} + int64_t count_range_rows(const std::vector& ranges) { int64_t rows = 0; for (const auto& range : ranges) { @@ -1637,6 +1669,154 @@ RowRange native_page_row_range(const tparquet::OffsetIndex& offset_index, size_t return {.start = start, .length = end - start}; } +class NativePageIndexPredicateEvaluator { +public: + NativePageIndexPredicateEvaluator( + const tparquet::FileMetaData& metadata, + const std::unordered_map& page_indexes, + const std::vector>& file_schema, + const format::FileScanRequest& request, int64_t row_group_rows, + ParquetPruningStats* pruning_stats, const cctz::time_zone* timezone) + : _metadata(metadata), + _page_indexes(page_indexes), + _file_schema(file_schema), + _request(request), + _row_group_rows(row_group_rows), + _pruning_stats(pruning_stats), + _timezone(timezone) {} + + std::optional> evaluate(const VExprSPtr& expr) const { + if (expr == nullptr || !expr->can_evaluate_zonemap_filter()) { + return std::nullopt; + } + if (expr->op() == TExprOpcode::COMPOUND_AND) { + return evaluate_compound(expr, true); + } + if (expr->op() == TExprOpcode::COMPOUND_OR) { + return evaluate_compound(expr, false); + } + return evaluate_leaf(expr); + } + +private: + struct SlotPageZoneMaps { + DataTypePtr data_type; + std::vector ranges; + std::vector> zone_maps; + }; + + std::optional> evaluate_compound(const VExprSPtr& expr, + bool is_and) const { + std::optional> ranges; + for (const auto& child : expr->children()) { + if (!child->can_evaluate_zonemap_filter()) { + if (!is_and) { + return std::nullopt; + } + continue; + } + auto child_ranges = evaluate(child); + if (!child_ranges.has_value()) { + // An unavailable AND child can be ignored, while an unavailable OR branch must + // retain the complete range so metadata pruning cannot create a false negative. + if (!is_and) { + return std::nullopt; + } + continue; + } + if (!ranges.has_value()) { + ranges = std::move(*child_ranges); + } else if (is_and) { + ranges = intersect_ranges(*ranges, *child_ranges); + } else { + ranges = union_ranges(*ranges, *child_ranges); + } + if (is_and && ranges->empty()) { + return ranges; + } + } + return ranges; + } + + std::optional> evaluate_leaf(const VExprSPtr& expr) const { + std::set slot_indexes; + expr->collect_slot_column_ids(slot_indexes); + if (slot_indexes.size() != 1) { + return std::nullopt; + } + const int slot_index = *slot_indexes.begin(); + const auto* pages = load_slot_pages(slot_index); + if (pages == nullptr) { + return std::nullopt; + } + + std::vector ranges; + for (size_t page_idx = 0; page_idx < pages->ranges.size(); ++page_idx) { + ZoneMapEvalContext ctx; + add_slot_zonemap(&ctx, slot_index, pages->data_type, pages->zone_maps[page_idx]); + if (expr->evaluate_zonemap_filter(ctx) != ZoneMapFilterResult::kNoMatch) { + append_row_range(pages->ranges[page_idx], &ranges); + } + accumulate_zonemap_stats(ctx, _pruning_stats); + } + return ranges; + } + + const SlotPageZoneMaps* load_slot_pages(int slot_index) const { + const auto cached = _slot_page_zone_maps.find(slot_index); + if (cached != _slot_page_zone_maps.end()) { + return cached->second.has_value() ? &*cached->second : nullptr; + } + const auto file_column_id = file_column_id_by_block_position(_request, slot_index); + if (!file_column_id.has_value()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto* column_schema = resolve_local_leaf_schema(_file_schema, *file_column_id); + if (column_schema == nullptr || column_schema->type == nullptr || + !native_metadata_predicate_is_type_safe(*column_schema) || + !detail::has_supported_type_defined_order(_metadata, column_schema->leaf_column_id)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + const auto index_it = _page_indexes.find(column_schema->leaf_column_id); + if (index_it == _page_indexes.end()) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + + const auto& indexes = index_it->second; + SlotPageZoneMaps pages; + pages.data_type = column_schema->type; + pages.ranges.reserve(indexes.offset_index.page_locations.size()); + pages.zone_maps.reserve(indexes.offset_index.page_locations.size()); + for (size_t page_idx = 0; page_idx < indexes.offset_index.page_locations.size(); + ++page_idx) { + const auto page_range = + native_page_row_range(indexes.offset_index, page_idx, _row_group_rows); + ParquetColumnStatistics statistics; + if (!build_native_page_statistics(indexes.column_index, *column_schema, page_idx, + page_range.length, &statistics, _timezone)) { + _slot_page_zone_maps.emplace(slot_index, std::nullopt); + return nullptr; + } + pages.ranges.push_back(page_range); + pages.zone_maps.push_back(ParquetStatisticsUtils::MakeZoneMap(statistics)); + } + const auto inserted = _slot_page_zone_maps.emplace(slot_index, std::move(pages)); + return &*inserted.first->second; + } + + const tparquet::FileMetaData& _metadata; + const std::unordered_map& _page_indexes; + const std::vector>& _file_schema; + const format::FileScanRequest& _request; + int64_t _row_group_rows; + ParquetPruningStats* _pruning_stats; + const cctz::time_zone* _timezone; + mutable std::unordered_map> _slot_page_zone_maps; +}; + } // namespace Status select_row_group_ranges_by_native_page_index( @@ -1665,10 +1845,15 @@ Status select_row_group_ranges_by_native_page_index( } std::map conjuncts_by_slot; + VExprContextSPtrs multi_slot_conjuncts; + // Compound predicates must honor the same metadata-pruning fence as single-slot predicates. for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct); if (slot_index >= 0) { conjuncts_by_slot[slot_index].push_back(conjunct); + } else if (conjunct != nullptr && conjunct->root() != nullptr && + conjunct->root()->can_evaluate_zonemap_filter()) { + multi_slot_conjuncts.push_back(conjunct); } } for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { @@ -1706,12 +1891,7 @@ Status select_row_group_ranges_by_native_page_index( ZoneMapFilterResult::kNoMatch) { append_row_range(page_range, &filter_ranges); } - if (pruning_stats != nullptr) { - pruning_stats->expr_zonemap_unusable_evals += ctx.stats.unusable_zonemap_eval_count; - pruning_stats->in_zonemap_point_check_count += - ctx.stats.in_zonemap_point_check_count; - pruning_stats->in_zonemap_range_only_count += ctx.stats.in_zonemap_range_only_count; - } + accumulate_zonemap_stats(ctx, pruning_stats); } if (!usable) { continue; @@ -1726,6 +1906,23 @@ Status select_row_group_ranges_by_native_page_index( } } + NativePageIndexPredicateEvaluator evaluator(metadata, page_indexes, file_schema, request, + row_group_rows, pruning_stats, timezone); + for (const auto& conjunct : multi_slot_conjuncts) { + auto conjunct_ranges = evaluator.evaluate(conjunct->root()); + if (!conjunct_ranges.has_value()) { + continue; + } + *selected_ranges = intersect_ranges(*selected_ranges, *conjunct_ranges); + if (selected_ranges->empty()) { + if (pruning_stats != nullptr) { + pruning_stats->filtered_page_rows += row_group_rows; + ++pruning_stats->filtered_row_groups_by_page_index; + } + return Status::OK(); + } + } + for (const auto& conjunct : metadata_pruning_conjuncts(request)) { const auto predicate = extract_variant_shredded_predicate(conjunct); if (!predicate.has_value()) { diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index eaca8f772c8c93..8f1f9711ae826a 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -1678,6 +1678,17 @@ void write_page_index_parquet_file(const std::string& file_path) { write_table(file_path, table, ids.size(), false, true); } +void write_multi_column_page_index_parquet_file(const std::string& file_path) { + std::vector ascending(128); + std::iota(ascending.begin(), ascending.end(), 0); + std::vector descending(ascending.rbegin(), ascending.rend()); + auto schema = arrow::schema({arrow::field("ascending", arrow::int32(), false), + arrow::field("descending", arrow::int32(), false)}); + auto table = arrow::Table::Make(schema, + {build_int32_array(ascending), build_int32_array(descending)}); + write_table(file_path, table, ascending.size(), false, true); +} + void write_multi_row_group_page_index_parquet_file(const std::string& file_path) { std::vector ids(384); std::iota(ids.begin(), ids.end(), 0); @@ -4195,6 +4206,52 @@ TEST_F(ParquetScanTest, ProfileCountersReflectPageIndexAndRangeGapPruning) { EXPECT_GT(profile.get_counter("RangeGapSkippedRows")->value(), 0); } +TEST_F(ParquetScanTest, MultiColumnOrUsesPageIndexAndResidualExpression) { + write_multi_column_page_index_parquet_file(_file_path); + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(0)).ok()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(1)).ok()); + const auto low_ascending = create_int32_function_conjunct(0, "lt", TExprOpcode::LT, 13); + const auto low_descending = create_int32_function_conjunct(1, "lt", TExprOpcode::LT, 13); + auto disjunction = create_compound_conjunct(TExprOpcode::COMPOUND_OR, low_ascending->root(), + low_descending->root()); + ASSERT_TRUE(disjunction->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(disjunction->open(&state).ok()); + request->conjuncts.push_back(disjunction); + ASSERT_TRUE(reader->open(request).ok()); + + std::vector selected; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& values = int32_data_column(*block.get_by_position(0).column); + for (size_t row = 0; row < rows; ++row) { + selected.push_back(values.get_element(row)); + } + } + + std::vector expected(13); + std::iota(expected.begin(), expected.end(), 0); + for (int32_t value = 115; value < 128; ++value) { + expected.push_back(value); + } + EXPECT_EQ(selected, expected); + EXPECT_GT(counter_value(profile, "FilteredRowsByPage"), 0); + EXPECT_LT(counter_value(profile, "RawRowsRead"), 128); + EXPECT_GT(counter_value(profile, "RowsFilteredByConjunct"), 0); + disjunction->close(); +} + TEST_F(ParquetScanTest, OpenDefersPageIndexProbeToCurrentRowGroup) { write_multi_row_group_page_index_parquet_file(_file_path); RuntimeProfile profile("lazy_page_index_profile"); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 533b526d613d9f..6b48b05d61cd2f 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -40,6 +40,7 @@ #include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" #include "exprs/vliteral.h" @@ -167,6 +168,41 @@ class MetadataInt32GreaterThanExpr final : public VExpr { const std::string _expr_name = "MetadataInt32GreaterThanExpr"; }; +class MetadataSlotInt32GreaterThanExpr final : public VExpr { +public: + MetadataSlotInt32GreaterThanExpr(int slot_index, int32_t value) + : VExpr(std::make_shared(), false), + _slot_index(slot_index), + _value(value) {} + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataSlotInt32GreaterThanExpr is metadata-only"); + } + bool can_evaluate_zonemap_filter() const override { return true; } + void collect_slot_column_ids(std::set& column_ids) const override { + column_ids.insert(_slot_index); + } + ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override { + const auto zone_map = ctx.zone_map(_slot_index); + if (zone_map == nullptr) { + return unsupported_zonemap_filter(ctx); + } + if (!zone_map->has_not_null) { + return ZoneMapFilterResult::kNoMatch; + } + return zone_map->max_value <= Field::create_field(_value) + ? ZoneMapFilterResult::kNoMatch + : ZoneMapFilterResult::kMayMatch; + } + +private: + int _slot_index; + int32_t _value; + const std::string _expr_name = "MetadataSlotInt32GreaterThanExpr"; +}; + class MetadataBoundsProbeExpr final : public VExpr { public: explicit MetadataBoundsProbeExpr(bool require_false_boolean = false) @@ -466,6 +502,141 @@ TEST(NativeParquetStatisticsTest, InvalidTimeAndPaddedBooleanPageBoundsCannotPru EXPECT_EQ(bool_ranges[0].start, 0); EXPECT_EQ(bool_ranges[0].length, 1); } + +TEST(NativeParquetStatisticsTest, MultiColumnOrUnionsPageIndexRanges) { + auto encode_int32 = [](int32_t value) { + std::string bytes(sizeof(value), '\0'); + memcpy(bytes.data(), &value, sizeof(value)); + return bytes; + }; + auto make_schema = [](int local_id, int leaf_column_id) { + auto column = std::make_unique(); + column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column->local_id = local_id; + column->leaf_column_id = leaf_column_id; + column->type = std::make_shared(); + column->type_descriptor.doris_type = column->type; + column->type_descriptor.physical_type = tparquet::Type::INT32; + return column; + }; + auto make_page_index = [&](const std::vector& values) { + format::parquet::NativeParquetPageIndex page_index; + std::vector encoded; + encoded.reserve(values.size()); + for (const auto value : values) { + encoded.push_back(encode_int32(value)); + } + page_index.column_index.__set_min_values(encoded); + page_index.column_index.__set_max_values(encoded); + page_index.column_index.__set_null_pages(std::vector(values.size(), false)); + page_index.column_index.__set_null_counts(std::vector(values.size(), 0)); + std::vector locations; + for (size_t page_idx = 0; page_idx < values.size(); ++page_idx) { + tparquet::PageLocation location; + location.__set_offset(static_cast(page_idx * 100)); + location.__set_compressed_page_size(100); + location.__set_first_row_index(static_cast(page_idx * 10)); + locations.push_back(location); + } + page_index.offset_index.__set_page_locations(std::move(locations)); + return page_index; + }; + + std::vector> schema; + schema.push_back(make_schema(0, 0)); + schema.push_back(make_schema(1, 1)); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_column_orders({order, order}); + + auto make_compound_expr = [](TExprOpcode::type opcode, VExprSPtr left, VExprSPtr right) { + TExprNode compound_node; + compound_node.__set_node_type(TExprNodeType::COMPOUND_PRED); + compound_node.__set_opcode(opcode); + compound_node.__set_type(std::make_shared()->to_thrift()); + compound_node.__set_num_children(2); + compound_node.__set_is_nullable(false); + auto compound = VCompoundPred::create_shared(compound_node); + compound->add_child(std::move(left)); + compound->add_child(std::move(right)); + return compound; + }; + auto make_compound = [&](TExprOpcode::type opcode) { + return VExprContext::create_shared(make_compound_expr( + opcode, std::make_shared(0, 50), + std::make_shared(1, 50))); + }; + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.local_positions.emplace(format::LocalColumnId(1), format::LocalIndex(1)); + request.predicate_columns = {format::LocalColumnIndex::top_level(format::LocalColumnId(0)), + format::LocalColumnIndex::top_level(format::LocalColumnId(1))}; + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_OR)}; + + std::unordered_map page_indexes; + page_indexes.emplace(0, make_page_index({100, 0, 0})); + page_indexes.emplace(1, make_page_index({0, 0, 100})); + std::vector selected_ranges; + std::map skip_plans; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 2); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + EXPECT_EQ(selected_ranges[1].start, 20); + EXPECT_EQ(selected_ranges[1].length, 10); + + // A compound predicate after the safety fence must not participate in metadata pruning. + request.metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + request.metadata_pruning_safe_conjunct_count = request.conjuncts.size(); + + page_indexes.erase(1); + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 30); + + request.conjuncts = {make_compound(TExprOpcode::COMPOUND_AND)}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 0); + EXPECT_EQ(selected_ranges[0].length, 10); + + page_indexes.emplace(1, make_page_index({0, 0, 100})); + auto first_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, 50), + std::make_shared(1, 50)); + auto second_branch = make_compound_expr( + TExprOpcode::COMPOUND_AND, std::make_shared(0, -1), + std::make_shared(1, 50)); + request.conjuncts = {VExprContext::create_shared(make_compound_expr( + TExprOpcode::COMPOUND_OR, std::move(first_branch), std::move(second_branch)))}; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, tparquet::RowGroup {}, page_indexes, schema, request, 30, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(selected_ranges.size(), 1); + EXPECT_EQ(selected_ranges[0].start, 20); + EXPECT_EQ(selected_ranges[0].length, 10); +} + TEST(ParquetBloomFilterPruningTest, NativeUint32BloomUsesPhysicalInt32Hash) { const auto column_schema = uint32_parquet_bloom_schema(); format::parquet::native::BlockSplitBloomFilter bloom_filter; diff --git a/docs/file-scanner-v2-parquet-scan-design.md b/docs/file-scanner-v2-parquet-scan-design.md index b81a779ac5edcc..29dfe66c6fac63 100644 --- a/docs/file-scanner-v2-parquet-scan-design.md +++ b/docs/file-scanner-v2-parquet-scan-design.md @@ -197,9 +197,9 @@ flowchart LR does not repeatedly interpret table-schema evolution. 3. **Capability checks:** ZoneMap, Dictionary, and Bloom use only expressions they can interpret safely. All others remain row-level residual predicates. -4. **Prefer safe single-column predicates:** Single-column predicates can drive indexes and staged - filtering. Multi-column, stateful, or error-sensitive expressions retain whole-expression - evaluation. +4. **Prefer safe single-column row filters:** Single-column predicates can drive staged dictionary + or raw filtering. Multi-column AND/OR trees may still combine conservative Row Group and Page + Index candidate ranges, but the complete expression remains in whole-expression row evaluation. 5. **Runtime Filters can refresh:** ScannerScheduler refreshes late Runtime Filters before reading. TableReader handles partition-range pruning during Split preparation, and passes file-pushable parts as localized conjuncts. @@ -285,9 +285,11 @@ sequenceDiagram ### How the plan drives physical skips ColumnIndex provides min/max/null semantics for each page. OffsetIndex maps pages to Row Group row -numbers and file offsets. Candidate ranges from multiple predicate columns are intersected into -`selected_ranges`; a `page_skip_plan` is then built for each leaf so its column reader can skip pages -that do not overlap surviving rows. +numbers and file offsets. Candidate ranges follow the predicate tree: AND nodes intersect child +ranges and OR nodes union them into `selected_ranges`. A missing or unusable AND child contributes +no pruning, while a missing or unusable OR branch retains the complete Row Group range. A +`page_skip_plan` is then built for each leaf so its column reader can skip pages that do not overlap +surviving rows. > `selected_ranges` represents logical row ranges, while `page_skip_plan` represents physical page > reads. Keeping them separate allows the scheduler to advance by row batch while each column skips @@ -826,7 +828,8 @@ split safely, or read anomalies must never change query semantics. | Bloom missing, disabled, or unreadable | Skip Bloom pruning and continue with later scan stages | | Incomplete dictionary page, mixed non-dictionary encoding, complex/repeated column | Disable dictionary pruning and Dictionary-ID Filter; use actual values | | Missing or inconsistent ColumnIndex/OffsetIndex | Disable fine-grained page pruning and read the full candidate range | -| Multi-column, OR, stateful, or error-order-sensitive expression | Preserve whole-expression evaluation to avoid changing SQL short-circuit or error semantics | +| Multi-column AND/OR expression | Combine only conservative metadata candidate ranges; preserve whole-expression row evaluation | +| Stateful or error-order-sensitive expression | Preserve whole-expression evaluation without metadata decomposition | | No stable file-version identity for Page Cache | Disable Parquet Page Cache to prevent stale-byte reads | | Incomplete Condition Cache coverage | Retain and recompute uncovered ranges | From 88e3c4b944290dc9464b57f1bdb18fcc4a68f7bc Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 08:42:21 +0800 Subject: [PATCH 12/20] [fix](variant) Support native Paimon Variant reads --- .../column/variant_v2/column_variant_v2.cpp | 423 +++++++++++- .../column/variant_v2/column_variant_v2.h | 17 + .../data_type_variant_v2_serde.cpp | 4 +- .../value/variant/variant_batch_builder.cpp | 12 +- .../function/function_variant_element_v2.cpp | 7 +- be/src/format_v2/file_reader.cpp | 5 +- be/src/format_v2/file_reader.h | 5 + .../format_v2/parquet/native_schema_desc.cpp | 67 +- be/src/format_v2/parquet/native_schema_desc.h | 5 + .../parquet/parquet_column_schema.cpp | 120 +++- .../format_v2/parquet/parquet_column_schema.h | 6 + be/src/format_v2/parquet/parquet_profile.cpp | 37 +- be/src/format_v2/parquet/parquet_profile.h | 26 +- be/src/format_v2/parquet/parquet_reader.cpp | 37 + .../parquet/reader/variant_column_reader.cpp | 202 +++++- be/src/format_v2/table/paimon_reader.cpp | 164 ++++- be/src/format_v2/table/paimon_reader.h | 9 + be/src/runtime/runtime_profile.cpp | 22 + be/src/runtime/runtime_profile.h | 9 +- .../core/column/column_variant_v2_test.cpp | 67 ++ .../format_v2/parquet/parquet_schema_test.cpp | 128 ++++ .../parquet/variant_column_reader_test.cpp | 631 +++++++++++++++++- .../table/paimon_variant_reader_test.cpp | 548 +++++++++++++++ .../variant/variant_batch_builder_test.cpp | 17 + .../paimon/run13.sql | 70 ++ .../iceberg/test_iceberg_variant_read.out | 6 + .../paimon/test_paimon_catalog_variant.out | 68 ++ .../iceberg/test_iceberg_variant_read.groovy | 89 +-- .../paimon/test_paimon_catalog_variant.groovy | 224 +++++++ 29 files changed, 2869 insertions(+), 156 deletions(-) create mode 100644 be/test/format_v2/table/paimon_variant_reader_test.cpp create mode 100644 docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql create mode 100644 regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out create mode 100644 regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp b/be/src/core/column/variant_v2/column_variant_v2.cpp index 0d5a7fb74bb6df..8157220d930493 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.cpp +++ b/be/src/core/column/variant_v2/column_variant_v2.cpp @@ -22,9 +22,11 @@ #include #include #include +#include #include #include #include +#include #include "common/check.h" #include "common/exception.h" @@ -313,6 +315,307 @@ ValidatedTypedInput validate_typed_input(ColumnPtr column, DataTypePtr scalar_ty "ColumnVariantV2::{} is intentionally unsupported for Variant values", method); } +class CompositeVariantShreddedState final : public VariantShreddedState { +public: + explicit CompositeVariantShreddedState( + std::vector> segments) + : _segments(std::move(segments)) { + DORIS_CHECK(std::ranges::all_of(_segments, [](const auto& segment) { + return segment != nullptr; + })) << "composite Variant shredded segments must not be null"; + for (const auto& segment : _segments) { + const size_t segment_rows = segment->size(); + DORIS_CHECK_LE(segment_rows, std::numeric_limits::max() - _rows) + << "composite Variant shredded row count overflows size_t"; + _rows += segment_rows; + } + } + + size_t size() const override { return _rows; } + + size_t recompute_size() const { + size_t rows = 0; + for (const auto& segment : _segments) { + const size_t segment_rows = segment->size(); + DORIS_CHECK_LE(segment_rows, std::numeric_limits::max() - rows) + << "composite Variant shredded row count overflows size_t"; + rows += segment_rows; + } + return rows; + } + + size_t byte_size() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->byte_size(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->byte_size() : 0) + + (_serialized ? _serialized->byte_size() : 0); + } + + size_t allocated_bytes() const override { + size_t bytes = 0; + for (const auto& segment : _segments) { + bytes += segment->allocated_bytes(); + } + std::lock_guard lock(_materialization_lock); + return bytes + (_materialized ? _materialized->allocated_bytes() : 0) + + (_serialized ? _serialized->allocated_bytes() : 0); + } + + void sanity_check() const override { + // Row count is read in per-row expression loops, so cache it and reserve the full segment + // walk for invariant checks instead of making extraction quadratic in segment count. + DORIS_CHECK_EQ(recompute_size(), _rows) + << "cached composite Variant shredded row count is stale"; + for (const auto& segment : _segments) { + segment->sanity_check(); + } + } + + void for_each_subcolumn(IColumn::ColumnCallback callback) const override { + for (const auto& segment : _segments) { + segment->for_each_subcolumn(callback); + } + } + + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t /*result_size_hint*/) const override { + DORIS_CHECK_EQ(filter.size(), size()) + << "composite Variant shredded filter size does not match row count"; + std::vector> selected; + selected.reserve(_segments.size()); + size_t offset = 0; + for (const auto& segment : _segments) { + IColumn::Filter segment_filter; + segment_filter.insert(filter.begin() + offset, + filter.begin() + offset + segment->size()); + auto filtered = segment->filter(segment_filter, -1); + if (filtered->size() != 0) { + selected.push_back(std::move(filtered)); + } + offset += segment->size(); + } + return pack(std::move(selected)); + } + + std::shared_ptr select_range(size_t start, size_t length) const override { + DORIS_CHECK_LE(start, size()) << "composite Variant range starts past source size"; + DORIS_CHECK_LE(length, size() - start) << "composite Variant range exceeds source size"; + std::vector> selected; + if (length == 0) { + return pack(std::move(selected)); + } + const size_t end = start + length; + size_t offset = 0; + for (const auto& segment : _segments) { + const size_t segment_end = offset + segment->size(); + const size_t overlap_begin = std::max(start, offset); + const size_t overlap_end = std::min(end, segment_end); + if (overlap_begin < overlap_end) { + selected.push_back( + segment->select_range(overlap_begin - offset, overlap_end - overlap_begin)); + } + offset = segment_end; + if (offset >= end) { + break; + } + } + return pack(std::move(selected)); + } + + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + if (indices_begin == indices_end) { + return pack({}); + } + DORIS_CHECK(indices_begin != nullptr && indices_end != nullptr && + indices_begin < indices_end) + << "composite Variant indices are invalid"; + + std::vector segment_ends; + segment_ends.reserve(_segments.size()); + size_t rows = 0; + for (const auto& segment : _segments) { + rows += segment->size(); + segment_ends.push_back(rows); + } + + std::vector> selected; + const uint32_t* cursor = indices_begin; + while (cursor != indices_end) { + DORIS_CHECK_LT(*cursor, rows) << "composite Variant source index is out of range"; + const size_t segment_index = + std::upper_bound(segment_ends.begin(), segment_ends.end(), *cursor) - + segment_ends.begin(); + const size_t segment_begin = segment_index == 0 ? 0 : segment_ends[segment_index - 1]; + DorisVector local_indices; + while (cursor != indices_end && *cursor >= segment_begin && + *cursor < segment_ends[segment_index]) { + local_indices.push_back(static_cast(*cursor - segment_begin)); + ++cursor; + } + selected.push_back(_segments[segment_index]->select_indices( + local_indices.data(), local_indices.data() + local_indices.size())); + } + return pack(std::move(selected)); + } + + bool can_materialize() const override { + return std::ranges::all_of(_segments, + [](const auto& segment) { return segment->can_materialize(); }); + } + + bool try_append(const VariantShreddedState& source) override { + const size_t source_rows = source.size(); + DORIS_CHECK_LE(source_rows, std::numeric_limits::max() - _rows) + << "composite Variant shredded row count overflows size_t"; + if (const auto* composite = dynamic_cast(&source)) { + for (const auto& segment : composite->_segments) { + append(segment); + } + } else { + append(source.select_range(0, source.size())); + } + std::lock_guard lock(_materialization_lock); + _materialized.reset(); + _serialized.reset(); + _rows += source_rows; + return true; + } + + std::optional find_typed_value( + std::span path) const override { + if (_segments.empty()) { + return std::nullopt; + } + std::vector matches; + matches.reserve(_segments.size()); + bool all_direct = true; + for (const auto& segment : _segments) { + auto match = segment->find_typed_value(path); + if (!match.has_value()) { + all_direct = false; + break; + } + matches.push_back(std::move(*match)); + } + + const bool homogeneous = all_direct && matches.front().column && matches.front().type && + std::ranges::all_of(matches, [&](const auto& match) { + return match.column && match.type && !match.normalized && + exact_typed_identity(matches.front().type, match.type); + }); + if (homogeneous) { + MutableColumnPtr combined = matches.front().column->clone_empty(); + for (const auto& match : matches) { + combined->insert_range_from(*match.column, 0, match.column->size()); + } + return VariantShreddedTypedValue {.column = std::move(combined), + .type = matches.front().type, + .normalized = nullptr}; + } + + auto normalized = find_normalized_value(path); + if (!normalized.has_value()) { + return std::nullopt; + } + return VariantShreddedTypedValue { + .column = nullptr, .type = nullptr, .normalized = std::move(*normalized)}; + } + + std::optional find_normalized_value( + std::span path) const override { + auto values = ColumnVariantV2::create(); + auto nulls = ColumnUInt8::create(); + nulls->reserve(size()); + for (const auto& segment : _segments) { + auto normalized = segment->find_normalized_value(path); + if (!normalized.has_value()) { + return std::nullopt; + } + const auto& nullable = assert_cast(**normalized); + const auto& variants = + assert_cast(nullable.get_nested_column()); + values->insert_range_from(variants, 0, variants.size()); + nulls->insert_range_from(nullable.get_null_map_column(), 0, nullable.size()); + } + return ColumnNullable::create(std::move(values), std::move(nulls)); + } + + const ColumnVariantV2& materialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_materialized) { + auto materialized = ColumnVariantV2::create(); + for (const auto& segment : _segments) { + const ColumnVariantV2& source = segment->materialized_column(); + materialized->insert_range_from(source, 0, source.size()); + } + _materialized = std::move(materialized); + } + return *_materialized; + } + + const ColumnVariantV2& serialized_column() const override { + std::lock_guard lock(_materialization_lock); + if (!_serialized) { + auto serialized = ColumnVariantV2::create(); + for (const auto& segment : _segments) { + const ColumnVariantV2& source = segment->serialized_column(); + serialized->insert_range_from(source, 0, source.size()); + } + _serialized = std::move(serialized); + } + return *_serialized; + } + +private: + static std::shared_ptr pack( + std::vector> segments) { + if (segments.size() == 1) { + return std::move(segments.front()); + } + return std::make_shared(std::move(segments)); + } + + void append(std::shared_ptr source) { + if (source->size() == 0) { + return; + } + if (const auto* composite = + dynamic_cast(source.get())) { + _segments.insert(_segments.end(), composite->_segments.begin(), + composite->_segments.end()); + return; + } + if (!_segments.empty()) { + auto& tail = _segments.back(); + if (tail.use_count() != 1) { + tail = tail->select_range(0, tail->size()); + } + if (tail->try_append(*source)) { + return; + } + } + _segments.push_back(std::move(source)); + } + + std::vector> _segments; + size_t _rows = 0; + mutable std::mutex _materialization_lock; + mutable ColumnVariantV2::MutablePtr _materialized; + mutable ColumnVariantV2::MutablePtr _serialized; +}; + +std::shared_ptr combine_shredded_states( + std::shared_ptr left, std::shared_ptr right) { + auto combined = std::make_shared( + std::vector> {std::move(left)}); + combined->try_append(*right); + return combined; +} + } // namespace #ifdef BE_TEST @@ -387,21 +690,21 @@ std::optional ColumnVariantV2::find_shredded_typed_va return _shredded->find_typed_value(path); } +const ColumnVariantV2& ColumnVariantV2::serialization_column() const { + if (!_shredded) { + return *this; + } + const ColumnVariantV2& serialized = _shredded->serialized_column(); + DORIS_CHECK(!serialized.is_shredded()) + << "shredded Variant wire materializer returned another shredded column"; + DORIS_CHECK_EQ(serialized.size(), size()) + << "shredded Variant wire materializer changed the row count"; + return serialized; +} + void ColumnVariantV2::ensure_encoded() { if (_shredded) { - const ColumnVariantV2& materialized = _shredded->materialized_column(); - DORIS_CHECK(!materialized.is_shredded()) - << "shredded state materializer returned another shredded column"; - // The shredded state may cache and share its canonical materialization across readers. - // Detach every mutable buffer before dropping that owner so later COW mutations stay legal. - _metadatas = materialized._metadatas->clone_resized(materialized._metadatas->size()); - _meta_ids = materialized._meta_ids->clone_resized(materialized._meta_ids->size()); - _values = materialized._values->clone_resized(materialized._values->size()); - _typed = materialized._typed == nullptr - ? nullptr - : materialized._typed->clone_resized(materialized._typed->size()); - _typed_type = materialized._typed_type; - _shredded.reset(); + _replace_shredded_state_with(_shredded->materialized_column()); } if (!_typed) { DCHECK(_typed_type == nullptr); @@ -426,6 +729,12 @@ void ColumnVariantV2::ensure_encoded() { _check_invariants(); } +void ColumnVariantV2::_ensure_serialized() { + if (_shredded) { + _replace_shredded_state_with(_shredded->serialized_column()); + } +} + std::string ColumnVariantV2::get_name() const { if (_shredded) { return "variant_v2(shredded)"; @@ -818,12 +1127,22 @@ void ColumnVariantV2::insert_range_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (!_shredded->can_materialize() || !selected_source->can_materialize()) { + // A projected segment cannot reconstruct omitted root fields. Preserve it beside any + // complete or projected neighbor so later path extraction can choose per-segment + // direct or canonical evaluation without forcing the incomplete state to materialize. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } } if (_shredded) { - ensure_encoded(); + // A merging exchange can combine a local projected state with its remotely serialized + // peer. Use the wire representation so omitted roots are never requested from either side. + _ensure_serialized(); } if (source._shredded) { - insert_range_from(source._shredded->materialized_column(), start, length); + insert_range_from(source.serialization_column(), start, length); return; } @@ -911,9 +1230,6 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) return; } - if (_shredded) { - ensure_encoded(); - } if (!_typed && empty() && _metadatas->empty() && source._shredded) { // Gather into the native shredded representation for the same reason as range selection: // row selection does not require, and may not have, a complete logical Variant value. @@ -921,8 +1237,29 @@ void ColumnVariantV2::insert_indices_from( // NOLINT(readability-function-size) _check_invariants(); return; } + if (_shredded && source._shredded) { + auto selected_source = source._shredded->select_indices(indices_begin, indices_end); + if (_shredded.use_count() != 1) { + _shredded = _shredded->select_range(0, size()); + } + if (_shredded->try_append(*selected_source)) { + _check_invariants(); + return; + } + if (!_shredded->can_materialize() || !selected_source->can_materialize()) { + // Exchange gathers may mix complete files with projected files. Keep their boundaries + // because only complete segments are allowed to reconstruct the full logical root. + _shredded = combine_shredded_states(std::move(_shredded), std::move(selected_source)); + _check_invariants(); + return; + } + } + if (_shredded) { + // Indexed gathers have the same local/remote representation boundary as range gathers. + _ensure_serialized(); + } if (source._shredded) { - insert_indices_from(source._shredded->materialized_column(), indices_begin, indices_end); + insert_indices_from(source.serialization_column(), indices_begin, indices_end); return; } @@ -1367,7 +1704,16 @@ MutableColumnPtr ColumnVariantV2::permute(const Permutation& permutation, size_t } if (_shredded) { - return _shredded->materialized_column().permute(permutation, limit); + DorisVector selected_indices(result_size); + for (size_t row = 0; row < result_size; ++row) { + DORIS_CHECK_LE(permutation[row], std::numeric_limits::max()) + << "shredded Variant permutation index exceeds uint32 domain"; + selected_indices[row] = static_cast(permutation[row]); + } + // Local TopN selection may run before exchange and projected states cannot reconstruct + // omitted root fields, so preserve the native shredded representation while gathering. + return ColumnVariantV2::create_shredded(_shredded->select_indices( + selected_indices.data(), selected_indices.data() + selected_indices.size())); } if (_typed) { @@ -1407,6 +1753,11 @@ MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { result->_check_invariants(); return result; } + if (new_size < size()) { + // LIMIT truncation is a row selection and must not require complete Variant roots from + // a projected scanner state. + return ColumnVariantV2::create_shredded(_shredded->select_range(0, new_size)); + } return _shredded->materialized_column().clone_resized(new_size); } if (_typed) { @@ -1447,7 +1798,18 @@ MutableColumnPtr ColumnVariantV2::clone_resized(size_t new_size) const { void ColumnVariantV2::resize(size_t new_size) { const size_t old_size = size(); - if (_shredded && new_size != old_size) { + if (_shredded && new_size < old_size) { + // LIMIT truncation only selects existing rows, so keep it physical: projected states may + // omit roots that cannot be reconstructed merely to reduce the row count. + if (new_size == 0) { + _shredded.reset(); + } else { + _shredded = _shredded->select_range(0, new_size); + } + _check_invariants(); + return; + } + if (_shredded && new_size > old_size) { ensure_encoded(); } if (_typed) { @@ -1505,6 +1867,25 @@ uint32_t ColumnVariantV2::_find_or_insert_metadata(StringRef metadata) { return id; } +void ColumnVariantV2::_replace_shredded_state_with(const ColumnVariantV2& replacement) { + DORIS_CHECK(_shredded != nullptr) << "replacing shredded state requires a shredded destination"; + DORIS_CHECK(!replacement.is_shredded()) + << "shredded state replacement must be a non-shredded column"; + DORIS_CHECK_EQ(replacement.size(), size()) + << "shredded state replacement changed the row count"; + // Format states cache and share materialized columns. Detach every mutable buffer before + // dropping the state owner so later COW mutations cannot modify a cached representation. + _metadatas = replacement._metadatas->clone_resized(replacement._metadatas->size()); + _meta_ids = replacement._meta_ids->clone_resized(replacement._meta_ids->size()); + _values = replacement._values->clone_resized(replacement._values->size()); + _typed = replacement._typed == nullptr + ? nullptr + : replacement._typed->clone_resized(replacement._typed->size()); + _typed_type = replacement._typed_type; + _shredded.reset(); + _check_invariants(); +} + void ColumnVariantV2::_adopt_state_from(ColumnVariantV2& replacement) { DORIS_CHECK(this != &replacement) << "cannot adopt ColumnVariantV2 state from itself"; _metadatas = std::move(replacement._metadatas); diff --git a/be/src/core/column/variant_v2/column_variant_v2.h b/be/src/core/column/variant_v2/column_variant_v2.h index abe4df4b8223d4..a34f52aa3ba6a1 100644 --- a/be/src/core/column/variant_v2/column_variant_v2.h +++ b/be/src/core/column/variant_v2/column_variant_v2.h @@ -53,6 +53,9 @@ struct VariantShreddedTypedValue { // retain the decoded leaf without copying it or depending on scanner lifetime. ColumnPtr column; DataTypePtr type; + // Physical identities such as binary annotations cannot use the typed scalar state. In that + // case the format reader may return an exact Nullable leaf instead. + ColumnPtr normalized; }; // Format readers keep their native shredded representation behind this interface. Core Variant @@ -76,15 +79,26 @@ class VariantShreddedState { size_t length) const = 0; virtual std::shared_ptr select_indices( const uint32_t* indices_begin, const uint32_t* indices_end) const = 0; + // False means the state contains only projected leaves and cannot reconstruct root values. + virtual bool can_materialize() const = 0; // Appends another state only when both format-owned physical layouts have identical semantics. // An incompatible source must leave this state unchanged and return false. virtual bool try_append(const VariantShreddedState& source) = 0; virtual std::optional find_typed_value( std::span path) const = 0; + // Produces an exact Variant representation of one requested path. Complete states may fall + // back to their canonical roots; projected states must preserve the format's physical scalar + // identity rather than inferring it from the decoded value. + virtual std::optional find_normalized_value( + std::span path) const = 0; // The returned column is cached and owned by this state, so borrowed VariantRef values remain // valid for the state lifetime. Implementations must not materialize before this is called. virtual const ColumnVariantV2& materialized_column() const = 0; + // Whole-column transport may encode only the retained projection because access-path planning + // guarantees that omitted fields have no downstream consumer. The returned column must be a + // self-contained, non-shredded wire representation with the same row count. + virtual const ColumnVariantV2& serialized_column() const = 0; }; // ColumnVariantV2 stores a whole column in exactly one state: encoded Variant bytes, one nullable @@ -146,6 +160,7 @@ class ColumnVariantV2 final : public COWHelper { const DataTypePtr& typed_type() const; std::optional find_shredded_typed_value( std::span path) const; + const ColumnVariantV2& serialization_column() const; void ensure_encoded(); ReadView read_view() const; @@ -235,6 +250,8 @@ class ColumnVariantV2 final : public COWHelper { ColumnVariantV2(const ColumnVariantV2& other); uint32_t _find_or_insert_metadata(StringRef metadata); + void _replace_shredded_state_with(const ColumnVariantV2& replacement); + void _ensure_serialized(); void _adopt_state_from(ColumnVariantV2& replacement); void _detach_metadata_for_write(); void _check_invariants() const; diff --git a/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp b/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp index a9a137e4fea85b..57521e7f34aa0d 100644 --- a/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp +++ b/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp @@ -181,7 +181,7 @@ DataTypeVariantV2SerDe::DataTypeVariantV2SerDe(int nesting_level) : DataTypeSerD int64_t DataTypeVariantV2SerDe::get_uncompressed_serialized_bytes(const IColumn& column, int be_exec_version) { - const auto& variant = get_variant_v2_column(column); + const auto& variant = get_variant_v2_column(column).serialization_column(); int64_t size = sizeof(bool) + sizeof(size_t) * 2 + sizeof(bool); if (variant.is_typed()) { const DataTypePtr nullable_type = make_nullable(variant._typed_type); @@ -199,7 +199,7 @@ char* DataTypeVariantV2SerDe::serialize(const IColumn& column, char* buf, int be const IColumn* physical = &column; size_t saved_rows = 0; buf = serialize_const_flag_and_row_num(&physical, buf, &saved_rows); - const auto& variant = assert_cast(*physical); + const auto& variant = assert_cast(*physical).serialization_column(); DCHECK_EQ(variant.size(), saved_rows); unaligned_store(buf, variant.is_typed()); buf += sizeof(bool); diff --git a/be/src/core/value/variant/variant_batch_builder.cpp b/be/src/core/value/variant/variant_batch_builder.cpp index ca03c1e670b10e..fedec1a5bd63f5 100644 --- a/be/src/core/value/variant/variant_batch_builder.cpp +++ b/be/src/core/value/variant/variant_batch_builder.cpp @@ -602,10 +602,18 @@ class VariantCollectionCore { add_bool(value.get_bool()); return; case VariantPrimitiveId::INT8: + add_scalar(VariantScalarRef::integer(value.get_int(), 1)); + return; case VariantPrimitiveId::INT16: + add_scalar(VariantScalarRef::integer(value.get_int(), 2)); + return; case VariantPrimitiveId::INT32: + add_scalar(VariantScalarRef::integer(value.get_int(), 4)); + return; case VariantPrimitiveId::INT64: - add_int(value.get_int()); + // Importing an existing VariantRef must retain its physical identity; external + // shredded schemas use the width to distinguish otherwise equal scalar values. + add_scalar(VariantScalarRef::integer(value.get_int(), 8)); return; case VariantPrimitiveId::FLOAT: add_scalar(VariantScalarRef::float32(value.get_float())); @@ -621,7 +629,7 @@ class VariantCollectionCore { throw Exception(ErrorCode::CORRUPTION, "Variant imported decimal exceeds precision 38"); } - add_scalar(VariantScalarRef::decimal(decimal.unscaled, decimal.scale)); + add_scalar(VariantScalarRef::decimal(decimal.unscaled, decimal.scale, decimal.width)); return; } case VariantPrimitiveId::DATE: diff --git a/be/src/exprs/function/function_variant_element_v2.cpp b/be/src/exprs/function/function_variant_element_v2.cpp index 90863fe86c5c72..2c506c05377e65 100644 --- a/be/src/exprs/function/function_variant_element_v2.cpp +++ b/be/src/exprs/function/function_variant_element_v2.cpp @@ -152,7 +152,8 @@ std::optional extract_shredded_typed_variant_element( if (!match.has_value()) { return std::nullopt; } - const auto& leaf = assert_cast(*match->column); + const ColumnPtr& matched_column = match->normalized ? match->normalized : match->column; + const auto& leaf = assert_cast(*matched_column); auto nulls = leaf.get_null_map_column().clone_resized(source.size()); auto& null_data = assert_cast(*nulls).get_data(); for (size_t row = 0; row < source.size(); ++row) { @@ -160,6 +161,10 @@ std::optional extract_shredded_typed_variant_element( static_cast(null_data[row] != 0 || is_outer_null(outer_nulls, row)); } + if (match->normalized) { + return ColumnNullable::create(leaf.get_nested_column_ptr(), std::move(nulls)); + } + // The typed ColumnVariantV2 retains the exact decoded Parquet leaf. Only the SQL result null // map is produced here, so predicates and casts can consume the leaf without reconstructing // canonical Variant rows. diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index a2ca4894044404..1b1f2f284405f9 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -74,7 +74,10 @@ std::string FileScanRequest::debug_string() const { out << column_id << ":" << block_position; } out << "}, conjunct_count=" << conjuncts.size() - << ", delete_conjunct_count=" << delete_conjuncts.size() + << ", delete_conjunct_count=" << delete_conjuncts.size() << ", variant_schema_overrides=" + << join_debug_strings( + variant_schema_overrides, + [](const LocalColumnIndex& projection) { return projection.debug_string(); }) << ", count_star_placeholder_columns={"; const char* delimiter = ""; for (const auto column_id : count_star_placeholder_columns) { diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index ddbabf6329864d..1f269cde3e280f 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -98,6 +98,11 @@ struct FileScanRequest { // predicate_columns, the value is semantically required and must still be validated and read. std::vector count_star_placeholder_columns; + // Table formats may assign semantics that legacy physical files do not encode. Each path here + // identifies an unannotated Parquet group that the physical reader must validate and decode as + // Variant. Keeping this explicit prevents generic Parquet scans from guessing based on names. + std::vector variant_schema_overrides; + bool is_count_star_placeholder(LocalColumnId column_id) const { return std::ranges::find(count_star_placeholder_columns, column_id) != count_star_placeholder_columns.end(); diff --git a/be/src/format_v2/parquet/native_schema_desc.cpp b/be/src/format_v2/parquet/native_schema_desc.cpp index b56afb6f7170fb..16f4623bb1bafb 100644 --- a/be/src/format_v2/parquet/native_schema_desc.cpp +++ b/be/src/format_v2/parquet/native_schema_desc.cpp @@ -65,6 +65,8 @@ enum class VariantPrimitiveAnnotation : uint8_t { NONE, INT8, INT16, + INT32, + INT64, DECIMAL, DATE, TIME_MICROS, @@ -91,6 +93,15 @@ static VariantPrimitiveAnnotation variant_logical_annotation( if (logical.INTEGER.bitWidth == 16) { return VariantPrimitiveAnnotation::INT16; } + // Iceberg 1.11 writes full-width signed INTEGER annotations even though the Variant + // specification represents these widths without an annotation. Keep the widths distinct + // so validation only accepts them with the matching physical type. + if (logical.INTEGER.bitWidth == 32) { + return VariantPrimitiveAnnotation::INT32; + } + if (logical.INTEGER.bitWidth == 64) { + return VariantPrimitiveAnnotation::INT64; + } return VariantPrimitiveAnnotation::UNSUPPORTED; } if (logical.__isset.DECIMAL) { @@ -136,6 +147,11 @@ static VariantPrimitiveAnnotation variant_converted_annotation( return VariantPrimitiveAnnotation::INT8; case tparquet::ConvertedType::INT_16: return VariantPrimitiveAnnotation::INT16; + // Parquet Java mirrors full-width logical annotations into these legacy converted types. + case tparquet::ConvertedType::INT_32: + return VariantPrimitiveAnnotation::INT32; + case tparquet::ConvertedType::INT_64: + return VariantPrimitiveAnnotation::INT64; case tparquet::ConvertedType::DECIMAL: return VariantPrimitiveAnnotation::DECIMAL; case tparquet::ConvertedType::DATE: @@ -215,11 +231,13 @@ static Status validate_variant_primitive_type(const NativeFieldSchema& typed) { valid = annotation == VariantPrimitiveAnnotation::NONE || annotation == VariantPrimitiveAnnotation::INT8 || annotation == VariantPrimitiveAnnotation::INT16 || + annotation == VariantPrimitiveAnnotation::INT32 || annotation == VariantPrimitiveAnnotation::DECIMAL || annotation == VariantPrimitiveAnnotation::DATE; break; case tparquet::Type::INT64: valid = annotation == VariantPrimitiveAnnotation::NONE || + annotation == VariantPrimitiveAnnotation::INT64 || annotation == VariantPrimitiveAnnotation::DECIMAL || annotation == VariantPrimitiveAnnotation::TIME_MICROS || annotation == VariantPrimitiveAnnotation::TIMESTAMP_MICROS || @@ -266,17 +284,22 @@ class ScopedBoolOverride { bool _original; }; -static Status validate_variant_layout(const tparquet::SchemaElement& group_schema, - const NativeFieldSchema& group_field) { - const auto& annotation = group_schema.logicalType.VARIANT; - if (annotation.__isset.specification_version && annotation.specification_version != 1) { +Status validate_variant_layout(const NativeFieldSchema& group_field, + std::optional specification_version, + bool allow_optional_shredded_metadata) { + if (specification_version.has_value() && *specification_version != 1) { return Status::NotSupported("Parquet Variant specification version {} is not supported", - annotation.specification_version); + *specification_version); + } + if (group_field.parquet_schema.__isset.repetition_type && + group_field.parquet_schema.repetition_type == tparquet::FieldRepetitionType::REPEATED) { + return Status::NotSupported("repeated Parquet Variant group {} is not supported", + group_field.name); } if (group_field.children.size() < 2 || group_field.children.size() > 3) { return Status::Corruption( "Parquet Variant {} must contain metadata, value, and optional typed_value", - group_schema.name); + group_field.name); } const NativeFieldSchema* metadata = nullptr; @@ -292,22 +315,29 @@ static Status validate_variant_layout(const tparquet::SchemaElement& group_schem target = &typed_value; } else { return Status::Corruption("Parquet Variant {} has unexpected child {}", - group_schema.name, child.name); + group_field.name, child.name); } if (*target != nullptr) { - return Status::Corruption("Parquet Variant {} has duplicate child {}", - group_schema.name, child.name); + return Status::Corruption("Parquet Variant {} has duplicate child {}", group_field.name, + child.name); } *target = &child; } if (metadata == nullptr || value == nullptr) { return Status::Corruption("Parquet Variant {} requires metadata and value children", - group_schema.name); - } + group_field.name); + } + const auto metadata_repetition = metadata->parquet_schema.repetition_type; + // Paimon makes every field in its shredded carrier optional. Restrict that compatibility to + // unannotated overrides; row materialization still rejects null metadata for a non-null value. + const bool valid_metadata_repetition = + metadata_repetition == tparquet::FieldRepetitionType::REQUIRED || + (allow_optional_shredded_metadata && typed_value != nullptr && + metadata_repetition == tparquet::FieldRepetitionType::OPTIONAL); if (!metadata->children.empty() || metadata->physical_type != tparquet::Type::BYTE_ARRAY || - metadata->parquet_schema.repetition_type != tparquet::FieldRepetitionType::REQUIRED) { + !valid_metadata_repetition) { return Status::Corruption("Parquet Variant {} metadata must be a required BYTE_ARRAY", - group_schema.name); + group_field.name); } const auto expected_value_repetition = typed_value == nullptr ? tparquet::FieldRepetitionType::REQUIRED @@ -317,13 +347,13 @@ static Status validate_variant_layout(const tparquet::SchemaElement& group_schem if (!value->children.empty() || value->physical_type != tparquet::Type::BYTE_ARRAY || value->parquet_schema.repetition_type != expected_value_repetition) { return Status::Corruption("Parquet Variant {} value must be a {} BYTE_ARRAY", - group_schema.name, + group_field.name, typed_value == nullptr ? "required" : "optional"); } if (typed_value != nullptr && typed_value->parquet_schema.repetition_type != tparquet::FieldRepetitionType::OPTIONAL) { return Status::Corruption("Parquet Variant {} typed_value must be optional", - group_schema.name); + group_field.name); } enum class WrapperContext : uint8_t { OBJECT_FIELD, ARRAY_ELEMENT }; @@ -945,7 +975,12 @@ Status NativeFieldDescriptor::parse_group_field( ScopedBoolOverride timestamp_tz_mapping(_enable_mapping_timestamp_tz, true); RETURN_IF_ERROR(parse_struct_field(t_schemas, curr_pos, group_field)); } - RETURN_IF_ERROR(validate_variant_layout(group_schema, *group_field)); + const auto& annotation = group_schema.logicalType.VARIANT; + const auto specification_version = + annotation.__isset.specification_version + ? std::optional(annotation.specification_version) + : std::nullopt; + RETURN_IF_ERROR(validate_variant_layout(*group_field, specification_version)); group_field->variant_physical_type = group_field->data_type; // Native page readers dispatch groups from data_type, so preserve the physical STRUCT // here. The public Parquet schema maps it to logical Variant without losing this shape. diff --git a/be/src/format_v2/parquet/native_schema_desc.h b/be/src/format_v2/parquet/native_schema_desc.h index 918be6d2c65c58..be0e739f80b6ff 100644 --- a/be/src/format_v2/parquet/native_schema_desc.h +++ b/be/src/format_v2/parquet/native_schema_desc.h @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -86,6 +87,10 @@ struct NativeFieldSchema { uint64_t get_max_column_id() const; }; +Status validate_variant_layout(const NativeFieldSchema& group_field, + std::optional specification_version = std::nullopt, + bool allow_optional_shredded_metadata = false); + // V2 owns this schema tree and parser so footer/schema planning never invokes the V1 reader path. class NativeFieldDescriptor { private: diff --git a/be/src/format_v2/parquet/parquet_column_schema.cpp b/be/src/format_v2/parquet/parquet_column_schema.cpp index 71416e17dc9209..984e0259165074 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.cpp +++ b/be/src/format_v2/parquet/parquet_column_schema.cpp @@ -178,6 +178,41 @@ void propagate_native_max_levels(ParquetColumnSchema* schema) { } } +void rebuild_logical_complex_type(ParquetColumnSchema* schema) { + DORIS_CHECK(schema != nullptr); + schema->contains_variant = schema->kind == ParquetColumnSchemaKind::VARIANT; + for (const auto& child : schema->children) { + schema->contains_variant |= child->contains_variant; + } + if (schema->kind == ParquetColumnSchemaKind::VARIANT || !schema->contains_variant) { + return; + } + + DataTypePtr logical_type; + if (schema->kind == ParquetColumnSchemaKind::LIST) { + DORIS_CHECK(schema->children.size() == 1); + logical_type = std::make_shared(schema->children[0]->type); + } else if (schema->kind == ParquetColumnSchemaKind::MAP) { + DORIS_CHECK(schema->children.size() == 2); + logical_type = std::make_shared(make_nullable(schema->children[0]->type), + make_nullable(schema->children[1]->type)); + } else { + DORIS_CHECK(schema->kind == ParquetColumnSchemaKind::STRUCT); + DataTypes child_types; + Strings child_names; + child_types.reserve(schema->children.size()); + child_names.reserve(schema->children.size()); + for (const auto& child : schema->children) { + child_types.push_back(child->type); + child_names.push_back(child->name); + } + logical_type = + std::make_shared(std::move(child_types), std::move(child_names)); + } + schema->type = schema->type->is_nullable() ? make_nullable(std::move(logical_type)) + : std::move(logical_type); +} + std::unique_ptr build_native_node_schema(const NativeFieldSchema& field, int32_t local_id) { auto result = std::make_unique(); @@ -228,34 +263,51 @@ std::unique_ptr build_native_node_schema(const NativeFieldS } // A nested Variant changes its public child type from the physical STRUCT carrier. Rebuild // every enclosing complex type so file-block columns keep the same logical shape as readers. - if (result->kind != ParquetColumnSchemaKind::VARIANT && result->contains_variant) { - DataTypePtr logical_type; - if (result->kind == ParquetColumnSchemaKind::LIST) { - DORIS_CHECK(result->children.size() == 1); - logical_type = std::make_shared(result->children[0]->type); - } else if (result->kind == ParquetColumnSchemaKind::MAP) { - DORIS_CHECK(result->children.size() == 2); - logical_type = std::make_shared(make_nullable(result->children[0]->type), - make_nullable(result->children[1]->type)); - } else { - DataTypes child_types; - Strings child_names; - child_types.reserve(result->children.size()); - child_names.reserve(result->children.size()); - for (const auto& child : result->children) { - child_types.push_back(child->type); - child_names.push_back(child->name); - } - logical_type = std::make_shared(std::move(child_types), - std::move(child_names)); - } - result->type = result->type->is_nullable() ? make_nullable(std::move(logical_type)) - : std::move(logical_type); - } + rebuild_logical_complex_type(result.get()); propagate_native_max_levels(result.get()); return result; } +Status apply_variant_schema_override(const NativeFieldSchema& native_schema, + const format::LocalColumnIndex& override, + ParquetColumnSchema* field) { + DORIS_CHECK(field != nullptr); + if (field->local_id != override.local_id()) { + return Status::InvalidArgument("Variant schema override local id {} does not match {}", + override.local_id(), field->local_id); + } + if (override.project_all_children) { + if (field->kind == ParquetColumnSchemaKind::VARIANT) { + return Status::OK(); + } + if (field->kind != ParquetColumnSchemaKind::STRUCT) { + return Status::Corruption("Parquet Variant {} must use a group carrier", + native_schema.name); + } + RETURN_IF_ERROR(validate_variant_layout(native_schema, std::nullopt, true)); + field->variant_physical_type = field->type; + DataTypePtr variant_type = std::make_shared(); + field->type = field->type->is_nullable() ? make_nullable(std::move(variant_type)) + : std::move(variant_type); + field->kind = ParquetColumnSchemaKind::VARIANT; + field->contains_variant = true; + return Status::OK(); + } + for (const auto& child_override : override.children) { + const auto child_idx = child_override.local_id(); + if (child_idx < 0 || child_idx >= static_cast(field->children.size()) || + child_idx >= static_cast(native_schema.children.size())) { + return Status::InvalidArgument("Invalid nested Variant schema override {} under {}", + child_idx, field->name); + } + RETURN_IF_ERROR(apply_variant_schema_override(native_schema.children[child_idx], + child_override, + field->children[child_idx].get())); + } + rebuild_logical_complex_type(field); + return Status::OK(); +} + } // namespace Status build_parquet_column_schema(const NativeFieldDescriptor& schema, @@ -277,4 +329,24 @@ Status build_parquet_column_schema(const NativeFieldDescriptor& schema, return Status::OK(); } +Status apply_variant_schema_overrides( + const NativeFieldDescriptor& native_schema, + const std::vector& variant_schema_overrides, + std::vector>* fields) { + if (fields == nullptr) { + return Status::InvalidArgument("fields is null"); + } + const auto& native_fields = native_schema.get_fields_schema(); + for (const auto& override : variant_schema_overrides) { + const auto local_id = override.local_id(); + if (local_id < 0 || local_id >= static_cast(fields->size()) || + local_id >= static_cast(native_fields.size())) { + return Status::InvalidArgument("Invalid Variant schema override root {}", local_id); + } + RETURN_IF_ERROR(apply_variant_schema_override(native_fields[local_id], override, + (*fields)[local_id].get())); + } + return Status::OK(); +} + } // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/parquet_column_schema.h b/be/src/format_v2/parquet/parquet_column_schema.h index 11b5d2f15a35b1..20a9bb50b301d2 100644 --- a/be/src/format_v2/parquet/parquet_column_schema.h +++ b/be/src/format_v2/parquet/parquet_column_schema.h @@ -22,6 +22,7 @@ #include "common/status.h" #include "core/data_type/data_type.h" +#include "format_v2/column_data.h" #include "format_v2/parquet/parquet_type.h" namespace doris::format::parquet { @@ -83,4 +84,9 @@ struct ParquetColumnSchema { Status build_parquet_column_schema(const NativeFieldDescriptor& schema, std::vector>* fields); +Status apply_variant_schema_overrides( + const NativeFieldDescriptor& native_schema, + const std::vector& variant_schema_overrides, + std::vector>* fields); + } // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index 5fd3100a2e5bcf..ec70db46840771 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -22,6 +22,19 @@ namespace doris::format::parquet { +namespace { + +std::shared_ptr add_persistent_counter(RuntimeProfile* profile, + const std::string& name, + TUnit::type type, + const std::string& parent) { + // A shredded Variant may be materialized after its scanner profile is destroyed. Keep the + // counter storage alive with the column state instead of retaining a dangling profile pointer. + return profile->add_shared_counter(name, type, parent, 1); +} + +} // namespace + void ParquetProfile::init(RuntimeProfile* profile) { if (profile == nullptr) { return; @@ -90,18 +103,18 @@ void ParquetProfile::init(RuntimeProfile* profile) { ADD_CHILD_TIMER_WITH_LEVEL(profile, "LevelOnlySkipTime", parquet_profile, 1); materialization_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "MaterializationTime", parquet_profile, 1); - variant_reconstruction_time = - ADD_CHILD_TIMER_WITH_LEVEL(profile, "VariantReconstructionTime", parquet_profile, 1); - variant_reconstructed_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantReconstructedRows", - TUnit::UNIT, parquet_profile, 1); - variant_direct_leaf_rows = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "VariantDirectLeafRows", - TUnit::UNIT, parquet_profile, 1); - variant_direct_leaf_path_misses = ADD_CHILD_COUNTER_WITH_LEVEL( - profile, "VariantDirectLeafPathMisses", TUnit::UNIT, parquet_profile, 1); - variant_direct_leaf_residual_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( - profile, "VariantDirectLeafResidualFallbacks", TUnit::UNIT, parquet_profile, 1); - variant_direct_leaf_unsupported_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( - profile, "VariantDirectLeafUnsupportedFallbacks", TUnit::UNIT, parquet_profile, 1); + variant_reconstruction_time = add_persistent_counter(profile, "VariantReconstructionTime", + TUnit::TIME_NS, parquet_profile); + variant_reconstructed_rows = add_persistent_counter(profile, "VariantReconstructedRows", + TUnit::UNIT, parquet_profile); + variant_direct_leaf_rows = + add_persistent_counter(profile, "VariantDirectLeafRows", TUnit::UNIT, parquet_profile); + variant_direct_leaf_path_misses = add_persistent_counter(profile, "VariantDirectLeafPathMisses", + TUnit::UNIT, parquet_profile); + variant_direct_leaf_residual_fallbacks = add_persistent_counter( + profile, "VariantDirectLeafResidualFallbacks", TUnit::UNIT, parquet_profile); + variant_direct_leaf_unsupported_fallbacks = add_persistent_counter( + profile, "VariantDirectLeafUnsupportedFallbacks", TUnit::UNIT, parquet_profile); hybrid_selection_batches = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionBatches", TUnit::UNIT, parquet_profile, 1); hybrid_selection_ranges = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "HybridSelectionRanges", diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index ed1faa8f935134..764fef1d80c190 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -15,6 +15,8 @@ #pragma once +#include + #include "runtime/runtime_profile.h" namespace doris::format::parquet { @@ -38,12 +40,12 @@ struct ParquetColumnReaderProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; // value materialization time (ns) - RuntimeProfile::Counter* variant_reconstruction_time = nullptr; - RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; + std::shared_ptr variant_reconstruction_time; + std::shared_ptr variant_reconstructed_rows; + std::shared_ptr variant_direct_leaf_rows; + std::shared_ptr variant_direct_leaf_path_misses; + std::shared_ptr variant_direct_leaf_residual_fallbacks; + std::shared_ptr variant_direct_leaf_unsupported_fallbacks; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; @@ -174,12 +176,12 @@ struct ParquetProfile { RuntimeProfile::Counter* level_only_read_time = nullptr; RuntimeProfile::Counter* level_only_skip_time = nullptr; RuntimeProfile::Counter* materialization_time = nullptr; - RuntimeProfile::Counter* variant_reconstruction_time = nullptr; - RuntimeProfile::Counter* variant_reconstructed_rows = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_rows = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_path_misses = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_residual_fallbacks = nullptr; - RuntimeProfile::Counter* variant_direct_leaf_unsupported_fallbacks = nullptr; + std::shared_ptr variant_reconstruction_time; + std::shared_ptr variant_reconstructed_rows; + std::shared_ptr variant_direct_leaf_rows; + std::shared_ptr variant_direct_leaf_path_misses; + std::shared_ptr variant_direct_leaf_residual_fallbacks; + std::shared_ptr variant_direct_leaf_unsupported_fallbacks; RuntimeProfile::Counter* hybrid_selection_batches = nullptr; RuntimeProfile::Counter* hybrid_selection_ranges = nullptr; RuntimeProfile::Counter* hybrid_selection_null_fallback_batches = nullptr; diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index dcb46d9b5ba643..1502616511a3a4 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -334,6 +334,15 @@ DataTypePtr apply_timestamp_tz_mapping(ParquetColumnSchema* column_schema) { } column_schema->type = nullable_like_original( column_schema->type, std::make_shared(child_types, child_names)); + } else if (column_schema->kind == ParquetColumnSchemaKind::VARIANT) { + Strings child_names; + child_names.reserve(column_schema->children.size()); + for (const auto& child : column_schema->children) { + child_names.push_back(child->name); + } + column_schema->variant_physical_type = + nullable_like_original(column_schema->variant_physical_type, + std::make_shared(child_types, child_names)); } return column_schema->type; } @@ -415,6 +424,19 @@ void apply_request_timestamp_semantics( } } +void apply_timestamp_tz_mapping_in_variants(ParquetColumnSchema* column_schema) { + DORIS_CHECK(column_schema != nullptr); + if (column_schema->kind == ParquetColumnSchemaKind::VARIANT) { + // Shredded Variant timestamps always represent instants, even when the surrounding table + // did not request catalog-level TIMESTAMPTZ mapping for ordinary Parquet columns. + apply_timestamp_tz_mapping(column_schema); + return; + } + for (auto& child : column_schema->children) { + apply_timestamp_tz_mapping_in_variants(child.get()); + } +} + static Status find_projected_minmax_leaf(const ParquetColumnSchema& column_schema, const format::LocalColumnIndex& projection, const ParquetColumnSchema** leaf_schema) { @@ -616,6 +638,21 @@ Status ParquetReader::open(std::shared_ptr request) { } auto request_snapshot = request; DORIS_CHECK(request_snapshot != nullptr); + if (!request_snapshot->variant_schema_overrides.empty()) { + // Apply table-format semantics before Variant projection planning. The override is the + // explicit proof that an otherwise ordinary Parquet group is a Variant carrier. + RETURN_IF_ERROR(apply_variant_schema_overrides( + _state->file_context.native_metadata->schema(), + request_snapshot->variant_schema_overrides, &_state->file_schema)); + for (auto& column_schema : _state->file_schema) { + apply_timestamp_tz_mapping_in_variants(column_schema.get()); + } + _state->file_context.contains_variant = + std::ranges::any_of(_state->file_schema, [](const auto& column) { + DORIS_CHECK(column != nullptr); + return column->contains_variant; + }); + } size_t retained_variant_leaf_projections = 0; if (_state->file_context.contains_variant) { retained_variant_leaf_projections = diff --git a/be/src/format_v2/parquet/reader/variant_column_reader.cpp b/be/src/format_v2/parquet/reader/variant_column_reader.cpp index da5ecbb8310a20..4600a6a9069191 100644 --- a/be/src/format_v2/parquet/reader/variant_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/variant_column_reader.cpp @@ -398,7 +398,7 @@ bool append_wrapper(const ParquetColumnSchema& schema, const IColumn& wrapper, s void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrapper, const ColumnNullable* outer_nullable, size_t begin, size_t end, - ColumnVariantV2& variants) { + bool require_metadata, ColumnVariantV2& variants) { try { VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = end - begin}); for (size_t row = begin; row < end; ++row) { @@ -408,14 +408,22 @@ void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrap output_row.finish(); continue; } - const Cell metadata_cell = struct_child_at(schema, wrapper, row, "metadata", nullptr); - if (metadata_cell.is_null) { - throw Exception(ErrorCode::CORRUPTION, - "Parquet Variant {} has null metadata at row {}", schema.name, row); + VariantMetadataRef metadata; + if (find_child(schema, "metadata", nullptr) != nullptr) { + const Cell metadata_cell = + struct_child_at(schema, wrapper, row, "metadata", nullptr); + if (metadata_cell.is_null) { + throw Exception(ErrorCode::CORRUPTION, + "Parquet Variant {} has null metadata at row {}", schema.name, + row); + } + const StringRef metadata_bytes = metadata_cell.column->get_data_at(row); + metadata = {metadata_bytes.data, metadata_bytes.size}; + metadata.validate(); + } else if (require_metadata) { + throw Exception(ErrorCode::CORRUPTION, "Parquet Variant {} has no root metadata", + schema.name); } - const StringRef metadata_bytes = metadata_cell.column->get_data_at(row); - const VariantMetadataRef metadata {metadata_bytes.data, metadata_bytes.size}; - metadata.validate(); (void)append_wrapper(schema, wrapper, row, metadata, output_row, WrapperContext::ROOT); output_row.finish(); } @@ -429,13 +437,16 @@ void encode_variant_range(const ParquetColumnSchema& schema, const IColumn& wrap // that dictionary, split without changing the destination column's already-valid batches. // Corrupt input still reaches a one-row range and propagates its original exception. const size_t middle = begin + (end - begin) / 2; - encode_variant_range(schema, wrapper, outer_nullable, begin, middle, variants); - encode_variant_range(schema, wrapper, outer_nullable, middle, end, variants); + encode_variant_range(schema, wrapper, outer_nullable, begin, middle, require_metadata, + variants); + encode_variant_range(schema, wrapper, outer_nullable, middle, end, require_metadata, + variants); } } ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& schema, - const IColumn& physical) { + const IColumn& physical, + bool require_metadata = true) { if (schema.kind != ParquetColumnSchemaKind::VARIANT) { throw Exception(ErrorCode::INVALID_ARGUMENT, "Parquet column {} is not Variant", schema.name); @@ -454,7 +465,7 @@ ColumnVariantV2::MutablePtr encode_variant_column(const ParquetColumnSchema& sch for (size_t begin = 0; begin < physical.size(); begin += MAX_RECONSTRUCTION_BATCH_ROWS) { encode_variant_range(schema, wrapper, outer_nullable, begin, std::min(physical.size(), begin + MAX_RECONSTRUCTION_BATCH_ROWS), - *variants); + require_metadata, *variants); } return variants; } @@ -553,6 +564,72 @@ bool supports_direct_typed_variant_state(const ParquetColumnSchema& schema) { } } +ColumnPtr normalize_projected_primitive_leaf(const ParquetColumnSchema& schema, + const ColumnPtr& typed) { + const auto& nullable = assert_cast(*typed); + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = nullable.size()}); + for (size_t row = 0; row < nullable.size(); ++row) { + auto output_row = builder.begin_row(); + if (nullable.get_null_map_data()[row] != 0) { + output_row.add_null(); + } else { + append_typed_scalar(schema, nullable.get_nested_column(), row, output_row); + } + output_row.finish(); + } + auto values = ColumnVariantV2::create(); + values->insert_encoded_batch(builder.finish_batch()); + auto nulls = nullable.get_null_map_column().clone_resized(nullable.size()); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + +bool find_materialized_path(VariantRef current, std::span path, + VariantRef* output) { + DORIS_CHECK(output != nullptr); + for (const auto& segment : path) { + if (segment.kind == VariantShreddedPathSegment::Kind::OBJECT_KEY) { + if (current.basic_type() != VariantBasicType::OBJECT || + !current.object_find(segment.key, ¤t)) { + return false; + } + continue; + } + if (current.basic_type() != VariantBasicType::ARRAY) { + return false; + } + const int64_t count = current.num_elements(); + const int64_t index = segment.index < 0 ? count + segment.index : segment.index; + if (index < 0 || index >= count) { + return false; + } + current = current.array_at(static_cast(index)); + } + *output = current; + return true; +} + +ColumnPtr normalize_materialized_path(const ColumnVariantV2& materialized, + std::span path) { + VariantBatchBuilder builder(VariantBatchBuilder::ReserveHint {.rows = materialized.size()}); + auto nulls = ColumnUInt8::create(); + nulls->reserve(materialized.size()); + for (size_t row = 0; row < materialized.size(); ++row) { + auto output_row = builder.begin_row(); + VariantRef value; + if (find_materialized_path(materialized.get_value_ref(row), path, &value)) { + output_row.add_value(value); + nulls->insert_value(0); + } else { + output_row.add_null(); + nulls->insert_value(1); + } + output_row.finish(); + } + auto values = ColumnVariantV2::create(); + values->insert_encoded_batch(builder.finish_batch()); + return ColumnNullable::create(std::move(values), std::move(nulls)); +} + bool same_data_type(const DataTypePtr& left, const DataTypePtr& right) { return (!left && !right) || (left && right && left->equals(*right)); } @@ -612,12 +689,14 @@ class ParquetVariantShreddedState final : public VariantShreddedState { size_t size() const override { return _physical->size(); } size_t byte_size() const override { std::lock_guard lock(_materialization_lock); - return _physical->byte_size() + (_materialized ? _materialized->byte_size() : 0); + return _physical->byte_size() + (_materialized ? _materialized->byte_size() : 0) + + (_serialized ? _serialized->byte_size() : 0); } size_t allocated_bytes() const override { std::lock_guard lock(_materialization_lock); return _physical->allocated_bytes() + - (_materialized ? _materialized->allocated_bytes() : 0); + (_materialized ? _materialized->allocated_bytes() : 0) + + (_serialized ? _serialized->allocated_bytes() : 0); } void sanity_check() const override { _physical->sanity_check(); } @@ -648,6 +727,8 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _complete, _profile); } + bool can_materialize() const override { return _complete; } + bool try_append(const VariantShreddedState& source) override { const auto* parquet_source = dynamic_cast(&source); if (parquet_source == nullptr || _complete != parquet_source->_complete || @@ -660,6 +741,7 @@ class ParquetVariantShreddedState final : public VariantShreddedState { _physical = std::move(mutable_physical); std::lock_guard lock(_materialization_lock); _materialized.reset(); + _serialized.reset(); return true; } @@ -705,15 +787,29 @@ class ParquetVariantShreddedState final : public VariantShreddedState { } if (position + 1 == path.size()) { if (typed_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || - check_and_get_column(*typed) == nullptr || - !supports_direct_typed_variant_state(*typed_schema)) { + check_and_get_column(*typed) == nullptr) { update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); return std::nullopt; } + if (!supports_direct_typed_variant_state(*typed_schema)) { + if (_complete) { + update_counter(_profile.variant_direct_leaf_unsupported_fallbacks, 1); + return std::nullopt; + } + // A partial projection cannot reconstruct its root. Normalize only the exact + // requested leaf so Parquet annotations survive heterogeneous file schemas. + update_counter(_profile.variant_direct_leaf_rows, + static_cast(typed->size())); + return VariantShreddedTypedValue { + .column = nullptr, + .type = nullptr, + .normalized = normalize_projected_primitive_leaf(*typed_schema, typed)}; + } update_counter(_profile.variant_direct_leaf_rows, static_cast(typed->size())); return VariantShreddedTypedValue {.column = std::move(typed), - .type = remove_nullable(typed_schema->type)}; + .type = remove_nullable(typed_schema->type), + .normalized = nullptr}; } if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { return path_miss(); @@ -722,6 +818,56 @@ class ParquetVariantShreddedState final : public VariantShreddedState { return std::nullopt; } + std::optional find_normalized_value( + std::span path) const override { + if (path.empty()) { + return std::nullopt; + } + + const ParquetColumnSchema* typed_schema = nullptr; + ColumnPtr typed = struct_child(*_schema, _physical, "typed_value", &typed_schema); + if (typed && typed_schema->kind == ParquetColumnSchemaKind::STRUCT) { + bool direct = true; + for (size_t position = 0; position < path.size(); ++position) { + if (path[position].kind != VariantShreddedPathSegment::Kind::OBJECT_KEY) { + direct = false; + break; + } + const std::string_view key(path[position].key.data, path[position].key.size); + const ParquetColumnSchema* wrapper_schema = nullptr; + ColumnPtr wrapper = struct_child(*typed_schema, typed, key, &wrapper_schema); + if (!wrapper) { + direct = false; + break; + } + ColumnPtr residual = struct_child(*wrapper_schema, wrapper, "value", nullptr); + if (residual && has_present_value(residual)) { + direct = false; + break; + } + typed = struct_child(*wrapper_schema, wrapper, "typed_value", &typed_schema); + if (!typed) { + direct = false; + break; + } + if (position + 1 == path.size()) { + direct = typed_schema->kind == ParquetColumnSchemaKind::PRIMITIVE && + check_and_get_column(*typed) != nullptr; + } else if (typed_schema->kind != ParquetColumnSchemaKind::STRUCT) { + direct = false; + break; + } + } + if (direct) { + return normalize_projected_primitive_leaf(*typed_schema, typed); + } + } + if (!_complete) { + return std::nullopt; + } + return normalize_materialized_path(materialized_column(), path); + } + const ColumnVariantV2& materialized_column() const override { std::lock_guard lock(_materialization_lock); if (!_complete) { @@ -730,7 +876,7 @@ class ParquetVariantShreddedState final : public VariantShreddedState { "A projected Parquet Variant can only serve its validated shredded leaves"); } if (!_materialized) { - SCOPED_TIMER(_profile.variant_reconstruction_time); + SCOPED_TIMER(_profile.variant_reconstruction_time.get()); _materialized = encode_variant_column(*_schema, *_physical); update_counter(_profile.variant_reconstructed_rows, static_cast(_physical->size())); @@ -738,10 +884,25 @@ class ParquetVariantShreddedState final : public VariantShreddedState { return *_materialized; } + const ColumnVariantV2& serialized_column() const override { + if (_complete) { + return materialized_column(); + } + std::lock_guard lock(_materialization_lock); + if (!_serialized) { + // Projected states intentionally omit root metadata, but an exchange buffer still + // needs self-contained bytes. Rebuild only retained paths; access-path planning is the + // invariant that prevents a downstream consumer from observing an omitted field. + _serialized = encode_variant_column(*_schema, *_physical, false); + } + return *_serialized; + } + private: - static void update_counter(RuntimeProfile::Counter* counter, int64_t value) { + static void update_counter(const std::shared_ptr& counter, + int64_t value) { if (counter != nullptr) { - COUNTER_UPDATE(counter, value); + COUNTER_UPDATE(counter.get(), value); } } @@ -751,6 +912,7 @@ class ParquetVariantShreddedState final : public VariantShreddedState { ParquetColumnReaderProfile _profile; mutable std::mutex _materialization_lock; mutable ColumnVariantV2::MutablePtr _materialized; + mutable ColumnVariantV2::MutablePtr _serialized; }; MutableColumnPtr build_variant_column(std::shared_ptr schema, diff --git a/be/src/format_v2/table/paimon_reader.cpp b/be/src/format_v2/table/paimon_reader.cpp index 7ec276ebe105a1..5534e5fe2d87fe 100644 --- a/be/src/format_v2/table/paimon_reader.cpp +++ b/be/src/format_v2/table/paimon_reader.cpp @@ -19,6 +19,8 @@ #include +#include +#include #include #include @@ -27,6 +29,7 @@ #include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" #include "exprs/vexpr_context.h" #include "format/table/deletion_vector_reader.h" #include "format/table/paimon_reader.h" @@ -79,6 +82,145 @@ DataTypePtr apply_paimon_timestamp_semantics(format::ColumnDefinition* column) { return column->type; } +ColumnDefinition* find_file_column(const ColumnDefinition& table_column, + std::vector* file_schema, + TableColumnMappingMode mode) { + DORIS_CHECK(file_schema != nullptr); + if (mode == TableColumnMappingMode::BY_FIELD_ID) { + if (!table_column.has_identifier_field_id()) { + return nullptr; + } + const auto field_id = table_column.get_identifier_field_id(); + const auto it = std::ranges::find_if(*file_schema, [&](const auto& file_column) { + return file_column.has_identifier_field_id() && + file_column.get_identifier_field_id() == field_id; + }); + return it == file_schema->end() ? nullptr : &*it; + } + const auto* matched = format::find_column_by_name(table_column, *file_schema); + return matched == nullptr ? nullptr : &(*file_schema)[matched - file_schema->data()]; +} + +void rebuild_complex_type(ColumnDefinition* column) { + DORIS_CHECK(column != nullptr); + const bool nullable = column->type->is_nullable(); + const auto primitive = remove_nullable(column->type)->get_primitive_type(); + DataTypePtr rebuilt; + if (primitive == TYPE_ARRAY && column->children.size() == 1) { + rebuilt = std::make_shared(column->children[0].type); + } else if (primitive == TYPE_MAP && column->children.size() == 2) { + rebuilt = std::make_shared(column->children[0].type, column->children[1].type); + } else if (primitive == TYPE_STRUCT) { + DataTypes child_types; + Strings child_names; + child_types.reserve(column->children.size()); + child_names.reserve(column->children.size()); + for (const auto& child : column->children) { + child_types.push_back(child.type); + child_names.push_back(child.name); + } + rebuilt = std::make_shared(std::move(child_types), std::move(child_names)); + } + if (rebuilt != nullptr) { + column->type = nullable ? make_nullable(std::move(rebuilt)) : std::move(rebuilt); + } +} + +Status add_variant_schema_override(const std::vector& path, + std::vector* overrides) { + DORIS_CHECK(!path.empty()); + DORIS_CHECK(overrides != nullptr); + auto projection = LocalColumnIndex::local(path.back()); + for (size_t path_idx = path.size() - 1; path_idx > 0; --path_idx) { + auto parent = LocalColumnIndex::partial_local(path[path_idx - 1]); + parent.children.push_back(std::move(projection)); + projection = std::move(parent); + } + const auto existing = std::ranges::find_if(*overrides, [&](const auto& override) { + return override.local_id() == projection.local_id(); + }); + if (existing == overrides->end()) { + overrides->push_back(std::move(projection)); + } else { + RETURN_IF_ERROR(merge_local_column_index(&*existing, projection)); + } + return Status::OK(); +} + +bool contains_variant_type(const ColumnDefinition& column) { + if (column.type != nullptr && + remove_nullable(column.type)->get_primitive_type() == TYPE_VARIANT) { + return true; + } + return std::ranges::any_of(column.children, contains_variant_type); +} + +Status annotate_matched_paimon_variant(const ColumnDefinition& table_column, + ColumnDefinition* file_column, TableColumnMappingMode mode, + const std::vector& prefix, + std::vector* overrides) { + DORIS_CHECK(file_column != nullptr); + if (!contains_variant_type(table_column) || table_column.type == nullptr || + file_column->type == nullptr) { + return Status::OK(); + } + auto path = prefix; + path.push_back(file_column->local_id); + const auto table_primitive = remove_nullable(table_column.type)->get_primitive_type(); + const auto file_primitive = remove_nullable(file_column->type)->get_primitive_type(); + if (table_primitive == TYPE_VARIANT) { + if (file_primitive == TYPE_STRUCT) { + // Paimon omits the Parquet VARIANT annotation, so only a matched table Variant may + // reinterpret this carrier; ordinary STRUCT must stay a STRUCT. + DataTypePtr variant = std::make_shared(); + file_column->type = file_column->type->is_nullable() ? make_nullable(std::move(variant)) + : std::move(variant); + RETURN_IF_ERROR(add_variant_schema_override(path, overrides)); + } + return Status::OK(); + } + if (table_column.children.empty() || file_column->children.empty() || + table_primitive != file_primitive) { + return Status::OK(); + } + if (table_primitive == TYPE_ARRAY || table_primitive == TYPE_MAP) { + const auto child_count = + std::min(table_column.children.size(), file_column->children.size()); + for (size_t child_idx = 0; child_idx < child_count; ++child_idx) { + // ARRAY/MAP child names are writer-specific structural labels, so match these nodes by + // position and reserve name/field-id matching for actual STRUCT members. + RETURN_IF_ERROR(annotate_matched_paimon_variant(table_column.children[child_idx], + &file_column->children[child_idx], mode, + path, overrides)); + } + } else if (table_primitive == TYPE_STRUCT) { + for (const auto& table_child : table_column.children) { + auto* file_child = find_file_column(table_child, &file_column->children, mode); + if (file_child != nullptr) { + RETURN_IF_ERROR(annotate_matched_paimon_variant(table_child, file_child, mode, path, + overrides)); + } + } + } + rebuild_complex_type(file_column); + return Status::OK(); +} + +Status annotate_paimon_variants(const std::vector& table_schema, + std::vector* file_schema, + TableColumnMappingMode mode, + std::vector* overrides) { + DORIS_CHECK(file_schema != nullptr); + for (const auto& table_column : table_schema) { + auto* file_column = find_file_column(table_column, file_schema, mode); + if (file_column != nullptr) { + RETURN_IF_ERROR(annotate_matched_paimon_variant(table_column, file_column, mode, {}, + overrides)); + } + } + return Status::OK(); +} + } // namespace Status PaimonReader::prepare_split(const format::SplitReadOptions& options) { @@ -115,19 +257,33 @@ format::TableColumnMappingMode PaimonReader::mapping_mode() const { Status PaimonReader::annotate_file_schema(std::vector* file_schema) { DORIS_CHECK(file_schema != nullptr); - if (mapping_mode() != format::TableColumnMappingMode::BY_FIELD_ID) { - return Status::OK(); + _variant_schema_overrides.clear(); + const auto mode = mapping_mode(); + if (mode == format::TableColumnMappingMode::BY_FIELD_ID) { + RETURN_IF_ERROR(format::annotate_file_schema_from_history(_scan_params, _split_schema_id, + file_schema)); } - RETURN_IF_ERROR( - format::annotate_file_schema_from_history(_scan_params, _split_schema_id, file_schema)); if (_format == format::FileFormat::PARQUET) { for (auto& column : *file_schema) { apply_paimon_timestamp_semantics(&column); } + const bool projects_variant = + std::ranges::any_of(_projected_columns, contains_variant_type); + if (projects_variant) { + RETURN_IF_ERROR(annotate_paimon_variants(_projected_columns, file_schema, mode, + &_variant_schema_overrides)); + } } return Status::OK(); } +Status PaimonReader::customize_file_scan_request(format::FileScanRequest* file_request) { + DORIS_CHECK(file_request != nullptr); + RETURN_IF_ERROR(format::TableReader::customize_file_scan_request(file_request)); + file_request->variant_schema_overrides = _variant_schema_overrides; + return Status::OK(); +} + Status PaimonReader::_parse_deletion_vector_file(const TTableFormatFileDesc& t_desc, DeleteFileDesc* desc, bool* has_delete_file) { DORIS_CHECK(desc != nullptr); diff --git a/be/src/format_v2/table/paimon_reader.h b/be/src/format_v2/table/paimon_reader.h index 8570f2efba624e..ed2b9e75c1c722 100644 --- a/be/src/format_v2/table/paimon_reader.h +++ b/be/src/format_v2/table/paimon_reader.h @@ -35,10 +35,17 @@ class PaimonReader final : public format::TableReader { #ifdef BE_TEST void TEST_set_scan_params(TFileScanRangeParams* params) { _scan_params = params; } + void TEST_set_projected_columns(std::vector columns) { + _projected_columns = std::move(columns); + } + void TEST_set_format(format::FileFormat format) { _format = format; } format::TableColumnMappingMode TEST_mapping_mode() const { return mapping_mode(); } Status TEST_annotate_file_schema(std::vector* file_schema) { return annotate_file_schema(file_schema); } + Status TEST_customize_file_scan_request(format::FileScanRequest* request) { + return customize_file_scan_request(request); + } Status TEST_parse_deletion_vector_file(const TTableFormatFileDesc& t_desc, DeleteFileDesc* desc, bool* has_delete_file) { return _parse_deletion_vector_file(t_desc, desc, has_delete_file); @@ -48,12 +55,14 @@ class PaimonReader final : public format::TableReader { protected: format::TableColumnMappingMode mapping_mode() const override; Status annotate_file_schema(std::vector* file_schema) override; + Status customize_file_scan_request(format::FileScanRequest* file_request) override; Status _parse_deletion_vector_file(const TTableFormatFileDesc& t_desc, DeleteFileDesc* desc, bool* has_delete_file) override; private: int64_t _split_schema_id = -1; + std::vector _variant_schema_overrides; }; // Paimon scans can contain both native data-file splits and serialized JNI splits in the same diff --git a/be/src/runtime/runtime_profile.cpp b/be/src/runtime/runtime_profile.cpp index 36bc79b82157b1..c2db5237fbed37 100644 --- a/be/src/runtime/runtime_profile.cpp +++ b/be/src/runtime/runtime_profile.cpp @@ -520,6 +520,28 @@ RuntimeProfile::Counter* RuntimeProfile::add_counter(const std::string& name, TU return counter; } +std::shared_ptr RuntimeProfile::add_shared_counter( + const std::string& name, TUnit::type type, const std::string& parent_counter_name, + int64_t level) { + std::lock_guard l(_counter_map_lock); + + if (auto it = _shared_counter_pool.find(name); it != _shared_counter_pool.end()) { + DCHECK_EQ(it->second->type(), type); + return it->second; + } + + // A raw counter with the same name cannot be safely upgraded because external users may + // already hold its profile-owned address. + DCHECK(_counter_map.find(name) == _counter_map.end()); + DCHECK(parent_counter_name == ROOT_COUNTER || + _counter_map.find(parent_counter_name) != _counter_map.end()); + auto counter = std::make_shared(type, 0, level); + _shared_counter_pool.emplace(name, counter); + _counter_map[name] = counter.get(); + _child_counter_map[parent_counter_name].insert(name); + return counter; +} + RuntimeProfile::NonZeroCounter* RuntimeProfile::add_nonzero_counter( const std::string& name, TUnit::type type, const std::string& parent_counter_name, int64_t level) { diff --git a/be/src/runtime/runtime_profile.h b/be/src/runtime/runtime_profile.h index 54f2e80c89e32d..a7e351c4c447a9 100644 --- a/be/src/runtime/runtime_profile.h +++ b/be/src/runtime/runtime_profile.h @@ -614,6 +614,13 @@ class RuntimeProfile { return add_counter(name, type, RuntimeProfile::ROOT_COUNTER, level); } + // Add a counter whose storage may outlive this profile. Repeated registration returns the same + // shared counter, matching add_counter() semantics for reused scanner profiles. + std::shared_ptr add_shared_counter( + const std::string& name, TUnit::type type, + const std::string& parent_counter_name = RuntimeProfile::ROOT_COUNTER, + int64_t level = 2); + NonZeroCounter* add_nonzero_counter( const std::string& name, TUnit::type type, const std::string& parent_counter_name = RuntimeProfile::ROOT_COUNTER, @@ -737,7 +744,7 @@ class RuntimeProfile { std::unique_ptr _pool; // Pool for allocated counters. These counters are shared with some other objects. - std::map> _shared_counter_pool; + std::map> _shared_counter_pool; // Name for this runtime profile. std::string _name; diff --git a/be/test/core/column/column_variant_v2_test.cpp b/be/test/core/column/column_variant_v2_test.cpp index 9937555b96d7c8..f496f02576525a 100644 --- a/be/test/core/column/column_variant_v2_test.cpp +++ b/be/test/core/column/column_variant_v2_test.cpp @@ -220,6 +220,56 @@ struct OwnedEncodedData { } }; +class CountingShreddedState final : public VariantShreddedState { +public: + CountingShreddedState(size_t rows, std::shared_ptr size_calls) + : _rows(rows), + _size_calls(std::move(size_calls)), + _serialized(ColumnVariantV2::create()) { + _serialized->insert_many_defaults(rows); + } + + size_t size() const override { + ++*_size_calls; + return _rows; + } + size_t byte_size() const override { return 0; } + size_t allocated_bytes() const override { return 0; } + void sanity_check() const override {} + void for_each_subcolumn(IColumn::ColumnCallback) const override {} + std::shared_ptr filter(const IColumn::Filter& filter, + ssize_t) const override { + return std::make_shared( + std::count(filter.begin(), filter.end(), UInt8 {1}), _size_calls); + } + std::shared_ptr select_range(size_t, size_t length) const override { + return std::make_shared(length, _size_calls); + } + std::shared_ptr select_indices( + const uint32_t* indices_begin, const uint32_t* indices_end) const override { + return std::make_shared(indices_end - indices_begin, _size_calls); + } + bool can_materialize() const override { return false; } + bool try_append(const VariantShreddedState&) override { return false; } + std::optional find_typed_value( + std::span) const override { + return std::nullopt; + } + std::optional find_normalized_value( + std::span) const override { + return std::nullopt; + } + const ColumnVariantV2& materialized_column() const override { + throw Exception(ErrorCode::INTERNAL_ERROR, "counting shredded state cannot materialize"); + } + const ColumnVariantV2& serialized_column() const override { return *_serialized; } + +private: + size_t _rows; + std::shared_ptr _size_calls; + ColumnVariantV2::MutablePtr _serialized; +}; + template void expect_not_implemented(Function&& function, std::string_view marker) { try { @@ -1250,6 +1300,23 @@ TEST(ColumnVariantV2Test, PermuteMatchesColumnStringAndRejectsInvalidInputs) { expect_values_match(*source, *reference); } +TEST(ColumnVariantV2Test, CompositeShreddedSizeDoesNotRecountSegments) { + auto size_calls = std::make_shared(0); + auto first = ColumnVariantV2::create_shredded( + std::make_shared(2, size_calls)); + auto second = ColumnVariantV2::create_shredded( + std::make_shared(3, size_calls)); + auto composite = ColumnVariantV2::create(); + composite->insert_range_from(*first, 0, first->size()); + composite->insert_range_from(*second, 0, second->size()); + + *size_calls = 0; + for (size_t iteration = 0; iteration < 32; ++iteration) { + EXPECT_EQ(composite->size(), 5); + } + EXPECT_EQ(*size_calls, 0); +} + TEST(ColumnVariantV2Test, PopBackAndResizeCoverBoundsShrinkAndGrowth) { auto pop_column = ColumnVariantV2::create(); auto pop_reference = ColumnString::create(); diff --git a/be/test/format_v2/parquet/parquet_schema_test.cpp b/be/test/format_v2/parquet/parquet_schema_test.cpp index acd5e8600789ec..a92a4dee22842e 100644 --- a/be/test/format_v2/parquet/parquet_schema_test.cpp +++ b/be/test/format_v2/parquet/parquet_schema_test.cpp @@ -227,6 +227,97 @@ TEST(ParquetSchemaTest, NativeSchemaRecognizesVariantLogicalGroup) { } } +TEST(ParquetSchemaTest, AppliesTableFormatVariantOverrideToUnannotatedGroup) { + auto schema = unshredded_variant_schema(); + schema[1].__isset.logicalType = false; + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + + std::vector> fields; + ASSERT_TRUE(build_parquet_column_schema(descriptor, &fields).ok()); + ASSERT_EQ(fields.size(), 1); + ASSERT_EQ(fields[0]->kind, ParquetColumnSchemaKind::STRUCT); + + const std::vector overrides {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + const auto status = apply_variant_schema_overrides(descriptor, overrides, &fields); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); + EXPECT_TRUE(fields[0]->contains_variant); + EXPECT_EQ(remove_nullable(fields[0]->type)->get_primitive_type(), TYPE_VARIANT); + EXPECT_NE(fields[0]->variant_physical_type, nullptr); +} + +TEST(ParquetSchemaTest, AppliesPaimonShreddedVariantOverrideWithOptionalMetadata) { + auto schema = shredded_object_variant_schema(); + schema[1].__isset.logicalType = false; + schema[2].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + + std::vector> fields; + ASSERT_TRUE(build_parquet_column_schema(descriptor, &fields).ok()); + const std::vector overrides {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + const auto status = apply_variant_schema_overrides(descriptor, overrides, &fields); + ASSERT_TRUE(status.ok()) << status; + EXPECT_EQ(fields[0]->kind, ParquetColumnSchemaKind::VARIANT); +} + +TEST(ParquetSchemaTest, RejectsMalformedUnannotatedVariantOverride) { + auto schema = unshredded_variant_schema(); + schema[1].__isset.logicalType = false; + schema[2].__set_name("unexpected"); + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + + std::vector> fields; + ASSERT_TRUE(build_parquet_column_schema(descriptor, &fields).ok()); + const std::vector overrides {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + const auto status = apply_variant_schema_overrides(descriptor, overrides, &fields); + EXPECT_TRUE(status.is()) << status; + EXPECT_NE(status.to_string().find("unexpected child"), std::string::npos); +} + +TEST(ParquetSchemaTest, RejectsOptionalMetadataOutsidePaimonShreddedOverride) { + auto annotated_shredded = shredded_object_variant_schema(); + annotated_shredded[2].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + NativeFieldDescriptor descriptor; + const auto annotated_status = descriptor.parse_from_thrift(annotated_shredded); + EXPECT_TRUE(annotated_status.is()) << annotated_status; + + auto unannotated_unshredded = unshredded_variant_schema(); + unannotated_unshredded[1].__isset.logicalType = false; + unannotated_unshredded[2].__set_repetition_type(tparquet::FieldRepetitionType::OPTIONAL); + ASSERT_TRUE(descriptor.parse_from_thrift(unannotated_unshredded).ok()); + std::vector> fields; + ASSERT_TRUE(build_parquet_column_schema(descriptor, &fields).ok()); + const std::vector overrides {format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + const auto override_status = apply_variant_schema_overrides(descriptor, overrides, &fields); + EXPECT_TRUE(override_status.is()) << override_status; +} + +TEST(ParquetSchemaTest, AppliesNestedTableFormatVariantOverride) { + auto schema = struct_with_variant_schema(); + schema[3].__isset.logicalType = false; + NativeFieldDescriptor descriptor; + ASSERT_TRUE(descriptor.parse_from_thrift(schema).ok()); + + std::vector> fields; + ASSERT_TRUE(build_parquet_column_schema(descriptor, &fields).ok()); + ASSERT_EQ(fields.size(), 1); + ASSERT_EQ(fields[0]->kind, ParquetColumnSchemaKind::STRUCT); + ASSERT_EQ(fields[0]->children.size(), 2); + ASSERT_EQ(fields[0]->children[1]->kind, ParquetColumnSchemaKind::STRUCT); + + auto root_override = format::LocalColumnIndex::partial_local(0); + root_override.children.push_back(format::LocalColumnIndex::local(1)); + const auto status = apply_variant_schema_overrides(descriptor, {root_override}, &fields); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(fields[0]->contains_variant); + EXPECT_EQ(fields[0]->children[1]->kind, ParquetColumnSchemaKind::VARIANT); + const auto& struct_type = assert_cast(*remove_nullable(fields[0]->type)); + EXPECT_EQ(remove_nullable(struct_type.get_element(1))->get_primitive_type(), TYPE_VARIANT); +} + TEST(ParquetSchemaTest, NativeSchemaAcceptsRequiredAndOptionalVariantGroups) { for (const auto repetition : {tparquet::FieldRepetitionType::REQUIRED, tparquet::FieldRepetitionType::OPTIONAL}) { @@ -358,6 +449,24 @@ TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedPrimitiveTypePairs) { mismatched_integer.logicalType.INTEGER.__set_isSigned(true); invalid_typed_values.push_back(mismatched_integer); + tparquet::SchemaElement mismatched_int32_annotation; + mismatched_int32_annotation.__set_type(tparquet::Type::INT64); + mismatched_int32_annotation.__set_logicalType(tparquet::LogicalType()); + mismatched_int32_annotation.logicalType.__set_INTEGER(tparquet::IntType()); + mismatched_int32_annotation.logicalType.INTEGER.__set_bitWidth(32); + mismatched_int32_annotation.logicalType.INTEGER.__set_isSigned(true); + mismatched_int32_annotation.__set_converted_type(tparquet::ConvertedType::INT_32); + invalid_typed_values.push_back(mismatched_int32_annotation); + + tparquet::SchemaElement mismatched_int64_annotation; + mismatched_int64_annotation.__set_type(tparquet::Type::INT32); + mismatched_int64_annotation.__set_logicalType(tparquet::LogicalType()); + mismatched_int64_annotation.logicalType.__set_INTEGER(tparquet::IntType()); + mismatched_int64_annotation.logicalType.INTEGER.__set_bitWidth(64); + mismatched_int64_annotation.logicalType.INTEGER.__set_isSigned(true); + mismatched_int64_annotation.__set_converted_type(tparquet::ConvertedType::INT_64); + invalid_typed_values.push_back(mismatched_int64_annotation); + tparquet::SchemaElement mismatched_decimal; mismatched_decimal.__set_type(tparquet::Type::INT32); mismatched_decimal.__set_logicalType(tparquet::LogicalType()); @@ -381,6 +490,25 @@ TEST(ParquetSchemaTest, NativeVariantRejectsUnsupportedPrimitiveTypePairs) { } } +TEST(ParquetSchemaTest, NativeVariantAcceptsIcebergFullWidthSignedIntegerAnnotations) { + for (const auto [physical_type, bit_width] : {std::pair {tparquet::Type::INT32, int8_t {32}}, + std::pair {tparquet::Type::INT64, int8_t {64}}}) { + tparquet::SchemaElement typed_value; + typed_value.__set_type(physical_type); + typed_value.__set_logicalType(tparquet::LogicalType()); + typed_value.logicalType.__set_INTEGER(tparquet::IntType()); + typed_value.logicalType.INTEGER.__set_bitWidth(bit_width); + typed_value.logicalType.INTEGER.__set_isSigned(true); + typed_value.__set_converted_type(bit_width == 32 ? tparquet::ConvertedType::INT_32 + : tparquet::ConvertedType::INT_64); + + NativeFieldDescriptor descriptor; + const auto status = descriptor.parse_from_thrift( + shredded_primitive_variant_schema(std::move(typed_value))); + EXPECT_TRUE(status.ok()) << status; + } +} + TEST(ParquetSchemaTest, NativeVariantRejectsRepeatedOuterGroup) { auto schema = unshredded_variant_schema(); schema[1].__set_repetition_type(tparquet::FieldRepetitionType::REPEATED); diff --git a/be/test/format_v2/parquet/variant_column_reader_test.cpp b/be/test/format_v2/parquet/variant_column_reader_test.cpp index 51a50319285c5b..9e2fc0774baf72 100644 --- a/be/test/format_v2/parquet/variant_column_reader_test.cpp +++ b/be/test/format_v2/parquet/variant_column_reader_test.cpp @@ -51,6 +51,8 @@ #include "core/value/variant/variant_parquet_encoding.h" #include "exprs/function/function_variant_element_v2.h" #include "format_v2/parquet/parquet_column_schema.h" +#include "format_v2/parquet/parquet_profile.h" +#include "runtime/runtime_profile.h" namespace doris::format::parquet { namespace { @@ -237,17 +239,57 @@ MutableColumnPtr projected_shredded_object_physical(const std::vector& return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); } +MutableColumnPtr projected_shredded_int32_object_physical(const std::vector& values) { + auto integers = ColumnInt32::create(); + integers->get_data().assign(values.begin(), values.end()); + MutableColumns wrapper_fields; + wrapper_fields.push_back( + ColumnNullable::create(std::move(integers), ColumnUInt8::create(values.size(), 0))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + +MutableColumnPtr projected_shredded_binary_object_physical( + const std::vector& values) { + std::vector refs; + refs.reserve(values.size()); + for (const auto value : values) { + refs.emplace_back(value.data(), value.size()); + } + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_strings(refs, std::vector(values.size(), 0))); + auto wrapper = ColumnStruct::create(std::move(wrapper_fields)); + MutableColumns object_fields; + object_fields.push_back( + ColumnNullable::create(std::move(wrapper), ColumnUInt8::create(values.size(), 0))); + auto object = ColumnStruct::create(std::move(object_fields)); + MutableColumns root_fields; + root_fields.push_back( + ColumnNullable::create(std::move(object), ColumnUInt8::create(values.size(), 0))); + auto root = ColumnStruct::create(std::move(root_fields)); + return ColumnNullable::create(std::move(root), ColumnUInt8::create(values.size(), 0)); +} + MutableColumnPtr root_wrapper(MutableColumns fields, NullMap root_nulls = {0}); MutableColumnPtr nullable_int64(const std::vector& values, const std::vector& nulls); MutableColumnPtr complete_shredded_object_physical(std::string_view residual_key, - int64_t residual_value, int64_t typed_value) { + int64_t residual_value, int64_t typed_value, + uint8_t residual_width = 0) { VariantBatchBuilder builder; auto row = builder.begin_row(); auto object = row.start_object(); object.add_key(StringRef(residual_key.data(), residual_key.size())); - row.add_int(residual_value); + row.add_scalar(VariantScalarRef::integer(residual_value, residual_width)); object.finish(); row.finish(); VariantBatchBuilder batch = builder.finish_batch(); @@ -268,6 +310,51 @@ MutableColumnPtr complete_shredded_object_physical(std::string_view residual_key return root_wrapper(std::move(root_fields)); } +MutableColumnPtr complete_shredded_decimal_object_physical(std::string_view residual_key, + __int128 residual_value, + uint8_t residual_scale, + uint8_t residual_width, + int64_t typed_value) { + VariantBatchBuilder builder; + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef(residual_key.data(), residual_key.size())); + row.add_scalar(VariantScalarRef::decimal(residual_value, residual_scale, residual_width)); + object.finish(); + row.finish(); + VariantBatchBuilder batch = builder.finish_batch(); + const VariantRef residual = batch.value_at(0); + + MutableColumns wrapper_fields; + wrapper_fields.push_back(nullable_int64({typed_value}, {0})); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(wrapper_fields)), + ColumnUInt8::create(1, 0))); + MutableColumns root_fields; + root_fields.push_back( + nullable_strings({StringRef(residual.metadata.data, residual.metadata.size)}, {0})); + root_fields.push_back( + nullable_strings({StringRef(residual.value.data, residual.value.size)}, {0})); + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return root_wrapper(std::move(root_fields)); +} + +MutableColumnPtr projected_shredded_decimal_object_physical(__int128 value, uint32_t scale) { + auto decimals = ColumnDecimal128V3::create(0, scale); + decimals->insert_value(Decimal128V3 {value}); + MutableColumns wrapper_fields; + wrapper_fields.push_back( + ColumnNullable::create(std::move(decimals), ColumnUInt8::create(1, 0))); + MutableColumns object_fields; + object_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(wrapper_fields)), + ColumnUInt8::create(1, 0))); + MutableColumns root_fields; + root_fields.push_back(ColumnNullable::create(ColumnStruct::create(std::move(object_fields)), + ColumnUInt8::create(1, 0))); + return root_wrapper(std::move(root_fields)); +} + MutableColumnPtr projected_two_field_object_physical(const std::vector& first, const std::vector& second) { DORIS_CHECK(first.size() == second.size()); @@ -343,6 +430,17 @@ MutableColumnPtr nullable_int64(const std::vector& values, return ColumnNullable::create(std::move(data), std::move(null_map)); } +MutableColumnPtr binary_round_trip(const ColumnVariantV2& source) { + DataTypeVariantV2 type; + const int64_t maximum_size = type.get_uncompressed_serialized_bytes(source, 10); + std::vector bytes(maximum_size); + char* end = type.serialize(source, bytes.data(), 10); + bytes.resize(end - bytes.data()); + MutableColumnPtr destination = type.create_column(); + EXPECT_EQ(type.deserialize(bytes.data(), &destination, 10), bytes.data() + bytes.size()); + return destination; +} + template MutableColumnPtr nullable_fixed(std::initializer_list values, std::initializer_list nulls) { @@ -840,6 +938,506 @@ TEST(VariantColumnReaderTest, AppendsProjectedShreddedBatchesWithoutMaterializin EXPECT_EQ(plan.variant_state_schema.use_count(), 3); } +TEST(VariantColumnReaderTest, GathersConsecutiveProjectedShreddedBatches) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto first = make_nullable(std::make_shared())->create_column(); + auto second = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), first) + .ok()); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({30}), second) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array first_indices {1, 0}; + const std::array second_indices {0}; + gathered->insert_indices_from(*first, first_indices.begin(), first_indices.end()); + gathered->insert_indices_from(*second, second_indices.begin(), second_indices.end()); + + const auto& variants = assert_cast( + assert_cast(*gathered).get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + const auto& values = assert_cast( + assert_cast(*match->column).get_nested_column()); + EXPECT_EQ(values.get_data(), ColumnInt64::Container({20, 10, 30})); + + auto restored = binary_round_trip(variants); + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr resolved_path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &resolved_path).ok()); + ColumnPtr extracted; + ASSERT_TRUE(extract_variant_element_v2(assert_cast(*restored), + *resolved_path, {}, &extracted) + .ok()); + const auto& restored_values = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(restored_values.get_value_ref(0).get_int(), 20); + EXPECT_EQ(restored_values.get_value_ref(1).get_int(), 10); + EXPECT_EQ(restored_values.get_value_ref(2).get_int(), 30); +} + +TEST(VariantColumnReaderTest, SelectsProjectedShreddedRowsWithoutMaterializing) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({10, 20, 30}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + const IColumn::Permutation permutation {2, 0, 1}; + MutableColumnPtr permuted = variants.permute(permutation, 2); + MutableColumnPtr truncated = variants.clone_resized(2); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + auto verify = [&](const IColumn& column, const ColumnInt64::Container& expected) { + const auto& selected = assert_cast(column); + ASSERT_TRUE(selected.is_shredded()); + const auto match = selected.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(assert_cast( + assert_cast(*match->column).get_nested_column()) + .get_data(), + expected); + }; + verify(*permuted, ColumnInt64::Container({30, 10})); + verify(*truncated, ColumnInt64::Container({10, 20})); +} + +TEST(VariantColumnReaderTest, GathersLocalProjectedAndRemoteSerializedRowsInEitherOrder) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE( + materialize_variant_columns(plan, projected_shredded_object_physical({10, 20}), output) + .ok()); + const auto& local = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_TRUE(local.is_shredded()); + MutableColumnPtr remote = binary_round_trip(local); + ASSERT_FALSE(assert_cast(*remote).is_shredded()); + + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + auto verify_result = [&](const ColumnVariantV2& gathered, + const std::array& expected) { + ASSERT_EQ(gathered.size(), expected.size()); + + ColumnPtr extracted; + ASSERT_TRUE(extract_variant_element_v2(gathered, *path, {}, &extracted).ok()); + const auto& values = assert_cast( + assert_cast(*extracted).get_nested_column()); + for (size_t row = 0; row < expected.size(); ++row) { + EXPECT_EQ(values.get_value_ref(row).get_int(), expected[row]); + } + }; + auto verify = [&](const std::vector& sources, + const std::vector& positions, + const std::array& expected) { + auto gathered = ColumnVariantV2::create(); + gathered->insert_from_multi_column(sources, positions); + verify_result(*gathered, expected); + }; + + verify({&local, remote.get()}, {0, 1}, {10, 20}); + verify({remote.get(), &local}, {1, 0}, {20, 10}); + + const std::array first_row {0}; + const std::array second_row {1}; + auto indexed = ColumnVariantV2::create(); + indexed->insert_indices_from(local, first_row.begin(), first_row.end()); + indexed->insert_indices_from(*remote, second_row.begin(), second_row.end()); + verify_result(*indexed, {10, 20}); + + indexed = ColumnVariantV2::create(); + indexed->insert_indices_from(*remote, second_row.begin(), second_row.end()); + indexed->insert_indices_from(local, first_row.begin(), first_row.end()); + verify_result(*indexed, {20, 10}); +} + +TEST(VariantColumnReaderTest, ShrinksProjectedShreddedStateWithoutMaterializing) { + auto schema = shredded_object_schema(); + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(plan, projected_shredded_object_physical({10, 20, 30}), + output) + .ok()); + const auto& variants = assert_cast( + assert_cast(*output).get_nested_column()); + ASSERT_TRUE(variants.is_shredded()); + + ColumnPtr shrink_source = variants.clone_resized(variants.size()); + ColumnPtr shrunk = shrink_source->shrink(2); + const auto& shrunk_variants = assert_cast(*shrunk); + ASSERT_TRUE(shrunk_variants.is_shredded()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + const auto match = shrunk_variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + EXPECT_EQ(assert_cast( + assert_cast(*match->column).get_nested_column()) + .get_data(), + ColumnInt64::Container({10, 20})); + + ColumnPtr empty_source = variants.clone_resized(variants.size()); + ColumnPtr empty = empty_source->shrink(0); + EXPECT_EQ(empty->size(), 0); + EXPECT_FALSE(assert_cast(*empty).is_shredded()); +} + +TEST(VariantColumnReaderTest, GathersCompleteAndProjectedShreddedBatches) { + auto projected_schema = shredded_object_schema(); + projected_schema.local_id = 0; + projected_schema.children[0]->local_id = 0; + projected_schema.children[1]->local_id = 1; + projected_schema.children[2]->local_id = 2; + projected_schema.children[2]->children[0]->local_id = 0; + projected_schema.children[2]->children[0]->children[0]->local_id = 0; + + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode projected_plan; + projected_plan.schema = &projected_schema; + projected_plan.contains_variant = true; + projected_plan.variant_projection = std::move(projection); + projected_plan.variant_state_schema = + create_variant_state_schema(projected_schema, &*projected_plan.variant_projection); + + auto complete_schema = shredded_named_object_schema("b"); + VariantMaterializationNode complete_plan; + complete_plan.schema = &complete_schema; + complete_plan.contains_variant = true; + complete_plan.variant_state_schema = create_variant_state_schema(complete_schema, nullptr); + + auto projected = make_nullable(std::make_shared())->create_column(); + auto complete = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(projected_plan, projected_shredded_object_physical({7}), + projected) + .ok()); + ASSERT_TRUE(materialize_variant_columns( + complete_plan, complete_shredded_object_physical("other", 9, 8), complete) + .ok()); + + const std::array selected {0}; + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + auto verify_order = [&](const IColumn& first, const IColumn& second, + const NullMap& expected_nulls, size_t value_row) { + auto gathered = make_nullable(std::make_shared())->create_column(); + gathered->insert_indices_from(first, selected.begin(), selected.end()); + gathered->insert_indices_from(second, selected.begin(), selected.end()); + + const auto& nullable = assert_cast(*gathered); + const auto& variants = assert_cast(nullable.get_nested_column()); + ColumnPtr extracted; + ASSERT_TRUE(extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), + &extracted) + .ok()); + const auto& extracted_nullable = assert_cast(*extracted); + EXPECT_EQ(extracted_nullable.get_null_map_data(), expected_nulls); + const auto& extracted_values = + assert_cast(extracted_nullable.get_nested_column()); + EXPECT_EQ(extracted_values.get_value_ref(value_row).get_int(), 7); + }; + + // Complete and projected files can alternate in either order; a field absent from the + // complete file must contribute NULL without forcing the projected file to materialize. + verify_order(*projected, *complete, NullMap({0, 1}), 0); + verify_order(*complete, *projected, NullMap({1, 0}), 1); +} + +TEST(VariantColumnReaderTest, PreservesPrimitiveWidthsAcrossProjectedFiles) { + auto int64_schema = shredded_object_schema(); + auto int32_schema = shredded_object_schema(); + int32_schema.children[2]->children[0]->children[0]->type = + make_nullable(std::make_shared()); + int32_schema.children[2]->children[0]->children[0]->type_descriptor.integer_bit_width = 32; + for (auto* schema : {&int64_schema, &int32_schema}) { + schema->local_id = 0; + schema->children[2]->local_id = 2; + schema->children[2]->children[0]->local_id = 0; + schema->children[2]->children[0]->children[0]->local_id = 0; + } + auto make_plan = [](const ParquetColumnSchema& schema) { + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + return plan; + }; + auto int64_plan = make_plan(int64_schema); + auto int32_plan = make_plan(int32_schema); + auto int64_rows = make_nullable(std::make_shared())->create_column(); + auto int32_rows = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(int64_plan, projected_shredded_object_physical({7}), + int64_rows) + .ok()); + ASSERT_TRUE(materialize_variant_columns( + int32_plan, projected_shredded_int32_object_physical({8}), int32_rows) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array selected {0}; + gathered->insert_indices_from(*int64_rows, selected.begin(), selected.end()); + gathered->insert_indices_from(*int32_rows, selected.begin(), selected.end()); + + const auto& nullable = assert_cast(*gathered); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + const auto& values = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(values.get_value_ref(0).primitive_id(), VariantPrimitiveId::INT64); + EXPECT_EQ(values.get_value_ref(1).primitive_id(), VariantPrimitiveId::INT32); +} + +TEST(VariantColumnReaderTest, PreservesWidthsAcrossMaterializedPathFallback) { + auto projected_int_schema = shredded_object_schema(); + auto projected_decimal_schema = shredded_object_schema(); + auto* decimal_leaf = projected_decimal_schema.children[2]->children[0]->children[0].get(); + decimal_leaf->type = make_nullable(std::make_shared(38, 2)); + decimal_leaf->type_descriptor.decimal_precision = 38; + decimal_leaf->type_descriptor.decimal_scale = 2; + auto prepare_projected = [](ParquetColumnSchema& schema) { + schema.local_id = 0; + schema.children[2]->local_id = 2; + schema.children[2]->children[0]->local_id = 0; + schema.children[2]->children[0]->children[0]->local_id = 0; + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + return plan; + }; + auto projected_int_plan = prepare_projected(projected_int_schema); + auto projected_decimal_plan = prepare_projected(projected_decimal_schema); + auto complete_schema = shredded_named_object_schema("b"); + VariantMaterializationNode complete_plan; + complete_plan.schema = &complete_schema; + complete_plan.contains_variant = true; + complete_plan.variant_state_schema = create_variant_state_schema(complete_schema, nullptr); + + auto verify = [&](VariantMaterializationNode& projected_plan, + MutableColumnPtr projected_physical, MutableColumnPtr complete_physical, + VariantPrimitiveId expected_id) { + auto projected = make_nullable(std::make_shared())->create_column(); + auto complete = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(projected_plan, std::move(projected_physical), + projected) + .ok()); + ASSERT_TRUE( + materialize_variant_columns(complete_plan, std::move(complete_physical), complete) + .ok()); + const std::array selected {0}; + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + for (const auto order : {std::array {projected.get(), complete.get()}, + std::array {complete.get(), projected.get()}}) { + auto gathered = make_nullable(std::make_shared())->create_column(); + for (const IColumn* source : order) { + gathered->insert_indices_from(*source, selected.begin(), selected.end()); + } + const auto& nullable = assert_cast(*gathered); + const auto& variants = + assert_cast(nullable.get_nested_column()); + ColumnPtr extracted; + ASSERT_TRUE(extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), + &extracted) + .ok()); + const auto& values = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(values.get_value_ref(0).primitive_id(), expected_id); + EXPECT_EQ(values.get_value_ref(1).primitive_id(), expected_id); + } + }; + + verify(projected_int_plan, projected_shredded_object_physical({7}), + complete_shredded_object_physical("a", 8, 9, 8), VariantPrimitiveId::INT64); + verify(projected_decimal_plan, projected_shredded_decimal_object_physical(7, 2), + complete_shredded_decimal_object_physical("a", 8, 2, 16, 9), + VariantPrimitiveId::DECIMAL16); +} + +TEST(VariantColumnReaderTest, GathersProjectedShreddedBatchesWithDifferentLeafTypes) { + auto integer_schema = shredded_object_schema(); + auto string_schema = shredded_binary_object_schema(); + for (auto* schema : {&integer_schema, &string_schema}) { + schema->local_id = 0; + schema->children[2]->local_id = 2; + schema->children[2]->children[0]->local_id = 0; + schema->children[2]->children[0]->children[0]->local_id = 0; + } + auto make_plan = [](const ParquetColumnSchema& schema) { + auto projection = format::LocalColumnIndex::partial_local(0); + projection.children.push_back(format::LocalColumnIndex::partial_local(2)); + projection.children.back().children.push_back(format::LocalColumnIndex::partial_local(0)); + projection.children.back().children.back().children.push_back( + format::LocalColumnIndex::local(0)); + VariantMaterializationNode plan; + plan.schema = &schema; + plan.contains_variant = true; + plan.variant_projection = std::move(projection); + plan.variant_state_schema = create_variant_state_schema(schema, &*plan.variant_projection); + return plan; + }; + auto integer_plan = make_plan(integer_schema); + auto string_plan = make_plan(string_schema); + auto integers = make_nullable(std::make_shared())->create_column(); + auto strings = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_columns(integer_plan, + projected_shredded_object_physical({7, 8}), integers) + .ok()); + ASSERT_TRUE(materialize_variant_columns( + string_plan, projected_shredded_binary_object_physical({"seven"}), strings) + .ok()); + + auto gathered = make_nullable(std::make_shared())->create_column(); + const std::array integer_rows {0, 1}; + const std::array string_rows {0}; + gathered->insert_indices_from(*integers, integer_rows.begin(), integer_rows.end()); + gathered->insert_indices_from(*strings, string_rows.begin(), string_rows.end()); + + const auto& nullable = assert_cast(*gathered); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path_segments {VariantElementV2PathSegment::object_key(StringRef("a"))}; + std::unique_ptr path; + ASSERT_TRUE(resolve_variant_element_v2_path(path_segments, &path).ok()); + ColumnPtr extracted; + ASSERT_TRUE( + extract_variant_element_v2(variants, *path, nullable.get_null_map_data(), &extracted) + .ok()); + const auto& extracted_variants = assert_cast( + assert_cast(*extracted).get_nested_column()); + EXPECT_EQ(extracted_variants.get_value_ref(0).get_int(), 7); + EXPECT_EQ(extracted_variants.get_value_ref(1).get_int(), 8); + EXPECT_EQ(extracted_variants.get_value_ref(2).get_binary(), StringRef("seven")); + + variants.sanity_check(); + EXPECT_GT(variants.byte_size(), 0); + EXPECT_GE(variants.allocated_bytes(), variants.byte_size()); + + auto ranged = ColumnVariantV2::create(); + ranged->insert_range_from(variants, 1, 2); + const auto ranged_match = + ranged->find_shredded_typed_value(std::array {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}); + ASSERT_TRUE(ranged_match.has_value()); + ASSERT_TRUE(ranged_match->normalized); + const auto& ranged_values = assert_cast( + assert_cast(*ranged_match->normalized).get_nested_column()); + EXPECT_EQ(ranged_values.get_value_ref(0).get_int(), 8); + EXPECT_EQ(ranged_values.get_value_ref(1).get_binary(), StringRef("seven")); + + const std::array shredded_path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("a")}}; + auto reordered = ColumnVariantV2::create(); + const std::array reversed {2, 1, 0}; + reordered->insert_indices_from(variants, reversed.begin(), reversed.end()); + const auto reordered_match = reordered->find_shredded_typed_value(shredded_path); + ASSERT_TRUE(reordered_match.has_value()); + ASSERT_TRUE(reordered_match->normalized); + const auto& reordered_values = assert_cast( + assert_cast(*reordered_match->normalized).get_nested_column()); + EXPECT_EQ(reordered_values.get_value_ref(0).get_binary(), StringRef("seven")); + EXPECT_EQ(reordered_values.get_value_ref(1).get_int(), 8); + EXPECT_EQ(reordered_values.get_value_ref(2).get_int(), 7); + + IColumn::Filter keep_integer {1, 0, 0}; + const auto filtered = variants.filter(keep_integer, 1); + const auto filtered_match = + assert_cast(*filtered).find_shredded_typed_value(shredded_path); + ASSERT_TRUE(filtered_match.has_value()); + ASSERT_TRUE(filtered_match->column); + EXPECT_EQ( + assert_cast( + assert_cast(*filtered_match->column).get_nested_column()) + .get_data()[0], + 7); +} + TEST(VariantColumnReaderTest, WideProjectionSharesSchemaAcrossBatchesAndSelections) { constexpr size_t width = 64; constexpr size_t batch_count = 16; @@ -1007,6 +1605,35 @@ TEST(VariantColumnReaderTest, MaterializedCacheParticipatesInMemoryAccounting) { EXPECT_GT(variants.allocated_bytes(), physical_allocated); } +TEST(VariantColumnReaderTest, ShreddedStateOutlivesScannerProfile) { + auto runtime_profile = std::make_unique("variant-reader-test"); + ParquetProfile parquet_profile; + parquet_profile.init(runtime_profile.get()); + ParquetProfile reused_profile; + reused_profile.init(runtime_profile.get()); + EXPECT_EQ(parquet_profile.variant_reconstructed_rows, + reused_profile.variant_reconstructed_rows); + + auto visible_output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({7}), + visible_output, parquet_profile.column_reader_profile()) + .ok()); + const auto& visible_variants = assert_cast( + assert_cast(*visible_output).get_nested_column()); + EXPECT_EQ(visible_variants.get_value_ref(0).get_int(), 7); + EXPECT_EQ(runtime_profile->get_counter("VariantReconstructedRows")->value(), 1); + + auto output = make_nullable(std::make_shared())->create_column(); + ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({42}), + output, parquet_profile.column_reader_profile()) + .ok()); + runtime_profile.reset(); + + const auto& nullable = assert_cast(*output); + const auto& variants = assert_cast(nullable.get_nested_column()); + EXPECT_EQ(variants.get_value_ref(0).get_int(), 42); +} + TEST(VariantColumnReaderTest, MaterializedShreddedCopiesDetachBeforeMutation) { auto first_output = make_nullable(std::make_shared())->create_column(); ASSERT_TRUE(materialize_variant_rows(shredded_int64_schema(), shredded_int64_physical({10, 20}), diff --git a/be/test/format_v2/table/paimon_variant_reader_test.cpp b/be/test/format_v2/table/paimon_variant_reader_test.cpp new file mode 100644 index 00000000000000..2cb8be17b80c8c --- /dev/null +++ b/be/test/format_v2/table/paimon_variant_reader_test.cpp @@ -0,0 +1,548 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "core/assert_cast.h" +#include "core/block/block.h" +#include "core/column/column_nullable.h" +#include "core/column/variant_v2/column_variant_v2.h" +#include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" +#include "core/data_type/data_type_variant_v2.h" +#include "core/value/variant/variant_batch_builder.h" +#include "format_v2/parquet/parquet_reader.h" +#include "format_v2/table/paimon_reader.h" +#include "io/io_common.h" +#include "runtime/runtime_state.h" + +namespace doris::format { +namespace { + +ColumnDefinition table_column(std::string name, DataTypePtr type) { + ColumnDefinition column; + column.name = std::move(name); + column.type = make_nullable(std::move(type)); + return column; +} + +ColumnDefinition file_column(int32_t local_id, std::string name, DataTypePtr type) { + ColumnDefinition column; + column.local_id = local_id; + column.name = std::move(name); + column.type = make_nullable(std::move(type)); + return column; +} + +std::shared_ptr binary_array(const std::vector& values) { + arrow::BinaryBuilder builder; + for (const auto value : values) { + EXPECT_TRUE(builder.Append(reinterpret_cast(value.data), + static_cast(value.size)) + .ok()); + } + std::shared_ptr result; + EXPECT_TRUE(builder.Finish(&result).ok()); + return result; +} + +std::shared_ptr null_binary_array(size_t rows) { + arrow::BinaryBuilder builder; + for (size_t row = 0; row < rows; ++row) { + EXPECT_TRUE(builder.AppendNull().ok()); + } + std::shared_ptr result; + EXPECT_TRUE(builder.Finish(&result).ok()); + return result; +} + +std::shared_ptr int32_array(const std::vector& values) { + arrow::Int32Builder builder; + EXPECT_TRUE(builder.AppendValues(values).ok()); + std::shared_ptr result; + EXPECT_TRUE(builder.Finish(&result).ok()); + return result; +} + +void write_unannotated_paimon_variant_file(const std::string& path, + const std::vector& values) { + VariantBatchBuilder builder; + for (const auto value : values) { + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef("n")); + row.add_int(value); + object.finish(); + row.finish(); + } + auto batch = builder.finish_batch(); + std::vector value_rows; + std::vector metadata_rows; + for (size_t row = 0; row < values.size(); ++row) { + const auto encoded = batch.value_at(row); + value_rows.push_back(encoded.value); + metadata_rows.emplace_back(encoded.metadata.data, encoded.metadata.size); + } + + const auto payload_type = arrow::struct_({arrow::field("value", arrow::binary(), false), + arrow::field("metadata", arrow::binary(), false)}); + auto payload_result = arrow::StructArray::Make( + {binary_array(value_rows), binary_array(metadata_rows)}, payload_type->fields()); + ASSERT_TRUE(payload_result.ok()) << payload_result.status(); + auto table = arrow::Table::Make(arrow::schema({arrow::field("payload", payload_type)}), + {*payload_result}); + + auto file_result = arrow::io::FileOutputStream::Open(path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + ::parquet::WriterProperties::Builder properties; + properties.version(::parquet::ParquetVersion::PARQUET_2_6); + properties.compression(::parquet::Compression::UNCOMPRESSED); + PARQUET_THROW_NOT_OK( + ::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), *file_result, + static_cast(values.size()), properties.build())); +} + +void write_unannotated_shredded_paimon_variant_file(const std::string& path, + const std::vector& ages) { + VariantBatchBuilder builder; + for (const auto age : ages) { + auto row = builder.begin_row(); + auto object = row.start_object(); + object.add_key(StringRef("age")); + row.add_int(age); + object.finish(); + row.finish(); + } + const auto batch = builder.finish_batch(); + std::vector metadata_rows; + for (size_t row = 0; row < ages.size(); ++row) { + const auto encoded = batch.value_at(row); + metadata_rows.emplace_back(encoded.metadata.data, encoded.metadata.size); + } + + const auto age_wrapper_type = arrow::struct_( + {arrow::field("value", arrow::binary()), arrow::field("typed_value", arrow::int32())}); + auto age_result = arrow::StructArray::Make({null_binary_array(ages.size()), int32_array(ages)}, + age_wrapper_type->fields()); + ASSERT_TRUE(age_result.ok()) << age_result.status(); + const auto typed_value_type = arrow::struct_({arrow::field("age", age_wrapper_type, false)}); + auto typed_value_result = arrow::StructArray::Make({*age_result}, typed_value_type->fields()); + ASSERT_TRUE(typed_value_result.ok()) << typed_value_result.status(); + const auto payload_type = arrow::struct_({arrow::field("metadata", arrow::binary(), false), + arrow::field("value", arrow::binary()), + arrow::field("typed_value", typed_value_type)}); + auto payload_result = arrow::StructArray::Make( + {binary_array(metadata_rows), null_binary_array(ages.size()), *typed_value_result}, + payload_type->fields()); + ASSERT_TRUE(payload_result.ok()) << payload_result.status(); + auto table = arrow::Table::Make(arrow::schema({arrow::field("payload", payload_type)}), + {*payload_result}); + + auto file_result = arrow::io::FileOutputStream::Open(path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + ::parquet::WriterProperties::Builder properties; + properties.version(::parquet::ParquetVersion::PARQUET_2_6); + properties.compression(::parquet::Compression::UNCOMPRESSED); + PARQUET_THROW_NOT_OK( + ::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), *file_result, + static_cast(ages.size()), properties.build())); +} + +// Scenario: Paimon 1.3/1.4 writes Variant as an unannotated Parquet group. Only the Paimon table +// schema can distinguish that carrier from an ordinary STRUCT, so the table reader must expose the +// matched file node as Variant while retaining its physical children for native decoding. +TEST(PaimonVariantReaderTest, AnnotatesUnmarkedParquetVariantFromTableSchema) { + const auto binary = std::make_shared(); + auto physical_type = std::make_shared(DataTypes {binary, binary}, + Strings {"value", "metadata"}); + auto payload = file_column(0, "payload", physical_type); + payload.children = {file_column(0, "value", binary), file_column(1, "metadata", binary)}; + std::vector file_schema {std::move(payload)}; + + paimon::PaimonReader reader; + reader.TEST_set_format(FileFormat::PARQUET); + reader.TEST_set_projected_columns( + {table_column("payload", std::make_shared())}); + + ASSERT_TRUE(reader.TEST_annotate_file_schema(&file_schema).ok()); + ASSERT_EQ(file_schema.size(), 1); + EXPECT_EQ(remove_nullable(file_schema[0].type)->get_primitive_type(), TYPE_VARIANT); + ASSERT_EQ(file_schema[0].children.size(), 2); + EXPECT_EQ(file_schema[0].children[0].name, "value"); + EXPECT_EQ(file_schema[0].children[1].name, "metadata"); + + FileScanRequest request; + ASSERT_TRUE(reader.TEST_customize_file_scan_request(&request).ok()); + ASSERT_EQ(request.variant_schema_overrides.size(), 1); + EXPECT_EQ(request.variant_schema_overrides[0].local_id(), 0); + EXPECT_TRUE(request.variant_schema_overrides[0].project_all_children); +} + +TEST(PaimonVariantReaderTest, DoesNotGuessOrdinaryStructWithVariantCarrierNames) { + const auto binary = std::make_shared(); + auto struct_type = std::make_shared(DataTypes {binary, binary}, + Strings {"value", "metadata"}); + auto payload = file_column(0, "payload", struct_type); + payload.children = {file_column(0, "value", binary), file_column(1, "metadata", binary)}; + std::vector file_schema {payload}; + + auto projected = table_column("payload", struct_type); + projected.children = payload.children; + paimon::PaimonReader reader; + reader.TEST_set_format(FileFormat::PARQUET); + reader.TEST_set_projected_columns({std::move(projected)}); + + ASSERT_TRUE(reader.TEST_annotate_file_schema(&file_schema).ok()); + EXPECT_EQ(remove_nullable(file_schema[0].type)->get_primitive_type(), TYPE_STRUCT); + FileScanRequest request; + ASSERT_TRUE(reader.TEST_customize_file_scan_request(&request).ok()); + EXPECT_TRUE(request.variant_schema_overrides.empty()); +} + +TEST(PaimonVariantReaderTest, AnnotatesNestedArrayVariantByStructuralPosition) { + const auto binary = std::make_shared(); + auto carrier_type = std::make_shared(DataTypes {binary, binary}, + Strings {"value", "metadata"}); + auto element = file_column(0, "element", carrier_type); + element.children = {file_column(0, "value", binary), file_column(1, "metadata", binary)}; + auto values = file_column(0, "values", std::make_shared(element.type)); + values.children = {std::move(element)}; + std::vector file_schema {std::move(values)}; + + auto item = table_column("item", std::make_shared()); + auto projected = table_column("values", std::make_shared(item.type)); + projected.children = {std::move(item)}; + paimon::PaimonReader reader; + reader.TEST_set_format(FileFormat::PARQUET); + reader.TEST_set_projected_columns({std::move(projected)}); + + ASSERT_TRUE(reader.TEST_annotate_file_schema(&file_schema).ok()); + const auto& array_type = + assert_cast(*remove_nullable(file_schema[0].type)); + EXPECT_EQ(remove_nullable(array_type.get_nested_type())->get_primitive_type(), TYPE_VARIANT); + ASSERT_EQ(file_schema[0].children.size(), 1); + EXPECT_EQ(remove_nullable(file_schema[0].children[0].type)->get_primitive_type(), TYPE_VARIANT); + + FileScanRequest request; + ASSERT_TRUE(reader.TEST_customize_file_scan_request(&request).ok()); + ASSERT_EQ(request.variant_schema_overrides.size(), 1); + EXPECT_FALSE(request.variant_schema_overrides[0].project_all_children); + ASSERT_EQ(request.variant_schema_overrides[0].children.size(), 1); + EXPECT_TRUE(request.variant_schema_overrides[0].children[0].project_all_children); +} + +TEST(PaimonVariantReaderTest, MergesSiblingNestedVariantOverrides) { + const auto binary = std::make_shared(); + auto carrier_type = std::make_shared(DataTypes {binary, binary}, + Strings {"value", "metadata"}); + auto carrier = [&](int32_t local_id, std::string name) { + auto column = file_column(local_id, std::move(name), carrier_type); + column.children = {file_column(0, "value", binary), file_column(1, "metadata", binary)}; + return column; + }; + auto row = file_column(0, "row", + std::make_shared(DataTypes {carrier_type, carrier_type}, + Strings {"left", "right"})); + row.children = {carrier(0, "left"), carrier(1, "right")}; + std::vector file_schema {std::move(row)}; + + auto projected = table_column( + "row", std::make_shared( + DataTypes {make_nullable(std::make_shared()), + make_nullable(std::make_shared())}, + Strings {"left", "right"})); + projected.children = {table_column("left", std::make_shared()), + table_column("right", std::make_shared())}; + paimon::PaimonReader reader; + reader.TEST_set_format(FileFormat::PARQUET); + reader.TEST_set_projected_columns({std::move(projected)}); + + ASSERT_TRUE(reader.TEST_annotate_file_schema(&file_schema).ok()); + FileScanRequest request; + ASSERT_TRUE(reader.TEST_customize_file_scan_request(&request).ok()); + ASSERT_EQ(request.variant_schema_overrides.size(), 1); + ASSERT_EQ(request.variant_schema_overrides[0].children.size(), 2); + EXPECT_TRUE(request.variant_schema_overrides[0].children[0].project_all_children); + EXPECT_TRUE(request.variant_schema_overrides[0].children[1].project_all_children); +} + +TEST(PaimonVariantReaderTest, ReadsUnannotatedPaimonVariantWithNativeParquetReader) { + const auto test_dir = + std::filesystem::temp_directory_path() / "doris_paimon_native_unannotated_variant_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "data.parquet").string(); + write_unannotated_paimon_variant_file(file_path, {1, 2, 3}); + + std::vector projected_columns {table_column("payload", std::make_shared())}; + TFileScanRangeParams scan_params; + scan_params.__set_file_type(TFileType::FILE_LOCAL); + scan_params.__set_format_type(TFileFormatType::FORMAT_PARQUET); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + io::FileReaderStats file_reader_stats; + io::FileCacheStatistics file_cache_stats; + auto io_ctx = std::make_shared(); + io_ctx->file_reader_stats = &file_reader_stats; + io_ctx->file_cache_stats = &file_cache_stats; + + paimon::PaimonReader reader; + ASSERT_TRUE(reader.init({.projected_columns = projected_columns, + .conjuncts = {}, + .format = FileFormat::PARQUET, + .scan_params = &scan_params, + .io_ctx = io_ctx, + .runtime_state = &state, + .scanner_profile = nullptr}) + .ok()); + SplitReadOptions split; + split.current_range.__set_path(file_path); + split.current_range.__set_file_size( + static_cast(std::filesystem::file_size(file_path))); + TTableFormatFileDesc table_format; + table_format.__set_table_format_type("paimon"); + table_format.__set_paimon_params(TPaimonFileDesc {}); + split.current_range.__set_table_format_params(std::move(table_format)); + ASSERT_TRUE(reader.prepare_split(split).ok()); + + std::vector actual; + bool eos = false; + while (!eos) { + Block block; + block.insert({projected_columns[0].type->create_column(), projected_columns[0].type, + projected_columns[0].name}); + ASSERT_TRUE(reader.get_block(&block, &eos).ok()); + if (block.rows() == 0) { + continue; + } + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + for (size_t row = 0; row < variants.size(); ++row) { + VariantRef n; + ASSERT_TRUE(variants.get_value_ref(row).object_find(StringRef("n"), &n)); + actual.push_back(n.get_int()); + } + } + EXPECT_EQ(actual, std::vector({1, 2, 3})); + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(PaimonVariantReaderTest, ParquetReaderAppliesExplicitVariantSchemaOverride) { + const auto test_dir = + std::filesystem::temp_directory_path() / "doris_paimon_parquet_variant_override_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "data.parquet").string(); + write_unannotated_paimon_variant_file(file_path, {7, 8}); + + auto system_properties = std::make_shared(); + system_properties->system_type = TFileType::FILE_LOCAL; + auto file_description = std::make_unique(); + file_description->path = file_path; + file_description->file_size = static_cast(std::filesystem::file_size(file_path)); + file_description->range_start_offset = 0; + file_description->range_size = -1; + auto reader = std::make_unique( + system_properties, file_description, std::shared_ptr {}, nullptr); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector file_schema; + ASSERT_TRUE(reader->get_schema(&file_schema).ok()); + ASSERT_EQ(file_schema.size(), 1); + ASSERT_EQ(remove_nullable(file_schema[0].type)->get_primitive_type(), TYPE_STRUCT); + + auto request = std::make_shared(); + request->non_predicate_columns.push_back( + LocalColumnIndex::top_level(LocalColumnId(file_schema[0].local_id))); + request->local_positions.emplace(LocalColumnId(file_schema[0].local_id), LocalIndex(0)); + request->variant_schema_overrides.push_back( + LocalColumnIndex::top_level(LocalColumnId(file_schema[0].local_id))); + ASSERT_TRUE(reader->open(request).ok()); + + const auto variant_type = make_nullable(std::make_shared()); + std::vector actual; + bool eof = false; + while (!eof) { + Block block; + block.insert({variant_type->create_column(), variant_type, "payload"}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + for (size_t row = 0; row < variants.size(); ++row) { + VariantRef n; + ASSERT_TRUE(variants.get_value_ref(row).object_find(StringRef("n"), &n)); + actual.push_back(n.get_int()); + } + } + EXPECT_EQ(actual, std::vector({7, 8})); + ASSERT_TRUE(reader->close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(PaimonVariantReaderTest, ReadsProjectedLeafFromUnannotatedShreddedVariant) { + const auto test_dir = std::filesystem::temp_directory_path() / + "doris_paimon_parquet_shredded_variant_override_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "data.parquet").string(); + write_unannotated_shredded_paimon_variant_file(file_path, {27, 42}); + + auto system_properties = std::make_shared(); + system_properties->system_type = TFileType::FILE_LOCAL; + auto file_description = std::make_unique(); + file_description->path = file_path; + file_description->file_size = static_cast(std::filesystem::file_size(file_path)); + file_description->range_start_offset = 0; + file_description->range_size = -1; + auto reader = std::make_unique( + system_properties, file_description, std::shared_ptr {}, nullptr); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + std::vector file_schema; + ASSERT_TRUE(reader->get_schema(&file_schema).ok()); + ASSERT_EQ(file_schema.size(), 1); + ASSERT_EQ(remove_nullable(file_schema[0].type)->get_primitive_type(), TYPE_STRUCT); + ASSERT_EQ(file_schema[0].children.size(), 3); + + auto find_child = [](const std::vector& children, + std::string_view name) -> const ColumnDefinition* { + const auto it = std::ranges::find_if( + children, [name](const auto& child) { return child.name == name; }); + return it == children.end() ? nullptr : &*it; + }; + const auto* typed_value = find_child(file_schema[0].children, "typed_value"); + ASSERT_NE(typed_value, nullptr); + const auto* age = find_child(typed_value->children, "age"); + ASSERT_NE(age, nullptr); + const auto* age_typed_value = find_child(age->children, "typed_value"); + ASSERT_NE(age_typed_value, nullptr); + + auto projection = LocalColumnIndex::partial_local(file_schema[0].local_id); + projection.children.push_back(LocalColumnIndex::partial_local(typed_value->local_id)); + projection.children.back().children.push_back(LocalColumnIndex::partial_local(age->local_id)); + projection.children.back().children.back().children.push_back( + LocalColumnIndex::local(age_typed_value->local_id)); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(std::move(projection)); + request->local_positions.emplace(LocalColumnId(file_schema[0].local_id), LocalIndex(0)); + request->variant_schema_overrides.push_back( + LocalColumnIndex::top_level(LocalColumnId(file_schema[0].local_id))); + ASSERT_TRUE(reader->open(request).ok()); + + const auto variant_type = make_nullable(std::make_shared()); + Block block; + block.insert({variant_type->create_column(), variant_type, "payload"}); + size_t rows = 0; + bool eof = false; + while (!eof) { + size_t batch_rows = 0; + ASSERT_TRUE(reader->get_block(&block, &batch_rows, &eof).ok()); + rows += batch_rows; + } + EXPECT_EQ(rows, 2); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + const std::array path {VariantShreddedPathSegment { + .kind = VariantShreddedPathSegment::Kind::OBJECT_KEY, .key = StringRef("age")}}; + const auto match = variants.find_shredded_typed_value(path); + ASSERT_TRUE(match.has_value()); + const auto& typed = assert_cast(*match->column); + const auto& values = assert_cast(typed.get_nested_column()); + ASSERT_EQ(values.size(), 2); + EXPECT_EQ(values.get_data()[0], 27); + EXPECT_EQ(values.get_data()[1], 42); + ASSERT_TRUE(reader->close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(PaimonVariantReaderTest, AppendsUnshreddedAndShreddedPaimonFilesInEitherOrder) { + const auto test_dir = std::filesystem::temp_directory_path() / + "doris_paimon_parquet_mixed_variant_override_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto unshredded_path = (test_dir / "unshredded.parquet").string(); + const auto shredded_path = (test_dir / "shredded.parquet").string(); + write_unannotated_paimon_variant_file(unshredded_path, {5}); + write_unannotated_shredded_paimon_variant_file(shredded_path, {27}); + + const auto assert_file_order = [&](const std::array& paths, + const std::array& fields, + const std::array& values) { + const auto variant_type = make_nullable(std::make_shared()); + Block block; + block.insert({variant_type->create_column(), variant_type, "payload"}); + const auto append_file = [&](const std::string& path) -> Status { + auto system_properties = std::make_shared(); + system_properties->system_type = TFileType::FILE_LOCAL; + auto file_description = std::make_unique(); + file_description->path = path; + file_description->file_size = static_cast(std::filesystem::file_size(path)); + file_description->range_start_offset = 0; + file_description->range_size = -1; + auto reader = std::make_unique( + system_properties, file_description, std::shared_ptr {}, + nullptr); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + RETURN_IF_ERROR(reader->init(&state)); + auto request = std::make_shared(); + request->non_predicate_columns.push_back(LocalColumnIndex::top_level(LocalColumnId(0))); + request->local_positions.emplace(LocalColumnId(0), LocalIndex(0)); + request->variant_schema_overrides.push_back( + LocalColumnIndex::top_level(LocalColumnId(0))); + RETURN_IF_ERROR(reader->open(request)); + bool eof = false; + while (!eof) { + size_t rows = 0; + RETURN_IF_ERROR(reader->get_block(&block, &rows, &eof)); + } + return reader->close(); + }; + + ASSERT_TRUE(append_file(paths[0]).ok()); + ASSERT_TRUE(append_file(paths[1]).ok()); + ASSERT_EQ(block.rows(), 2); + const auto& nullable = assert_cast(*block.get_by_position(0).column); + const auto& variants = assert_cast(nullable.get_nested_column()); + for (size_t row = 0; row < 2; ++row) { + VariantRef field; + ASSERT_TRUE(variants.get_value_ref(row).object_find(fields[row], &field)); + EXPECT_EQ(field.get_int(), values[row]); + } + }; + + assert_file_order({unshredded_path, shredded_path}, {StringRef("n"), StringRef("age")}, + {5, 27}); + assert_file_order({shredded_path, unshredded_path}, {StringRef("age"), StringRef("n")}, + {27, 5}); + + std::filesystem::remove_all(test_dir); +} + +} // namespace +} // namespace doris::format diff --git a/be/test/util/variant/variant_batch_builder_test.cpp b/be/test/util/variant/variant_batch_builder_test.cpp index f7e55ed1b3ad55..4e1f5817ab6515 100644 --- a/be/test/util/variant/variant_batch_builder_test.cpp +++ b/be/test/util/variant/variant_batch_builder_test.cpp @@ -567,6 +567,23 @@ TEST(VariantBatchBuilderTest, IntegerAndDecimalWidthsAreMinimal) { EXPECT_EQ(value.array_at(integers.size() + decimals.size()).get_decimal().width, 16); } +TEST(VariantBatchBuilderTest, AddValuePreservesExplicitPrimitiveWidths) { + const OwnedBuilderValue source = build_owned_value([](VariantBatchBuilder::Row& row) { + auto array = row.start_array(); + row.add_scalar(VariantScalarRef::integer(7, 8)); + row.add_scalar(VariantScalarRef::decimal(8, 2, 16)); + array.finish(); + }); + VariantBatchBuilder builder; + auto row = builder.begin_row(); + row.add_value(source.ref()); + row.finish(); + VariantBatchBuilder imported = builder.finish_batch(); + + EXPECT_EQ(imported.value_at(0).array_at(0).primitive_id(), VariantPrimitiveId::INT64); + EXPECT_EQ(imported.value_at(0).array_at(1).primitive_id(), VariantPrimitiveId::DECIMAL16); +} + TEST(VariantBatchBuilderTest, DecimalValidationLargeIntFallbackAndExplicitWidths) { VariantBatchBuilder builder; auto row = builder.begin_row(); diff --git a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql new file mode 100644 index 00000000000000..0f5804f610b6fe --- /dev/null +++ b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql @@ -0,0 +1,70 @@ +use paimon; +create database if not exists test_paimon_spark; +use test_paimon_spark; + +drop table if exists variant_smoke; +create table variant_smoke ( + id BIGINT, + payload VARIANT +) using paimon +tblproperties ( + 'file.format' = 'parquet' +); + +insert into variant_smoke values + (1, parse_json('{"name":"alice","age":18,"active":true,"score":98.5,"tags":["flink","paimon"],"profile":{"city":"beijing","zip":100000},"missing":null}')), + (2, parse_json('{"name":"bob","age":30,"active":false,"tags":["doris"],"profile":{"city":"shanghai"},"extra":{"levels":[1,2,3]}}')), + (3, parse_json('[1,"mixed",false,null,{"k":"v"}]')); + +drop table if exists variant_shredded; +create table variant_shredded ( + id BIGINT, + event_date DATE, + payload VARIANT +) using paimon +partitioned by (event_date) +tblproperties ( + 'file.format' = 'parquet', + 'parquet.variant.shreddingSchema' = '{"type":"ROW","fields":[{"name":"payload","type":{"type":"ROW","fields":[{"name":"name","type":"STRING"},{"name":"age","type":"INT"}]}}]}' +); + +insert into variant_shredded values + (1, date '2026-06-01', parse_json('{"name":"alice","age":18,"extra":"shredded"}')), + (2, date '2026-06-01', parse_json('{"name":"bob","age":30}')); + +drop table if exists variant_mixed_us; +create table variant_mixed_us ( + id BIGINT, + event_date DATE, + payload VARIANT +) using paimon +partitioned by (event_date) +tblproperties ('file.format' = 'parquet'); + +insert into variant_mixed_us values + (1, date '2026-06-01', parse_json('{"name":"alice","age":18,"layout":"unshredded"}')); +alter table variant_mixed_us set tblproperties ( + 'parquet.variant.shreddingSchema' = '{"type":"ROW","fields":[{"name":"payload","type":{"type":"ROW","fields":[{"name":"name","type":"STRING"},{"name":"age","type":"INT"}]}}]}' +); +insert into variant_mixed_us values + (2, date '2026-07-01', parse_json('{"name":"bob","age":30,"layout":"shredded"}')); + +drop table if exists variant_mixed_su; +create table variant_mixed_su ( + id BIGINT, + event_date DATE, + payload VARIANT +) using paimon +partitioned by (event_date) +tblproperties ( + 'file.format' = 'parquet', + 'parquet.variant.shreddingSchema' = '{"type":"ROW","fields":[{"name":"payload","type":{"type":"ROW","fields":[{"name":"name","type":"STRING"},{"name":"age","type":"INT"}]}}]}' +); + +insert into variant_mixed_su values + (1, date '2026-06-01', parse_json('{"name":"alice","age":18,"layout":"shredded"}')); +alter table variant_mixed_su set tblproperties ( + 'parquet.variant.shreddingSchema' = '' +); +insert into variant_mixed_su values + (2, date '2026-07-01', parse_json('{"name":"bob","age":30,"layout":"unshredded"}')); diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out index 6abb509f237f85..ca9e630f18b677 100644 --- a/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out @@ -66,6 +66,12 @@ 4 40 \N \N \N {"c":4,"shared":40} 5 50 \N 5 500 {"b":5,"new_field":{"k":500},"shared":50} +-- !variant_projected_remote_gather -- +4093 4093 +4094 4094 +4095 4095 +5000 5000 + -- !variant_type_matrix -- true -128 -32768 2147483647 -9223372036854775808 true true -1234567890.1234 1970-01-02 1970-01-01T00:00:01.234567 "YmluYXJ5" false diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out new file mode 100644 index 00000000000000..9334ed1344d8ac --- /dev/null +++ b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out @@ -0,0 +1,68 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !desc -- +id bigint Yes true \N +payload variant Yes true \N + +-- !full_variant -- +1 {"active":true,"age":18,"missing":null,"name":"alice","profile":{"city":"beijing","zip":100000},"score":98.5,"tags":["flink","paimon"]} +2 {"active":false,"age":30,"extra":{"levels":[1,2,3]},"name":"bob","profile":{"city":"shanghai"},"tags":["doris"]} +3 [1,"mixed",false,null,{"k":"v"}] + +-- !object_subpaths -- +1 alice 18 beijing true +2 bob 30 shanghai false +3 \N \N \N \N + +-- !null_and_missing -- +1 false true +2 true true +3 true true + +-- !root_array -- +3 1 mixed false null v + +-- !subpath_predicate -- +2 bob + +-- !native_full_variant -- +1 {"active":true,"age":18,"missing":null,"name":"alice","profile":{"city":"beijing","zip":100000},"score":98.5,"tags":["flink","paimon"]} +2 {"active":false,"age":30,"extra":{"levels":[1,2,3]},"name":"bob","profile":{"city":"shanghai"},"tags":["doris"]} +3 [1,"mixed",false,null,{"k":"v"}] + +-- !native_object_subpaths -- +1 alice 18 beijing true +2 bob 30 shanghai false +3 \N \N \N \N + +-- !native_null_and_missing -- +1 false true +2 true true +3 true true + +-- !native_root_array -- +3 1 mixed false null v + +-- !native_subpath_predicate -- +2 bob + +-- !native_shredded_projection -- +2 bob 30 \N + +-- !native_mixed_us_partitions -- +1 2026-06-01 alice 18 unshredded +2 2026-07-01 bob 30 shredded + +-- !native_mixed_us_root -- +1 {"age":18,"layout":"unshredded","name":"alice"} +2 {"age":30,"layout":"shredded","name":"bob"} + +-- !native_mixed_su_partitions -- +1 2026-06-01 alice 18 shredded +2 2026-07-01 bob 30 unshredded + +-- !native_mixed_su_root -- +1 {"age":18,"layout":"shredded","name":"alice"} +2 {"age":30,"layout":"unshredded","name":"bob"} + +-- !native_mixed_us_mtmv -- +alice 1 diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index ffc99236a1976f..b00bb5924b9f69 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -482,17 +482,6 @@ public class AppendVariantEqualityDelete { } return sum } - def profileInfoValues = { String profile, String infoName -> - Pattern pattern = Pattern.compile( - Pattern.quote(infoName) + ":\\s*\\[([^\\]]*)\\]") - Matcher matcher = pattern.matcher(profile) - if (!matcher.find()) { - return [] - } - return matcher.group(1).split(",").collect { String value -> value.trim() } - .findAll { String value -> !value.isEmpty() } - .collect { String value -> Long.parseLong(value.replace(",", "")) } - } def getProfileByToken = { String token, List positiveCounters = [] -> String lastProfile = profileAction.getProfileBySql(token, positiveCounters) if (positiveCounters.every { String counter -> counterSum(lastProfile, counter) > 0 }) { @@ -649,6 +638,9 @@ public class AppendVariantEqualityDelete { sql "set parallel_pipeline_task_num=4" sql "set max_file_scanners_concurrency=8" sql "set min_file_scanners_concurrency=4" + // Scanner tasks pull file ranges dynamically, so minimum concurrency does not guarantee that + // every scheduled scanner consumes rows. Validate parallel correctness without pinning the + // scheduler's nondeterministic range assignment. order_qt_variant_multi_file_parallel """ SELECT id, CAST(v['shared'] AS INT), @@ -660,35 +652,56 @@ public class AppendVariantEqualityDelete { WHERE v['shared'] >= 20 ORDER BY id """ - String parallelScanToken = - "iceberg_variant_parallel_scan_" + UUID.randomUUID().toString() - List> parallelScanRows = sql """ - SELECT '${parallelScanToken}', id, - CAST(v['shared'] AS INT), - CAST(v['a'] AS INT), - CAST(v['b'] AS INT), - CAST(v['new_field']['k'] AS INT), - CAST(v AS STRING) - FROM variant_multi_file - WHERE v['shared'] >= 20 + // The stable snapshot contributes a genuinely shredded file, while the appended file uses + // the unshredded fallback. More than four rows qualify, forcing local TopN overshoot to be + // truncated after the merge exchange while the mapper-eligible projected path crosses the wire. + explain { + sql """ + SELECT id, CAST(projected['n'] AS INT) + FROM ( + SELECT id, v AS projected + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + ORDER BY id DESC + LIMIT 4 + ) gathered + """ + contains "VMERGING-EXCHANGE" + contains "inputSplitNum=2" + contains "all access paths: [v(2).n]" + } + String projectedGatherToken = + "iceberg_variant_projected_remote_gather_" + UUID.randomUUID().toString() + List> projectedGatherRows = sql """ + SELECT '${projectedGatherToken}', id, CAST(projected['n'] AS INT) + FROM ( + SELECT id, v AS projected + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + ORDER BY id DESC + LIMIT 4 + ) gathered + ORDER BY id + """ + assertEquals(4, projectedGatherRows.size()) + String projectedGatherProfile = getProfileByToken(projectedGatherToken, + ["VariantLeafProjections", "VariantDirectLeafPathMisses"]).toString() + assertTrue(counterSum(projectedGatherProfile, "VariantLeafProjections") > 0, + "The projected TopN did not read a physical shredded Variant leaf") + assertTrue(counterSum(projectedGatherProfile, "VariantDirectLeafPathMisses") > 0, + "The projected TopN did not combine the unshredded fallback file") + order_qt_variant_projected_remote_gather """ + SELECT id, + CAST(projected['n'] AS INT) + FROM ( + SELECT id, v AS projected + FROM variant_page_pruning FOR VERSION AS OF ${mixedBeforeDeleteSnapshot} + WHERE CAST(v['n'] AS INT) > 3000 + ORDER BY id DESC + LIMIT 4 + ) gathered ORDER BY id """ - assertEquals(4, parallelScanRows.size(), - "The parallel Variant query must read rows from multiple data files") - String parallelScanProfile = profileAction.getProfileBySql( - parallelScanToken, ["PerScannerRowsRead"]) - if (profileInfoValues(parallelScanProfile, "PerScannerRowsRead") - .count { long rows -> rows > 0 } <= 1) { - parallelScanProfile = profileAction.waitProfile({ - String profile = profileAction.getProfileBySql( - parallelScanToken, ["PerScannerRowsRead"]) - return profileInfoValues(profile, "PerScannerRowsRead") - .count { long rows -> rows > 0 } > 1 ? profile : "" - }, [], "Completed parallel Variant profile with multiple non-empty scanners") - } - assertTrue(profileInfoValues(parallelScanProfile, "PerScannerRowsRead") - .count { long rows -> rows > 0 } > 1, - "The parallel Variant query did not use multiple non-empty scanners") sql "set min_file_scanners_concurrency=1" order_qt_variant_type_matrix """ diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy new file mode 100644 index 00000000000000..203d5788310aa2 --- /dev/null +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy @@ -0,0 +1,224 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_paimon_catalog_variant", "p0,external,doris,external_docker,external_docker_doris") { + String enabled = context.config.otherConfigs.get("enablePaimonTest") + if (enabled != null && enabled.equalsIgnoreCase("true")) { + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String catalogName = "test_paimon_variant" + + sql """drop catalog if exists ${catalogName}""" + sql """create catalog if not exists ${catalogName} properties ( + "type" = "paimon", + "paimon.catalog.type" = "filesystem", + "warehouse" = "s3://warehouse/wh", + "s3.endpoint" = "http://${externalEnvIp}:${minioPort}", + "s3.access_key" = "admin", + "s3.secret_key" = "password", + "s3.region" = "us-east-1", + "s3.path.style.access" = "true" + );""" + sql """use `${catalogName}`.`test_paimon_spark`""" + sql """set enable_variant_v2 = true""" + sql """set force_jni_scanner = true""" + + explain { + sql "select * from variant_smoke order by id" + contains "paimonNativeReadSplits=0/1" + } + + order_qt_desc """desc variant_smoke""" + + order_qt_full_variant """ + select id, payload + from variant_smoke + order by id + """ + + order_qt_object_subpaths """ + select id, + cast(payload['name'] as string), + cast(payload['age'] as int), + cast(payload['profile']['city'] as string), + cast(payload['active'] as boolean) + from variant_smoke + order by id + """ + + order_qt_null_and_missing """ + select id, + payload['missing'] is null, + payload['not_exist'] is null + from variant_smoke + order by id + """ + + order_qt_root_array """ + select id, + cast(payload[1] as int), + cast(payload[2] as string), + cast(payload[3] as boolean), + cast(payload[4] as string), + cast(payload[5]['k'] as string) + from variant_smoke + where id = 3 + order by id + """ + + order_qt_subpath_predicate """ + select id, cast(payload['name'] as string) + from variant_smoke + where cast(payload['age'] as int) >= 20 + order by id + """ + + sql """set force_jni_scanner = false""" + + explain { + sql "select * from variant_smoke order by id" + check { explainString -> + def nativeSplits = explainString =~ /paimonNativeReadSplits=(\d+)\/(\d+)/ + // Paimon can change the physical split count; every planned split must stay native. + return nativeSplits.find() + && nativeSplits.group(1).toInteger() > 0 + && nativeSplits.group(1) == nativeSplits.group(2) + } + } + + order_qt_native_full_variant """ + select id, payload + from variant_smoke + order by id + """ + + order_qt_native_object_subpaths """ + select id, + cast(payload['name'] as string), + cast(payload['age'] as int), + cast(payload['profile']['city'] as string), + cast(payload['active'] as boolean) + from variant_smoke + order by id + """ + + order_qt_native_null_and_missing """ + select id, + payload['missing'] is null, + payload['not_exist'] is null + from variant_smoke + order by id + """ + + order_qt_native_root_array """ + select id, + cast(payload[1] as int), + cast(payload[2] as string), + cast(payload[3] as boolean), + cast(payload[4] as string), + cast(payload[5]['k'] as string) + from variant_smoke + where id = 3 + order by id + """ + + order_qt_native_subpath_predicate """ + select id, cast(payload['name'] as string) + from variant_smoke + where cast(payload['age'] as int) >= 20 + order by id + """ + + ["variant_shredded", "variant_mixed_us", "variant_mixed_su"].each { tableName -> + explain { + sql "select * from ${tableName} order by id" + check { explainString -> + def nativeSplits = explainString =~ /paimonNativeReadSplits=(\d+)\/(\d+)/ + return nativeSplits.find() + && nativeSplits.group(1).toInteger() > 0 + && nativeSplits.group(1) == nativeSplits.group(2) + } + } + } + + order_qt_native_shredded_projection """ + select id, + cast(payload['name'] as string), + cast(payload['age'] as int), + cast(payload['extra'] as string) + from variant_shredded + where cast(payload['age'] as int) >= 20 + order by id + """ + + order_qt_native_mixed_us_partitions """ + select id, + cast(event_date as string), + cast(payload['name'] as string), + cast(payload['age'] as int), + cast(payload['layout'] as string) + from variant_mixed_us + order by id + """ + + order_qt_native_mixed_us_root """ + select id, cast(payload as string) + from variant_mixed_us + order by id + """ + + order_qt_native_mixed_su_partitions """ + select id, + cast(event_date as string), + cast(payload['name'] as string), + cast(payload['age'] as int), + cast(payload['layout'] as string) + from variant_mixed_su + order by id + """ + + order_qt_native_mixed_su_root """ + select id, cast(payload as string) + from variant_mixed_su + order by id + """ + + String internalDb = context.config.getDbNameByFile(context.file) + String mvName = "paimon_variant_mixed_mv" + sql """switch internal""" + sql """use `${internalDb}`""" + sql """drop materialized view if exists ${mvName}""" + try { + sql """ + create materialized view ${mvName} + build deferred refresh complete on manual + distributed by random buckets 1 + properties ('replication_num' = '1') + as + select cast(payload['name'] as string) as name, count(*) as row_count + from ${catalogName}.`test_paimon_spark`.variant_mixed_us + where event_date = '2026-06-01' + group by cast(payload['name'] as string) + """ + sql """refresh materialized view ${mvName} complete""" + waitingMTMVTaskFinishedByMvName(mvName) + order_qt_native_mixed_us_mtmv """select * from ${mvName} order by name""" + } finally { + sql """drop materialized view if exists ${mvName}""" + } + } +} From b08ccf245fd035469a15eca201d17abf59974800 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 08:42:45 +0800 Subject: [PATCH 13/20] [fix](scan) Preserve runtime filter refresh layout --- be/src/format_v2/table_reader.cpp | 60 ++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 17 deletions(-) diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 22cc6928762ad0..0ca97f527849b2 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -1079,31 +1079,57 @@ Status TableReader::_build_table_filters_from_conjuncts() { namespace { -bool same_scan_projections(const std::vector& lhs, - const std::vector& rhs) { - if (lhs.size() != rhs.size()) { +const LocalColumnIndex* scan_projection_at_position(const FileScanRequest& request, + LocalColumnId column_id, + bool deferred_non_predicate) { + const auto find_by_id = [column_id](const std::vector& projections) { + return std::ranges::find_if(projections, [column_id](const LocalColumnIndex& projection) { + return projection.column_id() == column_id; + }); + }; + if (deferred_non_predicate) { + const auto it = find_by_id(request.non_predicate_columns); + return it == request.non_predicate_columns.end() ? nullptr : &*it; + } + + auto it = find_by_id(request.predicate_columns); + if (it != request.predicate_columns.end()) { + return &*it; + } + it = find_by_id(request.non_predicate_columns); + return it == request.non_predicate_columns.end() ? nullptr : &*it; +} + +bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest& rhs) { + if (lhs.local_positions != rhs.local_positions || + lhs.non_predicate_positions != rhs.non_predicate_positions) { return false; } - for (const auto& lhs_projection : lhs) { - const auto rhs_it = std::ranges::find_if(rhs, [&](const LocalColumnIndex& rhs_projection) { - return rhs_projection.column_id() == lhs_projection.column_id(); - }); - if (rhs_it == rhs.end() || !same_local_column_index(lhs_projection, *rhs_it)) { + const auto same_projection = [&](LocalColumnId column_id, bool deferred_non_predicate) { + const auto* lhs_projection = + scan_projection_at_position(lhs, column_id, deferred_non_predicate); + const auto* rhs_projection = + scan_projection_at_position(rhs, column_id, deferred_non_predicate); + return lhs_projection != nullptr && rhs_projection != nullptr && + same_local_column_index(*lhs_projection, *rhs_projection); + }; + for (const auto& [column_id, _] : lhs.local_positions) { + // A late filter may reclassify a root as a predicate without moving its physical slot. + // Category membership is therefore not part of the immutable reader layout. + if (!same_projection(column_id, false)) { + return false; + } + } + for (const auto& [column_id, _] : lhs.non_predicate_positions) { + // Deferred complex roots have a second physical slot whose output projection must remain + // stable independently from the eager predicate projection above. + if (!same_projection(column_id, true)) { return false; } } return true; } -bool same_physical_scan_layout(const FileScanRequest& lhs, const FileScanRequest& rhs) { - // Deferred complex roots occupy independent output slots. Comparing only eager positions can - // accept a refresh whose second Variant root now aliases or overruns the active block layout. - return lhs.local_positions == rhs.local_positions && - lhs.non_predicate_positions == rhs.non_predicate_positions && - same_scan_projections(lhs.predicate_columns, rhs.predicate_columns) && - same_scan_projections(lhs.non_predicate_columns, rhs.non_predicate_columns); -} - } // namespace Status TableReader::refresh_conjuncts(VExprContextSPtrs conjuncts) { From 2369eecb082aa5c9696cd9291acebbbf25eafbd1 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 13:25:03 +0800 Subject: [PATCH 14/20] fix: restore null predicate helper after rebase --- be/test/format_v2/column_mapper_test.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index 0e06473abe06e0..c63121f41ea8c2 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -532,6 +532,14 @@ VExprSPtr binary_predicate(TExprOpcode::type opcode, const VExprSPtr& left, return expr; } +VExprSPtr null_predicate(const VExprSPtr& child, bool is_null) { + // Preserve the nested child expression so filter-only access paths remain discoverable. + auto expr = + std::make_shared(is_null ? "is_null_pred" : "is_not_null_pred", u8()); + expr->add_child(child); + return expr; +} + VExprSPtr cast_expr(const VExprSPtr& child, DataTypePtr target_type) { auto expr = Cast::create_shared(std::move(target_type)); expr->add_child(child); From 0323ef756cce1388a1f05a5539ac8c34c3e97dd5 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 13:38:47 +0800 Subject: [PATCH 15/20] test: align timestamp projection after rebase --- be/test/format_v2/column_mapper_test.cpp | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index c63121f41ea8c2..4c3f23ec262a4c 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -2572,13 +2572,17 @@ TEST(ColumnMapperScanRequestTest, FilterOnlyNestedTimestampRetainsTableFormatSem auto projected_table_struct = struct_col("s", 10, {table_payload}); auto table_ltz = field_id_col("ltz", 2, ltz_type); auto full_table_struct = struct_col("s", 10, {table_payload, table_ltz}); + // Parquet keeps the FE-provided predicate subtree independent from deferred output projection. + projected_table_struct.type = full_table_struct.type; + projected_table_struct.has_predicate_access_paths = true; + projected_table_struct.predicate_children = {table_ltz}; auto file_payload = field_id_col("payload", 1, int_type, 0); auto file_ltz = field_id_col("ltz", 2, ltz_type, 1); file_ltz.timestamp_is_adjusted_to_utc = true; auto file_struct = struct_col("s", 10, {file_payload, file_ltz}, 5); - TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_FIELD_ID}); ASSERT_TRUE(mapper.create_mapping({projected_table_struct}, {}, {file_struct}).ok()); auto filter_expr = null_predicate( @@ -2591,11 +2595,13 @@ TEST(ColumnMapperScanRequestTest, FilterOnlyNestedTimestampRetainsTableFormatSem ASSERT_EQ(request.predicate_columns.size(), 1); const auto& root_projection = request.predicate_columns[0]; - ASSERT_EQ(projection_ids(root_projection.children), std::vector({0, 1})); + ASSERT_EQ(projection_ids(root_projection.children), std::vector({1})); const auto* ltz_projection = find_child_projection(&root_projection, 1); ASSERT_NE(ltz_projection, nullptr); ASSERT_TRUE(ltz_projection->timestamp_is_adjusted_to_utc.has_value()); EXPECT_TRUE(*ltz_projection->timestamp_is_adjusted_to_utc); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + EXPECT_EQ(projection_ids(request.non_predicate_columns[0].children), std::vector({0})); } // Scenario: a filter references a top-level column that is not projected by the query; the mapper From da9d3d96feddaf0d312c08c09bc82c381f2913c2 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 15:54:05 +0800 Subject: [PATCH 16/20] fix: map Paimon Variant in plugin schema --- .../doris/connector/paimon/PaimonTypeMapping.java | 3 +++ .../connector/paimon/PaimonTypeMappingReadTest.java | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java index 1f29ccd1194df7..9f6139f5033a13 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java @@ -97,6 +97,9 @@ public static ConnectorType toConnectorType(DataType dataType, Options options) return toTimestampType(dataType); case TIMESTAMP_WITH_LOCAL_TIME_ZONE: return toTimestampTzType(dataType, options); + case VARIANT: + // Preserve the execution-only carrier shared by JNI and native Paimon readers. + return ConnectorType.of("VARIANT_COMPUTE_V2"); case ARRAY: return toArrayType((ArrayType) dataType, options); case MAP: diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingReadTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingReadTest.java index 81d5e305f841e1..6a13073c127e1b 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingReadTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingReadTest.java @@ -23,6 +23,7 @@ import org.apache.paimon.types.IntType; import org.apache.paimon.types.RowType; import org.apache.paimon.types.VarCharType; +import org.apache.paimon.types.VariantType; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -81,4 +82,13 @@ public void nestedStructFieldCommentAndNullabilityCarried() { Assertions.assertTrue(struct.isChildNullable(1), "a nullable nested struct field stays nullable"); } + + @Test + public void variantUsesExecutionCarrierOnRead() { + ConnectorType variant = PaimonTypeMapping.toConnectorType( + new VariantType(), PaimonTypeMapping.Options.DEFAULT); + + Assertions.assertEquals("VARIANT_COMPUTE_V2", variant.getTypeName(), + "Paimon Variant must stay queryable in Nereids and both JNI and native readers"); + } } From a1de39a3232393b868b8ebb4e8bd839ec0889c33 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Thu, 6 Aug 2026 18:57:20 +0800 Subject: [PATCH 17/20] [fix](regression) Align external Variant expectations with master ### What problem does this PR solve? Issue Number: None Related PR: #66413 Problem Summary: The forward-ported Iceberg case expected the branch-4.1 nested-path rendering, while plugin-driven scans on master render projected paths by logical column name. The Paimon DESC golden also omitted the Extra column that master returns for every schema row. Align both expectations with the observed master output without weakening the execution checks. ### Release note None ### Check List (For Author) - Test: Regression framework unit tests (3 tests passed) and static validation of the six-column DESC golden shape. - Behavior changed: No - Does this need documentation: No --- .../external_table_p0/paimon/test_paimon_catalog_variant.out | 4 ++-- .../iceberg/test_iceberg_variant_read.groovy | 3 ++- .../paimon/test_paimon_catalog_variant.groovy | 1 + 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out index 9334ed1344d8ac..830e466af48556 100644 --- a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out +++ b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out @@ -1,7 +1,7 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !desc -- -id bigint Yes true \N -payload variant Yes true \N +id bigint Yes true \N +payload variant Yes true \N -- !full_variant -- 1 {"active":true,"age":18,"missing":null,"name":"alice","profile":{"city":"beijing","zip":100000},"score":98.5,"tags":["flink","paimon"]} diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy index b00bb5924b9f69..b217de1fdc5861 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy @@ -668,7 +668,8 @@ public class AppendVariantEqualityDelete { """ contains "VMERGING-EXCHANGE" contains "inputSplitNum=2" - contains "all access paths: [v(2).n]" + // Plugin-driven EXPLAIN identifies projected paths by logical column name, independent of slot IDs. + contains "all access paths: [v.n]" } String projectedGatherToken = "iceberg_variant_projected_remote_gather_" + UUID.randomUUID().toString() diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy index 203d5788310aa2..becc0d267692cf 100644 --- a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy +++ b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy @@ -42,6 +42,7 @@ suite("test_paimon_catalog_variant", "p0,external,doris,external_docker,external contains "paimonNativeReadSplits=0/1" } + // Keep the complete DESC shape so the external column's Extra metadata remains covered. order_qt_desc """desc variant_smoke""" order_qt_full_variant """ From fc777db11cdc972308e84078dad751f628d262b3 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Fri, 7 Aug 2026 17:34:39 +0800 Subject: [PATCH 18/20] [fix](iceberg) Harden schema evolution and nested partition writes (#66529) ## Proposed changes - make Iceberg compatibility gates conservative and bounded while pinning column handles to historical schemas - reset connector metadata across INSERT retries and align V1/V2 defaults, required-field checks, and position-delete row projection - support primitive partition sources nested in structs, including nullable-parent propagation and regression coverage - cache the compact equality-delete field-ID projection by immutable table snapshot even when the optional full manifest cache is disabled - resolve nested partition sources through the top-level Nereids slot ID and fail closed when stable Iceberg IDs are unavailable ## Compatibility-gate trade-offs - Equality-delete fencing remains conservative across all delete manifests in the selected snapshot, including partition-pruned scans. The snapshot-scoped field-ID cache removes repeated manifest walks without weakening correctness; initial-load failures still fail closed and remain retryable. - Requiredness fencing intentionally uses bounded schema-history inspection rather than an O(snapshot-count) ancestry walk because snapshot schema IDs are optional. Once a projected requiredness hazard exists, every non-empty selected snapshot is fenced. This can reduce rolling-upgrade availability but cannot create a correctness false negative. ## Testing - `mvn -pl fe-core,fe-connector/fe-connector-iceberg -am -Dtest=PhysicalExternalRowLevelMergeSinkTest,IcebergManifestCacheTest,IcebergScanPlanProviderTest -Dsurefire.failIfNoSpecifiedTests=false test` - `mvn -pl fe-connector/fe-connector-iceberg -am -Dtest=IcebergScanPlanProviderTest,IcebergConnectorMetadataTest,IcebergWritePlanProviderTest -Dsurefire.failIfNoSpecifiedTests=false test` - `mvn -pl fe-core -am -Dtest=ConnectorStatementScopeTest,InsertIntoTableCommandTest -Dsurefire.failIfNoSpecifiedTests=false test` - `./run-be-ut.sh --run --filter=SchemaTest.*:VIcebergTableWriterTest.*:IcebergReaderTest.v1_materializes_non_finite_initial_defaults:IcebergV2ReaderTest.PreparesIcebergNonFiniteInitialDefaults:IcebergPositionDeleteSysTableV2ProfileTest.*` - FE Checkstyle for all affected modules - clang-format 16 check for all changed C/C++ files --- .../writer/iceberg/viceberg_table_writer.cpp | 112 ++++++++-- .../writer/iceberg/viceberg_table_writer.h | 7 + be/src/format/table/iceberg/schema.cpp | 29 ++- be/src/format/table/iceberg/schema.h | 4 + be/src/format/table/iceberg_default_value.h | 31 ++- .../iceberg_partition_function.cpp | 75 ++++++- .../transformer/iceberg_partition_function.h | 4 + ...eberg_position_delete_sys_table_reader.cpp | 6 + be/src/format_v2/table/iceberg_reader.cpp | 15 +- be/src/format_v2/table/iceberg_reader.h | 3 + be/test/core/value/merge_partitioner_test.cpp | 91 ++++++++ .../iceberg/viceberg_table_writer_test.cpp | 63 ++++++ .../table/iceberg/iceberg_reader_test.cpp | 17 ++ be/test/format/table/iceberg/schema_test.cpp | 31 +++ ..._position_delete_sys_table_reader_test.cpp | 25 +++ .../format_v2/table/iceberg_reader_test.cpp | 33 +++ .../iceberg/IcebergConnectorMetadata.java | 26 ++- .../iceberg/IcebergManifestCache.java | 54 ++++- .../iceberg/IcebergScanPlanProvider.java | 196 ++++++++---------- .../iceberg/IcebergWritePlanProvider.java | 34 ++- .../iceberg/IcebergWriteSchemaContext.java | 7 +- .../iceberg/IcebergConnectorMetadataTest.java | 36 ++++ .../iceberg/IcebergManifestCacheTest.java | 55 +++++ .../iceberg/IcebergScanPlanProviderTest.java | 135 +++++++++++- .../iceberg/IcebergWritePlanProviderTest.java | 34 +++ .../datasource/scan/PluginDrivenScanNode.java | 8 +- .../translator/PhysicalPlanTranslator.java | 3 +- .../properties/DistributionSpecMerge.java | 17 +- .../insert/InsertIntoTableCommand.java | 3 + .../PhysicalExternalRowLevelMergeSink.java | 99 ++++++++- .../apache/doris/planner/DataPartition.java | 10 + ...PluginDrivenScanNodeColumnPruningTest.java | 15 ++ ...PhysicalExternalRowLevelMergeSinkTest.java | 85 ++++++++ gensrc/thrift/Partitions.thrift | 2 + .../test_iceberg_write_complex_evolution.out | 12 +- ...est_iceberg_write_complex_evolution.groovy | 26 ++- 36 files changed, 1229 insertions(+), 174 deletions(-) diff --git a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp index e3f8ed645edb26..3519252d7ca940 100644 --- a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp +++ b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp @@ -17,14 +17,18 @@ #include "exec/sink/writer/iceberg/viceberg_table_writer.h" +#include + #include "common/exception.h" #include "core/block/block.h" #include "core/block/column_with_type_and_name.h" #include "core/block/materialize_block.h" #include "core/column/column_const.h" #include "core/column/column_nullable.h" +#include "core/column/column_struct.h" #include "core/column/column_vector.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_struct.h" #include "core/data_type_serde/data_type_serde.h" #include "exec/sink/writer/iceberg/iceberg_partition_path.h" #include "exec/sink/writer/iceberg/partition_transformers.h" @@ -121,31 +125,106 @@ std::vector VIcebergTableWriter::_to_iceberg_partition_columns() { std::vector partition_columns; - std::unordered_map id_to_column_idx; - id_to_column_idx.reserve(_schema->columns().size()); - for (int i = 0; i < _schema->columns().size(); i++) { - id_to_column_idx[_schema->columns()[i].field_id()] = i; - } for (const auto& partition_field : _partition_spec->fields()) { - auto column_idx_it = id_to_column_idx.find(partition_field.source_id()); - if (column_idx_it == id_to_column_idx.end()) { + const auto* field_path = _schema->find_field_path(partition_field.source_id()); + if (field_path == nullptr || field_path->empty()) { throw Exception( ErrorCode::INTERNAL_ERROR, "Iceberg partition field {} references source field {} outside writer schema", partition_field.field_id(), partition_field.source_id()); } - int column_idx = column_idx_it->second; + int column_idx = -1; + for (int i = 0; i < _schema->columns().size(); ++i) { + if (_schema->columns()[i].field_id() == field_path->front()->field_id()) { + column_idx = i; + break; + } + } + DORIS_CHECK(column_idx >= 0); + std::vector child_indices; + iceberg::Type* iceberg_type = field_path->front()->field_type(); + DataTypePtr source_type = _vec_output_expr_ctxs[column_idx]->root()->data_type(); + for (size_t depth = 1; depth < field_path->size(); ++depth) { + if (!iceberg_type->is_struct_type()) { + throw Exception(ErrorCode::INTERNAL_ERROR, + "Iceberg partition source field {} has a non-struct ancestor", + partition_field.source_id()); + } + const auto& fields = iceberg_type->as_struct_type()->fields(); + auto child = std::find_if(fields.begin(), fields.end(), [&](const auto& candidate) { + return candidate.field_id() == (*field_path)[depth]->field_id(); + }); + DORIS_CHECK(child != fields.end()); + const size_t child_idx = std::distance(fields.begin(), child); + const auto* struct_type = + check_and_get_data_type(remove_nullable(source_type).get()); + if (struct_type == nullptr || child_idx >= struct_type->get_elements().size()) { + throw Exception( + ErrorCode::INTERNAL_ERROR, + "Iceberg nested partition source field {} does not match writer type", + partition_field.source_id()); + } + child_indices.push_back(child_idx); + iceberg_type = child->field_type(); + source_type = struct_type->get_element(child_idx); + } + if (!iceberg_type->is_primitive_type()) { + throw Exception(ErrorCode::INTERNAL_ERROR, + "Iceberg partition source field {} is not primitive", + partition_field.source_id()); + } std::unique_ptr partition_column_transform = - PartitionColumnTransforms::create( - partition_field, _vec_output_expr_ctxs[column_idx]->root()->data_type()); + PartitionColumnTransforms::create(partition_field, source_type); partition_columns.emplace_back( - partition_field, - _vec_output_expr_ctxs[column_idx]->root()->data_type()->get_primitive_type(), - column_idx, std::move(partition_column_transform)); + partition_field, remove_nullable(source_type)->get_primitive_type(), column_idx, + std::move(child_indices), std::move(partition_column_transform)); } return partition_columns; } +ColumnWithTypeAndName VIcebergTableWriter::_nested_partition_source( + const Block& block, const IcebergPartitionColumn& partition_column) const { + ColumnWithTypeAndName source = block.get_by_position(partition_column.source_idx()); + if (partition_column.child_indices().empty()) { + return source; + } + ColumnPtr column = source.column->convert_to_full_column_if_const(); + DataTypePtr type = source.type; + auto combined_nulls = ColumnUInt8::create(block.rows(), 0); + bool nullable = false; + auto unwrap_nullable = [&]() { + if (const auto* nullable_column = check_and_get_column(column.get())) { + nullable = true; + const auto& nulls = nullable_column->get_null_map_data(); + auto& combined = combined_nulls->get_data(); + for (size_t row = 0; row < combined.size(); ++row) { + combined[row] |= nulls[row]; + } + column = nullable_column->get_nested_column_ptr(); + type = remove_nullable(type); + } + }; + for (size_t child_idx : partition_column.child_indices()) { + unwrap_nullable(); + const auto* struct_column = check_and_get_column(column.get()); + const auto* struct_type = check_and_get_data_type(type.get()); + if (struct_column == nullptr || struct_type == nullptr || + child_idx >= struct_column->tuple_size()) { + throw Exception(ErrorCode::INTERNAL_ERROR, + "Iceberg nested partition source does not match writer block"); + } + column = struct_column->get_column_ptr(child_idx); + type = struct_type->get_element(child_idx); + } + // Parent NULL masks the leaf even when the nested storage column contains a materialized value. + unwrap_nullable(); + if (nullable) { + column = ColumnNullable::create(column, std::move(combined_nulls)); + type = make_nullable(type); + } + return {std::move(column), std::move(type), source.name}; +} + void VIcebergTableWriter::_init_static_partition_values() { auto& iceberg_sink = _t_sink.iceberg_table_sink; if (!iceberg_sink.__isset.static_partition_values || @@ -358,9 +437,12 @@ Status VIcebergTableWriter::_write_prepared_block(Block& output_block) { transformed_block.insert( {std::move(col), result_type, iceberg_partition_columns.field().name()}); } else { + Block source_block; + source_block.insert( + _nested_partition_source(output_block, iceberg_partition_columns)); transformed_block.insert( - iceberg_partition_columns.partition_column_transform().apply( - output_block, iceberg_partition_columns.source_idx())); + iceberg_partition_columns.partition_column_transform().apply(source_block, + 0)); } } for (int i = 0; i < output_block.rows(); ++i) { diff --git a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.h b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.h index 2cb83f73ed0691..20d9562c35ee52 100644 --- a/be/src/exec/sink/writer/iceberg/viceberg_table_writer.h +++ b/be/src/exec/sink/writer/iceberg/viceberg_table_writer.h @@ -83,6 +83,7 @@ class VIcebergTableWriter final : public AsyncResultWriter { private: FRIEND_TEST(VIcebergTableWriterTest, RejectMissingPartitionSource); + FRIEND_TEST(VIcebergTableWriterTest, ResolvesNestedPartitionSource); // The currently active partition writer (may be VIcebergPartitionWriter or VIcebergSortWriter). // Updated during write() to track which writer received the most recent data. @@ -93,10 +94,12 @@ class VIcebergTableWriter final : public AsyncResultWriter { public: IcebergPartitionColumn(const iceberg::PartitionField& field, const PrimitiveType& source_type, int source_idx, + std::vector child_indices, std::unique_ptr partition_column_transform) : _field(field), _source_type(source_type), _source_idx(source_idx), + _child_indices(std::move(child_indices)), _partition_column_transform(std::move(partition_column_transform)) {} public: @@ -104,6 +107,7 @@ class VIcebergTableWriter final : public AsyncResultWriter { const PrimitiveType& source_type() const { return _source_type; } int source_idx() const { return _source_idx; } + const std::vector& child_indices() const { return _child_indices; } const PartitionColumnTransform& partition_column_transform() const { return *_partition_column_transform; @@ -117,10 +121,13 @@ class VIcebergTableWriter final : public AsyncResultWriter { const iceberg::PartitionField& _field; PrimitiveType _source_type; int _source_idx; + std::vector _child_indices; std::unique_ptr _partition_column_transform; }; std::vector _to_iceberg_partition_columns(); + ColumnWithTypeAndName _nested_partition_source( + const Block& block, const IcebergPartitionColumn& partition_column) const; std::string _partition_to_path(const doris::iceberg::StructLike& data); std::string _escape(const std::string& path); diff --git a/be/src/format/table/iceberg/schema.cpp b/be/src/format/table/iceberg/schema.cpp index 76dac166327985..a05456627bf980 100644 --- a/be/src/format/table/iceberg/schema.cpp +++ b/be/src/format/table/iceberg/schema.cpp @@ -17,6 +17,8 @@ #include "format/table/iceberg/schema.h" +#include + namespace doris::iceberg { const std::string Schema::ALL_COLUMNS = "*"; @@ -24,10 +26,26 @@ const int Schema::DEFAULT_SCHEMA_ID = 0; Schema::Schema(int schema_id, std::vector columns) : _schema_id(schema_id), _root_struct(std::move(columns)) { - _id_to_field.reserve(_root_struct.fields().size()); + FieldPath path; + std::function index_field = [&](const NestedField& field) { + path.push_back(&field); + _id_to_field[field.field_id()] = &field; + _id_to_field_path[field.field_id()] = path; + Type* type = field.field_type(); + if (type->is_struct_type()) { + for (const auto& child : type->as_struct_type()->fields()) { + index_field(child); + } + } else if (type->is_list_type()) { + index_field(type->as_list_type()->element_field()); + } else if (type->is_map_type()) { + index_field(type->as_map_type()->key_field()); + index_field(type->as_map_type()->value_field()); + } + path.pop_back(); + }; for (const auto& field : _root_struct.fields()) { - int field_id = field.field_id(); - _id_to_field[field_id] = &field; + index_field(field); } } Schema::Schema(std::vector columns) : Schema(DEFAULT_SCHEMA_ID, std::move(columns)) {} @@ -48,4 +66,9 @@ const NestedField* Schema::find_field(int id) const { return nullptr; } +const Schema::FieldPath* Schema::find_field_path(int id) const { + auto it = _id_to_field_path.find(id); + return it == _id_to_field_path.end() ? nullptr : &it->second; +} + } // namespace doris::iceberg diff --git a/be/src/format/table/iceberg/schema.h b/be/src/format/table/iceberg/schema.h index 0273a4450da324..5781b86b3fac4f 100644 --- a/be/src/format/table/iceberg/schema.h +++ b/be/src/format/table/iceberg/schema.h @@ -26,6 +26,7 @@ class StructType; class Schema { public: + using FieldPath = std::vector; Schema(int schema_id, std::vector columns); Schema(std::vector columns); @@ -40,6 +41,8 @@ class Schema { const NestedField* find_field(int id) const; + const FieldPath* find_field_path(int id) const; + private: static const char NEWLINE = '\n'; static const std::string ALL_COLUMNS; @@ -48,6 +51,7 @@ class Schema { int _schema_id; StructType _root_struct; std::unordered_map _id_to_field; + std::unordered_map _id_to_field_path; }; } // namespace doris::iceberg diff --git a/be/src/format/table/iceberg_default_value.h b/be/src/format/table/iceberg_default_value.h index ae75924336f676..5fe1834e3bcb23 100644 --- a/be/src/format/table/iceberg_default_value.h +++ b/be/src/format/table/iceberg_default_value.h @@ -24,6 +24,7 @@ #include #include +#include #include #include #include @@ -46,6 +47,28 @@ namespace doris::iceberg { namespace detail { +inline bool parse_non_finite_default(doris::PrimitiveType type, std::string_view value, + Field* result) { + DORIS_CHECK(result != nullptr); + if (type != TYPE_FLOAT && type != TYPE_DOUBLE) { + return false; + } + double parsed; + if (value == "NaN") { + parsed = std::numeric_limits::quiet_NaN(); + } else if (value == "Infinity") { + parsed = std::numeric_limits::infinity(); + } else if (value == "-Infinity") { + parsed = -std::numeric_limits::infinity(); + } else { + return false; + } + // Iceberg serializes non-finite defaults as strings, which generic Doris numeric parsers reject. + *result = type == TYPE_FLOAT ? Field::create_field(static_cast(parsed)) + : Field::create_field(parsed); + return true; +} + inline const schema::external::TField* get_field_ptr(const schema::external::TFieldPtr& field_ptr) { if (!field_ptr.__isset.field_ptr || field_ptr.field_ptr == nullptr) { return nullptr; @@ -135,7 +158,7 @@ inline std::string json_scalar_text(const rapidjson::Value& value) { return {buffer.GetString(), buffer.GetSize()}; } -inline void normalize_timestamp_for_doris(PrimitiveType primitive_type, std::string* value) { +inline void normalize_timestamp_for_doris(doris::PrimitiveType primitive_type, std::string* value) { if (primitive_type != TYPE_DATETIME && primitive_type != TYPE_DATETIMEV2 && primitive_type != TYPE_TIMESTAMPTZ) { return; @@ -344,6 +367,9 @@ inline Status build_json_scalar_default(const schema::external::TField& field, return Status::OK(); } normalize_timestamp_for_doris(primitive_type, &serialized_value); + if (parse_non_finite_default(primitive_type, serialized_value, result)) { + return Status::OK(); + } RETURN_IF_ERROR(value_type->get_serde()->from_fe_string(serialized_value, *result)); return Status::OK(); } @@ -422,6 +448,9 @@ inline Status build_initial_default_field(const schema::external::TField& field, return Status::OK(); } + if (parse_non_finite_default(primitive_type, field.initial_default_value, result)) { + return Status::OK(); + } RETURN_IF_ERROR(value_type->get_serde()->from_fe_string(field.initial_default_value, *result)); return Status::OK(); } diff --git a/be/src/format/transformer/iceberg_partition_function.cpp b/be/src/format/transformer/iceberg_partition_function.cpp index 3030fed3d600bd..4d134b062f3236 100644 --- a/be/src/format/transformer/iceberg_partition_function.cpp +++ b/be/src/format/transformer/iceberg_partition_function.cpp @@ -24,6 +24,8 @@ #include "core/column/column_const.h" #include "core/column/column_nullable.h" #include "core/column/column_struct.h" +#include "core/column/column_vector.h" +#include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_struct.h" #include "exec/sink/writer/iceberg/partition_transformers.h" #include "format/table/iceberg/partition_spec.h" @@ -88,6 +90,9 @@ Status IcebergInsertPartitionFunction::init(const std::vector& texprs) { insert_field.expr_ctx = std::move(ctx); insert_field.source_id = field.__isset.source_id ? field.source_id : 0; insert_field.name = field.__isset.name ? field.name : ""; + if (field.__isset.source_field_path) { + insert_field.source_field_path = field.source_field_path; + } _partition_fields.emplace_back(std::move(insert_field)); } } @@ -118,10 +123,21 @@ Status IcebergInsertPartitionFunction::open(RuntimeState* state) { RETURN_IF_ERROR(VExpr::open(field_ctxs, state)); for (auto& field : _partition_fields) { try { + DataTypePtr source_type = field.expr_ctx->root()->data_type(); + for (int32_t child_index : field.source_field_path) { + const auto* struct_type = check_and_get_data_type( + remove_nullable(source_type).get()); + if (child_index < 0 || struct_type == nullptr || + static_cast(child_index) >= struct_type->get_elements().size()) { + throw Exception(ErrorCode::INTERNAL_ERROR, + "Iceberg nested merge partition source does not match " + "expression type"); + } + source_type = struct_type->get_element(static_cast(child_index)); + } doris::iceberg::PartitionField partition_field(field.source_id, 0, field.name, field.transform); - field.transformer = PartitionColumnTransforms::create( - partition_field, field.expr_ctx->root()->data_type()); + field.transformer = PartitionColumnTransforms::create(partition_field, source_type); } catch (const doris::Exception& e) { LOG(WARNING) << "Merge partitioning fallback to RR: " << e.what(); _fallback_to_random = true; @@ -173,6 +189,7 @@ Status IcebergInsertPartitionFunction::clone(RuntimeState* state, field.expr_ctx = dst_field_ctxs[i]; field.source_id = _partition_fields[i].source_id; field.name = _partition_fields[i].name; + field.source_field_path = _partition_fields[i].source_field_path; new_function->_partition_fields.emplace_back(std::move(field)); } } @@ -180,6 +197,53 @@ Status IcebergInsertPartitionFunction::clone(RuntimeState* state, return Status::OK(); } +Status IcebergInsertPartitionFunction::_nested_partition_source( + size_t rows, const InsertPartitionField& field, ColumnWithTypeAndName* source) const { + if (field.source_field_path.empty()) { + return Status::OK(); + } + ColumnPtr column = source->column->convert_to_full_column_if_const(); + DataTypePtr type = source->type; + ColumnUInt8::MutablePtr combined_nulls; + bool nullable = false; + auto unwrap_nullable = [&]() { + if (const auto* nullable_column = check_and_get_column(column.get())) { + nullable = true; + if (!combined_nulls) { + combined_nulls = ColumnUInt8::create(rows, 0); + } + const auto& nulls = nullable_column->get_null_map_data(); + auto& combined = combined_nulls->get_data(); + for (size_t row = 0; row < combined.size(); ++row) { + combined[row] |= nulls[row]; + } + column = nullable_column->get_nested_column_ptr(); + type = remove_nullable(type); + } + }; + for (int32_t child_index : field.source_field_path) { + unwrap_nullable(); + const auto* struct_column = check_and_get_column(column.get()); + const auto* struct_type = check_and_get_data_type(type.get()); + if (child_index < 0 || struct_column == nullptr || struct_type == nullptr || + static_cast(child_index) >= struct_column->tuple_size()) { + return Status::InternalError( + "Iceberg nested merge partition source does not match input block"); + } + column = struct_column->get_column_ptr(static_cast(child_index)); + type = struct_type->get_element(static_cast(child_index)); + } + // A nullable parent masks a materialized child value; exchange routing must match the writer's partition. + unwrap_nullable(); + if (nullable) { + column = ColumnNullable::create(column, std::move(combined_nulls)); + type = make_nullable(type); + } + std::string name = source->name; + *source = {std::move(column), std::move(type), std::move(name)}; + return Status::OK(); +} + Status IcebergInsertPartitionFunction::_compute_hashes_with_transform( Block* block, std::vector& partitions) const { const size_t rows = block->rows(); @@ -202,8 +266,13 @@ Status IcebergInsertPartitionFunction::_compute_hashes_with_transform( if (_partition_fields[i].transformer == nullptr) { return Status::InternalError("Merge partitioning transform is not initialized"); } + ColumnWithTypeAndName source = block->get_by_position(results[i]); + if (!_partition_fields[i].source_field_path.empty()) { + RETURN_IF_ERROR(_nested_partition_source(rows, _partition_fields[i], &source)); + } + Block source_block({source}); ColumnWithTypeAndName transformed = - _partition_fields[i].transformer->apply(*block, results[i]); + _partition_fields[i].transformer->apply(source_block, 0); const auto& [column, is_const] = unpack_if_const(transformed.column); if (is_const) { // A const column has the same value for all rows in this block, diff --git a/be/src/format/transformer/iceberg_partition_function.h b/be/src/format/transformer/iceberg_partition_function.h index d2c1a25724bbfd..0ab36c91a0ee91 100644 --- a/be/src/format/transformer/iceberg_partition_function.h +++ b/be/src/format/transformer/iceberg_partition_function.h @@ -23,6 +23,7 @@ #include #include +#include "core/block/column_with_type_and_name.h" #include "exec/partitioner/partitioner.h" #include "exec/sink/writer/iceberg/partition_transformers.h" @@ -52,8 +53,11 @@ class IcebergInsertPartitionFunction final : public PartitionFunction { std::unique_ptr transformer; int32_t source_id = 0; std::string name; + std::vector source_field_path; }; + Status _nested_partition_source(size_t rows, const InsertPartitionField& field, + ColumnWithTypeAndName* source) const; Status _compute_hashes_with_transform(Block* block, std::vector& partitions) const; Status _compute_hashes_with_exprs(Block* block, std::vector& partitions) const; Status _clone_expr_ctxs(RuntimeState* state, const VExprContextSPtrs& src, diff --git a/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp b/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp index b2b37c1bf09226..94f489747e95bd 100644 --- a/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp +++ b/be/src/format_v2/table/iceberg_position_delete_sys_table_reader.cpp @@ -35,6 +35,7 @@ #include "core/types.h" #include "format/table/iceberg_delete_file_reader_helper.h" #include "format/table/parquet_utils.h" +#include "format_v2/table/iceberg_reader.h" #include "format_v2/table/iceberg_schema_utils.h" #include "runtime/descriptors.h" #include "runtime/runtime_state.h" @@ -147,6 +148,8 @@ class PositionDeleteFileTableReader final : public format::TableReader { void configure_mapper_options(format::TableColumnMapperOptions* options) const override { options->enable_row_lineage_virtual_columns = true; + // Position-delete row projection must reject a physically absent required field exactly like data scans. + options->reject_missing_required_field = supports_iceberg_scan_semantics_v2(_scan_params); // Parquet may preserve a selected complex wrapper without its own ID; position-delete row // projection must use the same descendant-ID fallback as ordinary Iceberg data scans. options->allow_idless_complex_wrapper_projection = @@ -591,6 +594,9 @@ Status IcebergPositionDeleteSysTableV2Reader::_build_delete_file_projected_colum columns->push_back(*it); columns->back().type = column.type; set_iceberg_delete_field_id(&columns->back()); + // The copied row tree bypasses IcebergTableReader::annotate_projected_column, so prepare its + // typed nested defaults before the generic inner reader builds the column mapper. + RETURN_IF_ERROR(prepare_iceberg_initial_default_exprs(&columns->back())); continue; } auto field = build_delete_file_column(column.name, column.type); diff --git a/be/src/format_v2/table/iceberg_reader.cpp b/be/src/format_v2/table/iceberg_reader.cpp index 1fc36a57c1ab69..7bc26348bde46b 100644 --- a/be/src/format_v2/table/iceberg_reader.cpp +++ b/be/src/format_v2/table/iceberg_reader.cpp @@ -47,6 +47,7 @@ #include "exprs/vliteral.h" #include "exprs/vslot_ref.h" #include "format/table/deletion_vector_reader.h" +#include "format/table/iceberg_default_value.h" #include "format_v2/expr/cast.h" #include "format_v2/expr/equality_delete_predicate.h" #include "format_v2/orc/orc_reader.h" @@ -445,6 +446,10 @@ static Status build_v2_json_scalar_default(const format::ColumnDefinition& field return Status::OK(); } normalize_iceberg_json_timestamp(primitive_type, &serialized_value); + if (doris::iceberg::detail::parse_non_finite_default(primitive_type, serialized_value, + result)) { + return Status::OK(); + } RETURN_IF_ERROR(value_type->get_serde()->from_fe_string(serialized_value, *result)); return Status::OK(); } @@ -520,6 +525,10 @@ static Status build_v2_initial_default_field(const format::ColumnDefinition& fie return Status::OK(); } + if (doris::iceberg::detail::parse_non_finite_default(primitive_type, + *field.initial_default_value, result)) { + return Status::OK(); + } RETURN_IF_ERROR(value_type->get_serde()->from_fe_string(*field.initial_default_value, *result)); return Status::OK(); } @@ -539,7 +548,7 @@ static Status build_initial_default_literal(const format::ColumnDefinition& tabl return Status::OK(); } -static Status build_initial_default_exprs(format::ColumnDefinition* column) { +Status prepare_iceberg_initial_default_exprs(format::ColumnDefinition* column) { DORIS_CHECK(column != nullptr); if (column->initial_default_value.has_value()) { VExprSPtr literal; @@ -547,7 +556,7 @@ static Status build_initial_default_exprs(format::ColumnDefinition* column) { column->default_expr = VExprContext::create_shared(std::move(literal)); } for (auto& child : column->children) { - RETURN_IF_ERROR(build_initial_default_exprs(&child)); + RETURN_IF_ERROR(prepare_iceberg_initial_default_exprs(&child)); } return Status::OK(); } @@ -859,7 +868,7 @@ Status IcebergTableReader::annotate_projected_column(const TFileScanSlotInfo& sl } auto& schema_column = *context->schema_column; - RETURN_IF_ERROR(build_initial_default_exprs(&schema_column)); + RETURN_IF_ERROR(prepare_iceberg_initial_default_exprs(&schema_column)); column->initial_default_value = schema_column.initial_default_value; column->initial_default_value_is_base64 = schema_column.initial_default_value_is_base64; column->is_optional = schema_column.is_optional; diff --git a/be/src/format_v2/table/iceberg_reader.h b/be/src/format_v2/table/iceberg_reader.h index 4118655dca46f0..c4069b5e840ac2 100644 --- a/be/src/format_v2/table/iceberg_reader.h +++ b/be/src/format_v2/table/iceberg_reader.h @@ -42,6 +42,8 @@ struct FileSystemProperties; namespace doris::format::iceberg { +Status prepare_iceberg_initial_default_exprs(format::ColumnDefinition* column); + // Iceberg table-level reader. // It reuses TableReader for split orchestration, dynamic partition pruning and table-block // finalization, while composing a FileReader for physical data-file reads instead of inheriting @@ -79,6 +81,7 @@ class IcebergTableReader : public format::TableReader { void configure_mapper_options(format::TableColumnMapperOptions* options) const override { options->enable_row_lineage_virtual_columns = true; + options->reject_missing_required_field = supports_iceberg_scan_semantics_v2(_scan_params); options->allow_idless_complex_wrapper_projection = supports_iceberg_scan_semantics_v1(_scan_params) && _format == FileFormat::PARQUET; } diff --git a/be/test/core/value/merge_partitioner_test.cpp b/be/test/core/value/merge_partitioner_test.cpp index f2682cf657aaf9..43c494832d0f93 100644 --- a/be/test/core/value/merge_partitioner_test.cpp +++ b/be/test/core/value/merge_partitioner_test.cpp @@ -83,6 +83,41 @@ class MergePartitionerTest : public ::testing::Test { return expr; } + TTypeDesc _nested_int_struct_type_desc() { + TTypeNode struct_node; + struct_node.__set_type(TTypeNodeType::STRUCT); + TStructField child; + child.__set_name("part"); + child.__set_contains_null(true); + struct_node.__set_struct_fields({child}); + + TTypeNode int_node; + int_node.__set_type(TTypeNodeType::SCALAR); + TScalarType scalar; + scalar.__set_type(TPrimitiveType::INT); + int_node.__set_scalar_type(scalar); + + TTypeDesc type_desc; + type_desc.__set_types({struct_node, int_node}); + type_desc.__set_is_nullable(true); + return type_desc; + } + + TExpr _make_nested_source_expr() { + TExprNode node; + node.__set_node_type(TExprNodeType::SLOT_REF); + node.__set_num_children(0); + TSlotRef slot_ref; + slot_ref.__set_slot_id(_nested_source_slot_id); + slot_ref.__set_tuple_id(_tuple_id); + node.__set_slot_ref(slot_ref); + node.__set_type(_nested_int_struct_type_desc()); + node.__set_is_nullable(true); + TExpr expr; + expr.nodes.emplace_back(std::move(node)); + return expr; + } + TMergePartitionInfo _make_base_merge_info(bool insert_random) { TMergePartitionInfo merge_info; merge_info.__set_operation_expr( @@ -180,6 +215,13 @@ class MergePartitionerTest : public ::testing::Test { .column_name("delete_key") .column_pos(4) .build()); + TTypeDesc nested_type = _nested_int_struct_type_desc(); + tuple_builder.add_slot(TSlotDescriptorBuilder() + .set_slotType(nested_type) + .nullable(true) + .column_name("nested_source") + .column_pos(5) + .build()); tuple_builder.build(&dtb); TDescriptorTable thrift_tbl = dtb.desc_tbl(); @@ -204,11 +246,13 @@ class MergePartitionerTest : public ::testing::Test { _row_id_slot_id = find_slot_id("row_id"); _insert_key_slot_id = find_slot_id("insert_key"); _delete_key_slot_id = find_slot_id("delete_key"); + _nested_source_slot_id = find_slot_id("nested_source"); ASSERT_GE(_operation_slot_id, 0); ASSERT_GE(_row_id_slot_id, 0); ASSERT_GE(_insert_key_slot_id, 0); ASSERT_GE(_delete_key_slot_id, 0); + ASSERT_GE(_nested_source_slot_id, 0); } ObjectPool _pool; @@ -219,6 +263,7 @@ class MergePartitionerTest : public ::testing::Test { TSlotId _row_id_slot_id = -1; TSlotId _insert_key_slot_id = -1; TSlotId _delete_key_slot_id = -1; + TSlotId _nested_source_slot_id = -1; }; TEST_F(MergePartitionerTest, TestInsertDeleteUpdatePartitioning) { @@ -317,6 +362,52 @@ TEST_F(MergePartitionerTest, TestInsertPartitionFieldsIdentity) { ASSERT_TRUE(partitioner.close(&_state).ok()); } +TEST_F(MergePartitionerTest, TestNestedInsertPartitionFieldPreservesParentNulls) { + ScopedConfigValue max_partition_guard( + config::table_sink_partition_write_max_partition_nums_per_writer, 0); + + TMergePartitionInfo merge_info = _make_base_merge_info(false); + TIcebergPartitionField field; + field.__set_transform("identity"); + field.__set_source_expr(_make_nested_source_expr()); + field.__set_name("payload_part"); + field.__set_source_id(3); + field.__set_source_field_path({0}); + merge_info.__set_insert_partition_fields({field}); + + MergePartitioner partitioner(8, merge_info, false); + ASSERT_TRUE(partitioner.init({}).ok()); + ASSERT_TRUE(partitioner.prepare(&_state, *_row_desc).ok()); + ASSERT_TRUE(partitioner.open(&_state).ok()); + + Block block = _build_block({1, 1, 1, 1}, {"p1", "p2", "p3", "p4"}, {1, 2, 3, 4}, + {10, 11, 12, 13}, {"d1", "d2", "d3", "d4"}); + auto values = ColumnInt32::create(); + values->insert_value(9); + values->insert_value(9); + values->insert_value(7); + values->insert_value(8); + auto child_nulls = ColumnUInt8::create(4, 0); + ColumnPtr child = ColumnNullable::create(std::move(values), std::move(child_nulls)); + auto struct_column = ColumnStruct::create(Columns {std::move(child)}); + auto parent_nulls = ColumnUInt8::create(); + parent_nulls->get_data().assign({0, 0, 1, 1}); + DataTypePtr child_type = make_nullable(std::make_shared()); + DataTypePtr struct_type = + std::make_shared(DataTypes {child_type}, Strings {"part"}); + block.insert(ColumnWithTypeAndName( + ColumnNullable::create(std::move(struct_column), std::move(parent_nulls)), + make_nullable(struct_type), "nested_source")); + + ASSERT_TRUE(partitioner.do_partitioning(&_state, &block).ok()); + const auto& channel_ids = partitioner.get_channel_ids(); + ASSERT_EQ(4, channel_ids.size()); + EXPECT_EQ(channel_ids[0], channel_ids[1]); + EXPECT_EQ(channel_ids[2], channel_ids[3]); + + ASSERT_TRUE(partitioner.close(&_state).ok()); +} + TEST_F(MergePartitionerTest, TestInvalidTransformFallbacksToRandom) { ScopedConfigValue threshold_guard( config::table_sink_non_partition_write_scaling_data_processed_threshold, 0); diff --git a/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp b/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp index 3800ef4f86e14f..06066de0fd7e66 100644 --- a/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp +++ b/be/test/exec/sink/writer/iceberg/viceberg_table_writer_test.cpp @@ -20,6 +20,15 @@ #include #include "common/exception.h" +#include "core/block/block.h" +#include "core/column/column_nullable.h" +#include "core/column/column_struct.h" +#include "core/column/column_vector.h" +#include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_struct.h" +#include "exprs/vexpr_context.h" +#include "exprs/vslot_ref.h" #include "format/table/iceberg/partition_spec_parser.h" #include "format/table/iceberg/schema.h" #include "format/table/iceberg/types.h" @@ -50,4 +59,58 @@ TEST(VIcebergTableWriterTest, RejectMissingPartitionSource) { } } +TEST(VIcebergTableWriterTest, ResolvesNestedPartitionSource) { + std::vector children; + children.emplace_back(true, 2, "part", std::make_unique(), std::nullopt); + std::vector columns; + columns.emplace_back(true, 1, "payload", + std::make_unique(std::move(children)), std::nullopt); + auto schema = std::make_shared(std::move(columns)); + const std::string spec_json = R"({"spec-id":1,"fields":[{"name":"part","transform":"identity",)" + R"("source-id":2,"field-id":1000}]})"; + auto child_type = make_nullable(std::make_shared()); + auto struct_type = make_nullable( + std::make_shared(DataTypes {child_type}, Strings {"part"})); + VExprContextSPtrs output_exprs { + VExprContext::create_shared(VSlotRef::create_shared(0, 0, -1, struct_type, "payload"))}; + + TIcebergTableSink iceberg_sink; + TDataSink data_sink; + data_sink.__set_iceberg_table_sink(iceberg_sink); + VIcebergTableWriter writer(data_sink, output_exprs, nullptr, nullptr); + writer._schema = schema; + writer._partition_spec = iceberg::PartitionSpecParser::from_json(schema, spec_json); + + auto partition_columns = writer._to_iceberg_partition_columns(); + ASSERT_EQ(partition_columns.size(), 1); + EXPECT_EQ(partition_columns[0].source_idx(), 0); + EXPECT_EQ(partition_columns[0].child_indices(), std::vector({0})); + EXPECT_EQ(partition_columns[0].source_type(), TYPE_INT); + + auto child_data = ColumnInt32::create(); + child_data->insert_value(7); + child_data->insert_value(8); + auto child_nulls = ColumnUInt8::create(2, 0); + auto child_column = ColumnNullable::create(std::move(child_data), std::move(child_nulls)); + Columns children_columns {std::move(child_column)}; + auto struct_column = ColumnStruct::create(std::move(children_columns)); + auto parent_nulls = ColumnUInt8::create(2, 0); + parent_nulls->get_data()[1] = 1; + auto parent_column = ColumnNullable::create(std::move(struct_column), std::move(parent_nulls)); + Block block; + block.insert({std::move(parent_column), struct_type, "payload"}); + + auto source = writer._nested_partition_source(block, partition_columns[0]); + const auto* nullable_source = check_and_get_column(source.column.get()); + ASSERT_NE(nullable_source, nullptr); + ASSERT_EQ(nullable_source->size(), 2); + EXPECT_EQ(nullable_source->get_null_map_data()[0], 0); + EXPECT_EQ(nullable_source->get_null_map_data()[1], 1); + const auto* source_data = + check_and_get_column(nullable_source->get_nested_column_ptr().get()); + ASSERT_NE(source_data, nullptr); + EXPECT_EQ(source_data->get_data()[0], 7); + EXPECT_EQ(source_data->get_data()[1], 8); +} + } // namespace doris diff --git a/be/test/format/table/iceberg/iceberg_reader_test.cpp b/be/test/format/table/iceberg/iceberg_reader_test.cpp index a212de6adfcf61..ef1c469ca23808 100644 --- a/be/test/format/table/iceberg/iceberg_reader_test.cpp +++ b/be/test/format/table/iceberg/iceberg_reader_test.cpp @@ -28,6 +28,7 @@ #include #include +#include #include #include #include @@ -1796,6 +1797,22 @@ TEST_F(IcebergReaderTest, initial_default_rejects_invalid_nullability) { EXPECT_TRUE(value.is_null()); } +TEST_F(IcebergReaderTest, v1_materializes_non_finite_initial_defaults) { + schema::external::TField field; + field.__set_name("value"); + field.__set_id(1); + field.__set_is_optional(false); + field.__set_initial_default_value("NaN"); + + ColumnPtr column; + ASSERT_TRUE(iceberg::create_initial_default_column(field, std::make_shared(), + &column) + .ok()); + Field value; + column->get(0, value); + EXPECT_TRUE(std::isnan(value.get())); +} + // GTest assertion macros inflate clang-tidy's cognitive-complexity score. // NOLINTNEXTLINE(readability-function-cognitive-complexity) TEST_F(IcebergReaderTest, v1_reuses_prepared_complex_initial_default_across_block_types) { diff --git a/be/test/format/table/iceberg/schema_test.cpp b/be/test/format/table/iceberg/schema_test.cpp index bccf0f1f3418b2..91c9ae778b308a 100644 --- a/be/test/format/table/iceberg/schema_test.cpp +++ b/be/test/format/table/iceberg/schema_test.cpp @@ -19,6 +19,10 @@ #include +#include + +#include "format/table/iceberg_default_value.h" + namespace doris { namespace iceberg { @@ -66,5 +70,32 @@ TEST(SchemaTest, test_find_field) { EXPECT_EQ(found_field2->field_id(), 2); } +TEST(SchemaTest, FindNestedFieldPath) { + std::vector children; + children.emplace_back(true, 2, "part", std::make_unique(), std::nullopt); + std::vector columns; + columns.emplace_back(true, 1, "payload", std::make_unique(std::move(children)), + std::nullopt); + Schema schema(1, std::move(columns)); + + const auto* path = schema.find_field_path(2); + ASSERT_NE(path, nullptr); + ASSERT_EQ(path->size(), 2); + EXPECT_EQ((*path)[0]->field_id(), 1); + EXPECT_EQ((*path)[1]->field_id(), 2); + EXPECT_EQ(schema.find_type(2)->type_id(), TypeID::INTEGER); +} + +TEST(SchemaTest, ParsesIcebergNonFiniteDefaults) { + Field value; + EXPECT_TRUE(detail::parse_non_finite_default(TYPE_FLOAT, "NaN", &value)); + EXPECT_TRUE(std::isnan(value.get())); + EXPECT_TRUE(detail::parse_non_finite_default(TYPE_DOUBLE, "Infinity", &value)); + EXPECT_TRUE(std::isinf(value.get())); + EXPECT_GT(value.get(), 0); + EXPECT_TRUE(detail::parse_non_finite_default(TYPE_DOUBLE, "-Infinity", &value)); + EXPECT_LT(value.get(), 0); +} + } // namespace iceberg } // namespace doris diff --git a/be/test/format_v2/table/iceberg_position_delete_sys_table_reader_test.cpp b/be/test/format_v2/table/iceberg_position_delete_sys_table_reader_test.cpp index a219ba37f3dd8d..6db534fa06d2a7 100644 --- a/be/test/format_v2/table/iceberg_position_delete_sys_table_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_position_delete_sys_table_reader_test.cpp @@ -19,6 +19,8 @@ #include +#include "core/data_type/data_type_number.h" +#include "core/data_type/data_type_struct.h" #include "runtime/runtime_profile.h" #include "runtime/runtime_state.h" @@ -59,5 +61,28 @@ TEST(IcebergPositionDeleteSysTableV2ProfileTest, UsesDistinctProfileForNestedPos reader._position_reader_profile); } +TEST(IcebergPositionDeleteSysTableV2ProfileTest, PreparesNestedRowInitialDefaults) { + IcebergPositionDeleteSysTableV2Reader reader; + const auto child_type = make_nullable(std::make_shared()); + const auto row_type = make_nullable( + std::make_shared(DataTypes {child_type}, Strings {"added"})); + ColumnDefinition row; + row.name = "row"; + row.type = row_type; + ColumnDefinition child; + child.name = "added"; + child.type = child_type; + child.initial_default_value = "7"; + row.children.push_back(std::move(child)); + reader._projected_columns = {row}; + reader._read_columns = {{"row", row_type}}; + + std::vector columns; + ASSERT_TRUE(reader._build_delete_file_projected_columns(&columns).ok()); + ASSERT_EQ(columns.size(), 1); + ASSERT_EQ(columns[0].children.size(), 1); + EXPECT_NE(columns[0].children[0].default_expr, nullptr); +} + } // namespace } // namespace doris::format::iceberg diff --git a/be/test/format_v2/table/iceberg_reader_test.cpp b/be/test/format_v2/table/iceberg_reader_test.cpp index b9fe813c9c68f0..980c81e17db3a4 100644 --- a/be/test/format_v2/table/iceberg_reader_test.cpp +++ b/be/test/format_v2/table/iceberg_reader_test.cpp @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -1476,6 +1477,38 @@ TEST(IcebergV2ReaderTest, AnnotateBuildsTypedNestedInitialDefault) { EXPECT_EQ(value.get(), 7); } +TEST(IcebergV2ReaderTest, PreparesIcebergNonFiniteInitialDefaults) { + struct Case { + DataTypePtr type; + std::string value; + bool nan; + bool negative; + }; + std::vector cases {{std::make_shared(), "NaN", true, false}, + {std::make_shared(), "Infinity", false, false}, + {std::make_shared(), "-Infinity", false, true}}; + for (const auto& test_case : cases) { + ColumnDefinition column; + column.name = "value"; + column.type = test_case.type; + column.initial_default_value = test_case.value; + ASSERT_TRUE(iceberg::prepare_iceberg_initial_default_exprs(&column).ok()); + ASSERT_NE(column.default_expr, nullptr); + const auto* literal = dynamic_cast(column.default_expr->root().get()); + ASSERT_NE(literal, nullptr); + Field value; + literal->get_column_ptr()->get(0, value); + const double number = test_case.type->get_primitive_type() == TYPE_FLOAT + ? value.get() + : value.get(); + EXPECT_EQ(std::isnan(number), test_case.nan); + if (!test_case.nan) { + EXPECT_TRUE(std::isinf(number)); + EXPECT_EQ(std::signbit(number), test_case.negative); + } + } +} + TEST(IcebergV2ReaderTest, AnnotateBuildsComplexInitialDefaults) { const auto required_int_type = std::make_shared(); const auto optional_string_type = make_nullable(std::make_shared()); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 9fb98d8f75411d..6e4d2c45eddc27 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -707,7 +707,31 @@ public Map getColumnHandles( // metadata-table columns (t$snapshots -> committed_at/...) so the generic scan node can look up // its pruned sys-table slots by name; a data handle resolves the base table's columns. Table table = iceHandle.isSystemTable() ? loadSysTable(session, iceHandle) : loadTable(session, iceHandle); - List fields = table.schema().columns(); + return buildColumnHandles(table.schema()); + } + + @Override + public Map getColumnHandles( + ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + if (iceHandle.isSystemTable() || snapshot == null || snapshot.getSchemaId() < 0) { + return getColumnHandles(session, handle); + } + Table table = loadTable(session, iceHandle); + Schema schema = table.currentSnapshot() == null + ? table.schema() : table.schemas().get((int) snapshot.getSchemaId()); + // Keep the handle-schema fallback identical to getTableSchema so slots and handles cannot diverge. + return buildColumnHandles(schema == null ? table.schema() : schema); + } + + @Override + public boolean supportsColumnHandleSnapshotPin(ConnectorSession session) { + return true; + } + + private static Map buildColumnHandles(Schema schema) { + List fields = schema.columns(); Map handles = new LinkedHashMap<>(fields.size()); for (Types.NestedField field : fields) { String name = field.name(); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java index f3c46732fb2fc1..7790d43062b5aa 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java @@ -30,12 +30,17 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Objects; +import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ForkJoinPool; import java.util.concurrent.TimeUnit; import java.util.function.LongSupplier; +import java.util.function.Supplier; /** * Per-catalog cache of an iceberg manifest's parsed files, keyed by {@link IcebergManifestEntryKey} @@ -46,8 +51,9 @@ *

Consumed by {@link IcebergScanPlanProvider}'s manifest-level planning path (gated by * {@code meta.cache.iceberg.manifest.enable}, default off — the default scan path is the iceberg SDK * {@code planFiles()}). The external enable-gate lives in the scan provider (which decides whether to take the - * manifest-planning path at all); this cache is unconditionally on when consulted. Within one catalog the same - * manifest file is parsed once and shared across queries (and across tables that reference it). + * full manifest-planning path); the compact equality-delete field-id projection is always reused per immutable + * snapshot. Within one catalog the same manifest file is parsed once and shared across queries (and across + * tables that reference it). * *

No TTL; capacity-bounded; cleared on REFRESH CATALOG. This mirrors the legacy entry's * {@code contextualOnly(CacheSpec.of(false, CACHE_NO_TTL, 100_000))} default spec: a manifest's content is @@ -68,6 +74,35 @@ final class IcebergManifestCache { private static final long DEFAULT_STATS_TTL_SECONDS = 300L; private final MetaCacheEntry entry; + private final MetaCacheEntry> equalityDeleteFieldIds; + + /** Immutable snapshot key for the compact equality-delete field-id projection. */ + private static final class SnapshotKey { + private final String tableLocation; + private final long snapshotId; + + private SnapshotKey(String tableLocation, long snapshotId) { + this.tableLocation = tableLocation; + this.snapshotId = snapshotId; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof SnapshotKey)) { + return false; + } + SnapshotKey that = (SnapshotKey) o; + return snapshotId == that.snapshotId && Objects.equals(tableLocation, that.tableLocation); + } + + @Override + public int hashCode() { + return Objects.hash(tableLocation, snapshotId); + } + } // Per-scan manifest-cache access tally, keyed by the statement's stable queryId // (ConnectorSession.getQueryId()), so VERBOSE EXPLAIN can report THIS scan's hits/misses/failures (the @@ -106,10 +141,24 @@ private static final class ScanStats { CacheSpec spec = CacheSpec.of(true, CacheSpec.CACHE_NO_TTL, Math.max(1, maxSize)); this.entry = new MetaCacheEntry<>("iceberg-manifest", null, spec, ForkJoinPool.commonPool(), false, true, 0L, true); + this.equalityDeleteFieldIds = new MetaCacheEntry<>("iceberg-equality-delete-field-ids", null, spec, + ForkJoinPool.commonPool(), false, true, 0L, true); this.statsTtlNanos = TimeUnit.SECONDS.toNanos(Math.max(1L, statsTtlSeconds)); this.nanoClock = nanoClock; } + /** + * Returns the equality-delete field ids for one immutable snapshot. Unlike the optional full manifest-file + * cache, this compact projection is always reused so scan properties do not re-read every delete manifest on + * every query. Loader failures are deliberately not cached, preserving retry after transient storage errors. + */ + Set getOrLoadEqualityDeleteFieldIds( + String tableLocation, long snapshotId, Supplier> loader) { + SnapshotKey key = new SnapshotKey(tableLocation, snapshotId); + return equalityDeleteFieldIds.get(key, + ignored -> Collections.unmodifiableSet(new HashSet<>(loader.get()))); + } + /** * Returns the parsed files for {@code manifest}, loading (and reading from storage) only on a miss. The * loader runs OUTSIDE Caffeine's compute lock (manual miss-load; single-flight per key), so a same-key @@ -223,6 +272,7 @@ private static List loadDeleteFiles(ManifestFile manifest, Table tab */ void invalidateAll() { entry.invalidateAll(); + equalityDeleteFieldIds.invalidateAll(); statsByQuery.clear(); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index f0718653d09f12..e1286b8ca54234 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -54,6 +54,8 @@ import org.apache.iceberg.HasTableOperations; import org.apache.iceberg.ManifestContent; import org.apache.iceberg.ManifestFile; +import org.apache.iceberg.ManifestFiles; +import org.apache.iceberg.ManifestReader; import org.apache.iceberg.MetadataColumns; import org.apache.iceberg.MetadataTableType; import org.apache.iceberg.MetadataTableUtils; @@ -65,7 +67,6 @@ import org.apache.iceberg.Schema; import org.apache.iceberg.SchemaParser; import org.apache.iceberg.Snapshot; -import org.apache.iceberg.SnapshotSummary; import org.apache.iceberg.SplittableScanTask; import org.apache.iceberg.Table; import org.apache.iceberg.TableOperations; @@ -99,10 +100,8 @@ import java.io.IOException; import java.nio.ByteBuffer; import java.time.ZoneId; -import java.util.ArrayDeque; import java.util.ArrayList; import java.util.Collections; -import java.util.Deque; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; @@ -231,8 +230,9 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // null in offline unit tests via the 2-arg ctor, in which case resolveTable resolves directly. private final ConnectorContext context; // T08: per-catalog manifest cache, owned by the long-lived IcebergConnector and injected via getScanPlanProvider. - // Nullable — null via the 2-/3-arg ctors (offline tests, default-disabled gate); when null the gate is - // forced off and planScan uses the SDK splitFiles path. + // Its compact equality-delete field-id projection is used regardless of the full-cache feature gate. Nullable + // via the 2-/3-arg ctors (offline tests); when null the projection is loaded directly, the full-cache gate is + // forced off, and planScan uses the SDK splitFiles path. private final IcebergManifestCache manifestCache; // PERF-01: cross-query RAW-table cache shared with the metadata layer, owned by the long-lived // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors and @@ -1612,11 +1612,13 @@ public Map getScanNodeProperties( boolean systemTable = iceHandle.isSystemTable(); Schema scanSchema = null; TableScan exactScan = null; + Set applicableEqualityDeleteFieldIds = Collections.emptySet(); boolean hasApplicableEqualityDeletes = false; if (!systemTable) { scanSchema = pinnedSchema(table, iceHandle); exactScan = buildScan(table, iceHandle, filter, session); - hasApplicableEqualityDeletes = hasApplicableEqualityDeletes(exactScan); + applicableEqualityDeleteFieldIds = cachedApplicableEqualityDeleteFieldIds(table, exactScan); + hasApplicableEqualityDeletes = !applicableEqualityDeleteFieldIds.isEmpty(); Optional>> nameMapping = IcebergSchemaUtils.extractNameMapping(table); if (requiresCurrentScanSemantics( table, exactScan, scanSchema, columns, hasApplicableEqualityDeletes, nameMapping)) { @@ -1714,7 +1716,7 @@ public Map getScanNodeProperties( // every branch so the default and current field type match BE's read. if (hasApplicableEqualityDeletes) { List equalityFields = schemaForPotentialEqualityDeletes( - table, exactScan, scanSchema); + table, scanSchema, applicableEqualityDeleteFieldIds); dict = IcebergSchemaUtils.encodeEqualitySchemaEvolutionProp( table, equalityFields, appendRowLineage, enableVarbinary, enableTimestampTz); @@ -1738,6 +1740,14 @@ public Map getScanNodeProperties( } props.put(SCHEMA_EVOLUTION_PROP, dict); } else if (isPositionDeletesSysTable(iceHandle)) { + // The native position-delete row reader depends on the current nested-default and requiredness + // semantics, so rolling upgrades must not route this system-table scan to an older backend. + // Metadata-only projections never materialize `row`, so fencing those scans needlessly reduces + // rolling-upgrade availability without preserving a reader invariant. + if (requestsColumn(columns, "row")) { + props.put(ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS, + "Current Iceberg position delete semantics"); + } // [D-065] narrowed: $position_deletes is the ONE system table BE reads with a NATIVE reader, so // the "schema rides inside the serialized FileScanTask" rationale above does not hold for it — no // FileScanTask is serialized on this path. Both native readers resolve the `row` column through @@ -1814,47 +1824,77 @@ private static List requestedLowerNames(List colu return names; } + private static boolean requestsColumn(List columns, String requestedName) { + if (columns == null || columns.isEmpty()) { + return true; + } + for (ConnectorColumnHandle column : columns) { + if (requestedName.equalsIgnoreCase(((IcebergColumnHandle) column).getName())) { + return true; + } + } + return false; + } + @VisibleForTesting - static boolean hasApplicableEqualityDeletes(TableScan scan) { + static Set applicableEqualityDeleteFieldIds(Table table, TableScan scan) { Snapshot snapshot = scan.snapshot(); - if (snapshot == null - || "0".equals(snapshot.summary().get(TOTAL_EQUALITY_DELETES))) { - return false; + if (snapshot == null || "0".equals(snapshot.summary().get(TOTAL_EQUALITY_DELETES))) { + return Collections.emptySet(); } - // planFiles binds delete files to the exact filtered data-file tasks after partition and sequence - // pruning. A snapshot summary of zero returns above without planning; a positive or missing summary - // needs this exact proof. Iterate whole-file tasks lazily and stop at the first equality delete: this - // keeps memory O(1), does not create or retain byte-split tasks, and avoids snapshot-wide delete - // counters forcing new-BE-only semantics when no dispatched task can consume an equality delete. - try (CloseableIterable tasks = scan.planFiles()) { - for (FileScanTask task : tasks) { - for (DeleteFile delete : task.deletes()) { - if (delete.content() == FileContent.EQUALITY_DELETES) { - return true; + Set fieldIds = new HashSet<>(); + for (ManifestFile manifest : snapshot.deleteManifests(table.io())) { + if (!manifest.hasAddedFiles() && !manifest.hasExistingFiles()) { + continue; + } + try (ManifestReader reader = ManifestFiles.readDeleteManifest( + manifest, table.io(), table.specs())) { + for (DeleteFile deleteFile : reader) { + if (deleteFile.content() == FileContent.EQUALITY_DELETES) { + fieldIds.addAll(deleteFile.equalityFieldIds()); } } + } catch (IOException e) { + throw new DorisConnectorException( + "Failed to read iceberg delete manifest " + manifest.path() + ": " + e.getMessage(), e); } - } catch (IOException e) { - throw new DorisConnectorException( - "Failed to inspect applicable Iceberg equality deletes: " + e.getMessage(), e); } - return false; + return fieldIds; + } + + private Set cachedApplicableEqualityDeleteFieldIds(Table table, TableScan scan) { + Snapshot snapshot = scan.snapshot(); + if (snapshot == null || "0".equals(snapshot.summary().get(TOTAL_EQUALITY_DELETES))) { + return Collections.emptySet(); + } + if (manifestCache == null) { + return applicableEqualityDeleteFieldIds(table, scan); + } + // Snapshot contents are immutable, so this compact projection can be shared even when the optional + // full manifest cache is disabled; a loader failure must still escape and remain retryable. + return manifestCache.getOrLoadEqualityDeleteFieldIds( + table.location(), snapshot.snapshotId(), () -> applicableEqualityDeleteFieldIds(table, scan)); } /** - * Build a schema carrier that can resolve any equality key reachable before the selected schema without - * enumerating data files, manifests, or byte-split tasks. Its retained state is bounded by table schema - * history rather than scan cardinality. At execution time BE looks fields up by the exact IDs on each - * {@link FileScanTask#deletes()}; unrelated carrier fields never participate in delete matching. + * Build a schema carrier that can resolve the field IDs referenced by live equality delete files. Reading + * delete manifests does not enumerate data files or byte-split tasks, and retained state is bounded by the + * number of equality keys rather than the table's entire schema history. * - *

The selected snapshot lineage wins when a field was renamed. The metadata schema list, in its actual - * chronology up to the selected schema (schema IDs are identifiers, not a sequence), fills schema-only - * changes and expired ancestors. Current fields remain first, so a dropped/re-added name still resolves the - * projected current field by name while a historical equality key resolves by its stable field ID.

+ *

The metadata schema list is searched in reverse chronology up to the selected schema (schema IDs are + * identifiers, not a sequence), so the latest definition of each stable field ID wins. Current fields remain + * first, allowing a dropped/re-added name to resolve the projected field by name while a live historical + * equality key resolves by its stable field ID.

*/ @VisibleForTesting static List schemaForPotentialEqualityDeletes( Table table, TableScan scan, Schema scanSchema) { + return schemaForPotentialEqualityDeletes( + table, scanSchema, applicableEqualityDeleteFieldIds(table, scan)); + } + + private static List schemaForPotentialEqualityDeletes( + Table table, Schema scanSchema, Set equalityFieldIds) { List metadataSchemas = metadataSchemaHistory(table); int selectedSchemaIndex = -1; for (int index = 0; index < metadataSchemas.size(); index++) { @@ -1864,45 +1904,19 @@ static List schemaForPotentialEqualityDeletes( } int lastRelevantIndex = selectedSchemaIndex >= 0 ? selectedSchemaIndex : metadataSchemas.size() - 1; - Set missing = new HashSet<>(); - for (int index = 0; index <= lastRelevantIndex; index++) { - Schema schema = metadataSchemas.get(index); - for (NestedField field : TypeUtil.indexById(schema.asStruct()).values()) { - if (field.type().isPrimitiveType()) { - missing.add(field.fieldId()); - } - } - } + Set missing = new HashSet<>(equalityFieldIds); missing.removeAll(TypeUtil.indexById(scanSchema.asStruct()).keySet()); if (missing.isEmpty()) { return scanSchema.columns(); } List fields = new ArrayList<>(scanSchema.columns()); - Map schemasById = table.schemas(); - Snapshot snapshot = scan.snapshot(); - while (snapshot != null && !missing.isEmpty()) { - Integer schemaId = snapshot.schemaId(); - if (schemaId != null) { - Schema historicalSchema = schemasById.get(schemaId); - if (historicalSchema == null) { - throw new IllegalStateException( - "Iceberg snapshot schema " + schemaId + " is absent from table metadata"); - } - addHistoricalEqualityFields(fields, missing, historicalSchema); - } - if (missing.isEmpty()) { - break; - } - Long parentId = snapshot.parentId(); - snapshot = parentId == null ? null : table.snapshot(parentId); - } for (int index = lastRelevantIndex; index >= 0 && !missing.isEmpty(); index--) { addHistoricalEqualityFields(fields, missing, metadataSchemas.get(index)); } if (!missing.isEmpty()) { throw new IllegalStateException( - "Iceberg historical primitive fields are absent from schema history: " + missing); + "Iceberg equality-delete fields are absent from schema history: " + missing); } return fields; } @@ -2094,62 +2108,22 @@ private static Set fieldAndDescendantIds(NestedField field) { static boolean selectedHistoryRequiresMissingRequiredFieldRejection( Table table, Schema scanSchema, Set projectedFieldIds, Snapshot selectedSnapshot) { - Map schemasById = table.schemas(); - Set relevantSchemaIds = schemaIdsRequiringMissingRequiredFieldRejection( - scanSchema, projectedFieldIds, schemasById.values()); - if (relevantSchemaIds.isEmpty()) { + if (!schemaHistoryRequiresMissingRequiredFieldRejection( + scanSchema, projectedFieldIds, table.schemas().values())) { return false; } - Deque snapshots = new ArrayDeque<>(); - if (selectedSnapshot != null) { - snapshots.add(selectedSnapshot); - } - Set visitedSnapshotIds = new HashSet<>(); - while (!snapshots.isEmpty()) { - Snapshot snapshot = snapshots.removeFirst(); - if (!visitedSnapshotIds.add(snapshot.snapshotId())) { - continue; - } - Integer schemaId = snapshot.schemaId(); - if (schemaId != null) { - Schema historical = schemasById.get(schemaId); - if (historical == null) { - throw new IllegalStateException( - "Iceberg snapshot schema " + schemaId + " is absent from table metadata"); - } - if (relevantSchemaIds.contains(schemaId)) { - return true; - } - } - Long parentId = snapshot.parentId(); - if (parentId != null) { - Snapshot parent = table.snapshot(parentId); - if (parent == null) { - return true; - } - snapshots.addLast(parent); - } - String sourceSnapshotId = - snapshot.summary().get(SnapshotSummary.SOURCE_SNAPSHOT_ID_PROP); - if (sourceSnapshotId != null) { - Snapshot source = table.snapshot(Long.parseLong(sourceSnapshotId)); - if (source == null) { - return true; - } - snapshots.addLast(source); - } - } - return false; + // Snapshot schema IDs are optional and proving ancestry is O(snapshot count). Once schema history + // exposes a requiredness hazard, conservatively fence every non-empty selected snapshot. + return selectedSnapshot != null; } - private static Set schemaIdsRequiringMissingRequiredFieldRejection( + private static boolean schemaHistoryRequiresMissingRequiredFieldRejection( Schema scanSchema, Set projectedFieldIds, Iterable historicalSchemas) { Map currentFields = TypeUtil.indexById(scanSchema.asStruct()); Map parentById = TypeUtil.indexParents(scanSchema.asStruct()); Set collectionWrapperIds = new HashSet<>(); collectCollectionWrapperFieldIds(scanSchema.asStruct(), collectionWrapperIds); - Set schemaIds = new HashSet<>(); for (Schema historicalSchema : historicalSchemas) { Map historicalFields = TypeUtil.indexById(historicalSchema.asStruct()); @@ -2162,8 +2136,7 @@ private static Set schemaIdsRequiringMissingRequiredFieldRejection( NestedField historicalField = historicalFields.get(fieldId); if (historicalField != null) { if (historicalField.isOptional()) { - schemaIds.add(historicalSchema.schemaId()); - break; + return true; } continue; } @@ -2177,12 +2150,11 @@ private static Set schemaIdsRequiringMissingRequiredFieldRejection( if (!collectionWrapperIds.contains(highestMissing.fieldId()) && highestMissing.isRequired() && highestMissing.initialDefault() == null) { - schemaIds.add(historicalSchema.schemaId()); - break; + return true; } } } - return schemaIds; + return false; } private static void collectCollectionWrapperFieldIds(Type type, Set result) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java index 38b57fa6e56dd9..733fc2b418e9a4 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java @@ -61,6 +61,7 @@ import org.apache.iceberg.SortOrder; import org.apache.iceberg.Table; import org.apache.iceberg.TableProperties; +import org.apache.iceberg.types.Type; import org.apache.iceberg.types.Types.NestedField; import org.apache.iceberg.util.LocationUtil; @@ -556,18 +557,39 @@ public ConnectorWritePartitionSpec getWritePartitioning(ConnectorSession session ? active.get().getSchema() : table.schema(); List fields = new ArrayList<>(); for (PartitionField field : spec.fields()) { - // sourceColumnName mirrors the legacy schema.findField(field.sourceId()).name() lookup the engine - // used to map a partition field back to a bound output expr id. transform/param mirror - // field.transform().toString() + parseTransformParam (kept connector-side so fe-core never parses). - NestedField sourceField = schema.findField(field.sourceId()); - String sourceColumnName = sourceField == null ? null : sourceField.name(); + // Bind a nested source to its top-level slot. fe-core recovers child indexes from the stable Iceberg + // field ids already carried by the Doris Column tree, so the public connector SPI stays unchanged. + String sourceColumnName = findPartitionSourceColumnName(schema, field.sourceId()); String transform = field.transform().toString(); fields.add(new ConnectorWritePartitionField( - transform, parseTransformParam(transform), sourceColumnName, field.name(), field.sourceId())); + transform, parseTransformParam(transform), sourceColumnName, + field.name(), field.sourceId())); } return new ConnectorWritePartitionSpec(spec.specId(), fields); } + private static String findPartitionSourceColumnName(Schema schema, int sourceId) { + List columns = schema.columns(); + for (NestedField column : columns) { + if (column.fieldId() == sourceId || containsStructField(column.type(), sourceId)) { + return column.name(); + } + } + return null; + } + + private static boolean containsStructField(Type type, int sourceId) { + if (!type.isStructType()) { + return false; + } + for (NestedField field : type.asStructType().fields()) { + if (field.fieldId() == sourceId || containsStructField(field.type(), sourceId)) { + return true; + } + } + return false; + } + @Override public List getSyntheticWriteColumns(ConnectorSession session, ConnectorTableHandle tableHandle) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java index ae16a1ee4fb0c2..b04c2fa42168ef 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriteSchemaContext.java @@ -202,13 +202,12 @@ private static SortOrder bindSortOrder(SortOrder sortOrder, Schema schema, Strin private static void validateWriterMetadataSources( Schema schema, PartitionSpec partitionSpec, SortOrder sortOrder, String tableName) { - Map topLevelFields = schema.columns().stream() - .collect(ImmutableMap.toImmutableMap(Types.NestedField::fieldId, field -> field)); for (PartitionField field : partitionSpec.fields()) { - if (!topLevelFields.containsKey(field.sourceId())) { + // Iceberg permits a nested primitive field as a partition source; field IDs are schema-wide. + if (schema.findField(field.sourceId()) == null) { throw new DorisConnectorException("Iceberg partition field " + field.fieldId() + " references source field " + field.sourceId() - + " outside pinned top-level schema " + schema.schemaId() + + " outside pinned schema " + schema.schemaId() + " for table " + tableName); } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java index 83a2303d95ac06..a6c2a9d5750403 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java @@ -25,13 +25,17 @@ import org.apache.doris.connector.spi.handle.ConnectorColumnHandle; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; import org.apache.doris.connector.spi.handle.WriteOperation; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.iceberg.DataFiles; import org.apache.iceberg.PartitionSpec; import org.apache.iceberg.RowLevelOperationMode; import org.apache.iceberg.Schema; import org.apache.iceberg.SortOrder; import org.apache.iceberg.TableProperties; import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.inmemory.InMemoryCatalog; import org.apache.iceberg.types.Types; import org.apache.iceberg.view.ImmutableSQLViewRepresentation; import org.apache.iceberg.view.ImmutableViewVersion; @@ -1313,6 +1317,38 @@ public void getColumnHandlesKeysByCasePreservedNameAndCarriesIcebergFieldId() { "getColumnHandles must load the table via the seam using the handle coordinates"); } + @Test + public void getColumnHandlesUsesPinnedHistoricalSchema() { + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + Schema oldSchema = new Schema( + Types.NestedField.required(7, "old_name", Types.IntegerType.get()), + Types.NestedField.optional(9, "survivor", Types.StringType.get())); + InMemoryCatalog catalog = new InMemoryCatalog(); + catalog.initialize("test", Collections.emptyMap()); + catalog.createNamespace(Namespace.of("db1")); + org.apache.iceberg.Table table = catalog.createTable( + TableIdentifier.of("db1", "t1"), oldSchema, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(DataFiles.builder(PartitionSpec.unpartitioned()) + .withPath("s3://bucket/db1/t1/old.parquet") + .withFileSizeInBytes(1).withRecordCount(1).build()).commit(); + Schema historicalSchema = table.schema(); + table.updateSchema().renameColumn("old_name", "new_name").commit(); + ops.table = table; + + ConnectorMvccSnapshot pin = ConnectorMvccSnapshot.builder() + .snapshotId(11L).schemaId(historicalSchema.schemaId()).build(); + IcebergConnectorMetadata metadata = metadataWith(ops); + Map handles = metadata.getColumnHandles( + null, new IcebergTableHandle("db1", "t1"), pin); + + Assertions.assertTrue(metadata.supportsColumnHandleSnapshotPin(null)); + Assertions.assertTrue(handles.containsKey("old_name")); + Assertions.assertTrue(handles.containsKey("survivor")); + Assertions.assertFalse(handles.containsKey("new_name")); + Assertions.assertEquals(historicalSchema.findField("old_name").fieldId(), + ((IcebergColumnHandle) handles.get("old_name")).getFieldId()); + } + // --------------------------------------------------------------------- // P6.3-T03: write transaction wiring (gate-closed / dormant) // --------------------------------------------------------------------- diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestCacheTest.java index 28fb5019decebb..920e5ae0c6d646 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergManifestCacheTest.java @@ -32,6 +32,8 @@ import java.util.Collections; import java.util.List; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; /** * Unit tests for {@link IcebergManifestCache} (T08). Uses a real {@link InMemoryCatalog} table so the cache is @@ -78,6 +80,49 @@ public void loadsDataFilesAndCachesByManifestPath() { Assertions.assertEquals(1, cache.size()); } + @Test + public void equalityDeleteFieldIdsLoadOncePerSnapshot() { + IcebergManifestCache cache = new IcebergManifestCache(); + AtomicInteger loads = new AtomicInteger(); + + Set first = cache.getOrLoadEqualityDeleteFieldIds("/warehouse/db/t", 10L, () -> { + loads.incrementAndGet(); + return Collections.singleton(7); + }); + Set second = cache.getOrLoadEqualityDeleteFieldIds("/warehouse/db/t", 10L, () -> { + loads.incrementAndGet(); + return Collections.singleton(8); + }); + Set nextSnapshot = cache.getOrLoadEqualityDeleteFieldIds("/warehouse/db/t", 11L, () -> { + loads.incrementAndGet(); + return Collections.singleton(9); + }); + + Assertions.assertEquals(Collections.singleton(7), first); + Assertions.assertEquals(first, second, "the same immutable snapshot must reuse its field-id set"); + Assertions.assertEquals(Collections.singleton(9), nextSnapshot); + Assertions.assertEquals(2, loads.get(), "only one manifest walk is allowed per snapshot"); + } + + @Test + public void equalityDeleteFieldIdFailureIsNotCached() { + IcebergManifestCache cache = new IcebergManifestCache(); + AtomicInteger loads = new AtomicInteger(); + + Assertions.assertThrows(IllegalStateException.class, + () -> cache.getOrLoadEqualityDeleteFieldIds("/warehouse/db/t", 10L, () -> { + loads.incrementAndGet(); + throw new IllegalStateException("transient manifest failure"); + })); + Set retry = cache.getOrLoadEqualityDeleteFieldIds("/warehouse/db/t", 10L, () -> { + loads.incrementAndGet(); + return Collections.singleton(7); + }); + + Assertions.assertEquals(Collections.singleton(7), retry); + Assertions.assertEquals(2, loads.get(), "a failed manifest walk must be retried, not memoized"); + } + @Test public void capacityOverflowFlushesWholesale() { Table table = tableWithTwoDataFiles(); @@ -97,11 +142,21 @@ public void invalidateAllClearsEveryEntry() { ManifestFile manifest = table.currentSnapshot().dataManifests(table.io()).get(0); IcebergManifestCache cache = new IcebergManifestCache(); cache.getManifestCacheValue(manifest, table); + AtomicInteger equalityLoads = new AtomicInteger(); + cache.getOrLoadEqualityDeleteFieldIds(table.location(), table.currentSnapshot().snapshotId(), () -> { + equalityLoads.incrementAndGet(); + return Collections.singleton(1); + }); Assertions.assertEquals(1, cache.size()); // REFRESH CATALOG hook (H-5): invalidateAll drops every cached manifest (legacy catalog-wide // group.invalidateAll parity). MUTATION: a no-op invalidateAll -> size stays 1 -> red. cache.invalidateAll(); Assertions.assertEquals(0, cache.size()); + cache.getOrLoadEqualityDeleteFieldIds(table.location(), table.currentSnapshot().snapshotId(), () -> { + equalityLoads.incrementAndGet(); + return Collections.singleton(1); + }); + Assertions.assertEquals(2, equalityLoads.get(), "catalog refresh must also clear the snapshot projection"); } @Test diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 0e5e2b9ed51e74..e7addabe0f6cc4 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -67,6 +67,7 @@ import org.apache.iceberg.io.SupportsStorageCredentials; import org.apache.iceberg.types.Conversions; import org.apache.iceberg.types.Types; +import org.apache.iceberg.types.Types.NestedField; import org.apache.iceberg.util.SerializationUtil; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -814,6 +815,31 @@ public void getScanNodePropertiesForcesEqualityDeleteKeyColumnIntoDict() throws "the unprojected equality-delete key column must be force-included (#65502), got " + top); } + @Test + public void getScanNodePropertiesCachesEqualityDeleteFieldIdsWithManifestCacheDisabled() { + Schema schema = new Schema( + Types.NestedField.required(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "value", Types.StringType.get())); + Table table = createTable("cached_eq_ids", schema, PartitionSpec.unpartitioned(), + Collections.singletonMap(TableProperties.FORMAT_VERSION, "2")); + table.newAppend().appendFile(dataFile(table.spec(), + "s3://b/db/cached_eq_ids/f1.parquet", 1024, null, null)).commit(); + table.newRowDelta().addDeletes(equalityDeleteFile( + "s3://b/db/cached_eq_ids/eq.parquet", FileFormat.PARQUET, 1)).commit(); + IcebergManifestCache cache = new IcebergManifestCache(); + // Keep construction behind the shared helper so catalog-property API migrations do not break this test. + IcebergScanPlanProvider provider = manifestProvider(Collections.emptyMap(), table, cache); + + provider.getScanNodeProperties(null, new IcebergTableHandle("db1", "cached_eq_ids"), + Collections.singletonList(new IcebergColumnHandle("value", 2)), Optional.empty()); + + Set cached = cache.getOrLoadEqualityDeleteFieldIds( + table.location(), table.currentSnapshot().snapshotId(), () -> { + throw new AssertionError("the provider must populate the snapshot-scoped projection cache"); + }); + Assertions.assertEquals(Collections.singleton(1), cached); + } + @Test public void equalityCarrierAllowsUnrelatedDropAndReaddNames() throws Exception { Schema schema = new Schema( @@ -833,8 +859,6 @@ public void equalityCarrierAllowsUnrelatedDropAndReaddNames() throws Exception { "s3://b/db/drop_readd/eq.parquet", FileFormat.PARQUET, 1)) .commit(); - int oldTopLevelId = table.schema().findField("same_name").fieldId(); - int oldNestedId = table.schema().findField("payload.same_name").fieldId(); table.updateSchema().deleteColumn("same_name").deleteColumn("payload.same_name").commit(); table.updateSchema() .addColumn("same_name", Types.IntegerType.get()) @@ -863,7 +887,8 @@ null, new IcebergTableHandle("db1", "drop_readd"), payload = field; } } - Assertions.assertEquals(Arrays.asList(currentTopLevelId, oldTopLevelId), sameNameIds); + Assertions.assertEquals(Collections.singletonList(currentTopLevelId), sameNameIds, + "unrelated historical fields must not inflate the equality-delete schema carrier"); Assertions.assertNotNull(payload); List nestedSameNameIds = new ArrayList<>(); for (TFieldPtr field : payload.getFieldPtr().getNestedField().getStructField().getFields()) { @@ -871,11 +896,60 @@ null, new IcebergTableHandle("db1", "drop_readd"), nestedSameNameIds.add(field.getFieldPtr().getId()); } } - Assertions.assertEquals(Arrays.asList(currentNestedId, oldNestedId), nestedSameNameIds); + Assertions.assertEquals(Collections.singletonList(currentNestedId), nestedSameNameIds, + "only field IDs referenced by live equality deletes may be retained"); + } + + @Test + public void equalityCarrierResolvesDroppedLiveEqualityKeyFromSchemaHistory() throws Exception { + Schema schema = new Schema( + Types.NestedField.required(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "old_key", Types.StringType.get())); + Table table = createTable("dropped_eq_key", schema, PartitionSpec.unpartitioned(), + Collections.singletonMap(TableProperties.FORMAT_VERSION, "2")); + table.newAppend().appendFile(dataFile(table.spec(), + "s3://b/db/dropped_eq_key/f1.parquet", 1024, null, null)).commit(); + table.newRowDelta().addDeletes(equalityDeleteFile( + "s3://b/db/dropped_eq_key/eq.parquet", FileFormat.PARQUET, 2)).commit(); + table.updateSchema().deleteColumn("old_key").commit(); + + IcebergScanPlanProvider provider = providerOver(table); + Map props = provider.getScanNodeProperties( + null, new IcebergTableHandle("db1", "dropped_eq_key"), + Collections.singletonList(new IcebergColumnHandle("id", 1)), Optional.empty()); + TFileScanRangeParams params = new TFileScanRangeParams(); + provider.populateScanLevelParams(params, props); + + List fields = params.getHistorySchemaInfo().get(0).getRootField().getFields(); + Assertions.assertTrue(fields.stream().anyMatch(field -> field.getFieldPtr().getId() == 2), + "a dropped field still referenced by a live equality delete must remain resolvable by ID"); } @Test - public void partitionPrunedEqualityDeleteDoesNotRequireCurrentBackendSemantics() { + public void equalityCarrierSizeDoesNotGrowWithUnrelatedSchemaHistory() throws Exception { + Schema schema = new Schema( + Types.NestedField.required(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "churn", Types.IntegerType.get())); + Table table = createTable("bounded_eq_carrier", schema, PartitionSpec.unpartitioned(), + Collections.singletonMap(TableProperties.FORMAT_VERSION, "2")); + table.newAppend().appendFile(dataFile(table.spec(), + "s3://b/db/bounded_eq_carrier/f1.parquet", 1024, null, null)).commit(); + table.newRowDelta().addDeletes(equalityDeleteFile( + "s3://b/db/bounded_eq_carrier/eq.parquet", FileFormat.PARQUET, 1)).commit(); + for (int index = 0; index < 64; index++) { + table.updateSchema().deleteColumn("churn").commit(); + table.updateSchema().addColumn("churn", Types.IntegerType.get()).commit(); + } + + List carrier = IcebergScanPlanProvider.schemaForPotentialEqualityDeletes( + table, table.newScan(), table.schema()); + Assertions.assertEquals(2, carrier.size(), + "unrelated schema churn must not increase FE heap or schema RPC size"); + Assertions.assertEquals(table.schema().columns(), carrier); + } + + @Test + public void partitionPrunedEqualityDeleteConservativelyRequiresCurrentBackendSemantics() { PartitionSpec spec = PartitionSpec.builderFor(PART_SCHEMA).identity("p").build(); Table table = createTable("partition_pruned_eqdel", PART_SCHEMA, spec, Collections.singletonMap(TableProperties.FORMAT_VERSION, "2")); @@ -901,9 +975,9 @@ public void partitionPrunedEqualityDeleteDoesNotRequireCurrentBackendSemantics() Map prunedProps = provider.getScanNodeProperties( null, new IcebergTableHandle("db1", "partition_pruned_eqdel"), columns, Optional.of(eqInt("p", 1))); - Assertions.assertFalse(prunedProps.containsKey( + Assertions.assertTrue(prunedProps.containsKey( ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS), - "an equality delete in a pruned partition must not gate the selected tasks"); + "delete-manifest inspection must stay conservative without enumerating data tasks"); Map applicableProps = provider.getScanNodeProperties( null, new IcebergTableHandle("db1", "partition_pruned_eqdel"), @@ -914,7 +988,7 @@ null, new IcebergTableHandle("db1", "partition_pruned_eqdel"), } @Test - public void sequencePrunedEqualityDeleteDoesNotRequireCurrentBackendSemantics() { + public void sequencePrunedEqualityDeleteConservativelyRequiresCurrentBackendSemantics() { Table table = createTable("sequence_pruned_eqdel", SCHEMA, PartitionSpec.unpartitioned(), Collections.singletonMap(TableProperties.FORMAT_VERSION, "2")); DataFile oldFile = dataFile(table.spec(), @@ -935,9 +1009,9 @@ public void sequencePrunedEqualityDeleteDoesNotRequireCurrentBackendSemantics() Map props = provider.getScanNodeProperties( null, new IcebergTableHandle("db1", "sequence_pruned_eqdel"), Collections.singletonList(new IcebergColumnHandle("id", 1)), Optional.empty()); - Assertions.assertFalse(props.containsKey( + Assertions.assertTrue(props.containsKey( ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS), - "an older equality delete must not gate a later-sequence replacement data file"); + "snapshot metadata must gate conservatively without synchronously replanning all files"); } @Test @@ -1032,6 +1106,28 @@ public void schemaHistoryPlanningSkipsSnapshotChainWhenOneSchemaResolvesEverythi "the equality carrier must not walk snapshots when no historical field is missing"); } + @Test + public void schemaHistoryPlanningStaysBoundedWhenHistoricalRequirednessNeedsTheFence() { + Schema oldSchema = new Schema( + Types.NestedField.optional(1, "id", Types.IntegerType.get())); + Table table = createTable("requiredness_history", oldSchema, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile(table.spec(), + "s3://b/db/requiredness_history/old.parquet", 128, null, null)).commit(); + table.updateSchema().allowIncompatibleChanges().requireColumn("id").commit(); + table.newAppend().appendFile(dataFile(table.spec(), + "s3://b/db/requiredness_history/new.parquet", 128, null, null)).commit(); + + FakeIcebergTable countingTable = new FakeIcebergTable( + table.name(), table.schema(), table.spec(), table.location(), table.properties()); + countingTable.setScanTable(table); + Assertions.assertTrue( + IcebergScanPlanProvider.selectedHistoryRequiresMissingRequiredFieldRejection( + countingTable, table.schema(), Collections.singleton(1), + table.currentSnapshot())); + Assertions.assertEquals(0, countingTable.getSnapshotLookupCount(), + "the upgrade fence must be conservative and bounded by schema history, not snapshot count"); + } + @Test public void getScanNodePropertiesEmitsSchemaEvolutionDictForPartitionedTableToo() { // The dict is emitted alongside path_partition_keys (it is unconditional, like legacy @@ -1361,10 +1457,29 @@ public void getScanNodePropertiesForPositionDeletesSysHandleEmitsDict() { Assertions.assertTrue(props.containsKey("iceberg.schema_evolution"), "position_deletes reads natively and needs the field-id dict to resolve `row`"); + Assertions.assertFalse(props.containsKey( + ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS), + "metadata-only projections do not depend on position_deletes.row semantics"); Assertions.assertFalse(props.containsKey("path_partition_keys"), "a metadata table is still not base-spec partitioned -> no path_partition_keys"); } + @Test + public void getScanNodePropertiesForPositionDeletesRowRequiresCurrentBackendSemantics() { + Table table = tableWithPositionDelete( + positionDeleteFile("s3://b/db/t1/pos.parquet", FileFormat.PARQUET, null, null)); + // The helper follows the active catalog-property wrapper while this test stays focused on scan semantics. + IcebergScanPlanProvider provider = providerOver(table); + + Map props = provider.getScanNodeProperties( + null, IcebergTableHandle.forSystemTable("db1", "t1", "position_deletes", -1L, null, -1L), + Collections.singletonList(new IcebergColumnHandle("row", 3)), Optional.empty()); + + Assertions.assertTrue(props.containsKey( + ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS), + "projecting position_deletes.row depends on current nested-default semantics"); + } + @Test public void getScanNodePropertiesForPositionDeletesLoadsTheBaseTableOnlyOnce() { // WHY: the dict branch needs the METADATA table, and the obvious way to get one is resolveSysTable(). diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java index 25b8e52e22591f..8cd0158e925e62 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java @@ -364,6 +364,20 @@ public void planWriteValidatesBoundColumnsAgainstPinnedBranchSchema() { Assertions.assertFalse(plan.getDataSink().getIcebergTableSink().getSchemaJson().contains("renamed_name")); } + @Test + public void writeSchemaAllowsNestedPrimitivePartitionSource() { + InMemoryCatalog catalog = freshCatalog(); + Schema schema = new Schema( + Types.NestedField.required(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "payload", Types.StructType.of( + Types.NestedField.optional(3, "part", Types.IntegerType.get())))); + Table table = catalog.createTable(TableIdentifier.of("db1", "nested_partition"), schema, + PartitionSpec.builderFor(schema).identity("payload.part").build()); + + Assertions.assertDoesNotThrow(() -> IcebergWriteSchemaContext.create( + table, "db1.nested_partition", Optional.empty(), false, false)); + } + @Test public void branchWriteRejectsCurrentRequiredFieldAbsentWithoutDefault() { InMemoryCatalog catalog = freshCatalog(); @@ -1305,6 +1319,26 @@ public void getWritePartitioningBucketTransformCarriesParamAndDistinctNames() { Assertions.assertEquals(table.schema().findField("id").fieldId(), f.getSourceId()); } + @Test + public void getWritePartitioningRoutesNestedSourceByItsTopLevelColumn() { + InMemoryCatalog catalog = freshCatalog(); + Schema schema = new Schema( + Types.NestedField.required(1, "id", Types.IntegerType.get()), + Types.NestedField.optional(2, "payload", Types.StructType.of( + Types.NestedField.optional(3, "part", Types.IntegerType.get())))); + Table table = catalog.createTable(TableIdentifier.of("db1", "nested_partition"), schema, + PartitionSpec.builderFor(schema).bucket("payload.part", 8).build()); + + ConnectorWritePartitionField field = providerFor(table, contextWithStorage()) + .getWritePartitioning(sessionFor(table, contextWithStorage()), + new IcebergTableHandle("db1", "nested_partition")) + .getFields().get(0); + + Assertions.assertEquals("payload", field.getSourceColumnName(), + "merge exchange must route a nested source through its bound top-level struct slot"); + Assertions.assertEquals(Integer.valueOf(3), field.getSourceId()); + } + // ───────────────────── getSyntheticWriteColumns (connector declares the row-id STRUCT, ③ C3b-core) ───────────────────── // // WHY: post-flip the iceberg DML hidden column __DORIS_ICEBERG_ROWID_COL__ that legacy diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 979826563dc2d4..06dfc29622c1fa 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -2278,7 +2278,7 @@ private List buildColumnHandles() throws UserException { ConnectorColumnHandle ch = allHandles.get(name); if (ch != null) { selected.add(withProjectedFieldIds(ch, slot)); - } else if (pinnedNames.contains(name)) { + } else if (requiresPinnedColumnHandle(slot.getColumn(), pinnedNames)) { throw new UserException("Column '" + name + "' of table " + getTargetTable().getName() + " resolves in the pinned time-travel schema" + " but has no connector column handle; refusing to silently drop it" @@ -2289,6 +2289,12 @@ private List buildColumnHandles() throws UserException { return selected; } + static boolean requiresPinnedColumnHandle(Column column, Set pinnedNames) { + // A connector-reserved passthrough column is generated by the scan provider rather than resolved + // from the physical table schema, so it may legitimately be absent from the column-handle map. + return pinnedNames.contains(column.getName()) && !column.isReservedPassthrough(); + } + static ConnectorColumnHandle withProjectedFieldIds( ConnectorColumnHandle handle, SlotDescriptor slot) { Set projectedFieldIds = new HashSet<>(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 5dc082b0635618..9760d615e8fb22 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -3390,7 +3390,8 @@ private DataPartition toDataPartition(DistributionSpec distributionSpec/* target field.getTransform(), field.getParam(), field.getName(), - field.getSourceId())); + field.getSourceId(), + field.getSourceFieldPath())); } return new DataPartition(TPartitionType.MERGE_PARTITIONED, operationExpr, insertPartitionExprs, deletePartitionExprs, mergeSpec.isInsertRandom(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecMerge.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecMerge.java index f1c567b251572f..6e722aba19d7a7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecMerge.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecMerge.java @@ -38,17 +38,25 @@ public static class MergePartitionField { private final Integer param; private final String name; private final Integer sourceId; + private final ImmutableList sourceFieldPath; /** * Create a partition field mapping for merge insert routing. */ public MergePartitionField(String transform, ExprId sourceExprId, Integer param, String name, Integer sourceId) { + this(transform, sourceExprId, param, name, sourceId, ImmutableList.of()); + } + + /** Create a partition field mapping whose source is nested below a top-level slot. */ + public MergePartitionField(String transform, ExprId sourceExprId, Integer param, + String name, Integer sourceId, List sourceFieldPath) { this.transform = Objects.requireNonNull(transform, "transform should not be null"); this.sourceExprId = Objects.requireNonNull(sourceExprId, "sourceExprId should not be null"); this.param = param; this.name = name; this.sourceId = sourceId; + this.sourceFieldPath = ImmutableList.copyOf(sourceFieldPath); } public String getTransform() { @@ -71,6 +79,10 @@ public Integer getSourceId() { return sourceId; } + public List getSourceFieldPath() { + return sourceFieldPath; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -84,12 +96,13 @@ public boolean equals(Object o) { && sourceExprId.equals(that.sourceExprId) && Objects.equals(param, that.param) && Objects.equals(name, that.name) - && Objects.equals(sourceId, that.sourceId); + && Objects.equals(sourceId, that.sourceId) + && sourceFieldPath.equals(that.sourceFieldPath); } @Override public int hashCode() { - return Objects.hash(transform, sourceExprId, param, name, sourceId); + return Objects.hash(transform, sourceExprId, param, name, sourceId, sourceFieldPath); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java index b40366aa3b9a0c..7c7a2c5003aa81 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java @@ -259,6 +259,9 @@ public AbstractInsertExecutor initPlan(ConnectContext ctx, StmtExecutor stmtExec int retryTimes = 0; ctx.getStatementContext().setIsInsert(true); while (++retryTimes < Math.max(ctx.getSessionVariable().dmlPlanRetryTimes, 3)) { + // Each internal attempt must repin connector metadata; retaining the previous writer schema can + // plan defaults and partition fields against the table version that triggered the retry. + ctx.getStatementContext().resetConnectorStatementScope(); TableIf targetTableIf = getTargetTableIf(ctx, qualifiedTargetTableName); DatabaseIf targetDatabase = getTargetDatabase(targetTableIf); // check auth diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java index bc528540523161..f58440d9279cdc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSink.java @@ -415,7 +415,8 @@ private InsertPartitionFieldResult buildInsertPartitionFieldsFromConnector( return new InsertPartitionFieldResult(false, false, null); } ConnectorWritePartitionSpec spec = writePlanProvider.getWritePartitioning(session, handle); - return reconstructPartitionFields(insertPartitionFields, spec, columnExprIdMap, columnIdToExprId); + return reconstructPartitionFields( + insertPartitionFields, spec, columnExprIdMap, columnIdToExprId, cols); } /** @@ -443,7 +444,7 @@ static InsertPartitionFieldResult reconstructPartitionFields( ConnectorWritePartitionSpec spec, Map columnExprIdMap) { return reconstructPartitionFields(insertPartitionFields, spec, columnExprIdMap, - java.util.Collections.emptyMap()); + java.util.Collections.emptyMap(), null); } static InsertPartitionFieldResult reconstructPartitionFields( @@ -451,6 +452,25 @@ static InsertPartitionFieldResult reconstructPartitionFields( ConnectorWritePartitionSpec spec, Map columnExprIdMap, Map columnIdToExprId) { + return reconstructPartitionFields( + insertPartitionFields, spec, columnExprIdMap, columnIdToExprId, null); + } + + static InsertPartitionFieldResult reconstructPartitionFields( + List insertPartitionFields, + ConnectorWritePartitionSpec spec, + Map columnExprIdMap, + List tableColumns) { + return reconstructPartitionFields(insertPartitionFields, spec, columnExprIdMap, + java.util.Collections.emptyMap(), tableColumns); + } + + static InsertPartitionFieldResult reconstructPartitionFields( + List insertPartitionFields, + ConnectorWritePartitionSpec spec, + Map columnExprIdMap, + Map columnIdToExprId, + List tableColumns) { if (spec == null) { return new InsertPartitionFieldResult(false, false, null); } @@ -470,16 +490,32 @@ static InsertPartitionFieldResult reconstructPartitionFields( } // Prefer the stable source field id carried by the bind-time schema. A same-name replacement // must not inherit the old output expression after concurrent Iceberg schema evolution. - ExprId exprId = columnIdToExprId.isEmpty() - ? columnExprIdMap.get(sourceColumnName) - : columnIdToExprId.get(field.getSourceId()); + Column sourceColumn = findSourceColumn(tableColumns, sourceColumnName); + ExprId exprId; + if (columnIdToExprId.isEmpty()) { + exprId = columnExprIdMap.get(sourceColumnName); + } else if (sourceColumn == null) { + // The id-only test seam has no column tree, so preserve its exact top-level-id lookup. + exprId = tableColumns == null ? columnIdToExprId.get(field.getSourceId()) : null; + } else { + // A nested Iceberg source id identifies a child, but the Nereids slot and its ExprId belong to + // the top-level struct. Resolve the slot by its root id and use sourceFieldPath for the child. + exprId = sourceColumn.getUniqueId() < 0 + ? null : columnIdToExprId.get(sourceColumn.getUniqueId()); + } if (exprId == null) { insertPartitionFields.clear(); return new InsertPartitionFieldResult(false, hasNonIdentity, spec.getSpecId()); } + List sourceFieldPath = resolveSourceFieldPath( + tableColumns, sourceColumnName, field.getSourceId()); + if (sourceFieldPath == null) { + insertPartitionFields.clear(); + return new InsertPartitionFieldResult(false, hasNonIdentity, spec.getSpecId()); + } insertPartitionFields.add(new DistributionSpecMerge.MergePartitionField( field.getTransform(), exprId, field.getTransformParam(), - field.getFieldName(), field.getSourceId())); + field.getFieldName(), field.getSourceId(), sourceFieldPath)); } if (insertPartitionFields.isEmpty()) { return new InsertPartitionFieldResult(false, hasNonIdentity, spec.getSpecId()); @@ -487,6 +523,57 @@ static InsertPartitionFieldResult reconstructPartitionFields( return new InsertPartitionFieldResult(true, hasNonIdentity, spec.getSpecId()); } + private static List resolveSourceFieldPath( + List tableColumns, String sourceColumnName, int sourceId) { + if (tableColumns == null) { + return ImmutableList.of(); + } + Column sourceColumn = findSourceColumn(tableColumns, sourceColumnName); + if (sourceColumn == null) { + return null; + } + if (sourceColumn.getUniqueId() < 0) { + // Without the root id an empty path cannot distinguish a top-level source from an unstamped child. + return null; + } + if (sourceColumn.getUniqueId() == sourceId) { + return ImmutableList.of(); + } + List path = new ArrayList<>(); + // Iceberg field ids are stable across rename/evolution; resolving by id avoids ambiguous dotted names + // and keeps exchange routing on the same nested value used by the writer. + return findSourceFieldPath(sourceColumn.getChildren(), sourceId, path) + ? ImmutableList.copyOf(path) : null; + } + + private static Column findSourceColumn(List tableColumns, String sourceColumnName) { + if (tableColumns == null) { + return null; + } + for (Column column : tableColumns) { + if (column.getName().equalsIgnoreCase(sourceColumnName)) { + return column; + } + } + return null; + } + + private static boolean findSourceFieldPath(List columns, int sourceId, List path) { + if (columns == null) { + return false; + } + for (int index = 0; index < columns.size(); index++) { + Column column = columns.get(index); + path.add(index); + if (column.getUniqueId() == sourceId + || findSourceFieldPath(column.getChildren(), sourceId, path)) { + return true; + } + path.remove(path.size() - 1); + } + return false; + } + // Package-private (not private) so the same-package parity test can assert on the reconstructed // result of {@link #reconstructPartitionFields} directly, without driving the full distribution. static class InsertPartitionFieldResult { diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/DataPartition.java b/fe/fe-core/src/main/java/org/apache/doris/planner/DataPartition.java index e156fa4336fe93..0ef85f8ee67170 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/DataPartition.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/DataPartition.java @@ -161,14 +161,21 @@ public static class MergePartitionField { private final Integer param; private final String name; private final Integer sourceId; + private final ImmutableList sourceFieldPath; public MergePartitionField(Expr sourceExpr, String transform, Integer param, String name, Integer sourceId) { + this(sourceExpr, transform, param, name, sourceId, ImmutableList.of()); + } + + public MergePartitionField(Expr sourceExpr, String transform, Integer param, + String name, Integer sourceId, List sourceFieldPath) { this.sourceExpr = Preconditions.checkNotNull(sourceExpr, "sourceExpr should not be null"); this.transform = Preconditions.checkNotNull(transform, "transform should not be null"); this.param = param; this.name = name; this.sourceId = sourceId; + this.sourceFieldPath = ImmutableList.copyOf(sourceFieldPath); } public TIcebergPartitionField toThrift() { @@ -184,6 +191,9 @@ public TIcebergPartitionField toThrift() { if (sourceId != null) { field.setSourceId(sourceId); } + if (!sourceFieldPath.isEmpty()) { + field.setSourceFieldPath(sourceFieldPath); + } return field; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeColumnPruningTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeColumnPruningTest.java index 4f7c8cbe9ca6c3..9eccf6102ed527 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeColumnPruningTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeColumnPruningTest.java @@ -31,6 +31,7 @@ import org.mockito.Mockito; import java.util.ArrayList; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -162,6 +163,20 @@ public void testSlotWithNoMatchingHandleIsDropped() { Assertions.assertSame(all.get("c1"), selected.get(0)); } + @Test + public void testPinnedHandleGuardExemptsConnectorReservedPassthroughColumn() { + // Iceberg v3 row-lineage columns are part of the bound Doris schema but are generated scan slots, + // not physical Iceberg schema columns. Requiring a connector handle for one rejects every pinned + // v3 scan before the scan provider can append the generated field to its schema dictionary. + Column rowId = new Column("_row_id", PrimitiveType.BIGINT); + rowId.setReservedPassthrough(true); + + Assertions.assertFalse(PluginDrivenScanNode.requiresPinnedColumnHandle( + rowId, Collections.singleton("_row_id"))); + Assertions.assertTrue(PluginDrivenScanNode.requiresPinnedColumnHandle( + new Column("physical_col", PrimitiveType.INT), Collections.singleton("physical_col"))); + } + @Test public void testEmptyTupleProjectsNothing() { // A tuple with no slots projects nothing — the ONLY input that makes the jdbc connector fall back to diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSinkTest.java index 793f14a3a834c6..970e80d380d128 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalExternalRowLevelMergeSinkTest.java @@ -19,6 +19,9 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.catalog.StructField; +import org.apache.doris.catalog.StructType; +import org.apache.doris.catalog.Type; import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.connector.spi.Connector; import org.apache.doris.connector.spi.ConnectorMetadata; @@ -132,6 +135,87 @@ public void reconstructCarriesNonIdentityTransformAndParam() { "transform param/name/sourceId must be carried verbatim from the connector field"); } + @Test + public void reconstructCarriesNestedSourcePathWithoutRandomFallback() { + ExprId payload = exprId("payload"); + List out = new ArrayList<>(); + ConnectorWritePartitionField nested = new ConnectorWritePartitionField( + "bucket[8]", 8, "payload", "payload_part_bucket", 3); + Column payloadColumn = new Column("payload", new StructType( + new StructField("part", Type.INT))); + payloadColumn.setUniqueId(2); + payloadColumn.getChildren().get(0).setUniqueId(3); + + InsertPartitionFieldResult result = PhysicalExternalRowLevelMergeSink.reconstructPartitionFields( + out, spec(4, nested), map("payload", payload), ImmutableList.of(payloadColumn)); + + Assertions.assertTrue(result.success); + Assertions.assertEquals(ImmutableList.of( + new MergePartitionField("bucket[8]", payload, 8, "payload_part_bucket", 3, + ImmutableList.of(0))), out); + } + + @Test + public void reconstructNestedSourceUsesTopLevelIdMapFromProductionPath() { + ExprId payload = exprId("payload"); + List out = new ArrayList<>(); + ConnectorWritePartitionField nested = new ConnectorWritePartitionField( + "bucket[8]", 8, "payload", "payload_part_bucket", 3); + Column payloadColumn = new Column("payload", new StructType( + new StructField("part", Type.INT))); + payloadColumn.setUniqueId(2); + payloadColumn.getChildren().get(0).setUniqueId(3); + + InsertPartitionFieldResult result = PhysicalExternalRowLevelMergeSink.reconstructPartitionFields( + out, spec(4, nested), map("payload", payload), + java.util.Collections.singletonMap(2, payload), ImmutableList.of(payloadColumn)); + + // Production maps expressions by top-level column id, while Iceberg partition sources carry the + // nested child id. The root id must select the slot and the child id must select the path within it. + Assertions.assertTrue(result.success); + Assertions.assertEquals(ImmutableList.of( + new MergePartitionField("bucket[8]", payload, 8, "payload_part_bucket", 3, + ImmutableList.of(0))), out); + } + + @Test + public void reconstructUnstampedNestedSourceHardFails() { + ExprId payload = exprId("payload"); + List out = new ArrayList<>(); + ConnectorWritePartitionField nested = new ConnectorWritePartitionField( + "bucket[8]", 8, "payload", "payload_part_bucket", 3); + Column payloadColumn = new Column("payload", new StructType( + new StructField("part", Type.INT))); + + InsertPartitionFieldResult result = PhysicalExternalRowLevelMergeSink.reconstructPartitionFields( + out, spec(4, nested), map("payload", payload), ImmutableList.of(payloadColumn)); + + // An unstamped tree cannot prove whether the requested id is top-level or nested. Treating it as an + // empty path would hash the whole struct and silently diverge from the writer's partition source. + Assertions.assertFalse(result.success); + Assertions.assertTrue(out.isEmpty()); + } + + @Test + public void reconstructMissingNestedSourceIdHardFails() { + Column payloadColumn = new Column("payload", new StructType( + new StructField("part", Type.INT))); + payloadColumn.setUniqueId(2); + payloadColumn.getChildren().get(0).setUniqueId(3); + List out = new ArrayList<>(); + + InsertPartitionFieldResult result = PhysicalExternalRowLevelMergeSink.reconstructPartitionFields( + out, + spec(4, new ConnectorWritePartitionField( + "bucket[8]", 8, "payload", "payload_part_bucket", 99)), + map("payload", exprId("payload")), + ImmutableList.of(payloadColumn)); + + Assertions.assertFalse(result.success, + "an evolved schema must not hash the whole struct when the nested source id disappeared"); + Assertions.assertTrue(out.isEmpty()); + } + @Test public void reconstructNullSourceColumnNameHardFailsAndClears() { // PARITY-1a: a null source-column-name field hard-fails the whole spec; the already-added prior @@ -220,6 +304,7 @@ public void postFlipMergeBuildsDistributionFromConnectorSpec() { // must flow into the DistributionSpecMerge: one identity partition column 'id' resolved to the // child's id slot, insertRandom=false, spec id carried. Column id = new Column("id", PrimitiveType.INT); + id.setUniqueId(1); SlotReference idSlot = new SlotReference("id", IntegerType.INSTANCE); SlotReference opSlot = new SlotReference(MergeOperation.OPERATION_COLUMN, IntegerType.INSTANCE); SlotReference rowidSlot = new SlotReference(Column.ICEBERG_ROWID_COL, IntegerType.INSTANCE); diff --git a/gensrc/thrift/Partitions.thrift b/gensrc/thrift/Partitions.thrift index 19ab0a17dc7de0..da172fac735c2b 100644 --- a/gensrc/thrift/Partitions.thrift +++ b/gensrc/thrift/Partitions.thrift @@ -183,6 +183,8 @@ struct TIcebergPartitionField { 3: required Exprs.TExpr source_expr 4: optional string name 5: optional i32 source_id + // Zero-based STRUCT child indexes below source_expr; empty/unset means a top-level source. + 6: optional list source_field_path } struct TMergePartitionInfo { diff --git a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out index 76bede82d3f5fb..673d6c879cb824 100644 --- a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out +++ b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.out @@ -3,8 +3,9 @@ 1 A [1, null, 3] {"x":10, "null-value":null} {"metric":10, "label":"old-a", "nested":{"count":1, "comment":null, "score":null}, "tags":null, "attributes":null} 2 N \N {"x":null} {"metric":20, "label":null, "nested":{"count":null, "comment":"old-null", "score":null}, "tags":null, "attributes":null} 3 B [] {} \N -4 A1 [4000000000, null] {"large":5000000000, "null-value":null} {"metric":6000000000, "label":"new-a", "nested":{"count":7000000000, "comment":"nested-new", "score":7.5}, "tags":["x", null, "z"], "attributes":{"a":8000000000, "b":null}} +4 A2 [4000000000, null] {"large":5000000000, "null-value":null} {"metric":6000000000, "label":"new-a", "nested":{"count":7000000000, "comment":"nested-new", "score":7.5}, "tags":["x", null, "z"], "attributes":{"a":8000000000, "b":null}} 5 N2 [null] \N {"metric":50, "label":null, "nested":{"count":5, "comment":null, "score":null}, "tags":null, "attributes":{"null-value":null}} +6 Z4 \N \N \N -- !complex_children -- 1 10 1 \N \N \N @@ -12,19 +13,24 @@ 3 \N \N \N \N \N 4 6000000000 7000000000 7.5 ["x", null, "z"] {"a":8000000000, "b":null} 5 50 5 \N \N {"null-value":null} +6 \N \N \N \N \N -- !complex_nulls -- 1 2 3 5 +6 -- !complex_partition_specs -- 0 3 -2 2 +3 5 + +-- !complex_nested_partition_pruning -- +4 +6 -- !complex_base_tag -- 1 [1, null, 3] {"x":10, "null-value":null} 10 old-a 1 \N 2 \N {"x":null} 20 \N \N old-null 3 [] {} \N \N \N \N - diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy index e5ad9e7c6ed19e..3460561577b036 100644 --- a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_complex_evolution.groovy @@ -116,6 +116,15 @@ suite("test_iceberg_write_complex_evolution", """ sql """alter table complex_evolution add partition key bucket(8, id) as id_bucket""" sql """alter table complex_evolution add partition key truncate(1, group_key) as group_prefix""" + // Iceberg permits a nested primitive source. Create it through Spark to verify Doris can plan and + // physically partition the following INSERT by the schema-wide nested field id. Invalidate Spark's + // cached table first so its commit requirement sees the partition ids assigned by the Doris DDLs. + spark_iceberg """refresh table demo.${dbName}.complex_evolution""" + spark_iceberg """ + alter table demo.${dbName}.complex_evolution + add partition field bucket(4, payload.nested.count) + """ + sql """refresh table complex_evolution""" sql """ insert into complex_evolution values @@ -135,7 +144,15 @@ suite("test_iceberg_write_complex_evolution", struct(cast(5 as bigint), null, null), null, map('null-value', null) - )) + )), + (6, 'Z3', null, null, null) + """ + + // Route an UPDATE insert image by the nested source and preserve the parent-NULL partition value. + sql """ + update complex_evolution + set group_key = case id when 4 then 'A2' else 'Z4' end + where id in (4, 6) """ // W02-S03: Current schema reads both old and new files without moving old child values. @@ -166,6 +183,13 @@ suite("test_iceberg_write_complex_evolution", group by spec_id order by spec_id """ + order_qt_complex_nested_partition_pruning """ + select id + from complex_evolution + where payload.nested.count = cast(7000000000 as bigint) + or (id = 6 and payload.nested.count is null) + order by id + """ assertSparkMatchesDoris() // W02-S04: A pre-evolution tag binds the old files to their historical complex schema. From 536657a1d18e1ca2c415f0681c9af93e78d2a4f7 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Fri, 7 Aug 2026 12:58:19 +0800 Subject: [PATCH 19/20] [fix](be) Preserve floating-point equality in Parquet pruning (#66470) ## Summary - backport the floating-point pruning correctness fixes to branch-4.1 for File Scanner V2 - preserve Doris NaN and signed-zero equality semantics in V2 Parquet Bloom, min/max, and IN-predicate pruning - keep pruning conservative when Parquet statistics omit NaNs or a Bloom encoding cannot safely prove absence - leave the legacy File Scanner V1 path unchanged ## Testing - clang-format 16 check on all affected C/C++ files - compiled every affected production and test translation unit after rebasing onto branch-4.1 - 121 focused BE unit tests passed, including expression, hybrid-set, and native V2 Parquet pruning coverage --- be/src/core/field.h | 13 + be/src/exprs/expr_zonemap_filter.cpp | 37 +- be/src/exprs/expr_zonemap_filter.h | 4 +- be/src/exprs/function/functions_comparison.h | 17 +- be/src/exprs/hybrid_set.h | 18 +- be/src/exprs/vdirect_in_predicate.h | 5 +- be/src/exprs/vin_predicate.cpp | 8 +- be/src/exprs/vin_predicate.h | 1 + .../format_v2/parquet/parquet_statistics.cpp | 3 + .../index/zone_map/zonemap_eval_context.cpp | 5 + .../index/zone_map/zonemap_eval_context.h | 3 + be/test/core/field_test.cpp | 11 + be/test/exprs/expr_zonemap_filter_test.cpp | 223 +++++++++++- be/test/exprs/hybrid_set_test.cpp | 29 ++ .../format_v2/parquet/parquet_scan_test.cpp | 241 +++++++++++++ .../parquet/parquet_statistics_test.cpp | 330 ++++++++++++++++++ 16 files changed, 921 insertions(+), 27 deletions(-) diff --git a/be/src/core/field.h b/be/src/core/field.h index d39d82087772ca..f418283d4e5652 100644 --- a/be/src/core/field.h +++ b/be/src/core/field.h @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -271,6 +272,18 @@ class Field { template const typename PrimitiveTypeTraits::CppType& get() const; + bool is_nan() const { + // Keep type dispatch with the value so callers cannot reinterpret Field storage using a + // mismatched PrimitiveType. + if (type == PrimitiveType::TYPE_FLOAT) { + return std::isnan(get()); + } + if (type == PrimitiveType::TYPE_DOUBLE) { + return std::isnan(get()); + } + return false; + } + bool operator==(const Field& rhs) const { return operator<=>(rhs) == std::strong_ordering::equal; } diff --git a/be/src/exprs/expr_zonemap_filter.cpp b/be/src/exprs/expr_zonemap_filter.cpp index 8cb999d08bafd4..ff60af4befac4c 100644 --- a/be/src/exprs/expr_zonemap_filter.cpp +++ b/be/src/exprs/expr_zonemap_filter.cpp @@ -18,7 +18,9 @@ #include "exprs/expr_zonemap_filter.h" #include +#include #include +#include #include #include "common/check.h" @@ -60,6 +62,24 @@ bool dictionary_contains(const DictionaryEvalContext::SlotDictionary& dictionary }); } +template +bool floating_point_bloom_filter_may_contain(const segment_v2::BloomFilter& bloom_filter, T value) { + static_assert(std::is_floating_point_v); + // Doris equality collapses NaN payloads and signed zeros, while Parquet Bloom hashes physical + // bytes. A negative probe is safe only after covering the entire Doris-equivalent class. + if (std::isnan(value)) { + return true; + } + const auto test_value = [&](T candidate) { + return bloom_filter.test_bytes(reinterpret_cast(&candidate), + sizeof(candidate)); + }; + if (test_value(value)) { + return true; + } + return value == T {0} && test_value(-value); +} + bool bloom_filter_may_contain(const BloomFilterEvalContext::SlotBloomFilter& slot_filter, const Field& value) { DORIS_CHECK(slot_filter.data_type != nullptr); @@ -84,13 +104,11 @@ bool bloom_filter_may_contain(const BloomFilterEvalContext::SlotBloomFilter& slo } case TYPE_FLOAT: { const float typed_value = value.get(); - return slot_filter.bloom_filter->test_bytes(reinterpret_cast(&typed_value), - sizeof(typed_value)); + return floating_point_bloom_filter_may_contain(*slot_filter.bloom_filter, typed_value); } case TYPE_DOUBLE: { const double typed_value = value.get(); - return slot_filter.bloom_filter->test_bytes(reinterpret_cast(&typed_value), - sizeof(typed_value)); + return floating_point_bloom_filter_may_contain(*slot_filter.bloom_filter, typed_value); } case TYPE_CHAR: case TYPE_VARCHAR: @@ -151,6 +169,7 @@ Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataT DORIS_CHECK(value_type != nullptr); result->contains_null = set.contain_null(); + result->contains_nan = false; result->values.clear(); result->min_value = Field(); result->max_value = Field(); @@ -165,6 +184,7 @@ Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataT auto literal = VLiteral::create_shared(literal_node); Field field; literal->get_column_ptr()->get(0, field); + result->contains_nan |= field.is_nan(); result->values.emplace_back(std::move(field)); } iterator->next(); @@ -244,7 +264,8 @@ ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExpr ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtr& slot_expr, bool is_not_in, const std::vector& values, - const Field& min_value, const Field& max_value) { + bool contains_nan, const Field& min_value, + const Field& max_value) { auto slot = std::dynamic_pointer_cast(slot_expr); DORIS_CHECK(slot != nullptr); // Empty IN has no candidate values, while NOT IN with an empty set cannot filter anything. @@ -277,6 +298,12 @@ ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSP return ZoneMapFilterResult::kNoMatch; } + if (ctx.floating_nan_count_unknown(slot->column_id()) && + ((!is_not_in && contains_nan) || (is_not_in && !contains_nan))) { + // Hidden Parquet NaNs can satisfy IN only when queried, and NOT IN only when omitted. + return unsupported_zonemap_filter(ctx); + } + if (!range_stats_usable_for_zonemap(zone_map, slot_type)) { return unsupported_zonemap_filter(ctx); } diff --git a/be/src/exprs/expr_zonemap_filter.h b/be/src/exprs/expr_zonemap_filter.h index 5b0df00e121547..10e6c7dbfbad77 100644 --- a/be/src/exprs/expr_zonemap_filter.h +++ b/be/src/exprs/expr_zonemap_filter.h @@ -46,6 +46,7 @@ namespace doris::expr_zonemap { struct InZonemapMaterializedSet { bool contains_null = false; + bool contains_nan = false; std::vector values; Field min_value; Field max_value; @@ -143,7 +144,8 @@ ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExpr ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtr& slot_expr, bool is_not_in, const std::vector& values, - const Field& min_value, const Field& max_value); + bool contains_nan, const Field& min_value, + const Field& max_value); ZoneMapFilterResult eval_eq_dictionary(const DictionaryEvalContext& ctx, const SlotLiteral& slot_literal); diff --git a/be/src/exprs/function/functions_comparison.h b/be/src/exprs/function/functions_comparison.h index a0789c259c790b..10fdc89f86c71f 100644 --- a/be/src/exprs/function/functions_comparison.h +++ b/be/src/exprs/function/functions_comparison.h @@ -314,6 +314,15 @@ inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const VExprSP const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op; const auto& literal = slot_literal->literal; + const bool literal_is_nan = literal.is_nan(); + const bool hidden_nan_can_match = (effective_op == Op::EQ && literal_is_nan) || + (effective_op == Op::NE && !literal_is_nan) || + (effective_op == Op::GT && !literal_is_nan) || + effective_op == Op::GE; + if (ctx.floating_nan_count_unknown(slot_literal->slot_index) && hidden_nan_can_match) { + // Parquet bounds omit NaNs, so only operators that cannot match a hidden NaN may prune. + return unsupported_zonemap_filter(ctx); + } switch (effective_op) { case Op::EQ: return literal < zone_map.min_value || zone_map.max_value < literal @@ -367,7 +376,13 @@ inline bool can_evaluate(const VExprSPtrs& arguments) { } inline bool can_evaluate_equality(const VExprSPtrs& arguments, Op op) { - return op == Op::EQ && can_evaluate(arguments); + if (op != Op::EQ || !can_evaluate(arguments)) { + return false; + } + const auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments); + DORIS_CHECK(slot_literal.has_value()); + // Bloom membership cannot disprove Doris NaN equality across different physical encodings. + return !slot_literal->literal.is_nan(); } inline bool dictionary_value_matches(const Field& value, const Field& literal, Op op) { diff --git a/be/src/exprs/hybrid_set.h b/be/src/exprs/hybrid_set.h index 0717e71fc2029a..55012b37c39ff7 100644 --- a/be/src/exprs/hybrid_set.h +++ b/be/src/exprs/hybrid_set.h @@ -144,6 +144,19 @@ struct IsBitSetContainer : std::false_type {}; template struct IsBitSetContainer> : std::true_type {}; +template +struct DynamicContainerHash { + size_t operator()(const T& value) const { + if constexpr (std::is_floating_point_v) { + T normalized = value; + // The hash must collapse NaN payloads and signed zeros exactly as Doris equality does. + NormalizeFloat(normalized); + return phmap::Hash {}(normalized); + } + return phmap::Hash {}(value); + } +}; + /** * Dynamic Container uses phmap::flat_hash_set. * @tparam T Element Type @@ -152,7 +165,8 @@ template class DynamicContainer { public: using Self = DynamicContainer; - using Iterator = typename flat_hash_set::iterator; + using Set = flat_hash_set>; + using Iterator = typename Set::iterator; using ElementType = T; DynamicContainer() = default; @@ -173,7 +187,7 @@ class DynamicContainer { size_t size() const { return _set.size(); } private: - flat_hash_set _set; + Set _set; }; // TODO Maybe change void* parameter to template parameter better. diff --git a/be/src/exprs/vdirect_in_predicate.h b/be/src/exprs/vdirect_in_predicate.h index 693571d25bf777..ff7971b941ddb3 100644 --- a/be/src/exprs/vdirect_in_predicate.h +++ b/be/src/exprs/vdirect_in_predicate.h @@ -41,6 +41,7 @@ class VDirectInPredicate final : public VExpr { std::once_flag materialize_once; Status materialization_status; bool zonemap_materialized = false; + bool seg_filter_contains_nan = false; std::vector seg_filter_values; Field seg_filter_min; Field seg_filter_max; @@ -101,7 +102,8 @@ class VDirectInPredicate final : public VExpr { ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override { return expr_zonemap::eval_in_zonemap( ctx, get_child(0), false, _pruning_state->seg_filter_values, - _pruning_state->seg_filter_min, _pruning_state->seg_filter_max); + _pruning_state->seg_filter_contains_nan, _pruning_state->seg_filter_min, + _pruning_state->seg_filter_max); } bool can_evaluate_zonemap_filter() const override { @@ -329,6 +331,7 @@ class VDirectInPredicate final : public VExpr { return; } pruning_state->seg_filter_values = std::move(materialized.values); + pruning_state->seg_filter_contains_nan = materialized.contains_nan; pruning_state->seg_filter_min = std::move(materialized.min_value); pruning_state->seg_filter_max = std::move(materialized.max_value); pruning_state->zonemap_materialized = true; diff --git a/be/src/exprs/vin_predicate.cpp b/be/src/exprs/vin_predicate.cpp index 0dbfb3f7e9a289..3fe26cff2d207a 100644 --- a/be/src/exprs/vin_predicate.cpp +++ b/be/src/exprs/vin_predicate.cpp @@ -157,6 +157,7 @@ Status VInPredicate::evaluate_inverted_index(VExprContext* context, uint32_t seg Status VInPredicate::_materialize_for_zonemap_filter(VExprContext* context) { _seg_filter_values.clear(); _seg_filter_contains_null = false; + _seg_filter_contains_nan = false; _zonemap_materialized = false; _direct_filter_set.reset(); if (_children.size() < 2 || !_children[0]->is_slot_ref()) { @@ -183,6 +184,7 @@ Status VInPredicate::_materialize_for_zonemap_filter(VExprContext* context) { RETURN_IF_ERROR(expr_zonemap::materialize_hybrid_set_for_zonemap_filter( *in_state->hybrid_set, data_type, &materialized)); _seg_filter_contains_null = materialized.contains_null; + _seg_filter_contains_nan = materialized.contains_nan; _seg_filter_values = std::move(materialized.values); _seg_filter_min = std::move(materialized.min_value); _seg_filter_max = std::move(materialized.max_value); @@ -195,7 +197,8 @@ ZoneMapFilterResult VInPredicate::evaluate_zonemap_filter(const ZoneMapEvalConte return ZoneMapFilterResult::kNoMatch; } return expr_zonemap::eval_in_zonemap(ctx, get_child(0), _is_not_in, _seg_filter_values, - _seg_filter_min, _seg_filter_max); + _seg_filter_contains_nan, _seg_filter_min, + _seg_filter_max); } bool VInPredicate::can_evaluate_zonemap_filter() const { @@ -217,7 +220,8 @@ ZoneMapFilterResult VInPredicate::evaluate_bloom_filter(const BloomFilterEvalCon } bool VInPredicate::can_evaluate_bloom_filter() const { - return _zonemap_materialized && !_is_not_in && + // A NaN member forces conservative retention regardless of the remaining finite probes. + return _zonemap_materialized && !_is_not_in && !_seg_filter_contains_nan && std::dynamic_pointer_cast(get_child(0)) != nullptr; } diff --git a/be/src/exprs/vin_predicate.h b/be/src/exprs/vin_predicate.h index 299a69f46a49b9..6e708915025424 100644 --- a/be/src/exprs/vin_predicate.h +++ b/be/src/exprs/vin_predicate.h @@ -102,6 +102,7 @@ class VInPredicate MOCK_REMOVE(final) : public VExpr { bool _is_args_all_constant = false; bool _zonemap_materialized = false; bool _seg_filter_contains_null = false; + bool _seg_filter_contains_nan = false; std::shared_ptr _direct_filter_set; std::vector _seg_filter_values; Field _seg_filter_min; diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index 30a8cc6da92600..b0c6fbc28eb1df 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -847,6 +847,9 @@ void add_slot_zonemap(ZoneMapEvalContext* ctx, int slot_index, const DataTypePtr ZoneMapEvalContext::SlotZoneMap slot_zone_map; slot_zone_map.data_type = data_type; slot_zone_map.zone_map = std::move(zone_map); + const auto primitive_type = remove_nullable(data_type)->get_primitive_type(); + slot_zone_map.floating_nan_count_unknown = + primitive_type == TYPE_FLOAT || primitive_type == TYPE_DOUBLE; ctx->slots.emplace(slot_index, std::move(slot_zone_map)); } diff --git a/be/src/storage/index/zone_map/zonemap_eval_context.cpp b/be/src/storage/index/zone_map/zonemap_eval_context.cpp index b65d4400e8af2c..2fb6864349c865 100644 --- a/be/src/storage/index/zone_map/zonemap_eval_context.cpp +++ b/be/src/storage/index/zone_map/zonemap_eval_context.cpp @@ -37,6 +37,11 @@ DataTypePtr ZoneMapEvalContext::data_type(int slot_index) const { return it->second.data_type; } +bool ZoneMapEvalContext::floating_nan_count_unknown(int slot_index) const { + auto it = slots.find(slot_index); + return it != slots.end() && it->second.floating_nan_count_unknown; +} + void ZoneMapEvalStats::merge_page_eval_stats(const ZoneMapEvalStats& src) { // Page-level evaluation repeats the same conjuncts for many pages. Keep structural // diagnostics once per column, while operation counters still reflect actual page checks. diff --git a/be/src/storage/index/zone_map/zonemap_eval_context.h b/be/src/storage/index/zone_map/zonemap_eval_context.h index c5a750b05914c1..7c1afab8780e0e 100644 --- a/be/src/storage/index/zone_map/zonemap_eval_context.h +++ b/be/src/storage/index/zone_map/zonemap_eval_context.h @@ -52,10 +52,13 @@ class ZoneMapEvalContext { struct SlotZoneMap { DataTypePtr data_type; std::shared_ptr zone_map; + // Parquet min/max does not expose whether a floating chunk also contains NaNs. + bool floating_nan_count_unknown = false; }; std::shared_ptr zone_map(int slot_index) const; DataTypePtr data_type(int slot_index) const; + bool floating_nan_count_unknown(int slot_index) const; phmap::flat_hash_map slots; diff --git a/be/test/core/field_test.cpp b/be/test/core/field_test.cpp index 694b292049c383..efe94cef96892d 100644 --- a/be/test/core/field_test.cpp +++ b/be/test/core/field_test.cpp @@ -20,6 +20,7 @@ #include #include +#include #include #include "core/column/column_string.h" @@ -32,6 +33,16 @@ #include "gtest/gtest_pred_impl.h" // IWYU pragma: keep namespace doris { +TEST(VFieldTest, detects_floating_point_nan) { + EXPECT_TRUE(Field::create_field(std::numeric_limits::quiet_NaN()).is_nan()); + EXPECT_TRUE( + Field::create_field(std::numeric_limits::quiet_NaN()).is_nan()); + EXPECT_FALSE(Field::create_field(0.0F).is_nan()); + EXPECT_FALSE(Field::create_field(-0.0).is_nan()); + EXPECT_FALSE(Field::create_field(0).is_nan()); + EXPECT_FALSE(Field().is_nan()); +} + TEST(VFieldTest, field_string) { Field f; diff --git a/be/test/exprs/expr_zonemap_filter_test.cpp b/be/test/exprs/expr_zonemap_filter_test.cpp index 72981c01275c25..d8164cb7248701 100644 --- a/be/test/exprs/expr_zonemap_filter_test.cpp +++ b/be/test/exprs/expr_zonemap_filter_test.cpp @@ -20,6 +20,9 @@ #include #include +#include +#include +#include #include #include #include @@ -427,6 +430,193 @@ TEST(ExprZonemapFilterTest, ComparisonDictionarySupportsTypedRangesWhileBloomUse {string_slot, make_string_literal("delta")})); } +TEST(ExprZonemapFilterTest, FloatingPointNanBloomProbeIsConservative) { + auto bloom_filter = std::make_unique(); + ASSERT_TRUE(bloom_filter->init(segment_v2::BloomFilter::MINIMUM_BYTES).ok()); + const double finite_value = 1.0; + bloom_filter->add_bytes(reinterpret_cast(&finite_value), sizeof(finite_value)); + + FunctionComparison equals; + const auto check_type = [&](const DataTypePtr& type, Field nan_field) { + auto slot = make_slot(0, type); + auto literal = std::make_shared(create_texpr_node_from( + nan_field, remove_nullable(type)->get_primitive_type(), 0, 0)); + auto bloom_ctx = make_bloom_filter_context(bloom_filter.get(), type); + + EXPECT_FALSE(equals.can_evaluate_bloom_filter({slot, literal})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + equals.evaluate_bloom_filter(bloom_ctx, {slot, literal})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + expr_zonemap::eval_in_bloom_filter(bloom_ctx, slot, false, {nan_field})); + + const auto primitive_type = remove_nullable(type)->get_primitive_type(); + const Field absent_finite = primitive_type == TYPE_FLOAT + ? Field::create_field(2.0F) + : Field::create_field(2.0); + auto finite_literal = std::make_shared( + create_texpr_node_from(absent_finite, primitive_type, 0, 0)); + EXPECT_TRUE(equals.can_evaluate_bloom_filter({slot, finite_literal})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + equals.evaluate_bloom_filter(bloom_ctx, {slot, finite_literal})); + }; + + check_type(std::make_shared(), + Field::create_field(std::numeric_limits::quiet_NaN())); + check_type(std::make_shared(), + Field::create_field(std::numeric_limits::quiet_NaN())); +} + +TEST(ExprZonemapFilterTest, FloatingPointInWithNanIsNotBloomEligible) { + auto type = std::make_shared(); + auto predicate = std::make_shared(make_in_predicate_node(false, 2)); + predicate->add_child(make_slot(0, type)); + predicate->_zonemap_materialized = true; + predicate->_seg_filter_contains_nan = true; + predicate->_seg_filter_values = { + Field::create_field(1.0), + Field::create_field(std::numeric_limits::quiet_NaN())}; + + EXPECT_FALSE(predicate->can_evaluate_bloom_filter()); + predicate->_seg_filter_contains_nan = false; + EXPECT_TRUE(predicate->can_evaluate_bloom_filter()); +} + +TEST(ExprZonemapFilterTest, FloatingPointNanEqualityIgnoresFiniteOnlyRangeBounds) { + const auto check_type = []( + UInt nan_bits) { + using T = typename PrimitiveTypeTraits::CppType; + auto type = std::make_shared(); + auto slot = make_slot(0, type); + const auto nan_field = Field::create_field(std::bit_cast(nan_bits)); + auto nan_literal = + std::make_shared(create_texpr_node_from(nan_field, Type, 0, 0)); + + segment_v2::ZoneMap zone_map; + zone_map.min_value = Field::create_field(T {0}); + zone_map.max_value = Field::create_field(T {0}); + zone_map.has_not_null = true; + auto ctx = make_context(std::move(zone_map), type); + ctx.slots.at(0).floating_nan_count_unknown = true; + + FunctionComparison equals; + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + equals.evaluate_zonemap_filter(ctx, {slot, nan_literal})); + + const auto finite_field = Field::create_field(T {10}); + const auto zero_field = Field::create_field(T {0}); + const auto one_field = Field::create_field(T {1}); + auto zero_literal = + std::make_shared(create_texpr_node_from(zero_field, Type, 0, 0)); + auto one_literal = + std::make_shared(create_texpr_node_from(one_field, Type, 0, 0)); + FunctionComparison not_equals; + FunctionComparison greater; + FunctionComparison greater_equal; + FunctionComparison less; + FunctionComparison less_equal; + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + not_equals.evaluate_zonemap_filter(ctx, {slot, zero_literal})); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + greater.evaluate_zonemap_filter(ctx, {slot, one_literal})); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + greater_equal.evaluate_zonemap_filter(ctx, {slot, one_literal})); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + less.evaluate_zonemap_filter(ctx, {one_literal, slot})); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + less_equal.evaluate_zonemap_filter(ctx, {one_literal, slot})); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + expr_zonemap::eval_in_zonemap(ctx, slot, false, {finite_field, nan_field}, true, + finite_field, nan_field)); + EXPECT_EQ(ZoneMapFilterResult::kUnsupported, + expr_zonemap::eval_in_zonemap(ctx, slot, true, {zero_field}, false, zero_field, + zero_field)); + + segment_v2::ZoneMap all_null_zone_map; + all_null_zone_map.min_value = zero_field; + all_null_zone_map.max_value = zero_field; + auto all_null_ctx = make_context(std::move(all_null_zone_map), type); + all_null_ctx.slots.at(0).floating_nan_count_unknown = true; + EXPECT_EQ( + ZoneMapFilterResult::kNoMatch, + expr_zonemap::eval_in_zonemap(all_null_ctx, slot, false, {finite_field, nan_field}, + true, finite_field, nan_field)); + + ctx.slots.at(0).floating_nan_count_unknown = false; + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + equals.evaluate_zonemap_filter(ctx, {slot, nan_literal})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + expr_zonemap::eval_in_zonemap(ctx, slot, false, {finite_field, nan_field}, true, + finite_field, nan_field)); + }; + + check_type.template operator()(uint32_t {0x7fc00002U}); + check_type.template operator()(uint64_t {0x7ff8000000000002ULL}); +} + +TEST(ExprZonemapFilterTest, DirectInRawFixedKeepsEqualNanPayloadFromLargeSet) { + const auto check_type = []( + UInt stored_bits, UInt probe_bits) { + using T = typename PrimitiveTypeTraits::CppType; + auto type = std::make_shared(); + std::shared_ptr filter(create_set(Type, false)); + for (int value = 0; value < FIXED_CONTAINER_MAX_SIZE; ++value) { + T finite = static_cast(value); + filter->insert(&finite); + } + const T stored_nan = std::bit_cast(stored_bits); + filter->insert(&stored_nan); + ASSERT_EQ(FIXED_CONTAINER_MAX_SIZE + 1, filter->size()); + + VDirectInPredicate predicate(make_in_predicate_node(false, 1), filter, true); + predicate.add_child(make_slot(0, type)); + ASSERT_TRUE(predicate.can_execute_on_raw_fixed_values(type, 0)); + + const T probe_nan = std::bit_cast(probe_bits); + uint8_t match = 1; + ASSERT_TRUE( + predicate + .execute_on_raw_fixed_values(reinterpret_cast(&probe_nan), + 1, sizeof(T), type, 0, &match) + .ok()); + EXPECT_EQ(1, match); + }; + + check_type.template operator()(uint32_t {0x7fc00001U}, + uint32_t {0x7fc00002U}); + check_type.template operator()(uint64_t {0x7ff8000000000001ULL}, + uint64_t {0x7ff8000000000002ULL}); +} + +TEST(ExprZonemapFilterTest, FloatingPointSignedZeroBloomProbeChecksBothEncodings) { + FunctionComparison equals; + const auto check_type = [&]( + const DataTypePtr& type, + typename PrimitiveTypeTraits::CppType stored_value, + typename PrimitiveTypeTraits::CppType predicate_value) { + auto bloom_filter = std::make_unique(); + ASSERT_TRUE(bloom_filter->init(segment_v2::BloomFilter::MINIMUM_BYTES).ok()); + bloom_filter->add_bytes(reinterpret_cast(&stored_value), sizeof(stored_value)); + ASSERT_FALSE(bloom_filter->test_bytes(reinterpret_cast(&predicate_value), + sizeof(predicate_value))); + + auto slot = make_slot(0, type); + const auto field = Field::create_field(predicate_value); + auto literal = std::make_shared(create_texpr_node_from(field, Type, 0, 0)); + auto bloom_ctx = make_bloom_filter_context(bloom_filter.get(), type); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + equals.evaluate_bloom_filter(bloom_ctx, {slot, literal})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + expr_zonemap::eval_in_bloom_filter(bloom_ctx, slot, false, {field})); + }; + + const auto float_type = std::make_shared(); + check_type.template operator()(float_type, -0.0F, 0.0F); + check_type.template operator()(float_type, 0.0F, -0.0F); + const auto double_type = std::make_shared(); + check_type.template operator()(double_type, -0.0, 0.0); + check_type.template operator()(double_type, 0.0, -0.0); +} + TEST(ExprZonemapFilterTest, DefaultFunctionForwardsDictionaryAndBloomEvaluation) { auto type = int_type(); auto slot = make_slot(0, type); @@ -478,7 +668,7 @@ TEST(ExprZonemapFilterTest, MissingSlotTypeCountsUnsupportedZonemapEvalOnce) { std::vector values {int_field(10)}; ZoneMapEvalContext in_ctx; EXPECT_EQ(ZoneMapFilterResult::kUnsupported, - expr_zonemap::eval_in_zonemap(in_ctx, slot, false, values, int_field(10), + expr_zonemap::eval_in_zonemap(in_ctx, slot, false, values, false, int_field(10), int_field(10))); EXPECT_EQ(1, in_ctx.stats.unusable_zonemap_eval_count); } @@ -541,20 +731,20 @@ TEST(ExprZonemapFilterTest, InZonemapSkipsZonesWithoutNonNullValues) { segment_v2::ZoneMap empty_zone; auto empty_ctx = make_context(empty_zone, type); EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(empty_ctx, slot, false, values, int_field(10), + expr_zonemap::eval_in_zonemap(empty_ctx, slot, false, values, false, int_field(10), int_field(10))); EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(empty_ctx, slot, true, values, int_field(10), + expr_zonemap::eval_in_zonemap(empty_ctx, slot, true, values, false, int_field(10), int_field(10))); segment_v2::ZoneMap only_null_zone; only_null_zone.has_null = true; auto only_null_ctx = make_context(only_null_zone, type); EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(only_null_ctx, slot, false, values, int_field(10), - int_field(10))); + expr_zonemap::eval_in_zonemap(only_null_ctx, slot, false, values, false, + int_field(10), int_field(10))); EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(only_null_ctx, slot, true, values, int_field(10), + expr_zonemap::eval_in_zonemap(only_null_ctx, slot, true, values, false, int_field(10), int_field(10))); } @@ -644,8 +834,9 @@ TEST(ExprZonemapFilterTest, CharZonemapUsesTrimmedLogicalBounds) { auto in_value = Field::create_field("gamma"); std::vector values {in_value}; auto in_ctx = make_context(zone_map, char_type); - EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(in_ctx, slot, false, values, in_value, in_value)); + EXPECT_EQ( + ZoneMapFilterResult::kNoMatch, + expr_zonemap::eval_in_zonemap(in_ctx, slot, false, values, false, in_value, in_value)); } TEST(ExprZonemapFilterTest, InZonemapFallsBackToRangeWhenPointListIsLarge) { @@ -658,7 +849,8 @@ TEST(ExprZonemapFilterTest, InZonemapFallsBackToRangeWhenPointListIsLarge) { values.emplace_back(int_field(value)); } EXPECT_EQ(ZoneMapFilterResult::kMayMatch, - expr_zonemap::eval_in_zonemap(ctx, slot, false, values, int_field(1), int_field(65))); + expr_zonemap::eval_in_zonemap(ctx, slot, false, values, false, int_field(1), + int_field(65))); EXPECT_EQ(0, ctx.stats.in_zonemap_point_check_count); EXPECT_EQ(1, ctx.stats.in_zonemap_range_only_count); @@ -671,7 +863,8 @@ TEST(ExprZonemapFilterTest, InZonemapUsesPointChecksUnderThreshold) { std::vector values {int_field(1), int_field(30)}; EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(ctx, slot, false, values, int_field(1), int_field(30))); + expr_zonemap::eval_in_zonemap(ctx, slot, false, values, false, int_field(1), + int_field(30))); EXPECT_EQ(1, ctx.stats.in_zonemap_point_check_count); } @@ -682,19 +875,19 @@ TEST(ExprZonemapFilterTest, InZonemapHandlesEmptyListAndNotInSingleValueRange) { std::vector empty_values; EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(ctx, slot, false, empty_values, {}, {})); + expr_zonemap::eval_in_zonemap(ctx, slot, false, empty_values, false, {}, {})); EXPECT_EQ(ZoneMapFilterResult::kMayMatch, - expr_zonemap::eval_in_zonemap(ctx, slot, true, empty_values, {}, {})); + expr_zonemap::eval_in_zonemap(ctx, slot, true, empty_values, false, {}, {})); auto single_value_ctx = make_context(make_int_zonemap(10, 10), type); std::vector values {int_field(10)}; EXPECT_EQ(ZoneMapFilterResult::kNoMatch, - expr_zonemap::eval_in_zonemap(single_value_ctx, slot, true, values, int_field(10), - int_field(10))); + expr_zonemap::eval_in_zonemap(single_value_ctx, slot, true, values, false, + int_field(10), int_field(10))); std::vector other_values {int_field(11)}; EXPECT_EQ(ZoneMapFilterResult::kMayMatch, - expr_zonemap::eval_in_zonemap(single_value_ctx, slot, true, other_values, + expr_zonemap::eval_in_zonemap(single_value_ctx, slot, true, other_values, false, int_field(11), int_field(11))); } diff --git a/be/test/exprs/hybrid_set_test.cpp b/be/test/exprs/hybrid_set_test.cpp index aed2103d66f34b..2d4177fb406a8c 100644 --- a/be/test/exprs/hybrid_set_test.cpp +++ b/be/test/exprs/hybrid_set_test.cpp @@ -19,6 +19,8 @@ #include +#include +#include #include #include @@ -395,6 +397,33 @@ TEST_F(HybridSetTest, double) { a = 5.1; EXPECT_FALSE(set->find(&a)); } + +TEST_F(HybridSetTest, DynamicFloatingSetFindsDorisEqualNanPayload) { + const auto check_type = [](UInt stored_bits, + UInt probe_bits) { + using T = typename PrimitiveTypeTraits::CppType; + std::unique_ptr set(create_set(Type, false)); + for (int value = 0; value < FIXED_CONTAINER_MAX_SIZE; ++value) { + T finite = static_cast(value); + set->insert(&finite); + } + const T stored_nan = std::bit_cast(stored_bits); + set->insert(&stored_nan); + ASSERT_EQ(FIXED_CONTAINER_MAX_SIZE + 1, set->size()); + + const T probe_nan = std::bit_cast(probe_bits); + EXPECT_TRUE(set->find(&probe_nan)); + uint8_t match = 1; + set->find_batch_raw_fixed(reinterpret_cast(&probe_nan), 1, sizeof(T), + &match); + EXPECT_EQ(1, match); + }; + + check_type.template operator()(uint32_t {0x7fc00001U}, uint32_t {0x7fc00002U}); + check_type.template operator()(uint64_t {0x7ff8000000000001ULL}, + uint64_t {0x7ff8000000000002ULL}); +} + TEST_F(HybridSetTest, string) { std::unique_ptr set(create_set(PrimitiveType::TYPE_VARCHAR, false)); StringRef a; diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index 8f1f9711ae826a..51d0283b172fce 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -66,6 +67,7 @@ #include "format_v2/file_reader.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/parquet_reader.h" +#include "format_v2/parquet/reader/native/block_split_bloom_filter.h" #include "format_v2/parquet/reader/native_column_reader.h" #include "gen_cpp/PlanNodes_types.h" #include "gen_cpp/Types_types.h" @@ -570,6 +572,47 @@ VExprContextSPtr create_string_in_conjunct(int column_id, const std::vector(); + } + DORIS_CHECK(type == TYPE_DOUBLE); + return std::make_shared(); +} + +VExprContextSPtr create_floating_function_conjunct(int column_id, PrimitiveType type, + const std::string& function_name, + TExprOpcode::type opcode, const Field& value) { + const auto data_type = floating_data_type(type); + auto root = create_binary_predicate( + function_name, opcode, + VSlotRef::create_shared(column_id, column_id, -1, make_nullable(data_type), + "floating_key"), + VLiteral::create_shared(data_type, value)); + return VExprContext::create_shared(std::move(root)); +} + +VExprContextSPtr create_floating_in_conjunct(int column_id, PrimitiveType type, + const std::vector& values, bool is_not_in) { + const auto data_type = floating_data_type(type); + const auto result_type = make_nullable(std::make_shared()); + TExprNode node; + node.__set_node_type(TExprNodeType::IN_PRED); + node.__set_type(result_type->to_thrift()); + node.__set_num_children(static_cast(values.size() + 1)); + node.__set_is_nullable(true); + TInPredicate in_predicate; + in_predicate.__set_is_not_in(is_not_in); + node.__set_in_predicate(in_predicate); + auto root = VInPredicate::create_shared(node); + root->add_child(VSlotRef::create_shared(column_id, column_id, -1, make_nullable(data_type), + "floating_key")); + for (const auto& value : values) { + root->add_child(VLiteral::create_shared(data_type, value)); + } + return VExprContext::create_shared(std::move(root)); +} + VExprContextSPtr create_null_conjunct(int column_id, const DataTypePtr& data_type, bool is_null) { const auto nullable_type = make_nullable(remove_nullable(data_type)); const auto result_type = std::make_shared(); @@ -1173,6 +1216,22 @@ std::shared_ptr build_int64_array(const std::vector& valu return finish_array(&builder); } +std::shared_ptr build_float_array(const std::vector& values) { + arrow::FloatBuilder builder; + for (const auto value : values) { + EXPECT_TRUE(builder.Append(value).ok()); + } + return finish_array(&builder); +} + +std::shared_ptr build_double_array(const std::vector& values) { + arrow::DoubleBuilder builder; + for (const auto value : values) { + EXPECT_TRUE(builder.Append(value).ok()); + } + return finish_array(&builder); +} + std::shared_ptr build_int8_array(const std::vector& values) { arrow::Int8Builder builder; for (const auto value : values) { @@ -1297,6 +1356,111 @@ void write_table(const std::string& file_path, const std::shared_ptr +std::vector build_parquet_bloom_filter(const T* values, size_t count) { + format::parquet::native::BlockSplitBloomFilter bloom_filter; + DORIS_CHECK(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + for (size_t index = 0; index < count; ++index) { + bloom_filter.add_bytes(reinterpret_cast(&values[index]), sizeof(T)); + } + + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader header; + header.__set_numBytes(static_cast(bloom_filter.size())); + header.__set_algorithm(algorithm); + header.__set_hash(hash); + header.__set_compression(compression); + + std::vector bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + DORIS_CHECK(serializer.serialize(&header, &bytes).ok()); + bytes.insert(bytes.end(), bloom_filter.data(), bloom_filter.data() + bloom_filter.size()); + return bytes; +} + +void append_floating_bloom_filters(const std::string& file_path, + const std::vector& float_values, + const std::vector& double_values) { + DORIS_CHECK(float_values.size() == double_values.size()); + std::ifstream input(file_path, std::ios::binary | std::ios::ate); + DORIS_CHECK(input.good()); + const auto input_size = static_cast(input.tellg()); + DORIS_CHECK(input_size >= static_cast(8)); + std::vector file_bytes(cast_set(input_size)); + input.seekg(0); + input.read(reinterpret_cast(file_bytes.data()), cast_set(input_size)); + DORIS_CHECK(input.good()); + DORIS_CHECK(memcmp(file_bytes.data() + file_bytes.size() - 4, "PAR1", 4) == 0); + + const uint32_t footer_size = decode_fixed32_le(file_bytes.data() + file_bytes.size() - 8); + DORIS_CHECK(footer_size <= file_bytes.size() - 8); + const size_t footer_offset = file_bytes.size() - 8 - footer_size; + uint32_t thrift_size = footer_size; + tparquet::FileMetaData metadata; + DORIS_CHECK( + deserialize_thrift_msg(file_bytes.data() + footer_offset, &thrift_size, true, &metadata) + .ok()); + + file_bytes.resize(footer_offset); + size_t first_row = 0; + for (auto& row_group : metadata.row_groups) { + const size_t row_count = cast_set(row_group.num_rows); + DORIS_CHECK(row_group.columns.size() >= 2); + DORIS_CHECK(first_row + row_count <= float_values.size()); + const auto append_column_bloom = [&](size_t column_id, + const std::vector& values) { + auto bytes = build_parquet_bloom_filter(values.data() + first_row, row_count); + auto& column = row_group.columns[column_id].meta_data; + column.__set_bloom_filter_offset(cast_set(file_bytes.size())); + column.__set_bloom_filter_length(cast_set(bytes.size())); + file_bytes.insert(file_bytes.end(), bytes.begin(), bytes.end()); + }; + append_column_bloom(0, float_values); + append_column_bloom(1, double_values); + first_row += row_count; + } + DORIS_CHECK(first_row == float_values.size()); + + std::vector footer; + ThriftSerializer serializer(/*compact=*/true, 1024); + DORIS_CHECK(serializer.serialize(&metadata, &footer).ok()); + file_bytes.insert(file_bytes.end(), footer.begin(), footer.end()); + std::array encoded_footer_size {}; + encode_fixed32_le(encoded_footer_size.data(), cast_set(footer.size())); + file_bytes.insert(file_bytes.end(), encoded_footer_size.begin(), encoded_footer_size.end()); + file_bytes.insert(file_bytes.end(), {'P', 'A', 'R', '1'}); + + std::ofstream output(file_path, std::ios::binary | std::ios::trunc); + output.write(reinterpret_cast(file_bytes.data()), file_bytes.size()); + output.close(); + DORIS_CHECK(output.good()); +} + +void write_floating_parquet_file_with_bloom_filters(const std::string& file_path) { + const std::vector float_values { + -1.0F, -0.0F, 1.0F, 0.5F, std::bit_cast(uint32_t {0x7fc00001U}), + -2.0F, 2.0F, 4.0F}; + const std::vector double_values { + -1.0, -0.0, 1.0, 0.5, std::bit_cast(uint64_t {0x7ff8000000000001ULL}), + -2.0, 2.0, 4.0}; + auto schema = arrow::schema({arrow::field("float_value", arrow::float32(), false), + arrow::field("double_value", arrow::float64(), false), + arrow::field("id", arrow::int32(), false)}); + auto table = arrow::Table::Make( + schema, {build_float_array(float_values), build_double_array(double_values), + build_int32_array({0, 1, 2, 3, 4, 5, 6, 7})}); + write_table(file_path, table, 4); + append_floating_bloom_filters(file_path, float_values, double_values); +} + void write_required_adjusted_time_parquet_file(const std::string& file_path) { auto file_result = arrow::io::FileOutputStream::Open(file_path); ASSERT_TRUE(file_result.ok()) << file_result.status(); @@ -1781,6 +1945,83 @@ class ParquetScanTest : public testing::Test { std::string _file_path; }; +TEST_F(ParquetScanTest, FloatingPredicatesPreserveDorisSemanticsWithRealBloomFilters) { + write_floating_parquet_file_with_bloom_filters(_file_path); + + const auto expect_ids = [&](int column_id, const VExprContextSPtr& conjunct, + const ColumnInt32::Container& expected, + bool expect_bloom_pruning = false) { + RuntimeProfile profile("profile"); + auto reader = create_reader(0, -1, &profile); + TQueryOptions options; + options.__set_enable_parquet_filter_by_bloom_filter(true); + RuntimeState state {options, TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + auto request = std::make_shared(); + use_schema_order_positions(request.get(), schema); + format::FileScanRequestBuilder request_builder(request.get()); + ASSERT_TRUE(request_builder.add_predicate_column(format::LocalColumnId(column_id)).ok()); + ASSERT_TRUE(request_builder.add_non_predicate_column(format::LocalColumnId(2)).ok()); + request->predicate_only_columns.push_back(format::LocalColumnId(column_id)); + ASSERT_TRUE(conjunct->prepare(&state, RowDescriptor()).ok()); + ASSERT_TRUE(conjunct->open(&state).ok()); + request->conjuncts.push_back(conjunct); + ASSERT_TRUE(reader->open(request).ok()); + + ColumnInt32::Container actual; + bool eof = false; + while (!eof) { + Block block = build_file_block(schema); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + const auto& ids = int32_data_column(*block.get_by_position(2).column).get_data(); + actual.insert(actual.end(), ids.begin(), ids.end()); + } + EXPECT_EQ(actual, expected); + if (expect_bloom_pruning) { + EXPECT_EQ(counter_value(profile, "RowGroupsFilteredByBloomFilter"), 1); + } + conjunct->close(); + }; + + const auto check_type = [&](int column_id) { + using CppType = typename PrimitiveTypeTraits::CppType; + const CppType query_nan = [] { + if constexpr (Type == TYPE_FLOAT) { + return std::bit_cast(uint32_t {0x7fc00002U}); + } else { + return std::bit_cast(uint64_t {0x7ff8000000000002ULL}); + } + }(); + const auto nan = Field::create_field(query_nan); + const auto zero = Field::create_field(CppType {0}); + const auto one = Field::create_field(CppType {1}); + const auto absent = Field::create_field(CppType {10}); + + expect_ids(column_id, + create_floating_function_conjunct(column_id, Type, "eq", TExprOpcode::EQ, nan), + {4}); + expect_ids(column_id, create_floating_in_conjunct(column_id, Type, {absent, nan}, false), + {4}); + expect_ids(column_id, create_floating_in_conjunct(column_id, Type, {zero}, true), + {0, 2, 3, 4, 5, 6, 7}); + expect_ids(column_id, + create_floating_function_conjunct(column_id, Type, "eq", TExprOpcode::EQ, zero), + {1}, true); + expect_ids(column_id, + create_floating_function_conjunct(column_id, Type, "gt", TExprOpcode::GT, one), + {4, 6, 7}); + }; + + // The file stores different NaN payloads and -0.0, so these scans exercise both semantic + // equivalence classes through the real V2 footer-statistics and Bloom-filter path. + check_type.template operator()(0); + check_type.template operator()(1); +} + TEST(ParquetScanSelectionTest, CompactFilterShrinksCurrentSelection) { format::parquet::SelectionVector selection(4); selection.set_index(0, 0); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 6b48b05d61cd2f..7f8fbbbcaca863 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -19,6 +19,7 @@ #include +#include #include #include #include @@ -40,6 +41,7 @@ #include "core/data_type/data_type_variant_v2.h" #include "core/field.h" #include "exprs/expr_zonemap_filter.h" +#include "exprs/function/functions_comparison.h" #include "exprs/vcompound_pred.h" #include "exprs/vexpr.h" #include "exprs/vexpr_context.h" @@ -115,6 +117,115 @@ class BloomInExpr final : public VExpr { const std::string _expr_name = "BloomInExpr"; }; +class BloomEqExpr final : public VExpr { +public: + BloomEqExpr(int column_id, DataTypePtr data_type, Field value) + : VExpr(std::make_shared(), false), + _slot(VSlotRef::create_shared(0, column_id, -1, std::move(data_type), "c0")), + _value(std::move(value)) {} + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("BloomEqExpr is only used by parquet statistics tests"); + } + bool can_evaluate_bloom_filter() const override { return true; } + ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx) const override { + return expr_zonemap::eval_eq_bloom_filter( + ctx, expr_zonemap::SlotLiteral {.slot_index = _slot->column_id(), + .slot_type = _slot->data_type(), + .literal = _value, + .literal_type = _slot->data_type(), + .literal_on_left = false}); + } + void collect_slot_column_ids(std::set& column_ids) const override { + _slot->collect_slot_column_ids(column_ids); + } + +private: + std::shared_ptr _slot; + Field _value; + const std::string _expr_name = "BloomEqExpr"; +}; + +class MetadataFloatingEqualityExpr final : public VExpr { +public: + enum class Mode { EQ, IN, NE, GT, GE, REVERSED_LT, REVERSED_LE, NOT_IN }; + + MetadataFloatingEqualityExpr(int column_id, DataTypePtr data_type, Field nan_value, Mode mode) + : VExpr(std::make_shared(), false), + _slot(VSlotRef::create_shared(0, column_id, -1, data_type, "c0")), + _nan_literal(VLiteral::create_shared(create_texpr_node_from( + nan_value, remove_nullable(data_type)->get_primitive_type(), 0, 0))), + _mode(mode), + _values {Field::create_field(10.0), std::move(nan_value)} { + const auto primitive_type = remove_nullable(data_type)->get_primitive_type(); + if (primitive_type == TYPE_FLOAT) { + _values[0] = Field::create_field(10.0F); + _zero_literal = VLiteral::create_shared(create_texpr_node_from( + Field::create_field(0.0F), TYPE_FLOAT, 0, 0)); + _one_literal = VLiteral::create_shared(create_texpr_node_from( + Field::create_field(1.0F), TYPE_FLOAT, 0, 0)); + _not_in_values = {Field::create_field(0.0F)}; + } else { + _zero_literal = VLiteral::create_shared(create_texpr_node_from( + Field::create_field(0.0), TYPE_DOUBLE, 0, 0)); + _one_literal = VLiteral::create_shared(create_texpr_node_from( + Field::create_field(1.0), TYPE_DOUBLE, 0, 0)); + _not_in_values = {Field::create_field(0.0)}; + } + } + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataFloatingEqualityExpr is metadata-only"); + } + bool can_evaluate_zonemap_filter() const override { return true; } + ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx) const override { + switch (_mode) { + case Mode::EQ: + return comparison_zonemap_detail::evaluate(ctx, {_slot, _nan_literal}, + comparison_zonemap_detail::Op::EQ); + case Mode::IN: + return expr_zonemap::eval_in_zonemap(ctx, _slot, false, _values, true, _values[0], + _values[1]); + case Mode::NE: + return comparison_zonemap_detail::evaluate(ctx, {_slot, _zero_literal}, + comparison_zonemap_detail::Op::NE); + case Mode::GT: + return comparison_zonemap_detail::evaluate(ctx, {_slot, _one_literal}, + comparison_zonemap_detail::Op::GT); + case Mode::GE: + return comparison_zonemap_detail::evaluate(ctx, {_slot, _one_literal}, + comparison_zonemap_detail::Op::GE); + case Mode::REVERSED_LT: + return comparison_zonemap_detail::evaluate(ctx, {_one_literal, _slot}, + comparison_zonemap_detail::Op::LT); + case Mode::REVERSED_LE: + return comparison_zonemap_detail::evaluate(ctx, {_one_literal, _slot}, + comparison_zonemap_detail::Op::LE); + case Mode::NOT_IN: + return expr_zonemap::eval_in_zonemap(ctx, _slot, true, _not_in_values, false, + _not_in_values[0], _not_in_values[0]); + } + __builtin_unreachable(); + } + void collect_slot_column_ids(std::set& column_ids) const override { + _slot->collect_slot_column_ids(column_ids); + } + +private: + VExprSPtr _slot; + VExprSPtr _nan_literal; + VExprSPtr _zero_literal; + VExprSPtr _one_literal; + Mode _mode; + std::vector _values; + std::vector _not_in_values; + const std::string _expr_name = "MetadataFloatingEqualityExpr"; +}; + class DictionaryStringInExpr final : public VExpr { public: DictionaryStringInExpr() : VExpr(std::make_shared(), false) {} @@ -340,6 +451,11 @@ VExprContextSPtrs bloom_conjuncts(DataTypePtr data_type, std::vector valu std::make_shared(0, std::move(data_type), std::move(values)))}; } +VExprContextSPtrs bloom_eq_conjunct(DataTypePtr data_type, Field value) { + return {VExprContext::create_shared( + std::make_shared(0, std::move(data_type), std::move(value)))}; +} + format::FileScanRequest request_with_bloom_conjunct(DataTypePtr data_type, std::vector values) { format::FileScanRequest request; @@ -376,6 +492,96 @@ TEST(NativeParquetStatisticsTest, InvalidNullableDateBoundsDisableMinMax) { EXPECT_FALSE(result.has_min_max); } +TEST(NativeParquetStatisticsTest, FloatingNanEqualityKeepsFiniteOnlyFooterAndPageRanges) { + const auto check_type = []( + tparquet::Type::type physical_type, UInt nan_bits) { + using T = typename PrimitiveTypeTraits::CppType; + auto column_schema = std::make_unique(); + column_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column_schema->local_id = 0; + column_schema->leaf_column_id = 0; + column_schema->type = std::make_shared(); + column_schema->type_descriptor.doris_type = column_schema->type; + column_schema->type_descriptor.physical_type = physical_type; + std::vector> schema; + schema.push_back(std::move(column_schema)); + + const T finite_bound = T {0}; + const std::string encoded_bound(reinterpret_cast(&finite_bound), sizeof(T)); + tparquet::Statistics statistics; + statistics.__set_min_value(encoded_bound); + statistics.__set_max_value(encoded_bound); + statistics.__set_null_count(0); + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(physical_type); + column_metadata.__set_num_values(2); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_statistics(statistics); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + tparquet::RowGroup row_group; + row_group.__set_columns({chunk}); + row_group.__set_num_rows(2); + tparquet::ColumnOrder order; + order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder()); + tparquet::FileMetaData metadata; + metadata.__set_column_orders({order}); + metadata.__set_row_groups({row_group}); + + format::parquet::NativeParquetPageIndex page_index; + page_index.column_index.__set_min_values({encoded_bound}); + page_index.column_index.__set_max_values({encoded_bound}); + page_index.column_index.__set_null_pages({false}); + page_index.column_index.__set_null_counts({0}); + tparquet::PageLocation location; + location.__set_offset(0); + location.__set_compressed_page_size(10); + location.__set_first_row_index(0); + page_index.offset_index.__set_page_locations({location}); + std::unordered_map page_indexes; + page_indexes.emplace(0, std::move(page_index)); + + const auto nan_field = Field::create_field(std::bit_cast(nan_bits)); + for (const auto mode : + {MetadataFloatingEqualityExpr::Mode::EQ, MetadataFloatingEqualityExpr::Mode::IN, + MetadataFloatingEqualityExpr::Mode::NE, MetadataFloatingEqualityExpr::Mode::GT, + MetadataFloatingEqualityExpr::Mode::GE, + MetadataFloatingEqualityExpr::Mode::REVERSED_LT, + MetadataFloatingEqualityExpr::Mode::REVERSED_LE, + MetadataFloatingEqualityExpr::Mode::NOT_IN}) { + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.predicate_columns = { + format::LocalColumnIndex::top_level(format::LocalColumnId(0))}; + request.conjuncts = { + VExprContext::create_shared(std::make_shared( + 0, schema[0]->type, nan_field, mode))}; + + std::vector selected_row_groups; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, false, + nullptr) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + + std::vector selected_ranges; + std::map skip_plans; + ASSERT_TRUE(format::parquet::select_row_group_ranges_by_native_page_index( + metadata, metadata.row_groups[0], page_indexes, schema, request, 2, + &selected_ranges, &skip_plans, nullptr) + .ok()); + ASSERT_EQ(1, selected_ranges.size()); + EXPECT_EQ(0, selected_ranges[0].start); + EXPECT_EQ(2, selected_ranges[0].length); + } + }; + + check_type.template operator()(tparquet::Type::FLOAT, + uint32_t {0x7fc00002U}); + check_type.template operator()(tparquet::Type::DOUBLE, + uint64_t {0x7ff8000000000002ULL}); +} + TEST(NativeParquetStatisticsTest, InvalidNullableDecimalBoundsDisableMinMax) { format::parquet::ParquetColumnSchema column_schema; column_schema.type = make_nullable(std::make_shared(2, 0)); @@ -661,6 +867,130 @@ TEST(ParquetBloomFilterPruningTest, NativeUint32BloomUsesPhysicalInt32Hash) { bloom_filter)); } +TEST(ParquetBloomFilterPruningTest, NativeFloatingBloomPreservesDorisEquality) { + const auto check_type = []( + tparquet::Type::type physical_type, + typename PrimitiveTypeTraits::CppType stored_value, + typename PrimitiveTypeTraits::CppType predicate_value) { + format::parquet::ParquetColumnSchema column_schema; + column_schema.type = std::make_shared(); + column_schema.type_descriptor.doris_type = column_schema.type; + column_schema.type_descriptor.physical_type = physical_type; + + format::parquet::native::BlockSplitBloomFilter bloom_filter; + ASSERT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + bloom_filter.add_bytes(reinterpret_cast(&stored_value), sizeof(stored_value)); + ASSERT_FALSE(bloom_filter.test_bytes(reinterpret_cast(&predicate_value), + sizeof(predicate_value))); + const auto field = Field::create_field(predicate_value); + + EXPECT_FALSE(format::parquet::ParquetStatisticsUtils::NativeBloomFilterExcludes( + column_schema, 0, bloom_eq_conjunct(column_schema.type, field), bloom_filter)); + EXPECT_FALSE(format::parquet::ParquetStatisticsUtils::NativeBloomFilterExcludes( + column_schema, 0, bloom_conjuncts(column_schema.type, {field}), bloom_filter)); + }; + + check_type.template operator()(tparquet::Type::FLOAT, -0.0F, 0.0F); + check_type.template operator()(tparquet::Type::FLOAT, 0.0F, -0.0F); + check_type.template operator()(tparquet::Type::DOUBLE, -0.0, 0.0); + check_type.template operator()(tparquet::Type::DOUBLE, 0.0, -0.0); + check_type.template operator()( + tparquet::Type::FLOAT, std::bit_cast(uint32_t {0x7fc00001U}), + std::bit_cast(uint32_t {0x7fc00002U})); + check_type.template operator()( + tparquet::Type::DOUBLE, std::bit_cast(uint64_t {0x7ff8000000000001ULL}), + std::bit_cast(uint64_t {0x7ff8000000000002ULL})); +} + +TEST(ParquetBloomFilterPruningTest, NativeRowGroupKeepsDorisEqualFloatingValues) { + const auto check_type = []( + tparquet::Type::type physical_type, + typename PrimitiveTypeTraits::CppType stored_value, + typename PrimitiveTypeTraits::CppType predicate_value) { + format::parquet::native::BlockSplitBloomFilter bloom_filter; + ASSERT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + bloom_filter.add_bytes(reinterpret_cast(&stored_value), sizeof(stored_value)); + + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader bloom_header; + bloom_header.__set_numBytes(static_cast(bloom_filter.size())); + bloom_header.__set_algorithm(algorithm); + bloom_header.__set_hash(hash); + bloom_header.__set_compression(compression); + std::vector bloom_bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + ASSERT_TRUE(serializer.serialize(&bloom_header, &bloom_bytes).ok()); + bloom_bytes.insert(bloom_bytes.end(), bloom_filter.data(), + bloom_filter.data() + bloom_filter.size()); + + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(physical_type); + column_metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + column_metadata.__set_num_values(1); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_data_page_offset(0); + column_metadata.__set_bloom_filter_offset(0); + column_metadata.__set_bloom_filter_length(static_cast(bloom_bytes.size())); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + tparquet::RowGroup row_group; + row_group.__set_columns({chunk}); + row_group.__set_total_byte_size(0); + row_group.__set_num_rows(1); + tparquet::FileMetaData metadata; + metadata.__set_version(1); + metadata.__set_num_rows(1); + metadata.__set_row_groups({row_group}); + + const auto field = Field::create_field(predicate_value); + for (const bool use_eq : {true, false}) { + auto column_schema = std::make_unique(); + column_schema->type = std::make_shared(); + column_schema->type_descriptor.doris_type = column_schema->type; + column_schema->type_descriptor.physical_type = physical_type; + column_schema->local_id = 0; + column_schema->leaf_column_id = 0; + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.conjuncts = use_eq ? bloom_eq_conjunct(column_schema->type, field) + : bloom_conjuncts(column_schema->type, {field}); + std::vector> schema; + schema.push_back(std::move(column_schema)); + format::parquet::ParquetFileContext file_context; + file_context.native_file = std::make_shared(bloom_bytes); + std::vector selected_row_groups; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, true, + &pruning_stats, nullptr, nullptr, &file_context) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + EXPECT_EQ(pruning_stats.filtered_row_groups_by_bloom_filter, 0); + } + }; + + check_type.template operator()(tparquet::Type::FLOAT, -0.0F, 0.0F); + check_type.template operator()(tparquet::Type::DOUBLE, 0.0, -0.0); + check_type.template operator()( + tparquet::Type::FLOAT, std::bit_cast(uint32_t {0x7fc00001U}), + std::bit_cast(uint32_t {0x7fc00002U})); + check_type.template operator()( + tparquet::Type::DOUBLE, std::bit_cast(uint64_t {0x7ff8000000000001ULL}), + std::bit_cast(uint64_t {0x7ff8000000000002ULL})); +} + TEST(ParquetBloomFilterPruningTest, NativeRowGroupKeepsPresentUint32AboveInt32Max) { auto column_schema = std::make_unique(uint32_parquet_bloom_schema()); From c9e0ad640723c75df5c6e74d364f55bceec29cd9 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Fri, 7 Aug 2026 17:37:51 +0800 Subject: [PATCH 20/20] [fix](be) Safely prune nested Parquet columns with Bloom filters (#66471) ## Summary - backport nested Parquet Bloom-filter pruning support to branch-4.1 - resolve struct and list leaf predicates for equality, null-safe equality, and IN probes - retain table-level evaluation when schema mapping, nullability, or expression localization makes early filtering unsafe - preserve filter order and merge deferred complex projections so rejected localization cannot bypass validation or drop residual-filter children - keep nested Variant leaf predicates eager while validating nullability only at mapped table-schema levels - add pruning diagnostics and focused coverage for Parquet, ORC, column mapping, and TableReader paths ## Testing - clang-format 16 check on all affected C/C++ files - 162 focused BE unit tests across ExprZonemapFilterTest, ColumnMapperScanRequestTest, ParquetBloomFilterPruningTest, and TableReaderTest - 38 focused mapper tests, including ColumnMapperTest.NestedVariantAllAccessPathKeepsPhysicalTypedLeaf from BE UT build 1016464 --- be/src/exprs/expr_zonemap_filter.cpp | 161 +++++- be/src/exprs/expr_zonemap_filter.h | 30 ++ .../function/comparison_equal_for_null.cpp | 16 +- be/src/exprs/function/functions_comparison.h | 5 +- be/src/exprs/vectorized_fn_call.cpp | 19 +- be/src/exprs/vin_predicate.cpp | 12 +- be/src/format_v2/column_mapper.cpp | 206 +++++++- be/src/format_v2/column_mapper_nested.cpp | 76 ++- be/src/format_v2/column_mapper_nested.h | 4 + be/src/format_v2/file_reader.cpp | 2 + be/src/format_v2/file_reader.h | 3 + be/src/format_v2/orc/orc_reader.cpp | 9 +- be/src/format_v2/parquet/parquet_profile.cpp | 18 + be/src/format_v2/parquet/parquet_profile.h | 4 + .../format_v2/parquet/parquet_statistics.cpp | 162 +++++- be/src/format_v2/parquet/parquet_statistics.h | 4 + be/src/format_v2/table_reader.h | 3 + be/test/exprs/expr_zonemap_filter_test.cpp | 181 +++++++ be/test/format_v2/column_mapper_test.cpp | 146 +++++- be/test/format_v2/orc/orc_reader_test.cpp | 30 ++ .../format_v2/parquet/parquet_scan_test.cpp | 42 ++ .../parquet/parquet_statistics_test.cpp | 482 +++++++++++++++++- be/test/format_v2/table_reader_test.cpp | 343 ++++++++++++- 23 files changed, 1868 insertions(+), 90 deletions(-) diff --git a/be/src/exprs/expr_zonemap_filter.cpp b/be/src/exprs/expr_zonemap_filter.cpp index ff60af4befac4c..7d0c3a3d34e744 100644 --- a/be/src/exprs/expr_zonemap_filter.cpp +++ b/be/src/exprs/expr_zonemap_filter.cpp @@ -19,6 +19,7 @@ #include #include +#include #include #include #include @@ -51,6 +52,33 @@ std::optional> field_from_literal_expr(const VExpr return std::make_pair(std::move(field), literal->get_data_type()); } +std::optional struct_field_ordinal(const Field& field) { + int64_t ordinal = -1; + switch (field.get_type()) { + case TYPE_BOOLEAN: + ordinal = field.get(); + break; + case TYPE_TINYINT: + ordinal = field.get(); + break; + case TYPE_SMALLINT: + ordinal = field.get(); + break; + case TYPE_INT: + ordinal = field.get(); + break; + case TYPE_BIGINT: + ordinal = field.get(); + break; + default: + return std::nullopt; + } + if (ordinal <= 0 || ordinal > std::numeric_limits::max()) { + return std::nullopt; + } + return static_cast(ordinal - 1); +} + bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) { return value >= min_value && value <= max_value; } @@ -80,6 +108,11 @@ bool floating_point_bloom_filter_may_contain(const segment_v2::BloomFilter& bloo return value == T {0} && test_value(-value); } +bool bloom_filter_probes_equal(const BloomFilterProbe& lhs, const BloomFilterProbe& rhs) { + return lhs.slot_index == rhs.slot_index && lhs.path == rhs.path && + data_types_compatible(lhs.value_type, rhs.value_type); +} + bool bloom_filter_may_contain(const BloomFilterEvalContext::SlotBloomFilter& slot_filter, const Field& value) { DORIS_CHECK(slot_filter.data_type != nullptr); @@ -233,6 +266,126 @@ std::optional extract_slot_and_literal(const VExprSPtrs& args) { return std::nullopt; } +std::optional extract_bloom_filter_probe(const VExprSPtr& expr) { + if (expr == nullptr || expr->data_type() == nullptr) { + return std::nullopt; + } + if (auto slot = std::dynamic_pointer_cast(expr); slot) { + return BloomFilterProbe { + .slot_index = slot->column_id(), .value_type = slot->data_type(), .path = {}}; + } + if ((expr->fn().name.function_name != "element_at" && + expr->fn().name.function_name != "struct_element") || + expr->get_num_children() != 2) { + return std::nullopt; + } + + auto probe = extract_bloom_filter_probe(expr->get_child(0)); + auto selector = field_from_literal_expr(expr->get_child(1)); + if (!probe.has_value() || !selector.has_value() || selector->first.is_null()) { + return std::nullopt; + } + const auto parent_type = remove_nullable(expr->get_child(0)->data_type()); + if (parent_type == nullptr) { + return std::nullopt; + } + + BloomFilterPathElement path_element; + switch (parent_type->get_primitive_type()) { + case TYPE_STRUCT: { + path_element.kind = BloomFilterPathKind::STRUCT_FIELD; + const auto selector_type = remove_nullable(selector->second); + if (selector_type == nullptr) { + return std::nullopt; + } + if (is_string_type(selector_type->get_primitive_type())) { + path_element.field_name = selector->first.get(); + } else { + auto ordinal = struct_field_ordinal(selector->first); + if (!ordinal.has_value()) { + return std::nullopt; + } + path_element.field_ordinal = *ordinal; + } + break; + } + case TYPE_ARRAY: + // Array element positions share one repeated Parquet leaf; membership in that leaf is a + // necessary condition for any element_at(array, constant) equality to match. + path_element.kind = BloomFilterPathKind::LIST_ELEMENT; + break; + default: + return std::nullopt; + } + probe->value_type = expr->data_type(); + probe->path.push_back(std::move(path_element)); + return probe; +} + +bool collect_unique_bloom_filter_probe(const VExprSPtr& expr, + std::optional* result) { + DORIS_CHECK(result != nullptr); + if (auto probe = extract_bloom_filter_probe(expr); probe.has_value()) { + if (result->has_value() && !bloom_filter_probes_equal(**result, *probe)) { + return false; + } + *result = std::move(probe); + return true; + } + if (expr == nullptr) { + return true; + } + for (uint16_t child_idx = 0; child_idx < expr->get_num_children(); ++child_idx) { + const auto& child = expr->get_child(child_idx); + if (child == nullptr || child->is_literal()) { + continue; + } + // Every Bloom-capable branch must bind to the same leaf; a conflicting subtree cannot be + // treated like a branch without a probe because the compound evaluator would use it. + if (!collect_unique_bloom_filter_probe(child, result)) { + return false; + } + } + return true; +} + +std::optional extract_bloom_filter_predicate_probe(const VExprSPtr& expr) { + std::optional result; + if (!collect_unique_bloom_filter_probe(expr, &result)) { + return std::nullopt; + } + return result; +} + +std::optional extract_bloom_filter_slot_and_literal(const VExprSPtrs& args) { + if (args.size() != 2) { + return std::nullopt; + } + for (size_t probe_idx = 0; probe_idx < args.size(); ++probe_idx) { + auto probe = extract_bloom_filter_probe(args[probe_idx]); + auto literal = field_from_literal_expr(args[1 - probe_idx]); + if (!probe.has_value() || !literal.has_value()) { + continue; + } + auto [literal_value, literal_type] = std::move(*literal); + return SlotLiteral {.slot_index = probe->slot_index, + .slot_type = probe->value_type, + .literal = std::move(literal_value), + .literal_type = std::move(literal_type), + .literal_on_left = probe_idx == 1}; + } + return std::nullopt; +} + +bool can_evaluate_bloom_filter_equality(const VExprSPtrs& args) { + auto slot_literal = extract_bloom_filter_slot_and_literal(args); + // Parquet Bloom hashes physical bytes, so it cannot disprove Doris NaN equality across + // different NaN payloads even when the probe targets a nested leaf. + return slot_literal.has_value() && !slot_literal->literal.is_null() && + !slot_literal->literal.is_nan() && + data_types_compatible(slot_literal->slot_type, slot_literal->literal_type); +} + bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map, const DataTypePtr& data_type) { if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf || @@ -401,14 +554,14 @@ ZoneMapFilterResult eval_in_bloom_filter(const BloomFilterEvalContext& ctx, if (is_not_in) { return ZoneMapFilterResult::kUnsupported; } - auto slot = std::dynamic_pointer_cast(slot_expr); - DORIS_CHECK(slot != nullptr); - auto slot_filter = ctx.slot(slot->column_id()); + auto probe = extract_bloom_filter_probe(slot_expr); + DORIS_CHECK(probe.has_value()); + auto slot_filter = ctx.slot(probe->slot_index); if (slot_filter == nullptr || slot_filter->data_type == nullptr || slot_filter->bloom_filter == nullptr) { return ZoneMapFilterResult::kUnsupported; } - DORIS_CHECK(data_types_compatible(slot_filter->data_type, slot->data_type())); + DORIS_CHECK(data_types_compatible(slot_filter->data_type, probe->value_type)); if (values.empty()) { return ZoneMapFilterResult::kNoMatch; } diff --git a/be/src/exprs/expr_zonemap_filter.h b/be/src/exprs/expr_zonemap_filter.h index 10e6c7dbfbad77..b32b513648352b 100644 --- a/be/src/exprs/expr_zonemap_filter.h +++ b/be/src/exprs/expr_zonemap_filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include #include "common/check.h" @@ -100,8 +101,37 @@ struct SlotLiteral { bool literal_on_left; }; +enum class BloomFilterPathKind { + STRUCT_FIELD, + LIST_ELEMENT, +}; + +struct BloomFilterPathElement { + BloomFilterPathKind kind; + std::string field_name; + int32_t field_ordinal = -1; + + bool operator==(const BloomFilterPathElement&) const = default; +}; + +struct BloomFilterProbe { + int slot_index; + DataTypePtr value_type; + std::vector path; + + bool operator==(const BloomFilterProbe&) const = default; +}; + std::optional extract_slot_and_literal(const VExprSPtrs& args); +std::optional extract_bloom_filter_probe(const VExprSPtr& expr); + +std::optional extract_bloom_filter_predicate_probe(const VExprSPtr& expr); + +std::optional extract_bloom_filter_slot_and_literal(const VExprSPtrs& args); + +bool can_evaluate_bloom_filter_equality(const VExprSPtrs& args); + TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type, int precision, int scale); diff --git a/be/src/exprs/function/comparison_equal_for_null.cpp b/be/src/exprs/function/comparison_equal_for_null.cpp index 78cd342f3c3ba1..32ffc7b1d88925 100644 --- a/be/src/exprs/function/comparison_equal_for_null.cpp +++ b/be/src/exprs/function/comparison_equal_for_null.cpp @@ -37,6 +37,7 @@ #include "core/data_type/data_type_number.h" #include "core/types.h" #include "exprs/aggregate/aggregate_function.h" +#include "exprs/expr_zonemap_filter.h" #include "exprs/function/function.h" #include "exprs/function/function_helpers.h" #include "exprs/function/simple_function_factory.h" @@ -64,6 +65,19 @@ class FunctionEqForNull : public IFunction { bool use_default_implementation_for_nulls() const override { return false; } + ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx, + const VExprSPtrs& arguments) const override { + auto slot_literal = expr_zonemap::extract_bloom_filter_slot_and_literal(arguments); + DORIS_CHECK(slot_literal.has_value()); + return expr_zonemap::eval_eq_bloom_filter(ctx, *slot_literal); + } + + bool can_evaluate_bloom_filter(const VExprSPtrs& arguments) const override { + // Parquet Bloom filters do not encode null membership, so null-safe equality can only use + // them when its literal is non-null and ordinary equality semantics apply. + return expr_zonemap::can_evaluate_bloom_filter_equality(arguments); + } + Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments, uint32_t result, size_t input_rows_count) const override { ColumnWithTypeAndName& col_left = block.get_by_position(arguments[0]); @@ -278,4 +292,4 @@ class FunctionEqForNull : public IFunction { void register_function_comparison_eq_for_null(SimpleFunctionFactory& factory) { factory.register_function(); } -} // namespace doris \ No newline at end of file +} // namespace doris diff --git a/be/src/exprs/function/functions_comparison.h b/be/src/exprs/function/functions_comparison.h index 10fdc89f86c71f..9876ecc9eac26d 100644 --- a/be/src/exprs/function/functions_comparison.h +++ b/be/src/exprs/function/functions_comparison.h @@ -431,7 +431,7 @@ inline ZoneMapFilterResult evaluate_dictionary(const DictionaryEvalContext& ctx, inline ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx, const VExprSPtrs& arguments, Op op) { DORIS_CHECK(op == Op::EQ); - auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments); + auto slot_literal = expr_zonemap::extract_bloom_filter_slot_and_literal(arguments); DORIS_CHECK(slot_literal.has_value()); return expr_zonemap::eval_eq_bloom_filter(ctx, *slot_literal); } @@ -673,7 +673,8 @@ class FunctionComparison : public IFunction { bool can_evaluate_bloom_filter(const VExprSPtrs& arguments) const override { auto op = comparison_zonemap_detail::op_from_name(name); - return op.has_value() && comparison_zonemap_detail::can_evaluate_equality(arguments, *op); + return op == comparison_zonemap_detail::Op::EQ && + expr_zonemap::can_evaluate_bloom_filter_equality(arguments); } /// Get result types by argument types. If the function does not apply to these arguments, throw an exception. diff --git a/be/src/exprs/vectorized_fn_call.cpp b/be/src/exprs/vectorized_fn_call.cpp index 6e9f25be48d864..ecbeeedaddbe07 100644 --- a/be/src/exprs/vectorized_fn_call.cpp +++ b/be/src/exprs/vectorized_fn_call.cpp @@ -671,11 +671,24 @@ bool VectorizedFnCall::is_deterministic() const { } bool VectorizedFnCall::is_safe_to_execute_on_selected_rows() const { - static const std::set TOTAL_PREDICATE_FUNCTIONS = { - "eq", "ne", "lt", "le", "gt", "ge", "in", "not_in", "is_null_pred", "is_not_null_pred"}; + static const std::set TOTAL_PREDICATE_FUNCTIONS = {"eq", + "eq_for_null", + "ne", + "lt", + "le", + "gt", + "ge", + "in", + "not_in", + "is_null_pred", + "is_not_null_pred", + "element_at", + "struct_element"}; // Selected-row execution may hide data-dependent errors in rows rejected by an earlier // predicate. Keep function calls unsafe by default and opt in only operations that are total - // for their input domain; child checks then reject expressions such as gt(mod(x, -1), 0). + // for their input domain. Accessors return NULL for absent elements, so admitting them keeps + // nested metadata predicates reachable without crossing an error-producing child such as + // gt(mod(x, -1), 0). return TOTAL_PREDICATE_FUNCTIONS.contains(_function_name) && VExpr::is_safe_to_execute_on_selected_rows(); } diff --git a/be/src/exprs/vin_predicate.cpp b/be/src/exprs/vin_predicate.cpp index 3fe26cff2d207a..73d7016c998a53 100644 --- a/be/src/exprs/vin_predicate.cpp +++ b/be/src/exprs/vin_predicate.cpp @@ -160,11 +160,17 @@ Status VInPredicate::_materialize_for_zonemap_filter(VExprContext* context) { _seg_filter_contains_nan = false; _zonemap_materialized = false; _direct_filter_set.reset(); - if (_children.size() < 2 || !_children[0]->is_slot_ref()) { + if (_children.size() < 2) { return Status::OK(); } - const auto data_type = remove_nullable(_children[0]->data_type()); + auto bloom_probe = expr_zonemap::extract_bloom_filter_probe(_children[0]); + if (!bloom_probe.has_value()) { + return Status::OK(); + } + // Materialization is shared by all pruning paths. Their capability checks keep ZoneMap, + // dictionary, and raw evaluation direct-slot-only while Bloom may consume a nested leaf. + const auto data_type = remove_nullable(bloom_probe->value_type); DORIS_CHECK(data_type != nullptr); if (is_complex_type(data_type->get_primitive_type())) { return Status::OK(); @@ -222,7 +228,7 @@ ZoneMapFilterResult VInPredicate::evaluate_bloom_filter(const BloomFilterEvalCon bool VInPredicate::can_evaluate_bloom_filter() const { // A NaN member forces conservative retention regardless of the remaining finite probes. return _zonemap_materialized && !_is_not_in && !_seg_filter_contains_nan && - std::dynamic_pointer_cast(get_child(0)) != nullptr; + expr_zonemap::extract_bloom_filter_probe(get_child(0)).has_value(); } bool VInPredicate::can_execute_on_raw_fixed_values(const DataTypePtr& data_type, diff --git a/be/src/format_v2/column_mapper.cpp b/be/src/format_v2/column_mapper.cpp index 1952c6839872ea..d864c477c1cc7f 100644 --- a/be/src/format_v2/column_mapper.cpp +++ b/be/src/format_v2/column_mapper.cpp @@ -317,6 +317,8 @@ struct FileSlotRewriteInfo { DataTypePtr file_type; DataTypePtr table_type; std::string file_column_name; + const ColumnMapping* root_mapping = nullptr; + LocalColumnIndex scan_projection; }; struct RewriteContext { @@ -524,6 +526,17 @@ static bool table_filter_has_only_local_entries( return true; } +static bool table_filter_has_only_constant_entries( + const TableFilter& table_filter, const std::map& filter_entries) { + for (const auto global_index : table_filter.global_indices) { + const auto entry_it = filter_entries.find(global_index); + if (entry_it == filter_entries.end() || !entry_it->second.is_constant()) { + return false; + } + } + return !table_filter.global_indices.empty(); +} + static VExprSPtr unwrap_literal_for_file_cast(const VExprSPtr& expr, const DataTypePtr& table_type) { if (expr == nullptr) { @@ -843,11 +856,21 @@ static bool needs_complex_file_slot_cast(const DataTypePtr& file_type, static bool collect_struct_element_chain(const VExprSPtr& expr, std::vector* chain) { DORIS_CHECK(chain != nullptr); - if (!is_struct_element_expr(expr)) { + const auto is_supported_element = [](const VExprSPtr& candidate) { + if (is_struct_element_expr(candidate)) { + return true; + } + return candidate != nullptr && candidate->get_num_children() == 2 && + candidate->fn().name.function_name == "element_at" && + candidate->children()[0]->data_type() != nullptr && + remove_nullable(candidate->children()[0]->data_type())->get_primitive_type() == + TYPE_ARRAY; + }; + if (!is_supported_element(expr)) { return false; } const auto& parent = expr->children()[0]; - if (is_struct_element_expr(parent)) { + if (is_supported_element(parent)) { if (!collect_struct_element_chain(parent, chain)) { return false; } @@ -880,6 +903,64 @@ static bool can_filter_before_table_nullability_alignment(const DataTypePtr& fil return !file_type->is_nullable() || table_type->is_nullable(); } +static const ColumnMapping* find_projected_child_mapping(const ColumnMapping& mapping, + int32_t file_local_id) { + const auto child_it = std::ranges::find_if( + mapping.child_mappings, [file_local_id](const ColumnMapping& child) { + return child.file_local_id.has_value() && *child.file_local_id == file_local_id; + }); + return child_it == mapping.child_mappings.end() ? nullptr : &*child_it; +} + +static bool projected_mapping_allows_file_filtering(const ColumnMapping& mapping, + const LocalColumnIndex* projection) { + if (!can_filter_before_table_nullability_alignment(mapping.file_type, mapping.table_type)) { + return false; + } + const auto file_type = remove_nullable(mapping.file_type); + const auto table_type = remove_nullable(mapping.table_type); + if (table_type->get_primitive_type() == TYPE_VARIANT) { + // Shredded Variant children describe physical encoding, not table-schema nullability + // contracts. The Variant root is therefore the only mapped level that can be validated. + return true; + } + + const auto file_primitive_type = file_type->get_primitive_type(); + const auto table_primitive_type = table_type->get_primitive_type(); + if (is_complex_type(file_primitive_type) || is_complex_type(table_primitive_type)) { + if (file_primitive_type != table_primitive_type) { + return false; + } + if (mapping.child_mappings.empty()) { + return file_type->equals(*table_type); + } + } else if (!file_type->equals(*table_type) && + !is_lossless_file_to_table_numeric_cast(mapping.file_type, mapping.table_type)) { + // A file-local filter can discard a row before TableReader casts a projected sibling. + // Require every projected scalar cast to preserve all source values so filtering cannot + // hide overflow or other materialization errors in that sibling. + return false; + } + + if (is_full_projection(projection)) { + for (const auto& child : mapping.child_mappings) { + if (child.file_local_id.has_value() && + !projected_mapping_allows_file_filtering(child, nullptr)) { + return false; + } + } + return true; + } + for (const auto& child_projection : projection->children) { + const auto* child = find_projected_child_mapping(mapping, child_projection.local_id()); + if (child == nullptr || + !projected_mapping_allows_file_filtering(*child, &child_projection)) { + return false; + } + } + return true; +} + static bool rewrite_struct_element_path_to_file_expr( const VExprSPtr& expr, const std::vector& mappings, const std::map& global_to_file_slot, @@ -892,7 +973,9 @@ static bool rewrite_struct_element_path_to_file_expr( std::vector struct_element_chain; if (!collect_struct_element_chain(expr, &struct_element_chain) || struct_element_chain.size() != resolved.file_child_names.size() || - struct_element_chain.size() != resolved.file_child_types.size()) { + struct_element_chain.size() != resolved.file_child_types.size() || + struct_element_chain.size() != resolved.table_child_types.size() || + struct_element_chain.size() != resolved.file_array_elements.size()) { return false; } @@ -906,18 +989,21 @@ static bool rewrite_struct_element_path_to_file_expr( return false; } - // Check every value-producing level, including the root struct. A nullable parent also makes - // a child access nullable even when the child type itself is required, so checking only the - // final leaf is insufficient. If any file level is more nullable than its table counterpart, - // keep the complete predicate above TableReader so schema validation observes all NULLs before - // row filtering. - if (!can_filter_before_table_nullability_alignment(rewrite_it->second.file_type, - rewrite_it->second.table_type)) { + DORIS_CHECK(rewrite_it->second.root_mapping != nullptr); + // File-local filtering cannot discard rows before every physically projected mapped child has + // reached TableReader's schema validation and casts. ARRAY access uses a full element + // projection on this branch, so validating only the selected STRUCT chain can miss an invalid + // required or narrowing sibling. + if (!projected_mapping_allows_file_filtering(*rewrite_it->second.root_mapping, + &rewrite_it->second.scan_projection)) { return false; } for (size_t idx = 0; idx < struct_element_chain.size(); ++idx) { - if (!can_filter_before_table_nullability_alignment( - resolved.file_child_types[idx], struct_element_chain[idx]->data_type())) { + // Accessor results become nullable for missing ARRAY indices and NULL parents. Compare the + // file child with the declared table child instead, or that execution-only wrapper can + // hide a nullable-file-to-required-table contract violation before alignment reports it. + if (!can_filter_before_table_nullability_alignment(resolved.file_child_types[idx], + resolved.table_child_types[idx])) { return false; } } @@ -934,8 +1020,10 @@ static bool rewrite_struct_element_path_to_file_expr( struct_element_chain.front()->set_children(std::move(root_children)); for (size_t idx = 0; idx < struct_element_chain.size(); ++idx) { auto children = struct_element_chain[idx]->children(); - children[1] = create_file_struct_child_name_literal(resolved.file_child_names[idx], - rewrite_context); + if (!resolved.file_array_elements[idx]) { + children[1] = create_file_struct_child_name_literal(resolved.file_child_names[idx], + rewrite_context); + } struct_element_chain[idx]->set_children(std::move(children)); // The selector name and the expression return type must be moved to file schema together. // Example: @@ -1008,6 +1096,8 @@ static bool rewrite_binary_struct_literal_predicate( .file_type = file_leaf_type, .table_type = table_leaf_type, .file_column_name = {}, + .root_mapping = nullptr, + .scan_projection = {}, }; auto file_literal = rewrite_literal_to_file_type(table_literal, leaf_rewrite_info, rewrite_context); @@ -1066,6 +1156,8 @@ static bool rewrite_in_struct_literal_predicate( .file_type = file_leaf_type, .table_type = table_leaf_type, .file_column_name = {}, + .root_mapping = nullptr, + .scan_projection = {}, }; VExprSPtrs file_literals; file_literals.reserve(table_literals.size()); @@ -2007,7 +2099,9 @@ static void rebuild_projection(ColumnMapping* mapping, LocalIndex block_position // file-reader expressions; constant and unset targets stay above the file reader. static std::map build_file_slot_rewrite_map( const std::vector& mappings, - const std::map& filter_entries) { + const std::vector& output_mappings, + const std::map& filter_entries, + const FileScanRequest& file_request) { std::map global_to_file_slot; for (const auto& mapping : mappings) { const auto entry_it = filter_entries.find(mapping.global_index); @@ -2015,12 +2109,28 @@ static std::map build_file_slot_rewrite_map( continue; } DORIS_CHECK(mapping.file_local_id.has_value()); + const auto file_column_id = LocalColumnId(*mapping.file_local_id); + const auto* scan_projection = + find_scan_projection(file_request.predicate_columns, file_column_id); + if (scan_projection == nullptr) { + scan_projection = + find_scan_projection(file_request.non_predicate_columns, file_column_id); + } + DORIS_CHECK(scan_projection != nullptr); + const auto output_mapping_it = + std::ranges::find_if(output_mappings, [&](const ColumnMapping& output_mapping) { + return output_mapping.global_index == mapping.global_index; + }); + const auto* physical_mapping = + output_mapping_it == output_mappings.end() ? &mapping : &*output_mapping_it; global_to_file_slot.emplace( mapping.global_index, FileSlotRewriteInfo {.block_position = entry_it->second.local_index().value(), .file_type = mapping.file_type, .table_type = mapping.table_type, - .file_column_name = mapping.file_column_name}); + .file_column_name = mapping.file_column_name, + .root_mapping = physical_mapping, + .scan_projection = *scan_projection}); } return global_to_file_slot; } @@ -2281,6 +2391,7 @@ Status TableColumnMapper::create_scan_request( } file_request->conjuncts.clear(); file_request->metadata_pruning_safe_conjunct_count = 0; + file_request->constant_pruning_safe_table_filter_count = 0; file_request->delete_conjuncts.clear(); _filter_entries.clear(); // 1. Build referenced non-predicate columns @@ -2448,8 +2559,11 @@ Status TableColumnMapper::localize_filters(const std::vector& table // Build the complete table-slot rewrite map after all predicate columns have been assigned. // This keeps expression localization independent from filter iteration order. filter_mappings = _filter_visible_mappings(); - const auto global_to_file_slot = build_file_slot_rewrite_map(filter_mappings, _filter_entries); - for (const auto& table_filter : table_filters) { + const auto global_to_file_slot = + build_file_slot_rewrite_map(filter_mappings, _mappings, _filter_entries, *file_request); + std::vector localized_table_filters(table_filters.size(), false); + for (size_t table_filter_idx = 0; table_filter_idx < table_filters.size(); ++table_filter_idx) { + const auto& table_filter = table_filters[table_filter_idx]; if (table_filter.conjunct != nullptr && table_filter.conjunct->root() != nullptr) { const auto root = table_filter.conjunct->root(); const auto impl = root->get_impl(); @@ -2507,9 +2621,7 @@ Status TableColumnMapper::localize_filters(const std::vector& table auto localized_conjunct = VExprContext::create_shared(std::move(localized_root)); RETURN_IF_ERROR(rewrite_context.prepare_created_exprs(localized_conjunct.get())); file_request->conjuncts.push_back(std::move(localized_conjunct)); - if (table_filter.metadata_pruning_safe) { - ++file_request->metadata_pruning_safe_conjunct_count; - } + localized_table_filters[table_filter_idx] = true; for (const auto global_index : table_filter.global_indices) { const auto* mapping = _find_filter_mapping(global_index); if (mapping != nullptr && mapping->file_local_id.has_value() && @@ -2520,17 +2632,44 @@ Status TableColumnMapper::localize_filters(const std::vector& table } } + bool in_metadata_pruning_safe_prefix = true; + bool in_constant_pruning_safe_prefix = true; + for (size_t table_filter_idx = 0; table_filter_idx < table_filters.size(); ++table_filter_idx) { + const auto& table_filter = table_filters[table_filter_idx]; + const bool constant_filter = + table_filter_has_only_constant_entries(table_filter, _filter_entries); + if (!table_filter.metadata_pruning_safe) { + in_metadata_pruning_safe_prefix = false; + } + if (constant_filter) { + // Safe constant filters are evaluated before opening the file and do not occupy a + // file-local conjunct position. + } else if (!localized_table_filters[table_filter_idx]) { + in_metadata_pruning_safe_prefix = false; + } else if (in_metadata_pruning_safe_prefix) { + ++file_request->metadata_pruning_safe_conjunct_count; + } + + if (in_constant_pruning_safe_prefix && + (constant_filter || localized_table_filters[table_filter_idx])) { + ++file_request->constant_pruning_safe_table_filter_count; + } else { + // A rejected localization must preserve all later filters for post-materialization + // evaluation, even if a later constant would otherwise prune the complete split. + in_constant_pruning_safe_prefix = false; + } + } + // Candidate columns are added before expression rewriting because their file-block positions // are needed to localize slot refs. If rewriting rejects every filter that references a visible - // column, move its all-access-path projection to the lazy non-predicate set instead of forcing - // it through the eager predicate path. + // column, merge any independent output/filter subtrees and move the result to the lazy + // non-predicate set instead of forcing it through the eager predicate path. for (auto& mapping : _mappings) { if (!mapping.file_local_id.has_value()) { continue; } const auto local_id = LocalColumnId(*mapping.file_local_id); - if (localized_predicate_columns.contains(local_id) || - file_request->has_deferred_non_predicate_column(local_id)) { + if (localized_predicate_columns.contains(local_id)) { continue; } const auto predicate_it = std::ranges::find_if( @@ -2540,8 +2679,23 @@ Status TableColumnMapper::localize_filters(const std::vector& table if (predicate_it == file_request->predicate_columns.end()) { continue; } - file_request->non_predicate_columns.push_back(std::move(*predicate_it)); + LocalColumnIndex demoted_projection = std::move(*predicate_it); file_request->predicate_columns.erase(predicate_it); + const auto output_it = std::ranges::find_if(file_request->non_predicate_columns, + [local_id](const LocalColumnIndex& projection) { + return projection.column_id() == local_id; + }); + if (output_it != file_request->non_predicate_columns.end()) { + // A rejected complex predicate still needs its filter-only subtree on Scanner's + // table-level path. Merge it with the deferred output before collapsing the two block + // positions, or branch-4.1 can silently drop the child used by the residual filter. + RETURN_IF_ERROR(merge_local_column_index(&demoted_projection, *output_it)); + file_request->non_predicate_columns.erase(output_it); + file_request->non_predicate_positions.erase(local_id); + std::erase(file_request->predicate_only_columns, local_id); + } + FileScanRequestBuilder builder(file_request); + RETURN_IF_ERROR(builder.add_non_predicate_column(std::move(demoted_projection))); } return Status::OK(); } diff --git a/be/src/format_v2/column_mapper_nested.cpp b/be/src/format_v2/column_mapper_nested.cpp index 61e04a0b7301de..f67596cc4991af 100644 --- a/be/src/format_v2/column_mapper_nested.cpp +++ b/be/src/format_v2/column_mapper_nested.cpp @@ -26,6 +26,7 @@ #include "common/cast_set.h" #include "common/exception.h" #include "core/assert_cast.h" +#include "core/data_type/data_type_array.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_struct.h" #include "core/data_type/primitive_type.h" @@ -88,7 +89,13 @@ static bool parse_struct_child_selector(const VExprSPtr& expr, StructChildSelect static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* path) { DORIS_CHECK(path != nullptr); - if (!is_struct_element_expr(expr)) { + const bool is_struct_element = is_struct_element_expr(expr); + const bool is_array_element = + expr != nullptr && expr->get_num_children() == 2 && + expr->fn().name.function_name == "element_at" && + expr->children()[0]->data_type() != nullptr && + remove_nullable(expr->children()[0]->data_type())->get_primitive_type() == TYPE_ARRAY; + if (!is_struct_element && !is_array_element) { return false; } @@ -97,11 +104,19 @@ static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* if (!parse_struct_child_selector(expr->children()[1], &selector)) { return false; } + if (is_array_element) { + if (selector.by_name) { + return false; + } + // Every array ordinal is represented by the same repeated Parquet element projection. + selector.is_array_element = true; + } const auto& parent = expr->children()[0]; if (parent->is_slot_ref()) { const auto* slot_ref = assert_cast(parent.get()); path->root_global_index = slot_ref_global_index(*slot_ref); + path->root_table_type = slot_ref->data_type(); path->selectors.clear(); path->selectors.push_back(std::move(selector)); return true; @@ -118,6 +133,9 @@ static bool extract_nested_struct_path(const VExprSPtr& expr, NestedStructPath* static const ColumnDefinition* resolve_file_child(const std::vector& children, const StructChildSelector& selector) { + if (selector.is_array_element) { + return children.size() == 1 ? &children[0] : nullptr; + } if (selector.by_name) { const auto child_it = std::ranges::find_if(children, [&](const ColumnDefinition& child) { return child.name == selector.name; @@ -141,8 +159,42 @@ static const DataTypeStruct* struct_type_or_null(const DataTypePtr& type) { return assert_cast(nested_type.get()); } +static DataTypePtr resolve_table_child_type(const DataTypePtr& parent_type, + const StructChildSelector& selector) { + if (parent_type == nullptr) { + return nullptr; + } + const auto nested_type = remove_nullable(parent_type); + if (selector.is_array_element) { + if (nested_type->get_primitive_type() != TYPE_ARRAY) { + return nullptr; + } + return assert_cast(nested_type.get())->get_nested_type(); + } + const auto* struct_type = struct_type_or_null(nested_type); + if (struct_type == nullptr) { + return nullptr; + } + if (selector.by_name) { + const auto position = struct_type->try_get_position_by_name(selector.name); + return position.has_value() ? struct_type->get_element(*position) : nullptr; + } + if (selector.ordinal == 0 || selector.ordinal > struct_type->get_elements().size()) { + return nullptr; + } + return struct_type->get_element(selector.ordinal - 1); +} + static std::optional struct_child_index(const ColumnMapping& mapping, const StructChildSelector& selector) { + if (selector.is_array_element) { + const auto nested_type = remove_nullable(mapping.table_type); + if (nested_type == nullptr || nested_type->get_primitive_type() != TYPE_ARRAY || + mapping.child_mappings.size() != 1) { + return std::nullopt; + } + return 0; + } const auto* struct_type = struct_type_or_null(mapping.table_type); if (struct_type == nullptr) { return std::nullopt; @@ -249,8 +301,10 @@ static NestedProjectionResolveResult resolve_nested_projection_with_mapping( return NestedProjectionResolveResult::RESOLVED; } -static bool table_root_is_struct(const ColumnMapping& mapping) { - return struct_type_or_null(mapping.table_type) != nullptr; +static bool table_root_is_nested_container(const ColumnMapping& mapping) { + const auto nested_type = remove_nullable(mapping.table_type); + return nested_type != nullptr && (nested_type->get_primitive_type() == TYPE_STRUCT || + nested_type->get_primitive_type() == TYPE_ARRAY); } static const std::vector& scan_file_children(const ColumnMapping& mapping) { @@ -344,7 +398,7 @@ bool resolve_nested_struct_path_for_file(const NestedStructPath& path, return false; } if (mapping_result == NestedProjectionResolveResult::NOT_REPRESENTED) { - if (!table_root_is_struct(*mapping_it)) { + if (!table_root_is_nested_container(*mapping_it)) { return false; } LocalColumnIndex child_projection; @@ -382,6 +436,20 @@ bool resolve_nested_struct_path_for_file(const NestedStructPath& path, *resolved = {}; return false; } + resolved->file_array_elements.reserve(path.selectors.size()); + resolved->table_child_types.reserve(path.selectors.size()); + auto table_child_type = path.root_table_type; + for (const auto& selector : path.selectors) { + resolved->file_array_elements.push_back(selector.is_array_element); + if (path.root_table_type != nullptr) { + table_child_type = resolve_table_child_type(table_child_type, selector); + if (table_child_type == nullptr) { + *resolved = {}; + return false; + } + resolved->table_child_types.push_back(table_child_type); + } + } return true; } diff --git a/be/src/format_v2/column_mapper_nested.h b/be/src/format_v2/column_mapper_nested.h index ab96512e1709ed..ba0d7f6c2e9fef 100644 --- a/be/src/format_v2/column_mapper_nested.h +++ b/be/src/format_v2/column_mapper_nested.h @@ -34,6 +34,7 @@ namespace doris::format { struct StructChildSelector { + bool is_array_element = false; bool by_name = true; std::string name; size_t ordinal = 0; @@ -41,6 +42,7 @@ struct StructChildSelector { struct NestedStructPath { GlobalIndex root_global_index; + DataTypePtr root_table_type; std::vector selectors; }; @@ -48,6 +50,8 @@ struct ResolvedNestedStructPath { LocalColumnIndex file_projection; std::vector file_child_names; std::vector file_child_types; + std::vector table_child_types; + std::vector file_array_elements; }; // A split-local literal produced by slot-literal predicate localization. This wrapper keeps the diff --git a/be/src/format_v2/file_reader.cpp b/be/src/format_v2/file_reader.cpp index 1b1f2f284405f9..8a9df5d7953454 100644 --- a/be/src/format_v2/file_reader.cpp +++ b/be/src/format_v2/file_reader.cpp @@ -74,6 +74,8 @@ std::string FileScanRequest::debug_string() const { out << column_id << ":" << block_position; } out << "}, conjunct_count=" << conjuncts.size() + << ", metadata_pruning_safe_conjunct_count=" << metadata_pruning_safe_conjunct_count + << ", constant_pruning_safe_table_filter_count=" << constant_pruning_safe_table_filter_count << ", delete_conjunct_count=" << delete_conjuncts.size() << ", variant_schema_overrides=" << join_debug_strings( variant_schema_overrides, diff --git a/be/src/format_v2/file_reader.h b/be/src/format_v2/file_reader.h index 1f269cde3e280f..02a722a0fa455e 100644 --- a/be/src/format_v2/file_reader.h +++ b/be/src/format_v2/file_reader.h @@ -87,6 +87,9 @@ struct FileScanRequest { // Only this leading subset may participate in footer/page metadata pruning. The boundary is // inherited from table-conjunct order so an omitted slotless unsafe expression remains a fence. size_t metadata_pruning_safe_conjunct_count = std::numeric_limits::max(); + // Constant split pruning may use only this table-filter prefix after mapping. A rejected + // file-local rewrite is a materialization barrier even when a later filter is constant. + size_t constant_pruning_safe_table_filter_count = std::numeric_limits::max(); // Delete predicates converted to file-local expressions. A TRUE result means that the row is // deleted, so readers must invert each result when building their keep filter. VExprContextSPtrs delete_conjuncts; diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index d21db0f57868bc..e7509ebd078364 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -1404,7 +1404,9 @@ Status OrcReader::_configure_row_reader_projection() { } Status OrcReader::_init_search_argument_from_local_filters() { - if (!_state->enable_filter_by_min_max || _request->conjuncts.empty()) { + const size_t safe_count = + std::min(_request->metadata_pruning_safe_conjunct_count, _request->conjuncts.size()); + if (!_state->enable_filter_by_min_max || safe_count == 0) { return Status::OK(); } @@ -1412,7 +1414,10 @@ Status OrcReader::_init_search_argument_from_local_filters() { auto builder = ::orc::SearchArgumentFactory::newBuilder(); bool has_pushdown = false; builder->startAnd(); - for (const auto& conjunct : _request->conjuncts) { + // ORC may omit unsupported expressions from a SARG, so a later predicate must not cross + // an earlier error-preserving barrier. + for (size_t i = 0; i < safe_count; ++i) { + const auto& conjunct = _request->conjuncts[i]; if (conjunct == nullptr) { continue; } diff --git a/be/src/format_v2/parquet/parquet_profile.cpp b/be/src/format_v2/parquet/parquet_profile.cpp index ec70db46840771..ef167847c18d76 100644 --- a/be/src/format_v2/parquet/parquet_profile.cpp +++ b/be/src/format_v2/parquet/parquet_profile.cpp @@ -250,6 +250,14 @@ void ParquetProfile::init(RuntimeProfile* profile) { TUnit::UNIT, parquet_profile, 1); rows_filtered_by_dict_filter = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "RowsFilteredByDictFilter", TUnit::UNIT, parquet_profile, 1); + bloom_filter_probe_attempts = ADD_CHILD_COUNTER_WITH_LEVEL(profile, "BloomFilterProbeAttempts", + TUnit::UNIT, parquet_profile, 1); + bloom_filter_probe_successes = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "BloomFilterProbeSuccesses", TUnit::UNIT, parquet_profile, 1); + bloom_filter_conservative_fallbacks = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "BloomFilterConservativeFallbacks", TUnit::UNIT, parquet_profile, 1); + bloom_filter_corrupt_rejections = ADD_CHILD_COUNTER_WITH_LEVEL( + profile, "BloomFilterCorruptRejections", TUnit::UNIT, parquet_profile, 1); bloom_filter_read_time = ADD_CHILD_TIMER_WITH_LEVEL(profile, "BloomFilterReadTime", parquet_profile, 1); } @@ -271,6 +279,11 @@ void ParquetProfile::update_pruning_stats(const ParquetPruningStats& pruning_sta COUNTER_UPDATE(filtered_bytes, pruning_stats.filtered_bytes); COUNTER_UPDATE(filtered_page_rows, pruning_stats.filtered_page_rows); COUNTER_UPDATE(page_index_read_calls, pruning_stats.page_index_read_calls); + COUNTER_UPDATE(bloom_filter_probe_attempts, pruning_stats.bloom_filter_probe_attempts); + COUNTER_UPDATE(bloom_filter_probe_successes, pruning_stats.bloom_filter_probe_successes); + COUNTER_UPDATE(bloom_filter_conservative_fallbacks, + pruning_stats.bloom_filter_conservative_fallbacks); + COUNTER_UPDATE(bloom_filter_corrupt_rejections, pruning_stats.bloom_filter_corrupt_rejections); COUNTER_UPDATE(bloom_filter_read_time, pruning_stats.bloom_filter_read_time); COUNTER_UPDATE(row_group_filter_time, pruning_stats.row_group_filter_time); COUNTER_UPDATE(page_index_filter_time, pruning_stats.page_index_filter_time); @@ -300,6 +313,11 @@ void ParquetProfile::update_deferred_pruning_stats(const ParquetPruningStats& pr COUNTER_UPDATE(filtered_bytes, pruning_stats.filtered_bytes); COUNTER_UPDATE(filtered_page_rows, pruning_stats.filtered_page_rows); COUNTER_UPDATE(page_index_read_calls, pruning_stats.page_index_read_calls); + COUNTER_UPDATE(bloom_filter_probe_attempts, pruning_stats.bloom_filter_probe_attempts); + COUNTER_UPDATE(bloom_filter_probe_successes, pruning_stats.bloom_filter_probe_successes); + COUNTER_UPDATE(bloom_filter_conservative_fallbacks, + pruning_stats.bloom_filter_conservative_fallbacks); + COUNTER_UPDATE(bloom_filter_corrupt_rejections, pruning_stats.bloom_filter_corrupt_rejections); COUNTER_UPDATE(bloom_filter_read_time, pruning_stats.bloom_filter_read_time); COUNTER_UPDATE(row_group_filter_time, pruning_stats.row_group_filter_time); COUNTER_UPDATE(page_index_filter_time, pruning_stats.page_index_filter_time); diff --git a/be/src/format_v2/parquet/parquet_profile.h b/be/src/format_v2/parquet/parquet_profile.h index 764fef1d80c190..cde385bdb726d8 100644 --- a/be/src/format_v2/parquet/parquet_profile.h +++ b/be/src/format_v2/parquet/parquet_profile.h @@ -258,6 +258,10 @@ struct ParquetProfile { RuntimeProfile::Counter* dict_filter_unsupported_columns = nullptr; RuntimeProfile::Counter* dict_filter_read_failures = nullptr; RuntimeProfile::Counter* rows_filtered_by_dict_filter = nullptr; + RuntimeProfile::Counter* bloom_filter_probe_attempts = nullptr; + RuntimeProfile::Counter* bloom_filter_probe_successes = nullptr; + RuntimeProfile::Counter* bloom_filter_conservative_fallbacks = nullptr; + RuntimeProfile::Counter* bloom_filter_corrupt_rejections = nullptr; RuntimeProfile::Counter* bloom_filter_read_time = nullptr; }; diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp b/be/src/format_v2/parquet/parquet_statistics.cpp index b0c6fbc28eb1df..254ae584c28692 100644 --- a/be/src/format_v2/parquet/parquet_statistics.cpp +++ b/be/src/format_v2/parquet/parquet_statistics.cpp @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -170,7 +171,12 @@ Status read_native_bloom_filter(const tparquet::ColumnMetaData& metadata, io_ctx)); tparquet::BloomFilterHeader header; uint32_t header_size = cast_set(bytes_read); - RETURN_IF_ERROR(deserialize_thrift_msg(header_buffer.data(), &header_size, true, &header)); + const auto deserialize_status = + deserialize_thrift_msg(header_buffer.data(), &header_size, true, &header); + if (!deserialize_status.ok()) { + // Keep invalid on-disk metadata distinguishable from transient read failures in profiles. + return Status::Corruption("Malformed Parquet Bloom filter header"); + } if (!header.algorithm.__isset.BLOCK || !header.compression.__isset.UNCOMPRESSED || !header.hash.__isset.XXHASH || header.numBytes <= 0) { return Status::NotSupported("Unsupported Parquet Bloom filter encoding"); @@ -449,6 +455,55 @@ const ParquetColumnSchema* resolve_local_leaf_schema( return column_schema; } +const ParquetColumnSchema* resolve_bloom_filter_leaf_schema( + const std::vector>& schema, + const format::LocalColumnId file_column_id, const expr_zonemap::BloomFilterProbe& probe) { + if (probe.path.empty()) { + return resolve_local_leaf_schema(schema, file_column_id); + } + if (!file_column_id.is_valid() || file_column_id.value() >= static_cast(schema.size())) { + return nullptr; + } + const ParquetColumnSchema* column_schema = schema[file_column_id.value()].get(); + // A nested predicate must bind to its exact localized primitive path. Falling back to a + // sibling leaf's Bloom filter could turn absence in that sibling into an invalid row-group skip. + for (const auto& path_element : probe.path) { + if (column_schema == nullptr) { + return nullptr; + } + if (path_element.kind == expr_zonemap::BloomFilterPathKind::STRUCT_FIELD) { + if (column_schema->kind != ParquetColumnSchemaKind::STRUCT) { + return nullptr; + } + const ParquetColumnSchema* field_schema = nullptr; + if (!path_element.field_name.empty()) { + auto field = std::ranges::find_if(column_schema->children, [&](const auto& child) { + return child != nullptr && child->name == path_element.field_name; + }); + if (field != column_schema->children.end()) { + field_schema = field->get(); + } + } else if (path_element.field_ordinal >= 0 && + path_element.field_ordinal < + static_cast(column_schema->children.size())) { + field_schema = column_schema->children[path_element.field_ordinal].get(); + } + column_schema = field_schema; + } else { + if (column_schema->kind != ParquetColumnSchemaKind::LIST || + column_schema->children.size() != 1) { + return nullptr; + } + column_schema = column_schema->children[0].get(); + } + } + if (column_schema == nullptr || column_schema->kind != ParquetColumnSchemaKind::PRIMITIVE || + column_schema->leaf_column_id < 0) { + return nullptr; + } + return column_schema; +} + std::optional file_column_id_by_block_position( const format::FileScanRequest& request, int block_position) { for (const auto& [file_column_id, local_index] : request.local_positions) { @@ -1198,39 +1253,108 @@ ParquetRowGroupPruneReason native_bloom_filter_prune_reason( if (file_context == nullptr || file_context->native_file == nullptr) { return ParquetRowGroupPruneReason::NONE; } - const auto conjuncts_by_slot = collect_conjuncts_by_single_slot( - metadata_pruning_conjuncts(request), expr_zonemap::single_slot_bloom_filter_index); - for (const auto& [slot_index, conjuncts] : conjuncts_by_slot) { - const auto file_column_id = file_column_id_by_block_position(request, slot_index); + struct BloomProbeGroup { + const ParquetColumnSchema* column_schema = nullptr; + int slot_index = -1; + VExprContextSPtrs conjuncts; + }; + struct LeafBloomProbeGroup { + int leaf_column_id = -1; + std::vector probes; + }; + // The vector preserves first-probe order, while the map only deduplicates repeated leaves. + // This avoids reading a potentially large later payload before an earlier probe can prune. + std::vector probes_by_first_use; + std::map group_index_by_leaf; + const auto add_probe = [&](const ParquetColumnSchema& column_schema, int slot_index, + VExprContextSPtrs conjuncts) { + if (column_schema.type == nullptr || + !native_metadata_predicate_is_type_safe(column_schema) || + !bloom_filter_supported(column_schema) || column_schema.leaf_column_id < 0 || + column_schema.leaf_column_id >= static_cast(row_group.columns.size())) { + return; + } + const auto [group_it, inserted] = group_index_by_leaf.try_emplace( + column_schema.leaf_column_id, probes_by_first_use.size()); + if (inserted) { + probes_by_first_use.push_back( + {.leaf_column_id = column_schema.leaf_column_id, .probes = {}}); + } + probes_by_first_use[group_it->second].probes.push_back({.column_schema = &column_schema, + .slot_index = slot_index, + .conjuncts = std::move(conjuncts)}); + }; + + const auto pruning_conjuncts = metadata_pruning_conjuncts(request); + // Resolve direct and nested probes in one conjunct-order pass. Deduplication happens only + // after first use so a later Bloom payload cannot be read before an earlier pruning probe. + for (const auto& conjunct : pruning_conjuncts) { + if (conjunct == nullptr || conjunct->root() == nullptr || + !conjunct->root()->can_evaluate_bloom_filter()) { + continue; + } + auto probe = expr_zonemap::extract_bloom_filter_predicate_probe(conjunct->root()); + if (!probe.has_value()) { + continue; + } + const auto file_column_id = file_column_id_by_block_position(request, probe->slot_index); if (!file_column_id.has_value()) { continue; } - const auto* column_schema = resolve_local_leaf_schema(file_schema, *file_column_id); - if (column_schema == nullptr || column_schema->type == nullptr || - !native_metadata_predicate_is_type_safe(*column_schema) || - !bloom_filter_supported(*column_schema) || - column_schema->leaf_column_id >= static_cast(row_group.columns.size())) { + const auto* column_schema = + probe->path.empty() + ? resolve_local_leaf_schema(file_schema, *file_column_id) + : resolve_bloom_filter_leaf_schema(file_schema, *file_column_id, *probe); + if (column_schema == nullptr || + !expr_zonemap::data_types_compatible(column_schema->type, probe->value_type)) { continue; } - const auto& chunk = row_group.columns[column_schema->leaf_column_id]; + add_probe(*column_schema, probe->slot_index, {conjunct}); + } + + for (const auto& leaf_group : probes_by_first_use) { + const int leaf_column_id = leaf_group.leaf_column_id; + if (pruning_stats != nullptr) { + ++pruning_stats->bloom_filter_probe_attempts; + } + const auto& chunk = row_group.columns[leaf_column_id]; if (!chunk.__isset.meta_data) { + if (pruning_stats != nullptr) { + ++pruning_stats->bloom_filter_conservative_fallbacks; + } continue; } std::unique_ptr bloom_filter; - Status status; + Status bloom_status; + int64_t timer_sink = 0; { - int64_t timer_sink = 0; SCOPED_RAW_TIMER(pruning_stats == nullptr ? &timer_sink : &pruning_stats->bloom_filter_read_time); - status = read_native_bloom_filter(chunk.meta_data, file_context->native_file, - file_context->native_io_ctx, &bloom_filter); + bloom_status = read_native_bloom_filter(row_group.columns[leaf_column_id].meta_data, + file_context->native_file, + file_context->native_io_ctx, &bloom_filter); + if (!bloom_status.ok()) { + bloom_filter.reset(); + } } - if (!status.ok() || bloom_filter == nullptr) { + if (bloom_filter == nullptr) { + if (pruning_stats != nullptr) { + ++pruning_stats->bloom_filter_conservative_fallbacks; + if (bloom_status.is()) { + ++pruning_stats->bloom_filter_corrupt_rejections; + } + } continue; } - if (ParquetStatisticsUtils::NativeBloomFilterExcludes(*column_schema, slot_index, conjuncts, - *bloom_filter)) { - return ParquetRowGroupPruneReason::BLOOM_FILTER; + if (pruning_stats != nullptr) { + ++pruning_stats->bloom_filter_probe_successes; + } + // Keep at most one decoded payload live while reusing it for every predicate on this leaf. + for (const auto& probe : leaf_group.probes) { + if (ParquetStatisticsUtils::NativeBloomFilterExcludes( + *probe.column_schema, probe.slot_index, probe.conjuncts, *bloom_filter)) { + return ParquetRowGroupPruneReason::BLOOM_FILTER; + } } } return ParquetRowGroupPruneReason::NONE; diff --git a/be/src/format_v2/parquet/parquet_statistics.h b/be/src/format_v2/parquet/parquet_statistics.h index 611ec5abe97ac2..cd758ebb90924d 100644 --- a/be/src/format_v2/parquet/parquet_statistics.h +++ b/be/src/format_v2/parquet/parquet_statistics.h @@ -80,6 +80,10 @@ struct ParquetPruningStats { int64_t selected_row_ranges = 0; // selected row range count int64_t page_index_read_calls = 0; // Page Index read count int64_t bloom_filter_read_time = 0; // Bloom filter read time (ns) + int64_t bloom_filter_probe_attempts = 0; // unique leaf Bloom probes attempted + int64_t bloom_filter_probe_successes = 0; // usable Bloom payloads decoded + int64_t bloom_filter_conservative_fallbacks = 0; // unavailable/unreadable Blooms retained + int64_t bloom_filter_corrupt_rejections = 0; // malformed Blooms rejected conservatively int64_t row_group_filter_time = 0; // row-group pruning time (ns) int64_t page_index_filter_time = 0; // page-index pruning time (ns) int64_t read_page_index_time = 0; // page-index read time (ns) diff --git a/be/src/format_v2/table_reader.h b/be/src/format_v2/table_reader.h index e98f96215e0e0f..c763a2301ba1d1 100644 --- a/be/src/format_v2/table_reader.h +++ b/be/src/format_v2/table_reader.h @@ -453,6 +453,9 @@ class TableReader { auto file_request = std::make_shared(); RETURN_IF_ERROR(_data_reader.column_mapper->create_scan_request( _table_filters, _projected_columns, file_request.get(), _runtime_state)); + _constant_pruning_safe_filter_count = + std::min(_constant_pruning_safe_filter_count, + file_request->constant_pruning_safe_table_filter_count); bool constant_filter_pruned_split = false; RETURN_IF_ERROR(_evaluate_constant_filters(&constant_filter_pruned_split)); if (constant_filter_pruned_split) { diff --git a/be/test/exprs/expr_zonemap_filter_test.cpp b/be/test/exprs/expr_zonemap_filter_test.cpp index d8164cb7248701..d76dd01af30eb0 100644 --- a/be/test/exprs/expr_zonemap_filter_test.cpp +++ b/be/test/exprs/expr_zonemap_filter_test.cpp @@ -32,11 +32,13 @@ #include "common/object_pool.h" #include "core/column/column_vector.h" +#include "core/data_type/data_type_array.h" #include "core/data_type/data_type_date_or_datetime_v2.h" #include "core/data_type/data_type_decimal.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" #include "core/field.h" #include "core/string_ref.h" #include "core/value/vdatetime_value.h" @@ -243,6 +245,48 @@ class FixedZonemapExpr final : public VExpr { std::string _expr_name = "fixed_zonemap_expr"; }; +class MetadataAccessorExpr final : public VExpr { +public: + MetadataAccessorExpr(std::string function_name, DataTypePtr result_type, VExprSPtr parent, + VExprSPtr selector) + : VExpr(std::move(result_type), false), _expr_name(std::move(function_name)) { + _fn.name.function_name = _expr_name; + add_child(std::move(parent)); + add_child(std::move(selector)); + } + + const std::string& expr_name() const override { return _expr_name; } + + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataAccessorExpr is metadata-only"); + } + +private: + std::string _expr_name; +}; + +class MetadataBloomPredicateExpr final : public VExpr { +public: + explicit MetadataBloomPredicateExpr(VExprSPtr probe) + : VExpr(std::make_shared(), false) { + add_child(std::move(probe)); + } + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataBloomPredicateExpr is metadata-only"); + } + bool can_evaluate_bloom_filter() const override { return true; } + ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext&) const override { + return ZoneMapFilterResult::kMayMatch; + } + +private: + const std::string _expr_name = "MetadataBloomPredicateExpr"; +}; + class UnsupportedSingleSlotExpr final : public VExpr { public: explicit UnsupportedSingleSlotExpr(const VExprSPtr& slot) { @@ -641,6 +685,105 @@ TEST(ExprZonemapFilterTest, DefaultFunctionForwardsDictionaryAndBloomEvaluation) equals->evaluate_bloom_filter(bloom_ctx, {slot, make_int_literal(3)})); } +TEST(ExprZonemapFilterTest, NullSafeEqualityUsesBloomOnlyForNonNullLiteral) { + auto type = int_type(); + auto slot = make_slot(0, type); + auto equals_for_null = SimpleFunctionFactory::instance().get_function( + "eq_for_null", + ColumnsWithTypeAndName {{nullptr, type, "slot"}, {nullptr, type, "literal"}}, + std::make_shared()); + ASSERT_NE(equals_for_null, nullptr); + + auto bloom_filter = make_int_bloom_filter({1, 3}); + auto bloom_ctx = make_bloom_filter_context(bloom_filter.get(), type); + EXPECT_TRUE(equals_for_null->can_evaluate_bloom_filter({slot, make_int_literal(2)})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + equals_for_null->evaluate_bloom_filter(bloom_ctx, {slot, make_int_literal(2)})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + equals_for_null->evaluate_bloom_filter(bloom_ctx, {slot, make_int_literal(3)})); + + EXPECT_FALSE(equals_for_null->can_evaluate_bloom_filter({slot, make_null_int_literal()})); +} + +TEST(ExprZonemapFilterTest, EqualityBloomAcceptsStructAndListLeafAccessors) { + auto leaf_type = int_type(); + auto bloom_filter = make_int_bloom_filter({1, 3}); + auto bloom_ctx = make_bloom_filter_context(bloom_filter.get(), leaf_type); + FunctionComparison equals; + + auto struct_type = std::make_shared(DataTypes {leaf_type}, Strings {"value"}); + auto struct_accessor = std::make_shared( + "element_at", leaf_type, make_slot(0, struct_type), make_string_literal("value")); + EXPECT_TRUE(equals.can_evaluate_bloom_filter({struct_accessor, make_int_literal(2)})); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + equals.evaluate_bloom_filter(bloom_ctx, {struct_accessor, make_int_literal(2)})); + + auto list_type = std::make_shared(leaf_type); + auto list_accessor = std::make_shared( + "element_at", leaf_type, make_slot(0, list_type), make_int_literal(1)); + EXPECT_TRUE(equals.can_evaluate_bloom_filter({list_accessor, make_int_literal(3)})); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + equals.evaluate_bloom_filter(bloom_ctx, {list_accessor, make_int_literal(3)})); + + auto nested_type = std::make_shared(DataTypes {list_type}, Strings {"items"}); + auto nested_list = std::make_shared( + "element_at", list_type, make_slot(0, nested_type), make_string_literal("items")); + auto nested_leaf = std::make_shared( + "element_at", leaf_type, std::move(nested_list), make_int_literal(1)); + auto nested_probe = expr_zonemap::extract_bloom_filter_probe(nested_leaf); + ASSERT_TRUE(nested_probe.has_value()); + ASSERT_EQ(nested_probe->path.size(), 2); + EXPECT_EQ(nested_probe->path[0].kind, expr_zonemap::BloomFilterPathKind::STRUCT_FIELD); + EXPECT_EQ(nested_probe->path[1].kind, expr_zonemap::BloomFilterPathKind::LIST_ELEMENT); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + equals.evaluate_bloom_filter(bloom_ctx, {nested_leaf, make_int_literal(2)})); +} + +TEST(ExprZonemapFilterTest, CompoundBloomProbeRequiresOneUniqueNestedLeaf) { + const auto make_accessor = [](const DataTypePtr& struct_type, const DataTypePtr& leaf_type, + std::string field_name) { + return std::make_shared("element_at", leaf_type, + make_slot(0, struct_type), + make_string_literal(std::move(field_name))); + }; + const auto compound_probe = [](const VExprSPtr& first, const VExprSPtr& second, + const VExprSPtr& outer) { + auto inner = + std::make_shared(make_compound_node(TExprOpcode::COMPOUND_AND, 2)); + inner->add_child(std::make_shared(first)); + inner->add_child(std::make_shared(second)); + auto root = + std::make_shared(make_compound_node(TExprOpcode::COMPOUND_OR, 2)); + root->add_child(std::move(inner)); + root->add_child(std::make_shared(outer)); + EXPECT_TRUE(root->can_evaluate_bloom_filter()); + return expr_zonemap::extract_bloom_filter_predicate_probe(root); + }; + + auto int_leaf = int_type(); + auto same_type_struct = + std::make_shared(DataTypes {int_leaf, int_leaf}, Strings {"a", "b"}); + EXPECT_FALSE(compound_probe(make_accessor(same_type_struct, int_leaf, "a"), + make_accessor(same_type_struct, int_leaf, "b"), + make_accessor(same_type_struct, int_leaf, "a")) + .has_value()); + + auto string_leaf = std::make_shared(); + auto mixed_type_struct = + std::make_shared(DataTypes {int_leaf, string_leaf}, Strings {"a", "b"}); + EXPECT_FALSE(compound_probe(make_accessor(mixed_type_struct, int_leaf, "a"), + make_accessor(mixed_type_struct, string_leaf, "b"), + make_accessor(mixed_type_struct, int_leaf, "a")) + .has_value()); + + auto same_leaf_probe = compound_probe(make_accessor(same_type_struct, int_leaf, "a"), + make_accessor(same_type_struct, int_leaf, "a"), + make_accessor(same_type_struct, int_leaf, "a")); + ASSERT_TRUE(same_leaf_probe.has_value()); + ASSERT_EQ(same_leaf_probe->path.size(), 1); + EXPECT_EQ(same_leaf_probe->path[0].field_name, "a"); +} + TEST(ExprZonemapFilterTest, MissingSlotTypeCountsUnsupportedZonemapEvalOnce) { auto type = int_type(); auto slot = make_slot(0, type); @@ -989,6 +1132,44 @@ TEST(ExprZonemapFilterTest, VInPredicateDictionaryAndBloomUseMaterializedValues) in_predicate->evaluate_bloom_filter(matching_bloom_ctx)); } +TEST(ExprZonemapFilterTest, VInPredicateMaterializesNestedBloomValuesDuringOpen) { + auto leaf_type = int_type(); + auto struct_type = std::make_shared(DataTypes {leaf_type}, Strings {"value"}); + auto slot = VSlotRef::create_shared(0, 0, -1, struct_type, "root"); + auto accessor = std::make_shared("element_at", leaf_type, std::move(slot), + make_string_literal("value")); + auto in_predicate = std::make_shared(make_in_predicate_node(false, 3)); + in_predicate->add_child(std::move(accessor)); + in_predicate->add_child(make_int_literal(2)); + in_predicate->add_child(make_int_literal(4)); + + ObjectPool obj_pool; + DescriptorTbl* desc_tbl = nullptr; + auto thrift_desc_tbl = make_k2_scan_desc_tbl(); + ASSERT_TRUE(DescriptorTbl::create(&obj_pool, thrift_desc_tbl, &desc_tbl).ok()); + RuntimeState runtime_state; + runtime_state.set_desc_tbl(desc_tbl); + RowDescriptor row_desc(runtime_state.desc_tbl(), {0}); + VExprContext in_context(in_predicate); + ASSERT_TRUE(in_context.prepare(&runtime_state, row_desc).ok()); + ASSERT_TRUE(in_context.open(&runtime_state).ok()); + + EXPECT_TRUE(in_predicate->_zonemap_materialized); + EXPECT_TRUE(in_predicate->can_evaluate_bloom_filter()); + EXPECT_FALSE(in_predicate->can_evaluate_zonemap_filter()); + EXPECT_FALSE(in_predicate->can_evaluate_dictionary_filter()); + EXPECT_FALSE(in_predicate->can_execute_on_raw_fixed_values(leaf_type, 0)); + + auto missing_bloom_filter = make_int_bloom_filter({1, 3}); + EXPECT_EQ(ZoneMapFilterResult::kNoMatch, + in_predicate->evaluate_bloom_filter( + make_bloom_filter_context(missing_bloom_filter.get(), leaf_type))); + auto matching_bloom_filter = make_int_bloom_filter({4}); + EXPECT_EQ(ZoneMapFilterResult::kMayMatch, + in_predicate->evaluate_bloom_filter( + make_bloom_filter_context(matching_bloom_filter.get(), leaf_type))); +} + TEST(ExprZonemapFilterTest, DirectInPredicateMaterializesStringSetForZonemap) { auto type = std::make_shared(); std::shared_ptr filter(create_set(PrimitiveType::TYPE_STRING, false)); diff --git a/be/test/format_v2/column_mapper_test.cpp b/be/test/format_v2/column_mapper_test.cpp index 4c3f23ec262a4c..8111902a6468f6 100644 --- a/be/test/format_v2/column_mapper_test.cpp +++ b/be/test/format_v2/column_mapper_test.cpp @@ -2419,10 +2419,9 @@ TEST(ColumnMapperScanRequestTest, StructProjectionPrunesChildrenByName) { EXPECT_EQ(projected_type->get_element_name(0), "b"); } -// Scenario: a row filter reaches a struct child through an array wrapper -// (`items.item.a > 5`). The mapper cannot localize the filter, so it keeps the full array root in -// the lazy non-predicate set for table-level evaluation. -TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) { +// Scenario: a row filter reaches a struct child through an array element +// (`items[1].a > 5`). Identical table/file schemas can safely localize the complete accessor path. +TEST(ColumnMapperScanRequestTest, ArrayStructPathBuildsNestedPredicateFilter) { const auto int_type = i32(); const auto string_type = str(); @@ -2435,7 +2434,7 @@ TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) auto table_array = file_array; const auto item_type = file_element.type; - auto item_expr = struct_element(table_slot(0, 0, table_array.type, "items"), item_type, "item"); + auto item_expr = array_element_at(table_slot(0, 0, table_array.type, "items"), item_type, 1); auto filter_expr = int_gt(struct_element(item_expr, int_type, "a"), 5); TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), .global_indices = {GlobalIndex(0)}}; @@ -2443,15 +2442,132 @@ TEST(ColumnMapperScanRequestTest, ArrayWrapperDoesNotBuildNestedPredicateFilter) TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {}, &request).ok()); + + ASSERT_EQ(request.conjuncts.size(), 1); + ASSERT_EQ(request.predicate_columns.size(), 1); + EXPECT_TRUE(request.non_predicate_columns.empty()); + const auto& projection = request.predicate_columns[0]; + EXPECT_EQ(projection.column_id(), LocalColumnId(0)); + // Array indexing still needs the complete repeated sequence, while the localized accessor lets + // Parquet metadata pruning resolve the selected struct leaf. + EXPECT_TRUE(projection.project_all_children); + EXPECT_TRUE(projection.children.empty()); +} + +// Production ARRAY and STRUCT accessors make their result nullable for missing indices and NULL +// parents. That execution nullability must not hide a required table child when deciding whether +// the file predicate may run before TableReader's schema validation. +TEST(ColumnMapperScanRequestTest, ArrayStructPathKeepsNullableFileLeafAboveRequiredTableLeaf) { + const auto required_int_type = i32(); + const auto nullable_int_type = make_nullable(required_int_type); + + auto table_a = name_col("a", required_int_type); + auto table_element = struct_name_col("element", {table_a}, 0); + auto table_array = array_col("items", -1, table_element, 0); + set_name_identifiers(&table_array, 0); + + auto file_a = name_col("a", nullable_int_type, 0); + auto file_element = struct_name_col("element", {file_a}, 0); + auto file_array = array_col("items", -1, file_element, 0); + set_name_identifiers(&file_array, 0); + + auto item_expr = array_element_at(table_slot(0, 0, table_array.type, "items"), + make_nullable(table_element.type), 1); + auto leaf_expr = struct_element(item_expr, nullable_int_type, "a"); + auto filter_expr = int_gt(leaf_expr, 10); + TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), + .global_indices = {GlobalIndex(0)}}; + + TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); + ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + FileScanRequest request; ASSERT_TRUE(mapper.create_scan_request({filter}, {table_array}, &request).ok()); + EXPECT_TRUE(request.conjuncts.empty()); +} + +// ARRAY access projects every element child, so an unrelated narrowing sibling conversion must +// remain visible to TableReader before any file-local predicate can discard its source row. +TEST(ColumnMapperScanRequestTest, ArrayStructPathKeepsLossyProjectedSiblingAtTableLevel) { + const auto int_type = i32(); + const auto bigint_type = i64(); + + auto table_a = name_col("a", int_type, 0); + auto table_b = name_col("b", int_type, 1); + auto table_element = struct_name_col("element", {table_a, table_b}, 0); + auto table_array = array_col("items", -1, table_element, 0); + set_name_identifiers(&table_array, 0); + + auto file_a = name_col("a", int_type, 0); + auto file_b = name_col("b", bigint_type, 1); + auto file_element = struct_name_col("element", {file_a, file_b}, 0); + auto file_array = array_col("items", -1, file_element, 0); + set_name_identifiers(&file_array, 0); + auto item_expr = array_element_at(table_slot(0, 0, table_array.type, "items"), + make_nullable(table_element.type), 1); + auto leaf_expr = struct_element(item_expr, make_nullable(int_type), "a"); + auto filter_expr = int_gt(leaf_expr, 5); + TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), + .global_indices = {GlobalIndex(0)}}; + + TableColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); + ASSERT_TRUE(mapper.create_mapping({table_array}, {}, {file_array}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_array}, &request).ok()); EXPECT_TRUE(request.conjuncts.empty()); +} + +// Scenario: a map value struct projects child `b`, while a row filter reads value child `a`. +// The filter is too complex to become a file-local nested predicate. Lazy demotion must move the +// merged projection to the non-predicate set without dropping either physical value child. +TEST(ColumnMapperScanRequestTest, MapFilterOnlyValueChildMergesWithOutputProjection) { + const auto key_type = i32(); + const auto int_type = i32(); + const auto string_type = str(); + + auto table_value_b = name_col("b", string_type); + auto table_value = struct_name_col("value", {table_value_b}); + auto table_map = map_col("m", -1, {table_value}, key_type, table_value.type); + auto predicate_value_a = name_col("a", int_type); + auto predicate_value = struct_name_col("value", {predicate_value_a}); + table_map.has_predicate_access_paths = true; + table_map.predicate_children = {std::move(predicate_value)}; + set_name_identifiers(&table_map, 0); + + auto file_key = name_col("key", key_type, 0); + auto file_value_a = name_col("a", int_type, 0); + auto file_value_b = name_col("b", string_type, 1); + auto file_value = struct_name_col("value", {file_value_a, file_value_b}, 1); + auto file_map = map_col("m", -1, {file_key, file_value}, key_type, file_value.type, 0); + set_name_identifiers(&file_map, 0); + + auto full_value_type = + std::make_shared(DataTypes {int_type, string_type}, Strings {"a", "b"}); + auto full_map_type = std::make_shared(key_type, full_value_type); + auto value_expr = + struct_element(table_slot(0, 0, full_map_type, "m"), full_value_type, "value"); + auto filter_expr = int_gt(struct_element(value_expr, int_type, "a"), 5); + TableFilter filter {.conjunct = VExprContext::create_shared(filter_expr), + .global_indices = {GlobalIndex(0)}}; + + ParquetColumnMapper mapper({.mode = TableColumnMappingMode::BY_NAME}); + ASSERT_TRUE(mapper.create_mapping({table_map}, {}, {file_map}).ok()); + + FileScanRequest request; + ASSERT_TRUE(mapper.create_scan_request({filter}, {table_map}, &request).ok()); + EXPECT_TRUE(request.predicate_columns.empty()); ASSERT_EQ(request.non_predicate_columns.size(), 1); - EXPECT_EQ(request.non_predicate_columns[0].column_id(), LocalColumnId(0)); - EXPECT_TRUE(request.non_predicate_columns[0].project_all_children); - EXPECT_TRUE(request.non_predicate_columns[0].children.empty()); + const auto& projection = request.non_predicate_columns[0]; + EXPECT_EQ(projection.column_id(), LocalColumnId(0)); + ASSERT_FALSE(projection.project_all_children); + ASSERT_EQ(projection.children.size(), 1); + EXPECT_EQ(projection.children[0].local_id(), 1); + EXPECT_EQ(projection_ids(projection.children[0].children), std::vector({0, 1})); } // Scenario: when projected struct children are an in-order prefix of the file struct, the mapper can @@ -2562,8 +2678,9 @@ TEST(ColumnMapperScanRequestTest, MissingPredicateAccessPathsDoNotInferStructPro } // Scenario: Paimon projects one struct child but filters on an unprojected TIMESTAMP_LTZ(9) -// child. The filter-only file projection must retain the history-schema timestamp semantic so an -// unannotated INT96 leaf is materialized as TIMESTAMPTZ instead of DATETIMEV2. +// child. When the filter cannot be localized, its projection is merged into the deferred output; +// that merged projection must retain the history-schema timestamp semantic so an unannotated +// INT96 leaf is materialized as TIMESTAMPTZ instead of DATETIMEV2. TEST(ColumnMapperScanRequestTest, FilterOnlyNestedTimestampRetainsTableFormatSemantic) { const auto int_type = i32(); const auto ltz_type = timestamptz(9); @@ -2593,15 +2710,14 @@ TEST(ColumnMapperScanRequestTest, FilterOnlyNestedTimestampRetainsTableFormatSem FileScanRequest request; ASSERT_TRUE(mapper.create_scan_request({filter}, {projected_table_struct}, &request).ok()); - ASSERT_EQ(request.predicate_columns.size(), 1); - const auto& root_projection = request.predicate_columns[0]; - ASSERT_EQ(projection_ids(root_projection.children), std::vector({1})); + EXPECT_TRUE(request.predicate_columns.empty()); + ASSERT_EQ(request.non_predicate_columns.size(), 1); + const auto& root_projection = request.non_predicate_columns[0]; + ASSERT_EQ(projection_ids(root_projection.children), std::vector({0, 1})); const auto* ltz_projection = find_child_projection(&root_projection, 1); ASSERT_NE(ltz_projection, nullptr); ASSERT_TRUE(ltz_projection->timestamp_is_adjusted_to_utc.has_value()); EXPECT_TRUE(*ltz_projection->timestamp_is_adjusted_to_utc); - ASSERT_EQ(request.non_predicate_columns.size(), 1); - EXPECT_EQ(projection_ids(request.non_predicate_columns[0].children), std::vector({0})); } // Scenario: a filter references a top-level column that is not projected by the query; the mapper diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 683a5aa90ca8d8..f08bb2770c192e 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -10080,6 +10080,36 @@ TEST_F(NewOrcReaderTest, SargConjunctReturnsEofWhenAllStripesArePruned) { EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 400); } +TEST_F(NewOrcReaderTest, SargSafePrefixPreservesEarlierRowFilterError) { + const auto multi_stripe_file_path = (_test_dir / "sarg_safe_prefix.orc").string(); + write_multi_stripe_orc_int_file(multi_stripe_file_path); + ASSERT_EQ(get_orc_stripe_count(multi_stripe_file_path), 2); + + auto reader = create_reader_for_path(multi_stripe_file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared())); + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared(0, 5000))); + request->metadata_pruning_safe_conjunct_count = 0; + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + const Status status = reader->get_block(&block, &rows, &eof); + EXPECT_NE(status.to_string().find("synthetic row filter failure"), std::string::npos) << status; + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); +} + TEST_F(NewOrcReaderTest, CloseClearsFileLocalState) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; diff --git a/be/test/format_v2/parquet/parquet_scan_test.cpp b/be/test/format_v2/parquet/parquet_scan_test.cpp index 51d0283b172fce..635355826a0d3e 100644 --- a/be/test/format_v2/parquet/parquet_scan_test.cpp +++ b/be/test/format_v2/parquet/parquet_scan_test.cpp @@ -2094,6 +2094,48 @@ TEST(ParquetScanAdaptivePredicateTest, ThrowingNestedFunctionDisablesSelectedRow EXPECT_FALSE(throwing_comparison->root()->is_safe_to_execute_on_selected_rows()); } +TEST(ParquetScanAdaptivePredicateTest, TotalNestedAccessorsAndNullSafeEqualityAreSafe) { + const auto int_type = std::make_shared(); + const auto nullable_int_type = make_nullable(int_type); + const auto struct_type = make_nullable( + std::make_shared(DataTypes {nullable_int_type}, Strings {"value"})); + const auto array_type = std::make_shared(struct_type); + + const auto function_call = [](const std::string& name, const DataTypePtr& result_type, + VExprSPtrs children) { + TFunctionName function_name; + function_name.__set_function_name(name); + TFunction function; + function.__set_name(function_name); + TExprNode node; + node.__set_node_type(TExprNodeType::FUNCTION_CALL); + node.__set_type(result_type->to_thrift()); + node.__set_fn(function); + node.__set_num_children(cast_set(children.size())); + node.__set_is_nullable(result_type->is_nullable()); + auto expr = VectorizedFnCall::create_shared(node); + expr->set_children(std::move(children)); + return expr; + }; + + auto array_element = + function_call("element_at", struct_type, + {VSlotRef::create_shared(0, 0, -1, array_type, "items"), + VLiteral::create_shared(int_type, Field::create_field(1))}); + auto struct_element = function_call( + "struct_element", nullable_int_type, + {array_element, VLiteral::create_shared(std::make_shared(), + Field::create_field("value"))}); + auto null_safe_eq = function_call( + "eq_for_null", std::make_shared(), + {struct_element, + VLiteral::create_shared(nullable_int_type, Field::create_field(7))}); + + EXPECT_TRUE(array_element->is_safe_to_execute_on_selected_rows()); + EXPECT_TRUE(struct_element->is_safe_to_execute_on_selected_rows()); + EXPECT_TRUE(null_safe_eq->is_safe_to_execute_on_selected_rows()); +} + TEST(ParquetScanSmallFileTest, StagesOnlyBoundedHttpObjects) { using format::parquet::detail::should_stage_small_http_file; EXPECT_TRUE(should_stage_small_http_file("http://host/tiny.parquet", 512, 1024)); diff --git a/be/test/format_v2/parquet/parquet_statistics_test.cpp b/be/test/format_v2/parquet/parquet_statistics_test.cpp index 7f8fbbbcaca863..878c0cfe5e51bb 100644 --- a/be/test/format_v2/parquet/parquet_statistics_test.cpp +++ b/be/test/format_v2/parquet/parquet_statistics_test.cpp @@ -32,11 +32,13 @@ #include #include +#include "core/data_type/data_type_array.h" #include "core/data_type/data_type_date.h" #include "core/data_type/data_type_decimal.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" #include "core/data_type/data_type_string.h" +#include "core/data_type/data_type_struct.h" #include "core/data_type/data_type_time.h" #include "core/data_type/data_type_variant_v2.h" #include "core/field.h" @@ -59,8 +61,10 @@ namespace { class StatisticsMemoryFileReader final : public io::FileReader { public: - explicit StatisticsMemoryFileReader(std::vector bytes) - : _bytes(std::move(bytes)), _path("native-bloom-filter.parquet") {} + explicit StatisticsMemoryFileReader(std::vector bytes, bool fail_reads = false) + : _bytes(std::move(bytes)), + _path("native-bloom-filter.parquet"), + _fail_reads(fail_reads) {} Status close() override { _closed = true; @@ -70,10 +74,15 @@ class StatisticsMemoryFileReader final : public io::FileReader { size_t size() const override { return _bytes.size(); } bool closed() const override { return _closed; } int64_t mtime() const override { return 1; } + int read_count() const { return _read_count; } protected: Status read_at_impl(size_t offset, Slice result, size_t* bytes_read, const io::IOContext*) override { + ++_read_count; + if (_fail_reads) { + return Status::IOError("injected native Bloom read failure"); + } if (offset > _bytes.size() || result.size > _bytes.size() - offset) { return Status::IOError("native Bloom test read exceeds memory file"); } @@ -86,13 +95,19 @@ class StatisticsMemoryFileReader final : public io::FileReader { std::vector _bytes; io::Path _path; bool _closed = false; + bool _fail_reads = false; + int _read_count = 0; }; class BloomInExpr final : public VExpr { public: BloomInExpr(int column_id, DataTypePtr data_type, std::vector values) - : VExpr(std::make_shared(), false), - _slot(VSlotRef::create_shared(0, column_id, -1, std::move(data_type), "c0")), - _values(std::move(values)) {} + : BloomInExpr(VSlotRef::create_shared(0, column_id, -1, std::move(data_type), "c0"), + std::move(values)) {} + + BloomInExpr(VExprSPtr probe, std::vector values) + : VExpr(std::make_shared(), false), _values(std::move(values)) { + add_child(std::move(probe)); + } const std::string& expr_name() const override { return _expr_name; } @@ -104,15 +119,14 @@ class BloomInExpr final : public VExpr { bool can_evaluate_bloom_filter() const override { return true; } ZoneMapFilterResult evaluate_bloom_filter(const BloomFilterEvalContext& ctx) const override { - return expr_zonemap::eval_in_bloom_filter(ctx, _slot, false, _values); + return expr_zonemap::eval_in_bloom_filter(ctx, get_child(0), false, _values); } void collect_slot_column_ids(std::set& column_ids) const override { - _slot->collect_slot_column_ids(column_ids); + get_child(0)->collect_slot_column_ids(column_ids); } private: - VExprSPtr _slot; std::vector _values; const std::string _expr_name = "BloomInExpr"; }; @@ -226,6 +240,26 @@ class MetadataFloatingEqualityExpr final : public VExpr { const std::string _expr_name = "MetadataFloatingEqualityExpr"; }; +class MetadataAccessorExpr final : public VExpr { +public: + MetadataAccessorExpr(DataTypePtr result_type, VExprSPtr parent, VExprSPtr selector) + : VExpr(std::move(result_type), false) { + _fn.name.function_name = "element_at"; + add_child(std::move(parent)); + add_child(std::move(selector)); + } + + const std::string& expr_name() const override { return _expr_name; } + + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("MetadataAccessorExpr is metadata-only"); + } + +private: + const std::string _expr_name = "MetadataAccessorExpr"; +}; + class DictionaryStringInExpr final : public VExpr { public: DictionaryStringInExpr() : VExpr(std::make_shared(), false) {} @@ -446,6 +480,21 @@ VExprContextSPtr variant_path_string_gt_conjunct(std::string literal_value) { return VExprContext::create_shared(std::move(gt)); } +class UnsafeMetadataExpr final : public VExpr { +public: + UnsafeMetadataExpr() : VExpr(std::make_shared(), false) {} + + const std::string& expr_name() const override { return _expr_name; } + Status execute_column_impl(VExprContext*, const Block*, const Selector*, size_t, + ColumnPtr&) const override { + return Status::InternalError("UnsafeMetadataExpr is metadata-only"); + } + bool is_safe_to_execute_on_selected_rows() const override { return false; } + +private: + const std::string _expr_name = "UnsafeMetadataExpr"; +}; + VExprContextSPtrs bloom_conjuncts(DataTypePtr data_type, std::vector values) { return {VExprContext::create_shared( std::make_shared(0, std::move(data_type), std::move(values)))}; @@ -882,6 +931,8 @@ TEST(ParquetBloomFilterPruningTest, NativeFloatingBloomPreservesDorisEquality) { .init(segment_v2::BloomFilter::MINIMUM_BYTES, segment_v2::HashStrategyPB::XX_HASH_64) .ok()); + // These raw PLAIN bytes model an external writer; the predicate uses a different physical + // representation from the same Doris equality class. bloom_filter.add_bytes(reinterpret_cast(&stored_value), sizeof(stored_value)); ASSERT_FALSE(bloom_filter.test_bytes(reinterpret_cast(&predicate_value), sizeof(predicate_value))); @@ -916,7 +967,6 @@ TEST(ParquetBloomFilterPruningTest, NativeRowGroupKeepsDorisEqualFloatingValues) segment_v2::HashStrategyPB::XX_HASH_64) .ok()); bloom_filter.add_bytes(reinterpret_cast(&stored_value), sizeof(stored_value)); - tparquet::BloomFilterAlgorithm algorithm; algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); tparquet::BloomFilterHash hash; @@ -991,6 +1041,420 @@ TEST(ParquetBloomFilterPruningTest, NativeRowGroupKeepsDorisEqualFloatingValues) std::bit_cast(uint64_t {0x7ff8000000000002ULL})); } +TEST(ParquetBloomFilterPruningTest, NativeBloomResolvesStructAndListLeaves) { + const auto run_case = [](format::parquet::ParquetColumnSchemaKind root_kind, + int32_t predicate_value, bool path_exists, int conjunct_count, + bool expected_pruned, int expected_reads, + bool add_unsafe_barrier = false) { + auto leaf_type = std::make_shared(); + DataTypePtr root_type; + VExprSPtr selector; + if (root_kind == format::parquet::ParquetColumnSchemaKind::STRUCT) { + root_type = std::make_shared(DataTypes {leaf_type}, Strings {"value"}); + selector = VLiteral::create_shared(std::make_shared(), + Field::create_field("value")); + } else { + root_type = std::make_shared(leaf_type); + selector = VLiteral::create_shared(leaf_type, Field::create_field(1)); + } + + auto root_schema = std::make_unique(); + root_schema->kind = root_kind; + root_schema->local_id = 0; + root_schema->name = "root"; + root_schema->type = root_type; + auto leaf_schema = std::make_unique(); + leaf_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + leaf_schema->local_id = 0; + leaf_schema->name = root_kind == format::parquet::ParquetColumnSchemaKind::STRUCT + ? (path_exists ? "value" : "renamed_value") + : "element"; + leaf_schema->leaf_column_id = 0; + leaf_schema->type = leaf_type; + leaf_schema->type_descriptor.doris_type = leaf_type; + leaf_schema->type_descriptor.physical_type = tparquet::Type::INT32; + if (root_kind == format::parquet::ParquetColumnSchemaKind::LIST) { + root_schema->max_repetition_level = 1; + leaf_schema->max_repetition_level = 1; + } + root_schema->children.push_back(std::move(leaf_schema)); + + format::parquet::native::BlockSplitBloomFilter bloom_filter; + ASSERT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + const int32_t present_value = 1; + bloom_filter.add_bytes(reinterpret_cast(&present_value), + sizeof(present_value)); + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader bloom_header; + bloom_header.__set_numBytes(static_cast(bloom_filter.size())); + bloom_header.__set_algorithm(algorithm); + bloom_header.__set_hash(hash); + bloom_header.__set_compression(compression); + std::vector bloom_bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + ASSERT_TRUE(serializer.serialize(&bloom_header, &bloom_bytes).ok()); + bloom_bytes.insert(bloom_bytes.end(), bloom_filter.data(), + bloom_filter.data() + bloom_filter.size()); + + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(tparquet::Type::INT32); + column_metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + column_metadata.__set_num_values(1); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_data_page_offset(0); + column_metadata.__set_bloom_filter_offset(0); + column_metadata.__set_bloom_filter_length(static_cast(bloom_bytes.size())); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + tparquet::RowGroup row_group; + row_group.__set_columns({chunk}); + row_group.__set_total_byte_size(0); + row_group.__set_num_rows(1); + tparquet::FileMetaData metadata; + metadata.__set_version(1); + metadata.__set_num_rows(1); + metadata.__set_row_groups({row_group}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + if (add_unsafe_barrier) { + request.conjuncts.push_back( + VExprContext::create_shared(std::make_shared())); + request.metadata_pruning_safe_conjunct_count = 0; + } + for (int conjunct_idx = 0; conjunct_idx < conjunct_count; ++conjunct_idx) { + auto slot = VSlotRef::create_shared(0, 0, -1, root_type, "root"); + auto accessor = + std::make_shared(leaf_type, std::move(slot), selector); + request.conjuncts.push_back(VExprContext::create_shared(std::make_shared( + std::move(accessor), + std::vector {Field::create_field(predicate_value)}))); + } + std::vector> schema; + schema.push_back(std::move(root_schema)); + format::parquet::ParquetFileContext file_context; + auto file_reader = std::make_shared(std::move(bloom_bytes)); + file_context.native_file = file_reader; + std::vector selected_row_groups; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, true, + &pruning_stats, nullptr, nullptr, &file_context) + .ok()); + EXPECT_EQ(selected_row_groups.empty(), expected_pruned); + EXPECT_EQ(pruning_stats.filtered_row_groups_by_bloom_filter, expected_pruned ? 1 : 0); + EXPECT_EQ(file_reader->read_count(), expected_reads); + EXPECT_EQ(pruning_stats.bloom_filter_probe_attempts, expected_reads == 0 ? 0 : 1); + EXPECT_EQ(pruning_stats.bloom_filter_probe_successes, expected_reads == 0 ? 0 : 1); + EXPECT_EQ(pruning_stats.bloom_filter_conservative_fallbacks, 0); + EXPECT_EQ(pruning_stats.bloom_filter_corrupt_rejections, 0); + }; + + run_case(format::parquet::ParquetColumnSchemaKind::STRUCT, 2, true, 1, true, 2); + run_case(format::parquet::ParquetColumnSchemaKind::STRUCT, 1, true, 1, false, 2); + run_case(format::parquet::ParquetColumnSchemaKind::STRUCT, 2, false, 1, false, 0); + run_case(format::parquet::ParquetColumnSchemaKind::LIST, 2, true, 1, true, 2); + run_case(format::parquet::ParquetColumnSchemaKind::LIST, 1, true, 1, false, 2); + run_case(format::parquet::ParquetColumnSchemaKind::STRUCT, 1, true, 2, false, 2); + run_case(format::parquet::ParquetColumnSchemaKind::STRUCT, 2, true, 1, false, 0, true); +} + +TEST(ParquetBloomFilterPruningTest, NativeBloomReportsConservativeReadOutcomes) { + const auto make_valid_bloom = [] { + format::parquet::native::BlockSplitBloomFilter bloom_filter; + EXPECT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + const int32_t present_value = 1; + bloom_filter.add_bytes(reinterpret_cast(&present_value), + sizeof(present_value)); + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader header; + header.__set_numBytes(static_cast(bloom_filter.size())); + header.__set_algorithm(algorithm); + header.__set_hash(hash); + header.__set_compression(compression); + std::vector bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + EXPECT_TRUE(serializer.serialize(&header, &bytes).ok()); + bytes.insert(bytes.end(), bloom_filter.data(), bloom_filter.data() + bloom_filter.size()); + return bytes; + }; + + const auto run_case = [&](std::vector bytes, bool has_offset, bool fail_reads, + int64_t expected_corrupt_rejections) { + auto type = std::make_shared(); + auto column_schema = std::make_unique(); + column_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column_schema->local_id = 0; + column_schema->leaf_column_id = 0; + column_schema->type = type; + column_schema->type_descriptor.doris_type = type; + column_schema->type_descriptor.physical_type = tparquet::Type::INT32; + + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(tparquet::Type::INT32); + column_metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + column_metadata.__set_num_values(1); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_data_page_offset(0); + if (has_offset) { + column_metadata.__set_bloom_filter_offset(0); + column_metadata.__set_bloom_filter_length(static_cast(bytes.size())); + } + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + tparquet::RowGroup row_group; + row_group.__set_columns({chunk}); + row_group.__set_total_byte_size(0); + row_group.__set_num_rows(1); + tparquet::FileMetaData metadata; + metadata.__set_version(1); + metadata.__set_num_rows(1); + metadata.__set_row_groups({row_group}); + + auto request = request_with_bloom_conjunct(type, {Field::create_field(2)}); + std::vector> schema; + schema.push_back(std::move(column_schema)); + format::parquet::ParquetFileContext file_context; + file_context.native_file = + std::make_shared(std::move(bytes), fail_reads); + std::vector selected_row_groups; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, true, + &pruning_stats, nullptr, nullptr, &file_context) + .ok()); + EXPECT_EQ(selected_row_groups, std::vector({0})); + EXPECT_EQ(pruning_stats.bloom_filter_probe_attempts, 1); + EXPECT_EQ(pruning_stats.bloom_filter_probe_successes, 0); + EXPECT_EQ(pruning_stats.bloom_filter_conservative_fallbacks, 1); + EXPECT_EQ(pruning_stats.bloom_filter_corrupt_rejections, expected_corrupt_rejections); + }; + + run_case({}, false, false, 0); // missing metadata offset + run_case({0xff, 0xff, 0xff}, true, false, 1); // malformed header + auto truncated = make_valid_bloom(); + truncated.resize(truncated.size() - 16); + run_case(std::move(truncated), true, false, 1); // truncated payload + run_case(make_valid_bloom(), true, true, 0); // I/O failure +} + +TEST(ParquetBloomFilterPruningTest, NativeBloomPreservesFirstLogicalProbeOrder) { + const auto make_bloom = [] { + format::parquet::native::BlockSplitBloomFilter bloom_filter; + EXPECT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + const int32_t present_value = 1; + bloom_filter.add_bytes(reinterpret_cast(&present_value), + sizeof(present_value)); + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader header; + header.__set_numBytes(static_cast(bloom_filter.size())); + header.__set_algorithm(algorithm); + header.__set_hash(hash); + header.__set_compression(compression); + std::vector bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + EXPECT_TRUE(serializer.serialize(&header, &bytes).ok()); + bytes.insert(bytes.end(), bloom_filter.data(), bloom_filter.data() + bloom_filter.size()); + return bytes; + }; + const auto bloom = make_bloom(); + std::vector file_bytes = bloom; + file_bytes.insert(file_bytes.end(), bloom.begin(), bloom.end()); + + const auto type = std::make_shared(); + std::vector> schema; + for (int local_id = 0; local_id < 2; ++local_id) { + auto column = std::make_unique(); + column->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + column->local_id = local_id; + column->leaf_column_id = 1 - local_id; + column->type = type; + column->type_descriptor.doris_type = type; + column->type_descriptor.physical_type = tparquet::Type::INT32; + schema.push_back(std::move(column)); + } + + std::vector chunks; + for (int leaf_id = 0; leaf_id < 2; ++leaf_id) { + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(tparquet::Type::INT32); + column_metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + column_metadata.__set_num_values(1); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_data_page_offset(0); + column_metadata.__set_bloom_filter_offset(leaf_id * bloom.size()); + column_metadata.__set_bloom_filter_length(static_cast(bloom.size())); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + chunks.push_back(std::move(chunk)); + } + tparquet::RowGroup row_group; + row_group.__set_columns(std::move(chunks)); + row_group.__set_total_byte_size(0); + row_group.__set_num_rows(1); + tparquet::FileMetaData metadata; + metadata.__set_version(1); + metadata.__set_num_rows(1); + metadata.__set_row_groups({row_group}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.local_positions.emplace(format::LocalColumnId(1), format::LocalIndex(1)); + request.conjuncts = {VExprContext::create_shared(std::make_shared( + 1, type, std::vector {Field::create_field(2)})), + VExprContext::create_shared(std::make_shared( + 0, type, std::vector {Field::create_field(1)}))}; + + format::parquet::ParquetFileContext file_context; + auto file_reader = std::make_shared(std::move(file_bytes)); + file_context.native_file = file_reader; + std::vector selected_row_groups; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, true, + &pruning_stats, nullptr, nullptr, &file_context) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + EXPECT_EQ(file_reader->read_count(), 2); + EXPECT_EQ(pruning_stats.bloom_filter_probe_attempts, 1); + EXPECT_EQ(pruning_stats.bloom_filter_probe_successes, 1); +} + +TEST(ParquetBloomFilterPruningTest, NativeBloomPreservesOrderAcrossNestedAndDirectProbes) { + const auto make_bloom = [] { + format::parquet::native::BlockSplitBloomFilter bloom_filter; + EXPECT_TRUE(bloom_filter + .init(segment_v2::BloomFilter::MINIMUM_BYTES, + segment_v2::HashStrategyPB::XX_HASH_64) + .ok()); + const int32_t present_value = 1; + bloom_filter.add_bytes(reinterpret_cast(&present_value), + sizeof(present_value)); + tparquet::BloomFilterAlgorithm algorithm; + algorithm.__set_BLOCK(tparquet::SplitBlockAlgorithm()); + tparquet::BloomFilterHash hash; + hash.__set_XXHASH(tparquet::XxHash()); + tparquet::BloomFilterCompression compression; + compression.__set_UNCOMPRESSED(tparquet::Uncompressed()); + tparquet::BloomFilterHeader header; + header.__set_numBytes(static_cast(bloom_filter.size())); + header.__set_algorithm(algorithm); + header.__set_hash(hash); + header.__set_compression(compression); + std::vector bytes; + ThriftSerializer serializer(/*compact=*/true, 64); + EXPECT_TRUE(serializer.serialize(&header, &bytes).ok()); + bytes.insert(bytes.end(), bloom_filter.data(), bloom_filter.data() + bloom_filter.size()); + return bytes; + }; + const auto bloom = make_bloom(); + std::vector file_bytes = bloom; + file_bytes.insert(file_bytes.end(), bloom.begin(), bloom.end()); + + const auto int_type = std::make_shared(); + const auto struct_type = + std::make_shared(DataTypes {int_type}, Strings {"value"}); + auto direct_schema = std::make_unique(); + direct_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + direct_schema->local_id = 0; + direct_schema->leaf_column_id = 1; + direct_schema->type = int_type; + direct_schema->type_descriptor.doris_type = int_type; + direct_schema->type_descriptor.physical_type = tparquet::Type::INT32; + + auto struct_schema = std::make_unique(); + struct_schema->kind = format::parquet::ParquetColumnSchemaKind::STRUCT; + struct_schema->local_id = 1; + struct_schema->name = "nested"; + struct_schema->type = struct_type; + auto nested_leaf_schema = std::make_unique(); + nested_leaf_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE; + nested_leaf_schema->local_id = 0; + nested_leaf_schema->name = "value"; + nested_leaf_schema->leaf_column_id = 0; + nested_leaf_schema->type = int_type; + nested_leaf_schema->type_descriptor.doris_type = int_type; + nested_leaf_schema->type_descriptor.physical_type = tparquet::Type::INT32; + struct_schema->children.push_back(std::move(nested_leaf_schema)); + + std::vector chunks; + for (int leaf_id = 0; leaf_id < 2; ++leaf_id) { + tparquet::ColumnMetaData column_metadata; + column_metadata.__set_type(tparquet::Type::INT32); + column_metadata.__set_codec(tparquet::CompressionCodec::UNCOMPRESSED); + column_metadata.__set_num_values(1); + column_metadata.__set_total_compressed_size(0); + column_metadata.__set_data_page_offset(0); + column_metadata.__set_bloom_filter_offset(leaf_id * bloom.size()); + column_metadata.__set_bloom_filter_length(static_cast(bloom.size())); + tparquet::ColumnChunk chunk; + chunk.__set_meta_data(column_metadata); + chunks.push_back(std::move(chunk)); + } + tparquet::RowGroup row_group; + row_group.__set_columns(std::move(chunks)); + row_group.__set_total_byte_size(0); + row_group.__set_num_rows(1); + tparquet::FileMetaData metadata; + metadata.__set_version(1); + metadata.__set_num_rows(1); + metadata.__set_row_groups({row_group}); + + format::FileScanRequest request; + request.local_positions.emplace(format::LocalColumnId(0), format::LocalIndex(0)); + request.local_positions.emplace(format::LocalColumnId(1), format::LocalIndex(1)); + auto nested_slot = VSlotRef::create_shared(0, 1, -1, struct_type, "nested"); + auto selector = VLiteral::create_shared(std::make_shared(), + Field::create_field("value")); + auto nested_accessor = + std::make_shared(int_type, std::move(nested_slot), selector); + request.conjuncts = { + VExprContext::create_shared(std::make_shared( + std::move(nested_accessor), + std::vector {Field::create_field(2)})), + VExprContext::create_shared(std::make_shared( + 0, int_type, std::vector {Field::create_field(1)}))}; + + std::vector> schema; + schema.push_back(std::move(direct_schema)); + schema.push_back(std::move(struct_schema)); + format::parquet::ParquetFileContext file_context; + auto file_reader = std::make_shared(std::move(file_bytes)); + file_context.native_file = file_reader; + std::vector selected_row_groups; + format::parquet::ParquetPruningStats pruning_stats; + ASSERT_TRUE(format::parquet::select_row_groups_by_metadata( + metadata, schema, request, nullptr, &selected_row_groups, true, + &pruning_stats, nullptr, nullptr, &file_context) + .ok()); + EXPECT_TRUE(selected_row_groups.empty()); + EXPECT_EQ(file_reader->read_count(), 2); +} + TEST(ParquetBloomFilterPruningTest, NativeRowGroupKeepsPresentUint32AboveInt32Max) { auto column_schema = std::make_unique(uint32_parquet_bloom_schema()); diff --git a/be/test/format_v2/table_reader_test.cpp b/be/test/format_v2/table_reader_test.cpp index b4a90f29030c17..800f0040dd6f6d 100644 --- a/be/test/format_v2/table_reader_test.cpp +++ b/be/test/format_v2/table_reader_test.cpp @@ -280,6 +280,28 @@ VExprSPtr table_struct_int32_child_greater_than_expr(int slot_id, int column_id, return predicate; } +VExprSPtr table_array_struct_int_greater_than_expr(int column_id, const std::string& column_name, + const DataTypePtr& array_type, + const DataTypePtr& element_type, + const DataTypePtr& accessor_type, + const std::string& child_name, int32_t value) { + const auto int_type = std::make_shared(); + auto array_element = table_function_expr("element_at", element_type, {array_type, int_type}); + array_element->add_child(VSlotRef::create_shared(0, column_id, -1, array_type, column_name)); + array_element->add_child(table_int32_literal(1)); + auto struct_element = table_function_expr("element_at", accessor_type, + {element_type, std::make_shared()}); + struct_element->add_child(std::move(array_element)); + struct_element->add_child(VLiteral::create_shared( + std::make_shared(), Field::create_field(child_name))); + auto greater_than = table_function_expr("gt", make_nullable(std::make_shared()), + {accessor_type, int_type}, TExprNodeType::BINARY_PRED, + TExprOpcode::GT); + greater_than->add_child(std::move(struct_element)); + greater_than->add_child(table_int32_literal(value)); + return greater_than; +} + VExprSPtr runtime_filter_wrapper_expr(VExprSPtr impl) { TExprNode node; node.__set_node_type(TExprNodeType::SLOT_REF); @@ -701,6 +723,84 @@ void write_list_struct_parquet_file(const std::string& file_path) { writer_builder.build())); } +void write_nullable_list_struct_parquet_file(const std::string& file_path, bool first_a_is_null) { + auto struct_type = arrow::struct_( + {arrow::field("a", arrow::int32(), true), arrow::field("b", arrow::int32(), true)}); + std::vector> field_builders; + field_builders.push_back(std::make_shared()); + field_builders.push_back(std::make_shared()); + auto struct_builder = std::make_shared( + struct_type, arrow::default_memory_pool(), std::move(field_builders)); + auto list_type = arrow::list(arrow::field("element", struct_type, true)); + arrow::ListBuilder builder(arrow::default_memory_pool(), struct_builder, list_type); + auto* a_builder = assert_cast(struct_builder->field_builder(0)); + auto* b_builder = assert_cast(struct_builder->field_builder(1)); + + EXPECT_TRUE(builder.Append().ok()); + EXPECT_TRUE(struct_builder->Append().ok()); + if (first_a_is_null) { + EXPECT_TRUE(a_builder->AppendNull().ok()); + } else { + EXPECT_TRUE(a_builder->Append(0).ok()); + } + EXPECT_TRUE(b_builder->AppendNull().ok()); + + EXPECT_TRUE(builder.Append().ok()); + EXPECT_TRUE(struct_builder->Append().ok()); + EXPECT_TRUE(a_builder->Append(20).ok()); + EXPECT_TRUE(b_builder->Append(1).ok()); + + auto schema = arrow::schema({arrow::field("items", list_type, false)}); + auto table = arrow::Table::Make(schema, {finish_array(&builder)}); + auto file_result = arrow::io::FileOutputStream::Open(file_path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + std::shared_ptr out = *file_result; + + ::parquet::WriterProperties::Builder writer_builder; + writer_builder.version(::parquet::ParquetVersion::PARQUET_2_6); + writer_builder.data_page_version(::parquet::ParquetDataPageVersion::V2); + writer_builder.compression(::parquet::Compression::UNCOMPRESSED); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), out, 2, + writer_builder.build())); +} + +void write_narrowing_list_struct_parquet_file(const std::string& file_path) { + auto struct_type = arrow::struct_( + {arrow::field("a", arrow::int32(), false), arrow::field("b", arrow::int64(), false)}); + std::vector> field_builders; + field_builders.push_back(std::make_shared()); + field_builders.push_back(std::make_shared()); + auto struct_builder = std::make_shared( + struct_type, arrow::default_memory_pool(), std::move(field_builders)); + auto list_type = arrow::list(arrow::field("element", struct_type, true)); + arrow::ListBuilder builder(arrow::default_memory_pool(), struct_builder, list_type); + auto* a_builder = assert_cast(struct_builder->field_builder(0)); + auto* b_builder = assert_cast(struct_builder->field_builder(1)); + + EXPECT_TRUE(builder.Append().ok()); + EXPECT_TRUE(struct_builder->Append().ok()); + EXPECT_TRUE(a_builder->Append(0).ok()); + EXPECT_TRUE(b_builder->Append(2147483648LL).ok()); + + EXPECT_TRUE(builder.Append().ok()); + EXPECT_TRUE(struct_builder->Append().ok()); + EXPECT_TRUE(a_builder->Append(20).ok()); + EXPECT_TRUE(b_builder->Append(1).ok()); + + auto schema = arrow::schema({arrow::field("items", list_type, false)}); + auto table = arrow::Table::Make(schema, {finish_array(&builder)}); + auto file_result = arrow::io::FileOutputStream::Open(file_path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + std::shared_ptr out = *file_result; + + ::parquet::WriterProperties::Builder writer_builder; + writer_builder.version(::parquet::ParquetVersion::PARQUET_2_6); + writer_builder.data_page_version(::parquet::ParquetDataPageVersion::V2); + writer_builder.compression(::parquet::Compression::UNCOMPRESSED); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), out, 2, + writer_builder.build())); +} + void write_map_struct_parquet_file(const std::string& file_path) { auto key_builder = std::make_shared(); auto struct_type = arrow::struct_( @@ -1514,7 +1614,13 @@ TEST(TableReaderTest, UnsafePredicateStaysOnScannerPath) { FakeTableReader reader(file_schema, fake_state); ASSERT_TRUE(reader.init({ .projected_columns = projected_columns, - .conjuncts = {prepared_conjunct(&state, unsafe_predicate)}, + .conjuncts = + { + prepared_conjunct(&state, unsafe_predicate), + prepared_conjunct(&state, + table_int32_greater_than_expr( + 0, 0, 10)), + }, .format = FileFormat::PARQUET, .scan_params = nullptr, .io_ctx = nullptr, @@ -1530,7 +1636,8 @@ TEST(TableReaderTest, UnsafePredicateStaysOnScannerPath) { bool eos = false; ASSERT_TRUE(reader.get_block(&block, &eos).ok()); ASSERT_NE(fake_state->last_request, nullptr); - EXPECT_TRUE(fake_state->last_request->conjuncts.empty()); + ASSERT_EQ(fake_state->last_request->conjuncts.size(), 1); + EXPECT_EQ(fake_state->last_request->metadata_pruning_safe_conjunct_count, 0); EXPECT_FALSE(predicate_executed); ASSERT_TRUE(reader.close().ok()); } @@ -4090,6 +4197,238 @@ TEST(TableReaderTest, ProjectedListStructReadsSelectedElementChild) { std::filesystem::remove_all(test_dir); } +TEST(TableReaderTest, NestedEqualityReachesParquetBloomProbe) { + const auto test_dir = + std::filesystem::temp_directory_path() / "doris_table_reader_nested_bloom_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "split.parquet").string(); + write_list_struct_parquet_file(file_path); + + const auto int_type = std::make_shared(); + const auto nullable_int_type = make_nullable(int_type); + auto element_type = make_nullable(std::make_shared( + DataTypes {nullable_int_type, nullable_int_type}, Strings {"a", "b"})); + auto list_column = make_table_column(100, "xs", std::make_shared(element_type)); + std::vector projected_columns = {list_column}; + set_name_identifiers(&projected_columns); + + const auto root_type = projected_columns[0].type; + auto array_element = table_function_expr("element_at", element_type, {root_type, int_type}); + array_element->add_child(VSlotRef::create_shared(0, 0, -1, root_type, "xs")); + array_element->add_child(table_int32_literal(1)); + auto struct_element = table_function_expr("element_at", nullable_int_type, + {element_type, std::make_shared()}); + struct_element->add_child(std::move(array_element)); + struct_element->add_child(VLiteral::create_shared(std::make_shared(), + Field::create_field("a"))); + auto equality = table_function_expr("eq", make_nullable(std::make_shared()), + {nullable_int_type, int_type}, TExprNodeType::BINARY_PRED, + TExprOpcode::EQ); + equality->add_child(std::move(struct_element)); + equality->add_child(table_int32_literal(10)); + + RuntimeProfile profile("profile"); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + TableReader reader; + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {prepared_conjunct(&state, equality)}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = &profile, + }) + .ok()); + ASSERT_TRUE(reader.prepare_split(build_split_options(file_path)).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + const auto status = reader.get_block(&block, &eos); + ASSERT_TRUE(status.ok()) << status; + auto* attempts = profile.get_counter("BloomFilterProbeAttempts"); + auto* fallbacks = profile.get_counter("BloomFilterConservativeFallbacks"); + ASSERT_NE(attempts, nullptr); + ASSERT_NE(fallbacks, nullptr); + EXPECT_EQ(attempts->value(), 1); + EXPECT_EQ(fallbacks->value(), 1); + + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(TableReaderTest, ArrayAccessorDoesNotHideRequiredUnreferencedSibling) { + const auto test_dir = std::filesystem::temp_directory_path() / + "doris_table_reader_array_sibling_nullability_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "split.parquet").string(); + write_nullable_list_struct_parquet_file(file_path, false); + + const auto int_type = std::make_shared(); + const auto nullable_int_type = make_nullable(int_type); + const auto table_element = make_nullable(std::make_shared( + DataTypes {nullable_int_type, int_type}, Strings {"a", "b"})); + const auto table_array = std::make_shared(table_element); + auto a_child = make_table_column(0, "a", nullable_int_type); + ColumnDefinition b_child; + b_child.name = "b"; + b_child.type = int_type; + ColumnDefinition element_child; + element_child.name = "element"; + element_child.type = table_element; + element_child.children = {std::move(a_child), std::move(b_child)}; + ColumnDefinition list_column; + list_column.name = "items"; + list_column.type = make_nullable(table_array); + list_column.children = {std::move(element_child)}; + std::vector projected_columns = {std::move(list_column)}; + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + const auto root_type = projected_columns[0].type; + auto predicate = table_array_struct_int_greater_than_expr(0, "items", root_type, table_element, + nullable_int_type, "a", 5); + TableReader reader; + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {prepared_conjunct(&state, predicate)}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + ASSERT_TRUE(reader.prepare_split(build_split_options(file_path)).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + const auto status = reader.get_block(&block, &eos); + EXPECT_FALSE(status.ok()) << "A file-local filter must not hide a required sibling NULL"; + + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(TableReaderTest, ArrayAccessorDoesNotHideLossyUnreferencedSiblingConversion) { + const auto test_dir = std::filesystem::temp_directory_path() / + "doris_table_reader_array_sibling_conversion_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "split.parquet").string(); + write_narrowing_list_struct_parquet_file(file_path); + + const auto int_type = std::make_shared(); + const auto table_element = make_nullable( + std::make_shared(DataTypes {int_type, int_type}, Strings {"a", "b"})); + const auto table_array = std::make_shared(table_element); + auto a_child = make_table_column(0, "a", int_type); + auto b_child = make_table_column(1, "b", int_type); + ColumnDefinition element_child; + element_child.name = "element"; + element_child.type = table_element; + element_child.children = {std::move(a_child), std::move(b_child)}; + ColumnDefinition list_column; + list_column.name = "items"; + list_column.type = make_nullable(table_array); + list_column.children = {std::move(element_child)}; + std::vector projected_columns = {std::move(list_column)}; + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + const auto root_type = projected_columns[0].type; + auto predicate = table_array_struct_int_greater_than_expr(0, "items", root_type, table_element, + make_nullable(int_type), "a", 5); + TableReader reader; + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {prepared_conjunct(&state, predicate)}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + ASSERT_TRUE(reader.prepare_split(build_split_options(file_path)).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + const auto status = reader.get_block(&block, &eos); + EXPECT_FALSE(status.ok()) << "A file-local filter must not hide a lossy sibling conversion"; + + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all(test_dir); +} + +TEST(TableReaderTest, RejectedArrayAccessorFencesLaterDefaultConstantPruning) { + const auto test_dir = std::filesystem::temp_directory_path() / + "doris_table_reader_array_constant_pruning_barrier_test"; + std::filesystem::remove_all(test_dir); + std::filesystem::create_directories(test_dir); + const auto file_path = (test_dir / "split.parquet").string(); + write_nullable_list_struct_parquet_file(file_path, true); + + const auto int_type = std::make_shared(); + const auto nullable_int_type = make_nullable(int_type); + const auto table_element = make_nullable(std::make_shared( + DataTypes {int_type, nullable_int_type}, Strings {"a", "b"})); + const auto table_array = std::make_shared(table_element); + ColumnDefinition a_child; + a_child.name = "a"; + a_child.type = int_type; + auto b_child = make_table_column(1, "b", nullable_int_type); + ColumnDefinition element_child; + element_child.name = "element"; + element_child.type = table_element; + element_child.children = {std::move(a_child), std::move(b_child)}; + ColumnDefinition list_column; + list_column.name = "items"; + list_column.type = make_nullable(table_array); + list_column.children = {std::move(element_child)}; + auto missing_default = make_table_column(101, "z", nullable_int_type); + missing_default.default_expr = VExprContext::create_shared( + VLiteral::create_shared(nullable_int_type, Field::create_field(0))); + std::vector projected_columns = {std::move(list_column), + std::move(missing_default)}; + set_name_identifiers(&projected_columns); + + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + const auto root_type = projected_columns[0].type; + auto accessor_predicate = table_array_struct_int_greater_than_expr( + 0, "items", root_type, table_element, nullable_int_type, "a", 10); + auto default_predicate = table_function_expr( + "eq", make_nullable(std::make_shared()), {nullable_int_type, int_type}, + TExprNodeType::BINARY_PRED, TExprOpcode::EQ); + default_predicate->add_child(VSlotRef::create_shared(1, 1, -1, nullable_int_type, "z")); + default_predicate->add_child(table_int32_literal(7)); + + TableReader reader; + ASSERT_TRUE(reader.init({ + .projected_columns = projected_columns, + .conjuncts = {prepared_conjunct(&state, accessor_predicate), + prepared_conjunct(&state, default_predicate)}, + .format = FileFormat::PARQUET, + .scan_params = nullptr, + .io_ctx = nullptr, + .runtime_state = &state, + .scanner_profile = nullptr, + }) + .ok()); + ASSERT_TRUE(reader.prepare_split(build_split_options(file_path)).ok()); + + Block block = build_table_block(projected_columns); + bool eos = false; + const auto status = reader.get_block(&block, &eos); + EXPECT_FALSE(status.ok()) + << "A later false default predicate must not bypass required-child validation"; + + ASSERT_TRUE(reader.close().ok()); + std::filesystem::remove_all(test_dir); +} + TEST(TableReaderTest, ProjectedListStructReordersRenamedAndMissingElementChildren) { const auto test_dir = std::filesystem::temp_directory_path() / "doris_table_reader_list_schema_evolution_test";