Iceberg ADD COLUMN of nested types fails commit-unknown recovery with Column already exists

[DimensionWieldr]

Describe the bug

Iceberg ALTER retry after a “commit state unknown” catalog response does not detect that an ADD COLUMN of a nested type (Array / Map / Tuple) already landed. The retry then fails with Column already exists.

Primitive ADD COLUMN … Nullable(String) recovers correctly. Nested types do not.

Found while testing PR #2157. Split out of this comment.

To Reproduce

On 26.6.2 (PR #2157), REST catalog, failpoint iceberg_alter_catalog_commit_reported_as_failed (ONCE):

SET allow_insert_into_iceberg = 1, enable_nullable_tuple_type = 1;
ALTER TABLE datalake.`namespace.table` ADD COLUMN tuple_col Nullable(Tuple(Int32, String));

The failpoint applies the catalog commit then returns false (Iceberg “commit state unknown”).

Expected behavior

Retry should see the nested column already in the schema and succeed, same as primitive ADD COLUMN.

Actual behaviour

Code: 36. DB::Exception: Column tuple_col already exists. (BAD_ARGUMENTS)

The catalog already committed the column. Recovery missed it.

Key information

• Project Antalya 26.6.2 (PR Antalya 26:6 Multiple fixes for Iceberg operations #2157)
• Iceberg catalog: REST

Additional context

isAddColumnApplied rebuilds the expected Iceberg type via getIcebergType(type, last_column_id) from the current last-column-id (already advanced by the successful commit).

For primitives that is a string ("string", "long", …) and the compare succeeds.

For Array / Map / Tuple, getIcebergType allocates new nested field ids. icebergTypesEqual stringifies JSON, so element-id / key-id / value-id / struct field ids do not match → false → retry → Column already exists.

[subkanthi]

Unable to reproduce in 26.6 @DimensionWieldr

 SET allow_insert_into_iceberg = 1;
SET enable_nullable_tuple_type = 1;

SET allow_insert_into_iceberg = 1

Query id: 85e1ee06-cc8e-42ac-a2d2-de6dce7a3a55

Ok.

0 rows in set. Elapsed: 0.002 sec. 


SET enable_nullable_tuple_type = 1

Query id: 344f2f8c-797d-4cbb-9380-dcbf57955916

Ok.

0 rows in set. Elapsed: 0.001 sec. 

Ubuntu-2404-noble-amd64-base :) SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed;


SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed

Query id: a31e3de3-21a9-4b8c-a165-27bde86e1089

Ok.

0 rows in set. Elapsed: 0.002 sec. 

Ubuntu-2404-noble-amd64-base :) SELECT name, type, enabled 
FROM system.fail_points 
WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed';

SELECT
    name,
    type,
    enabled
FROM system.fail_points
WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed'

Query id: df71199d-d00c-44ec-a310-f38a26833580

   ┌─name────────────────────────────────────────────┬─type─┬─enabled─┐
1. │ iceberg_alter_catalog_commit_reported_as_failed │ once │       1 │
   └─────────────────────────────────────────────────┴──────┴─────────┘

1 row in set. Elapsed: 0.002 sec. 

Ubuntu-2404-noble-amd64-base :) alter table ice.`nyc.taxis_p_by_day` add column tuple_col_4 Nullable(Tuple(Int32, String));

ALTER TABLE ice.`nyc.taxis_p_by_day`
    (ADD COLUMN `tuple_col_4` Nullable(Tuple(
        Int32,
        String)))

Query id: 95284155-b76c-4e37-8352-b1838461b145

Ok.

0 rows in set. Elapsed: 0.182 sec. 

[DimensionWieldr]

Unable to reproduce in 26.6

@subkanthi Try Nullable(Tuple(Array(Int32))).

It looks like this was partially fixed. Nullable(Tuple(Int32, String)) no longer fails, but I was able to reproduce it with Nullable(Tuple(Array(Int32))).

[subkanthi]

@DimensionWieldr not able to reproduce.

alter table ice.`nyc.taxis_p_by_day` add column tuple_col_15 Nullable(Tuple(Array(Int32)));

ALTER TABLE ice.`nyc.taxis_p_by_day`
    (ADD COLUMN `tuple_col_15` Nullable(Tuple(Array(Int32))))

Query id: 0705e80c-eae5-43e9-b7d4-debb5418e490

Ok.


show create table ice.`nyc.taxis_p_by_day`;

SHOW CREATE TABLE ice.`nyc.taxis_p_by_day`

Query id: 6847a708-4d56-494b-8859-7f1200058a6b

   ┌─statement─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`nyc.taxis_p_by_day`                                                                                    ↴│
   │↳(                                                                                                                        ↴│
   │↳    `VendorID` Nullable(Int32),                                                                                          ↴│
   │↳    `tpep_pickup_datetime` Nullable(DateTime64(6)),                                                                      ↴│
   │↳    `tpep_dropoff_datetime` Nullable(DateTime64(6)),                                                                     ↴│
   │↳    `passenger_count` Nullable(Int64),                                                                                   ↴│
   │↳    `trip_distance` Nullable(Float64),                                                                                   ↴│
   │↳    `RatecodeID` Nullable(Int64),                                                                                        ↴│
   │↳    `store_and_fwd_flag` Nullable(String),                                                                               ↴│
   │↳    `PULocationID` Nullable(Int64),                                                                                      ↴│
   │↳    `DOLocationID` Nullable(Int32),                                                                                      ↴│
   │↳    `payment_type` Nullable(Int64),                                                                                      ↴│
   │↳    `fare_amount` Nullable(Float64),                                                                                     ↴│
   │↳    `trip_extra3` Nullable(Float64),                                                                                     ↴│
   │↳    `mta_tax` Nullable(Float64),                                                                                         ↴│
   │↳    `tip_amount` Nullable(Float64),                                                                                      ↴│
   │↳    `tolls_amount` Nullable(Float64),                                                                                    ↴│
   │↳    `improvement_surcharge` Nullable(Float64),                                                                           ↴│
   │↳    `total_amount` Nullable(Float64),                                                                                    ↴│
   │↳    `congestion_surcharge` Nullable(Float64),                                                                            ↴│
   │↳    `Airport_fee` Nullable(Float64),                                                                                     ↴│
   │↳    `cbd_congestion_fee` Nullable(Float64),                                                                              ↴│
   │↳    `tuple_col` Tuple(`1` Int32, `2` String),                                                                            ↴│
   │↳    `tuple_col_2` Tuple(`1` Int32, `2` String),                                                                          ↴│
   │↳    `tuple_col_3` Tuple(`1` Int32, `2` String),                                                                          ↴│
   │↳    `tuple_col_4` Tuple(`1` Int32, `2` String),                                                                          ↴│
   │↳    `tuple_col_5` Tuple(`1` Array(Int32)),                                                                               ↴│
   │↳    `tuple_col_15` Tuple(`1` Array(Int32))                                                                               ↴│
   │↳)                                                                                                                        ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/nyc/taxis_p_by_day/metadata/v2.metadata.json/metadata/v3.metadata.json/') │
   └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.030 sec. 

[DimensionWieldr]

@subkanthi It works fine if the catalog commit is acknowledged, but on the off chance that the client is told the commit failed (which is reproducible using a system failpoint), the retry is supposed to recognize the nested column that already landed. See SQL below:

SET allow_insert_into_iceberg = 1;
SET enable_nullable_tuple_type = 1;

SYSTEM ENABLE FAILPOINT iceberg_alter_catalog_commit_reported_as_failed;

SELECT name, type, enabled
FROM system.fail_points
WHERE name = 'iceberg_alter_catalog_commit_reported_as_failed';
-- enabled must be 1 here

ALTER TABLE ice.`<namespace>.<table>`
    ADD COLUMN array_col Nullable(Tuple(Array(Int32)));