Skip to content

[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compaction Test #2265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rionmonster wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compaction Test #2265

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ...iceberg/src/test/java/org/apache/fluss/lake/iceberg/maintenance/IcebergRewriteITCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,9 @@ void testLogTableCompaction() throws Exception {
t1, t1Bucket, ++i, true, Collections.singletonList(row(1, "v1"))));
checkFileStatusInIcebergTable(t1, 3, false);

// Ensure tiering job has fully processed the previous writes
assertReplicaStatus(t1Bucket, i);
Copy link
Contributor

@luoyuxia luoyuxia Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious about it. If the files count already is 3 in iceberg, shouldn't it also mean that the tiering has already tiered all records since we only write 3 records.

Copy link
Contributor Author

@rionmonster rionmonster Dec 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

I agree that it’s a bit odd. The additional replica assertion seems to alleviate any race conditions within this specific test case that we were seeing arise within the failing CI instances (at least based on my local testing to reproduce the issue).

It may simply be a testing artifact as opposed to a legitimate issue, but if it doesn’t seem like a fix, we can explore a few other avenues. It seemed like one of those common race conditions, so leveraging an existing function to help seemed like a decent approach. Happy to explore some additional avenues though, if we feel that we need a bit more exhaustive checks.

Copy link
Contributor

@luoyuxia luoyuxia Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rionmonster I tried to reporduce it in my local. But fails. I think it'll be better to explore the root cause for I'm afraid of it is caused by another critical issue.

Copy link
Contributor Author

@rionmonster rionmonster Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

Good to know! It seemed to be pretty consistent regarding pass/fail on my end, but clearly there’s something else at play.

I’ll do some more exploration and see what I find. Thanks for the feedback!

Copy link
Contributor Author

@rionmonster rionmonster Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

I've been digging into this a bit further and it seems there's some disparity between the actual files being written and retrieving the latest offsets for those (specifically after any types of asynchronous operations, such as compaction, etc.). I think we may need some mechanism to improve the consistency, at least within the bounds of the tests.

I wrote a little monitor to run through several iterations of the tests to see what the state of various bits looked like during each iteration. Here's a sample of those:

Successful Test (Successfully Performs Compaction as Expected)
[MONITOR] log_table_33 - TableBucket{tableId=32, bucket=0}

IcebergFiles IcebergSnapshotId LakeSnapshotId ReplicaLakeSnapId ReplicaLakeLogEndOff Timestamp
0 -1 -1 -1 -1 1767113560800
0 -1 -1 -1 -1 1767113561046
0 -1 -1 -1 -1 1767113561294
0 -1 -1 -1 -1 1767113561547
0 -1 -1 -1 -1 1767113561795
0 -1 -1 -1 -1 1767113562045
0 -1 -1 -1 -1 1767113562299
0 -1 -1 -1 -1 1767113562547
0 -1 -1 -1 -1 1767113562797
0 -1 -1 -1 -1 1767113563054
0 -1 -1 -1 -1 1767113563298
0 -1 -1 -1 -1 1767113563546
0 -1 -1 -1 -1 1767113563796
0 -1 -1 -1 -1 1767113564048
0 -1 -1 -1 -1 1767113564295
0 -1 -1 -1 -1 1767113564544
0 -1 -1 -1 -1 1767113564796
0 -1 -1 -1 -1 1767113565047
0 -1 -1 -1 -1 1767113565295
0 -1 -1 -1 -1 1767113565545
0 -1 -1 -1 -1 1767113565797
0 -1 -1 -1 -1 1767113566046
0 -1 -1 -1 -1 1767113566294
0 -1 -1 -1 -1 1767113566545
0 -1 -1 -1 -1 1767113566795
0 -1 -1 -1 -1 1767113567047
0 -1 -1 -1 -1 1767113567296
0 -1 -1 -1 -1 1767113567547
0 -1 -1 -1 -1 1767113567793
0 -1 -1 -1 -1 1767113568048
0 -1 -1 -1 -1 1767113568296
0 -1 -1 -1 -1 1767113568546
0 -1 -1 -1 -1 1767113568797
0 -1 -1 -1 -1 1767113569048
0 -1 -1 -1 -1 1767113569297
0 -1 -1 -1 -1 1767113569547
0 -1 -1 -1 -1 1767113569796
0 -1 -1 -1 -1 1767113570045
0 -1 -1 -1 -1 1767113570298
0 -1 -1 -1 -1 1767113570547
0 -1 -1 -1 -1 1767113570799
0 -1 -1 -1 -1 1767113571045
0 -1 -1 -1 -1 1767113571294
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113571547
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113571799
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113572047
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113572301
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113572549
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113572800
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113573044
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113573301
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113573549
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113573798
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113574048
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113574298
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113574551
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113574801
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113575047
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113575300
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113575549
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113575796
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113576051
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113576300
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113576547
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113576800
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113577048
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113577304
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113577551
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113577800
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113578046
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113578300
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113578550
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113578800
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113579050
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113579301
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113579548
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113579799
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113580048
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113580298
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113580550
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113580798
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113581056
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113581297
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113581550
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113581801
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113582049
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113582301
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113582548
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113582798
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113583046
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113583298
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113583548
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113583800
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113584054
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113584303
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113584550
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113584804
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113585051
1 5443797100773076340 5443797100773076340 5443797100773076340 1 1767113585294
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113585558
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113585810
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113586062
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113586310
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113586559
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113586805
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113587059
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113587307
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113587559
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113587810
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113588056
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113588310
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113588558
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113588809
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113589059
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113589310
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113589558
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113589813
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113590057
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113590309
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113590557
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113590807
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113591058
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113591311
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113591557
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113591811
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113592058
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113592310
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113592560
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113592809
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113593058
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113593310
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113593559
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113593810
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113594057
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113594312
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113594560
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113594808
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113595063
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113595309
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113595559
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113595810
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113596060
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113596316
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113596558
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113596808
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113597060
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113597312
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113597557
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113597809
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113598057
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113598307
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113598558
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113598817
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113599062
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113599311
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113599562
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113599809
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113600058
2 5171148310110237921 5171148310110237921 5443797100773076340 2 1767113600306
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113600560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113600811
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113601060
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113601309
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113601560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113601807
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113602059
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113602307
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113602560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113602809
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113603057
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113603307
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113603561
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113603806
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113604057
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113604307
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113604553
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113604812
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113605056
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113605311
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113605559
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113605809
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113606058
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113606308
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113606559
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113606811
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113607060
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113607314
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113607557
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113607808
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113608059
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113608310
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113608561
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113608811
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113609056
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113609309
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113609560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113609810
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113610060
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113610312
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113610560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113610811
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113611061
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113611308
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113611554
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113611808
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113612060
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113612311
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113612561
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113612811
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113613062
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113613311
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113613561
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113613808
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113614061
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113614310
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113614560
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113614809
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113615059
3 5182733673261799288 5182733673261799288 5443797100773076340 3 1767113615307
2 2575057976625237982 2575057976625237982 5443797100773076340 4 1767113615557
2 2575057976625237982 2575057976625237982 5443797100773076340 4 1767113615808

Failing Test (File appeared to never be properly written before expected offset)
[MONITOR] log_table_34 - TableBucket{tableId=33, bucket=0}

IcebergFiles IcebergSnapshotId LakeSnapshotId ReplicaLakeSnapId ReplicaLakeLogEndOff Timestamp
0 -1 -1 -1 -1 1767113616327
0 -1 -1 -1 -1 1767113616573
0 -1 -1 -1 -1 1767113616823
0 -1 -1 -1 -1 1767113617072
0 -1 -1 -1 -1 1767113617326
0 -1 -1 -1 -1 1767113617569
0 -1 -1 -1 -1 1767113617826
0 -1 -1 -1 -1 1767113618068
0 -1 -1 -1 -1 1767113618326
0 -1 -1 -1 -1 1767113618574
0 -1 -1 -1 -1 1767113618823
0 -1 -1 -1 -1 1767113619071
0 -1 -1 -1 -1 1767113619321
0 -1 -1 -1 -1 1767113619572
0 -1 -1 -1 -1 1767113619825
0 -1 -1 -1 -1 1767113620069
0 -1 -1 -1 -1 1767113620322
0 -1 -1 -1 -1 1767113620576
0 -1 -1 -1 -1 1767113620822
0 -1 -1 -1 -1 1767113621069
0 -1 -1 -1 -1 1767113621324
0 -1 -1 -1 -1 1767113621572
0 -1 -1 -1 -1 1767113621826
0 -1 -1 -1 -1 1767113622072
0 -1 -1 -1 -1 1767113622321
0 -1 -1 -1 -1 1767113622574
0 -1 -1 -1 -1 1767113622823
0 -1 -1 -1 -1 1767113623069
0 -1 -1 -1 -1 1767113623324
0 -1 -1 -1 -1 1767113623573
0 -1 -1 -1 -1 1767113623821
0 -1 -1 -1 -1 1767113624070
0 -1 -1 -1 -1 1767113624320
0 -1 -1 -1 -1 1767113624569
0 -1 -1 -1 -1 1767113624821
0 -1 -1 -1 -1 1767113625073
0 -1 -1 -1 -1 1767113625324
0 -1 -1 -1 -1 1767113625573
0 -1 -1 -1 -1 1767113625824
0 -1 -1 -1 -1 1767113626071
0 -1 -1 -1 -1 1767113626319
0 -1 -1 -1 -1 1767113626573
0 -1 -1 -1 -1 1767113626824
0 -1 -1 -1 -1 1767113627070
0 -1 -1 -1 -1 1767113627324
0 -1 -1 -1 -1 1767113627574
0 -1 -1 -1 -1 1767113627818
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113628077
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113628325
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113628580
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113628828
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113629073
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113629330
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113629577
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113629829
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113630076
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113630324
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113630578
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113630828
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113631075
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113631326
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113631580
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113631827
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113632076
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113632326
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113632577
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113632826
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113633074
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113633325
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113633578
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113633829
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113634077
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113634327
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113634573
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113634826
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113635072
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113635326
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113635582
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113635828
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113636072
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113636327
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113636576
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113636826
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113637076
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113637327
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113637576
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113637829
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113638076
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113638326
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113638580
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113638827
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113639077
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113639327
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113639575
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113639829
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113640073
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113640326
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113640575
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113640827
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113641071
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113641323
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113641576
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113641834
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113642073
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113642330
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113642574
1 6752171114485570420 6752171114485570420 6752171114485570420 1 1767113642820
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113643084
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113643343
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113643586
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113643835
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113644086
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113644333
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113644589
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113644833
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113645090
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113645339
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113645588
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113645838
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113646087
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113646340
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113646591
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113646842
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113647089
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113647340
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113647593
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113647847
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113648089
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113648337
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113648593
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113648839
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113649086
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113649339
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113649593
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113649843
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113650094
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113650345
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113650597
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113650842
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113651089
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113651339
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113859238
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113859394
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113859633
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113859872
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113860134
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113860382
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113860628
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113860879
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861129
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861384
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861627
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113861882
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862135
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862381
2 7273969972093574431 7273969972093574431 7273969972093574431 2 1767113862633 
[ASSERTION FAILURE] Expected offset 3 but got 2 for bucket TableBucket{tableId=33, bucket=0}
  Replica Lake Snapshot ID: 7273969972093574431
  Current State:
    Iceberg Files: 2
    Iceberg Snapshot ID: 7273969972093574431
    Lake Snapshot ID (from admin): 7273969972093574431
    Replica Lake Snapshot ID: 7273969972093574431
    Replica Lake Log End Offset: 2

I'm not sure if this is more of an artifact of the tests themselves or a legitimate issue. Any thoughts? Happy to continue digging. It feels like a race-condition due to the inconsistency, either on writing the files or reading stale offsets from the data lake directly. I'm sure in a real-world environment, this might be tolerable (as I suspect it's just a minor latency spike which would eventually resolve), but in the confines of a test, it's flaky.

Copy link
Contributor Author

@rionmonster rionmonster Jan 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia

I added some additional checks around these to try and better identify the underlying issues. It doesn't look like the files are being written or recognized by Iceberg, which I suspect is simply a race-condition from the additional diagnostics:

[ASSERTION FAILURE] Expected offset 3 but got 2 for bucket TableBucket{tableId=21, bucket=0}
  Replica Lake Snapshot ID: 6090342046054561204
  Current State:
    Iceberg Files: 2
    Iceberg Snapshot ID: 6090342046054561204
    Lake Snapshot ID (from admin): 6090342046054561204
    Replica Lake Snapshot ID: 6090342046054561204
    Replica Lake Log End Offset: 2 (expected: 3, diff: 1)

Said differently, because the replica offset is off (by one) it means either the tiering job hasn't processed all writes yet (check monitor output above), the offset update is lagging behind file commits (race condition), or the tiering job itself may be stuck or running slowly.

I think we could help resolve this in two ways, which I'm testing through now and will update the PR:

  1. Introduce a waitForTieringToProcess helper function that would check to verify that the tiering had been executed before continuing to validate the replica statuses.
  2. Reduce the test-specific freshness and/or tiering intervals (e.g., POLL_TIERING_TABLE_INTERVAL) from their current values of 500ms to 100ms.

Thoughts? I’d probably err on the side of applying both since it’s a flaky test and the more consistency, the better.


// Write should trigger compaction now since the current data file count is greater or
// equal MIN_FILES_TO_COMPACT
flussRows.addAll(
Expand Down