Skip to content

Core: Make Metadata Timestamp Clock Skew Configurable #17286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chandrasekhar-188k wants to merge 1 commit into
base: main
Choose a base branch
from
+33 −4
Open

Core: Make Metadata Timestamp Clock Skew Configurable #17286

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 33 additions & 4 deletions core/src/main/java/org/apache/iceberg/TableMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,16 @@

private static final long ONE_MINUTE = TimeUnit.MINUTES.toMillis(1);

/**
* System property that controls the allowed clock-skew tolerance (in milliseconds) used when
* validating that snapshot-log and metadata-log entries are monotonically increasing in time.
* Defaults to 1 minute if unset or invalid.
*/
static final String CLOCK_SKEW_TOLERANCE_MS_PROPERTY =
"iceberg.table-metadata.clock-skew-tolerance-ms";

private static final long DEFAULT_CLOCK_SKEW_TOLERANCE_MS = ONE_MINUTE;
Comment on lines +74 to +77

@singhpk234 singhpk234 Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whats the motivation for doing this ?

heads up in v4 we need to have monotonic timestamps

cc @stevenzwu

@chandrasekhar-188k chandrasekhar-188k Jul 19, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in deployments where multiple compute engines (for example, Spark, Flink, Trino, or custom applications) operate on the same Iceberg table from different clusters, temporary clock skew greater than one minute can occur despite otherwise healthy deployments. In such cases, valid commits may fail because the metadata timestamp appears to be too far in the future relative to the committing process. we have observed this issue in some of our deployments.


public static TableMetadata newTableMetadata(
Schema schema,
PartitionSpec spec,
Expand Down Expand Up @@ -368,12 +378,13 @@

// row lineage
this.nextRowId = nextRowId;
long clockSkewToleranceMillis = getClockSkewToleranceMillis();

HistoryEntry last = null;
for (HistoryEntry logEntry : snapshotLog) {
if (last != null) {
Preconditions.checkArgument(
(logEntry.timestampMillis() - last.timestampMillis()) >= -ONE_MINUTE,
(logEntry.timestampMillis() - last.timestampMillis()) >= -clockSkewToleranceMillis,
"[BUG] Expected sorted snapshot log entries.");
}
last = logEntry;
Expand All @@ -382,7 +393,7 @@
Preconditions.checkArgument(
// commits can happen concurrently from different machines.
// A tolerance helps us avoid failure for small clock skew
lastUpdatedMillis - last.timestampMillis() >= -ONE_MINUTE,
lastUpdatedMillis - last.timestampMillis() >= -clockSkewToleranceMillis,
"Invalid update timestamp %s: before last snapshot log entry at %s",
lastUpdatedMillis,
last.timestampMillis());
Expand All @@ -394,7 +405,8 @@
Preconditions.checkArgument(
// commits can happen concurrently from different machines.
// A tolerance helps us avoid failure for small clock skew
(metadataEntry.timestampMillis() - previous.timestampMillis()) >= -ONE_MINUTE,
(metadataEntry.timestampMillis() - previous.timestampMillis())
>= -clockSkewToleranceMillis,
"[BUG] Expected sorted previous metadata log entries.");
}
previous = metadataEntry;
Expand All @@ -404,7 +416,7 @@
Preconditions.checkArgument(
// commits can happen concurrently from different machines.
// A tolerance helps us avoid failure for small clock skew
lastUpdatedMillis - previous.timestampMillis >= -ONE_MINUTE,
lastUpdatedMillis - previous.timestampMillis >= -clockSkewToleranceMillis,
"Invalid update timestamp %s: before the latest metadata log entry timestamp %s",
lastUpdatedMillis,
previous.timestampMillis);
Expand Down Expand Up @@ -1842,11 +1854,11 @@
Set<Long> addedSnapshotIds = Sets.newHashSet();
Set<Long> intermediateSnapshotIds = Sets.newHashSet();
for (MetadataUpdate update : changes) {
if (update instanceof MetadataUpdate.AddSnapshot) {

Check warning on line 1857 in core/src/main/java/org/apache/iceberg/TableMetadata.java

View workflow job for this annotation

GitHub Actions / build-checks (17, pull_request) 

[PatternMatchingInstanceof] This code can be simplified to use a pattern-matching instanceof.

Check warning on line 1857 in core/src/main/java/org/apache/iceberg/TableMetadata.java

View workflow job for this annotation

GitHub Actions / check-runtime-deps 

[PatternMatchingInstanceof] This code can be simplified to use a pattern-matching instanceof.
// adds must always come before set current snapshot
MetadataUpdate.AddSnapshot addSnapshot = (MetadataUpdate.AddSnapshot) update;
addedSnapshotIds.add(addSnapshot.snapshot().snapshotId());
} else if (update instanceof MetadataUpdate.SetSnapshotRef) {

Check warning on line 1861 in core/src/main/java/org/apache/iceberg/TableMetadata.java

View workflow job for this annotation

GitHub Actions / build-checks (17, pull_request) 

[PatternMatchingInstanceof] This code can be simplified to use a pattern-matching instanceof.

Check warning on line 1861 in core/src/main/java/org/apache/iceberg/TableMetadata.java

View workflow job for this annotation

GitHub Actions / check-runtime-deps 

[PatternMatchingInstanceof] This code can be simplified to use a pattern-matching instanceof.
MetadataUpdate.SetSnapshotRef setRef = (MetadataUpdate.SetSnapshotRef) update;
long snapshotId = setRef.snapshotId();
if (addedSnapshotIds.contains(snapshotId)
Expand Down Expand Up @@ -1920,4 +1932,21 @@
return changes.stream().filter(updateClass::isInstance).map(updateClass::cast);
}
}

private static long getClockSkewToleranceMillis() {
String configured = System.getProperty(CLOCK_SKEW_TOLERANCE_MS_PROPERTY);
if (configured == null) {
return DEFAULT_CLOCK_SKEW_TOLERANCE_MS;
}

try {
long value = Long.parseLong(configured.trim());
if (value > 0) {
return value;
}
return DEFAULT_CLOCK_SKEW_TOLERANCE_MS;
} catch (NumberFormatException e) {
return DEFAULT_CLOCK_SKEW_TOLERANCE_MS;
}
}
}
Loading