HDDS-14876. Make snapshot diff progress more granular by rhalm · Pull Request #10901 · apache/ozone

rhalm · 2026-07-29T18:42:09Z

What changes were proposed in this pull request?

Improves snapshot diff progress granularity by letting clients track which stage a diff job is in and adding more informative logs. Main changes:

Exposes substatus and progress percentage (when available) to clients in snapshot diff responses, and fixes minor bugs causing incorrect values:
- Wires subStatus and progressPercent through the proto layer and CLI output (with a minor formatting fix)
- Adds a new substatus for the FSO path resolution stage (PATH_RESOLUTION_FSO)
- Fixes progress percentage carrying over between stages by resetting progress to 0 at the start
- Fixes a bug where the substatus would remain stuck on SST_FILE_DELTA_* instead of switching to OBJECT_ID_MAP_GEN_* during object ID map generation
Improves logging for snapshot diff processing:
- Adds one summary log per stage with item count and elapsed time
- Adds a warn log when DAG-based delta computation returns no result and falls back to full diff

Before / After

Example CLI output when polling an in-progress snapshot diff before this change:

$ ozone sh snapshot diff --get-report /vol/bucket snap1 snap2
Snapshot diff job is IN_PROGRESS. Please retry after 60000 ms.

$ ozone sh snapshot diff --get-report --json /vol/bucket snap1 snap2
{
  "snapshotDiffReport" : {
    ...
  },
  "jobStatus" : "IN_PROGRESS",
  "waitTimeInMs" : 60000
}

And after:

$ ozone sh snapshot diff --get-report /vol/bucket snap1 snap2
Snapshot diff job is IN_PROGRESS. Please retry after 60000 ms.
SubStatus : OBJECT_ID_MAP_GEN_FSO
Keys Processed Estimated Percentage : 25.0

$ ozone sh snapshot diff --get-report --json /vol/bucket snap1 snap2
{
  "snapshotDiffReport" : {
    ...
  },
  "jobStatus" : "IN_PROGRESS",
  "waitTimeInMs" : 60000,
  "subStatus" : "OBJECT_ID_MAP_GEN_FSO",
  "progressPercent" : 25.0
}

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14876

How was this patch tested?

Updated and added unit tests
Manually verified the changes on a local deployment
CI pass: https://github.com/rhalm/ozone/actions/runs/30462314731

Copilot

Pull request overview

This PR improves snapshot diff progress observability across OM, client, and CLI by adding stage-level substatus reporting and (when applicable) progress percentage, plus additional per-stage logging to help operators understand runtime behavior.

Changes:

Extends the SnapshotDiffResponse wire format to include subStatus and optional progressPercent, and wires these through OM server handling, client-side translation, and CLI JSON/text rendering.
Adds a new FSO-specific stage (PATH_RESOLUTION_FSO) and refines snapshot diff stage transitions/logging inside SnapshotDiffManager.
Adds/updates unit tests to validate proto serialization behavior and SnapshotDiffResponse rendering.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/protocolPB/TestOzoneManagerRequestHandler.java	Adds tests asserting subStatus/progress are serialized into the OM protocol response (or omitted when subStatus is null).
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDiffManager.java	Updates snapshot diff manager tests and adds coverage for in-progress responses including subStatus/progress details and new PATH_RESOLUTION_FSO state.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java	Serializes SnapshotDiffResponse subStatus/progress into the protobuf response.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java	Adds PATH_RESOLUTION_FSO stage, improves stage logging, and resets progress at the start of object-ID-map generation.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/CompositeDeltaDiffComputer.java	Adds a warning when DAG-based delta computation yields no result and the code falls back to full diff.
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto	Adds PATH_RESOLUTION_FSO enum value and new optional `progressPercent` field to SnapshotDiffResponse.
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/snapshot/TestSnapshotDiffResponse.java	Expands response-string rendering tests for map-gen vs non-map-gen substatuses and null subStatus behavior.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotDiffResponse.java	Adds getters, refines toString formatting, introduces SubStatus.hasProgress(), and adds PATH_RESOLUTION_FSO.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java	Deserializes subStatus/progress from protobuf into the client SnapshotDiffResponse.
hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotDiffHandler.java	Includes subStatus and (only when applicable) progressPercent in CLI JSON output.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

SaketaChalamchala · 2026-07-30T20:48:15Z

@@ -1312,6 +1334,10 @@ void addToObjectIdMap(Table<String, ? extends WithParentObjectId> fsTable,
        }
      }
    }


In most cases Total keysProcessed < totalEstimatedKeysToProcess. So, the progress may never reach 100% inside the loop. You can set it after the loop to indicate that the stage has finished.

Suggested change

}

}

updateProgress(jobKey, 1.0)

One edge case appears to remain: when deltaFiles.isEmpty(), the method returns after resetting progress to 0.0, so it never reaches the terminal 1.0 update or emits a stage summary. An empty stage could go directly to 100% and log that it was skipped:

- updateProgress(jobKey, 0.0); if (deltaFiles.isEmpty()) { + updateProgress(jobKey, 1.0); + if (LOG.isDebugEnabled()) { + LOG.debug("Skipped object ID map generation for table '{}' because there are no delta files, jobId: {}", + fsTable.getName(), jobId); + } return; } + updateProgress(jobKey, 0.0);

SaketaChalamchala · 2026-07-30T20:52:26Z

          },
          () -> {
            if (bucketLayout.isFileSystemOptimized()) {
+              recordActivity(jobKey, PATH_RESOLUTION_FSO);


Progress capture for PATH_RESOLUTION_FSO and DIFF_REPORT_GEN stages is not consistent with the ObjectID generation stages. Could you also explore how we can capture the progress of these stages as well?

Good idea, opened HDDS-16058 as a follow-up

smengcl

Thanks @rhalm . The substatus and progress propagation looks good overall. I left two small comments in SnapshotDiffManager. (The second comment is above)

smengcl · 2026-07-31T22:51:39Z

        }
      }
    }
+    updateProgress(jobKey, 1.0);


Could we keep the debug log in the same 0-100 unit as the persisted and RPC value? updateProgress(1.0) stores 100.0, but the helper logs the raw 1.0 with a % suffix. The final update therefore logs 1.0%, and a 10% checkpoint logs 0.1%. This appears to be a one-line behavioral fix:

- LOG.debug("Completed processing {}% of keys for snapshot diff job {}", pct, jobKey); + LOG.debug("Completed processing {}% of keys for snapshot diff job {}", + snapshotDiffJob.getKeysProcessedPct(), jobKey);

HDDS-14876. Make snapshot diff progress more granular

e301dbc

Copilot AI review requested due to automatic review settings July 29, 2026 18:42

Copilot started reviewing on behalf of rhalm July 29, 2026 18:47 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

SaketaChalamchala added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jul 29, 2026

SaketaChalamchala requested review from SaketaChalamchala, jojochuang, sadanand48 and smengcl July 29, 2026 18:59

sadanand48 reviewed Jul 30, 2026

View reviewed changes

Comment thread ...one/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java Outdated

Address review comments

c48eb5a

rhalm requested a review from sadanand48 July 30, 2026 13:58

SaketaChalamchala reviewed Jul 30, 2026

View reviewed changes

Comment thread ...one/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java Outdated

SaketaChalamchala reviewed Jul 30, 2026

View reviewed changes

Comment thread ...one/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java Outdated

SaketaChalamchala reviewed Jul 30, 2026

View reviewed changes

Comment thread hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

Split OBJECT_ID_MAP_GEN_FSO substatus, address remaining comments

8840eef

rhalm requested a review from SaketaChalamchala July 31, 2026 16:13

smengcl reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-14876. Make snapshot diff progress more granular - #10901

HDDS-14876. Make snapshot diff progress more granular#10901
rhalm wants to merge 3 commits into
apache:masterfrom
rhalm:HDDS-14876

rhalm commented Jul 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SaketaChalamchala Jul 30, 2026

Uh oh!

rhalm Jul 31, 2026

Uh oh!

smengcl Jul 31, 2026

Uh oh!

SaketaChalamchala Jul 30, 2026 •

edited

Loading

Uh oh!

rhalm Jul 31, 2026

Uh oh!

Uh oh!

smengcl left a comment •

edited

Loading

Uh oh!

smengcl Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

rhalm commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SaketaChalamchala Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

rhalm Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

smengcl Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhalm Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smengcl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smengcl Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rhalm commented Jul 29, 2026 •

edited

Loading

SaketaChalamchala Jul 30, 2026 •

edited

Loading

smengcl left a comment •

edited

Loading