HDDS-16057. OM bootstrap fails when checkpoint install is rejected during BOOTSTRAPPING. - #10917
Conversation
…ring BOOTSTRAPPING.
smengcl
left a comment
smengcl
left a comment
There was a problem hiding this comment.
lgtm. Just one test issue above
Change-Id: Idb44c34e74470f118d4adef71d68318a4f5116f3
…essage Wait for bootstrapFuture to finish and assert the new OM joined the Ratis peer list. Update installSnapshotFromLeader abort log per review. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I9581d000c2868cfb843263d775a6d2910290ca6d
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
Addressed review feedback in 22f07bc: wait for bootstrapFuture to complete, assert the new OM joined the Ratis peer list, and updated the installSnapshotFromLeader abort log message.
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I1dcf1ba282663bf6c1e1806fa49de066d4809f70
|
Addressed review feedback in 22f07bc and e2f7d3:
Checkstyle passes locally. Marked ready for review. |
There was a problem hiding this comment.
Pull request overview
This PR fixes an OM HA bootstrap stall where Ratis may request snapshot/checkpoint installation while the joining OM is still in BOOTSTRAPPING (not yet RUNNING), particularly when the leader has already purged early Ratis logs and forces checkpoint install.
Changes:
- Allow
installSnapshotFromLeader()to proceed when OM isBOOTSTRAPPING(in addition toRUNNING). - Add an integration regression test that bootstraps a 3rd OM into a 2-OM cluster with purged logs and asserts checkpoint download begins during
BOOTSTRAPPING.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java | Broaden snapshot-install guard to permit installs during BOOTSTRAPPING. |
| hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java | Add integration test covering bootstrap + leader log purge + checkpoint install start. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
aryangupta1998
left a comment
aryangupta1998
left a comment
There was a problem hiding this comment.
Thanks for the fix, LGTM. Pending CI
ChenSammi
left a comment
ChenSammi
left a comment
There was a problem hiding this comment.
Thanks @sadanand48 . The change LGTM.
|
Thanks @smengcl @jojochuang @aryangupta1998 @ChenSammi for the reviews |
| IOUtils.closeQuietly(client); | ||
| if (cluster != null) { | ||
| cluster.shutdown(); | ||
| } |
There was a problem hiding this comment.
This test does not use the cluster started in @BeforeEach, but starts its own. This adds unnecessary test execution time. Please move out to a separate class.
There was a problem hiding this comment.
Thanks.
Before:
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.9 s -- in org.apache.hadoop.ozone.om.TestOMRatisSnapshots
After:
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 213.8 s -- in org.apache.hadoop.ozone.om.TestOMRatisSnapshots
What changes were proposed in this pull request?
Fixes OM bootstrap failure when the leader has purged Ratis logs and the new OM must install a checkpoint while still in BOOTSTRAPPING.
When a 3rd OM is added with --bootstrap, Ratis can notify the new node to install a snapshot during setConfiguration staging. At that point omState is BOOTSTRAPPING, not RUNNING. installSnapshotFromLeader() rejected install because it only allowed isRunning(), so checkpoint download never started. Bootstrap then stalled in a retry loop (SNAPSHOT_UNAVAILABLE) and the new OM never joined the ring.
This change allows checkpoint install during BOOTSTRAPPING as well as RUNNING. Install is still rejected during INITIALIZED and other pre-ready states (HDDS-15103). The existing testInstallSnapshot test hook is unchanged.
Changes:
OzoneManager.java: Replace isRunning() with isRunningOrBootstrapping() in the installSnapshotFromLeader() guard.
TestOMRatisSnapshots.java: Add testBootstrapInstallSnapshotDuringBootstrapping — 2-OM cluster with purged logs, bootstrap a 3rd OM, assert install is accepted and v2 checkpoint download starts, then shut down before later install stages.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16057
How was this patch tested?
integration test