HDDS-14754. Add ZDU upgrade action to clean up leftover state

[errose28]

What changes were proposed in this pull request?

OM Prepare for upgrade will be a no-op once ZDU is supported. There may be leftover state in the DB and VERSION file indicating that the OM was put in prepare mode in the prior version. We should remove this state in an upgrade action tied to OM version 100 (ZDU support) which will run when the OM is finalized. Similarly, SCM will no longer have a finalization-in-progress marker in its DB. This should be completely cleared out by the old version when SCM finishes finalizing, but we can also add an upgrade action to ensure it is removed.

What is the link to the Apache JIRA

HDDS-14754

How was this patch tested?

Unit tests for the upgrade actions were added, but they are difficult to test in a real cluster during the upgrade.

The existing mock upgrade actions for ZDU that were used in testing have been removed since we have concrete actions to test with now.

[dombizita]

Thanks @errose28, overall looks good to me, one minor question commented.

[dombizita]

I believe this is really unlikely to happen, but this is different from SCM, to throw this exception. If this happens the marker file deletion won't happen until finalization called again? Will the finalization fail?

[errose28]

Ideally OM and SCM should crash if there is an error during finalization. We can't have the ratis group in a mixed finalization state. The existing tests only check that an exception is propagated out of the version manager, but it looks like neither component will actually crash. I'll add this to the PR.

[errose28]

It turns out OM and SCM were crashing if upgrade actions failed, because an UpgradeException is mapped to an INTERNAL_ERROR result code which will crash each service. I added tests for this, although they look different due to the differences in how Ratis requests are processed on each service.

DNs are expected to keep retrying finalization as instructed by SCM, since they do not need to match with a quorum.

[errose28]

I also added and tested that failure to load an upgrade action will throw an unchecked exception. Previously it was just logging an error which could result in a misconfigured upgrade action not running.

[dombizita]

The CI failure is newly added test, otherwise the latest change make sense, once the test is sorted out

[ERROR] org.apache.hadoop.hdds.scm.ha.TestSCMStateMachine.testUpgradeExceptionDuringApplyTerminates -- Time elapsed: 1.352 s <<< ERROR!
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hdds.scm.ha.SCMHAManager.getRatisServer()" because the return value of "org.apache.hadoop.hdds.scm.server.StorageContainerManager.getScmHAManager()" is null
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.close(SCMStateMachine.java:566)
	at org.apache.hadoop.hdds.scm.ha.TestSCMStateMachine.testUpgradeExceptionDuringApplyTerminates(TestSCMStateMachine.java:102)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

[ERROR] org.apache.hadoop.hdds.scm.ha.TestSCMStateMachine.testRatisEventsRecording -- Time elapsed: 0.067 s <<< ERROR!
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.hdds.scm.ha.SCMHAManager.getRatisServer()" because the return value of "org.apache.hadoop.hdds.scm.server.StorageContainerManager.getScmHAManager()" is null
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.close(SCMStateMachine.java:566)
	at org.apache.hadoop.hdds.scm.ha.TestSCMStateMachine.testRatisEventsRecording(TestSCMStateMachine.java:56)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

[errose28]

I had wrapped the SCMStateMachine in try-with-resources to address a warning. Although the SCMStateMachine instance is a real object, its contents are mocked. This was causing the NPE when SCMStateMachine#close was invoked, since it is trying to do clean up on the mocks. I added a comment explaining this. Mocking the nested fields so close succeeds is fragile.

[dombizita]

Thanks for updating, sounds good!