HDDS-14645. Speed up TestBlockDeletingService

[adoroszlai]

What changes were proposed in this pull request?

PeriodicalTask runs in BackgroundService's executor, and wants to lock BackgroundService before spawning background tasks. BackgroundService.shutdown should not awaitTermination of the executor while holding the same lock: if shutdown is initiated while the task is before the critical section, the task cannot complete, causing awaitTermination to timeout after 1 minute.

"BlockDeletingService#0" #223 [893291] daemon prio=5 os_prio=0 cpu=1.69ms elapsed=42.58s allocated=49192B defined_classes=0 tid=0x0000721a69ac9790 nid=893291 waiting for monitor entry  [0x0000721a4082d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.run(BackgroundService.java:165)

...

"main" #1 [891422] prio=5 os_prio=0 cpu=2925.67ms elapsed=247.75s allocated=431M defined_classes=6151 tid=0x0000721a6801c170 nid=891422 waiting on condition  [0x0000721a6f7fa000]
	...
	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(java.base@21.0.10/ThreadPoolExecutor.java:1475)
	at org.apache.hadoop.hdds.utils.BackgroundService.shutdown(BackgroundService.java:198)
	- locked <0x000000060b86a0d0> (a org.apache.hadoop.ozone.container.common.impl.BlockDeletingService)

https://issues.apache.org/jira/browse/HDDS-14645

How was this patch tested?

Before:

Tests run: 72, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 325.3 s -- in org.apache.hadoop.ozone.container.common.TestBlockDeletingService

After:

Tests run: 72, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 25.42 s -- in org.apache.hadoop.ozone.container.common.TestBlockDeletingService

https://github.com/adoroszlai/ozone/actions/runs/26556849091

[SaketaChalamchala]

Thanks for the patch @adoroszlai. LGTM

[jojochuang]

is it necessary to synchronize here and assign current = exec? I mean the reference is essentially immutable after initialization.

[adoroszlai]

Spotbugs may complain about inconsistent synchronization. That's why shutdown was changed to synchronized in HDDS-13320 (#8677 (comment), 57f39f9).

[adoroszlai]

Thanks @jojochuang, @SaketaChalamchala for the review.