CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change by beobal · Pull Request #4979 · apache/cassandra

beobal · 2026-07-30T10:05:02Z

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:

<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

…change

…ed before allowing startup to proceed

…og processing is suspended

…iscovery

If two nodes restart with new broadcast addresses concurrently and one is a CMS member, the non-member may send a TCM_COMMIT_REQ containing its STARTUP transform to the CMS member's old address. If this is no longer reachable, the sender should time out quickly so it can resend to another CMS member, or try to discover the new CMS address(es).

…CEP-21, pre V9 version

…tructing a CMSLookup

…p messages

…ter upgrade

… response

krummas · 2026-07-31T13:01:55Z

        public CandidateIterator(Collection<InetAddressAndPort> initialContacts, boolean checkLive)
        {
            this.candidates = new ConcurrentLinkedDeque<>(initialContacts);
+            this.elements = new HashSet<>(initialContacts);


I think this needs to be a concurrent set

krummas · 2026-07-31T13:28:42Z

+        // recalculate CMS placement using endpoint mappings from lookup
+        cmsDataPlacement = calculateCMSPlacement(placements, cmsMembership, proposedLookup);
+
+        // We shouldn't need to null out the other lazily initialized CMS fields when we refresh the CMSLookup as the


should we assert that they are null?

krummas · 2026-07-31T14:11:57Z

                logger.debug("Received {} responses. {} required.", responses.size(), requiredResponses);
                if (responses.size() >= requiredResponses)
                {
                    isDone = true;


markDone() instead to get the logging?

krummas · 2026-07-31T14:32:20Z

+        this(ClusterMetadata.current().metadataIdentifier, MessagingService::instance);
+    }
+
+    public SurveyRequestHandler(int metadataId, Supplier<MessageDelivery> messaging)


nit, @ VisibleForTesting

krummas · 2026-07-31T14:43:22Z

+        //   During first initialization, placements are hardcoded to the local address after the PRE_INITIALIZE_CMS is
+        //   committed so that the subsequent INITIALIZE_CMS can be.
+        //
+        // * Some changes to the addresses of the CMS members have been changed and whilst this is known (i.e. has been


nit, Some addresses of the CMS members have been changed...

krummas · 2026-07-31T14:46:59Z

+            @Override
+            public InetAddressAndPort endpoint(NodeId id)
+            {
+                if (overrides.containsKey(id))


nit, avoid looking up the key twice:

Pair<InetAddressAndPort, InetAddressAndPort> override = overrides.get(id); if (override != null) return override.right; return lookup.endpoint(id); ...

krummas · 2026-08-03T08:01:15Z


            Result.Success success = result.success();
            Directory directory = directorySupplier.get();
+            EndpointLookup lookup = lookupSupplier.get();


is there a possibility that we get "mismatching" directory/lookup here since we call suppliers twice? Maybe the supplier should be something like Supplier<Pair<Directory, EndpointLookup>> to make sure it's backed by the same ClusterMetadata?

krummas · 2026-08-03T08:24:10Z

+                        {
+                            logger.info("Waiting for pending CMS address changes to complete {}", replayed.cmsLookup);
+                            TimeUnit.MILLISECONDS.sleep(1000);
+                            replayed = ClusterMetadata.current();


Is there a chance we miss a Startup and get stuck forever here? Should we do an explicit log fetch periodically? Maybe not every second, but once a minute or so, just to make sure?

I think there is a slight chance of that happening, but I don't think we need to wait for the commit/enactment anyway. By this point we must have already successfully committed our own address change, so we can just exit and allow any other inflight changes to be committed/propagated/enacted in the background.

krummas · 2026-08-03T11:47:07Z

    TCM_DISCOVER_REQ       (813, P0, rpcTimeout,      INTERNAL_METADATA,    () -> NoPayload.serializer,                         () -> Discovery.instance.requestHandler,    TCM_DISCOVER_RSP       ),
    TCM_FETCH_PEER_LOG_RSP (818, P0, shortTimeout,    FETCH_METADATA,       MessageSerializers::logStateSerializer,             RESPONSE_HANDLER                                 ),
    TCM_FETCH_PEER_LOG_REQ (819, P0, rpcTimeout,      FETCH_METADATA,       () -> FetchPeerLog.serializer,                      () -> FetchPeerLog.Handler.instance,        TCM_FETCH_PEER_LOG_RSP ),
+    TCM_DISCOVER_PEERS_RSP (820, P0, rpcTimeout,      INTERNAL_METADATA,    () -> Discovery.serializer,                         () -> ResponseVerbHandler.instance                                 ),


looks like this should now be RESPONSE_HANDLER

beobal and others added 27 commits July 29, 2026 14:58

[CASSANDRA-20476] Add dtest for CMS rediscovery

b030fc1

[CASSANDRA-20476] Introduce CMSLookup

e67e83f

[CASSANDRA-20476] Perform rediscovery of CMS at startup if addresses …

ccd89e4

…change

[CASSANDRA-20476] Attempt to wait for all address changes to be enact…

b51da1b

…ed before allowing startup to proceed

[CASSANDRA-20476] Some minor logging additions

8ae56a4

[CASSANDRA-20476] Make sure to start messaging service

c18ba33

[CASSANDRA-20476] Don't attempt to catch up from peers or CMS while l…

3ffee0d

…og processing is suspended

[CASSANDRA-20476] Remove unused CMSLookup.addressMap

d918d3e

[CASSANDRA-20476] Remove unnecessary map retrievals

46e6890

[CASSANDRA-20476] Make CMSLookup.overrides an ImmutableMap

7747808

[CASSANDRA-20476] When CMSLookup is RETIRED, remove log listener

12bda58

[CASSANDRA-20476] Include seeds in initial rediscovery candidates

571df66

[CASSANDRA-20476] Make number of discovery rounds configurable

1ecc2cd

[CASSANDRA-20476] Improve comment about target quorum size during red…

f43f1cb

…iscovery

[CASSANDRA-20476] Simplify Discovery.state to a boolean

1a39507

[CASSANDRA-20476] Include node ids in CMS description

65d7fc6

[CASSANDRA-20476] Don't attempt rediscovery if upgrading from a post …

0945944

…CEP-21, pre V9 version

[CASSANDRA-20476] Fix ip changes while still in gossip mode

0048124

[CASSANDRA-20476] Don't return CMS_ONLY Discovery responses when cons…

3188c09

…tructing a CMSLookup

[CASSANDRA-20476] Minor logging changes

45af389

[CASSANDRA-20476] Relocate Discovery to new package

561ae8d

[CASSANDRA-20476] Include ClusterMetadata.metadataId in survey req/rs…

42fc3ee

…p messages

[CASSANDRA-20476] Lower ProgressBarrier backoff in DiscoverNewCMSTest

fc802b7

[CASSANDRA-20476] Improvements to RemoteProcessor.CandidateIterator

718e4a4

[CASSANDRA-20476] Break up new long running test

1cafe66

[CASSANDRA-20476] Ensure proper exit from NewGossiper shadow round af…

665839a

…ter upgrade

beobal mentioned this pull request Jul 30, 2026

CASSANDRA-20476 & CASSANDRA-20736 Handle CMS member addresses changing concurrently #4613

Closed

[CASSANDRA-20476] Include current CMS member endpoints in rediscovery…

b2b0c8a

… response

krummas requested changes Aug 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change - #4979

CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change#4979
beobal wants to merge 28 commits into
apache:cassandra-6.0from
beobal:samt/CASSANDRA-20476

beobal commented Jul 30, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Jul 31, 2026

Uh oh!

krummas Aug 3, 2026

Uh oh!

krummas Aug 3, 2026

Uh oh!

beobal Aug 3, 2026

Uh oh!

Uh oh!

krummas Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

beobal commented Jul 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants