Skip to content

CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change - #4980

Closed
beobal wants to merge 41 commits into
apache:trunkfrom
beobal:samt/CASSANDRA-20476-trunk
Closed

CASSANDRA-20476 Cluster is unable to recover after shutdown if IPs change#4980
beobal wants to merge 41 commits into
apache:trunkfrom
beobal:samt/CASSANDRA-20476-trunk

Conversation

@beobal

@beobal beobal commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Thanks for sending a pull request! Here are some tips if you're new here:

  • Ensure you have added or run the appropriate tests for your PR.
  • Be sure to keep the PR description updated to reflect all changes.
  • Write your PR title to summarize what this PR proposes.
  • If possible, provide a concise example to reproduce the issue for a faster review.
  • Read our contributor guidelines
  • If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:

<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

@beobal beobal mentioned this pull request Jul 30, 2026
Closed
beobal and others added 28 commits August 6, 2026 13:02
@beobal
[CASSANDRA-20476] Add dtest for CMS rediscovery
97801e4
@beobal
[CASSANDRA-20476] Introduce CMSLookup
20cd9b6
@beobal
[CASSANDRA-20476] Perform rediscovery of CMS at startup if addresses …
0b97d84 
…change
@beobal
[CASSANDRA-20476] Attempt to wait for all address changes to be enact…
c4229f7 
…ed before allowing startup to proceed
@beobal
[CASSANDRA-20476] Some minor logging additions
b99a3f8
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Make sure to start messaging service
ab5623e
@beobal
[CASSANDRA-20476] Don't attempt to catch up from peers or CMS while l…
34371ce 
…og processing is suspended
@beobal
[CASSANDRA-20476] Remove unused CMSLookup.addressMap
01ee234
@beobal
[CASSANDRA-20476] Remove unnecessary map retrievals
a85024e
@beobal
[CASSANDRA-20476] Make CMSLookup.overrides an ImmutableMap
908c046
@beobal
[CASSANDRA-20476] When CMSLookup is RETIRED, remove log listener
d888294
@beobal
[CASSANDRA-20476] Include seeds in initial rediscovery candidates
2818b30
@beobal
[CASSANDRA-20476] Make number of discovery rounds configurable
0324148
@beobal
[CASSANDRA-20476] Improve comment about target quorum size during red…
b7a10be 
…iscovery
@beobal
[CASSANDRA-20476] Simplify Discovery.state to a boolean
5c06468
@beobal
[CASSANDRA-20476] Include node ids in CMS description
a9220e1
@beobal
[CASSANDRA-20476] Reduce message timeout when sending Startup transform
14d3aed 
If two nodes restart with new broadcast addresses concurrently
and one is a CMS member, the non-member may send a TCM_COMMIT_REQ
containing its STARTUP transform to the CMS member's old address.
If this is no longer reachable, the sender should time out quickly
so it can resend to another CMS member, or try to discover the new
CMS address(es).
@beobal
[CASSANDRA-20476] Don't attempt rediscovery if upgrading from a post …
390129a 
…CEP-21, pre V9 version
Marcus Eriksson @beobal
[CASSANDRA-20476] Fix ip changes while still in gossip mode
223d269
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Don't return CMS_ONLY Discovery responses when cons…
08057d9 
…tructing a CMSLookup
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Minor logging changes
c18c986
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Relocate Discovery to new package
e0885c5
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Include ClusterMetadata.metadataId in survey req/rs…
7e3c65a 
…p messages
Sam Tunnicliffe @beobal
[CASSANDRA-20476] Lower ProgressBarrier backoff in DiscoverNewCMSTest
ebc164d
@beobal
[CASSANDRA-20476] Improvements to RemoteProcessor.CandidateIterator
e78792c
@beobal
[CASSANDRA-20476] Break up new long running test
d69d61a
@beobal
[CASSANDRA-20476] Ensure proper exit from NewGossiper shadow round af…
9a9f0a3 
…ter upgrade
@beobal
[CASSANDRA-20476] Fix (unrelated) upgrade test issues
4275e07
beobal added 13 commits August 6, 2026 13:03
@beobal
[CASSANDRA-20476] Include current CMS member endpoints in rediscovery…
4a50bdd 
… response
@beobal
[CASSANDRA-20476] Survey request/response improvements
36db396 
* Add broadcast_address to SurveyResponse instead of inferring from
  message header.
* Add supplier for metadataId rather that memoizing the current value
  from ClusterMetadata to avoid any initialization races.
@beobal
[CASSANDRA-20476] Exit startup once local address changes have been c…
cf14fe8 
…ommitted
@beobal
[CASSANDRA-20476] Make CandidateIterator element set thread safe
0d0043e
@beobal
[CASSANDRA-20476] Assert that lazily initialized CMS membership field…
8d2ce55 
…s are null when refreshing CMSLookup
@beobal
[CASSANDRA-20476] ShadowRoundHandler uses markDone() after receiving …
d1d4036 
…sufficient acks
@beobal
[CASSANDRA-20476] Reword comment
d115267
@beobal
[CASSANDRA-20476] Avoid double lookup in CMSLookup internal EndpointL…
79fd3b4 
…ookup impl
@beobal
[CASSANDRA-20476] In DefaultReplicator, ensure node ids and endpoint …
1d74f45 
…lookup are based on the same ClusterMetadata instance
@beobal
[CASSANDRA-20476] Update response handler definition for new Discover…
ba52fcb 
…y verbs
@beobal
[CASSANDRA-20476] add tests for CMSLookup::rebuild
5ae8708
@beobal
[CASSANDRA-20476] tweak comment
8e9a9e9
@beobal
[CASSANDRA-20476] Fix upgrade versions on upgrade dtest
624f84a
@beobal
beobal force-pushed the samt/CASSANDRA-20476-trunk branch from c4c33f0 to 624f84a Compare August 6, 2026 15:27
@beobal beobal closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@krummas krummas krummas approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@beobal @krummas