-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[fix][broker] Close protocol handlers channel #25111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
| try { | ||
| List<CompletableFuture<Void>> channelCloseFutures = | ||
| brokerService.closeProtocolHandlerChannels(); | ||
| // Wait for all protocol handler channels to close before closing protocol handlers | ||
| FutureUtil.waitForAll(channelCloseFutures).get(); | ||
| LOG.info("Protocol handler channels closed successfully"); | ||
| } catch (Exception e) { | ||
| LOG.warn("Failed to close protocol handler channels", e); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be called after protocolHandlers.close() because the close method of protocol handler might do some cleanup jobs that rely on the connection.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@BewareMyPower I think that it should be fine to close the channels before protocol handlers. The reason for this is that the channels that are being closed are ServerChannel instances and wouldn't impact existing connections. Closing the ServerChannels will prevent new connections being opened.
BewareMyPower
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Protocol handlers close before the ownership change in BrokerService#unloadNamespaceBundlesGracefully. Even if you close the channels before closing the protocol handler, the requests could still be sent to the broker and rejected. It's actually worse because no logs can be found at broker side.
lhotari
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Motivation
During the shutdown process of Pulsar brokers, protocol handlers' channels are not properly closed, which allows new client requests to continue arriving even after the shutdown has been initiated. This can lead to:
The broker's
closeAsyncmethod properly handles closing all channels by callingchannel.close()on all channels inlistChannels. However, the same pattern is not applied to protocol handlers, leaving their channels open during the shutdown process.Modifications
closeProtocolHandlerChannels()method to properly close all protocol handler channels before callingprotocolHandler.close()closeAsyncVerifying this change
This change is already covered by existing tests, such as:
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
[
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
gaozhangmin#14