KAFKA-20976: Make max partitions per request configurable by andrewgrantcflt · Pull Request #23245 · apache/kafka

andrewgrantcflt · 2026-08-23T12:42:21Z

Description

In ReplicationControlManager there is a max of 10,000 partitions
allowed in a single create topics request. In an app I am seeing this
limit being hit. I am trying to reproduce the error in an integration
test, but by having to create so many partitions the test is slow and
flaky. I have to create up to 10,000 partitions to reproduce the issue
because the limit is currently hard-coded.

So here I am simply making the limit a config. By having it as a config,
we can configure the limit to be lower and reproduce the issue in a test
without having to create so many partitions.

In fact, the current limit of 10,000 comes from the fact that we should
only add 10,000 records to a single batch that gets persisted to the
metadata log. We use that same 10,000 limit in a few other places, such
as when handling DELETE_TOPICS. There we use MAX_RECORDS_PER_USER_OP
when creating a bounded array. Ultimately all of these limits are
related in that we want to not add more than 10,000 records to a single
batch. So I refactor some code that uses that same limit to use the same
config I have added.

https: //issues.apache.org/jira/browse/KAFKA-20976 Reviewers: Andrew
Schofield aschofield@confluent.io

Reviewers: Andrew Schofield aschofield@confluent.io

andrewgrantcflt · 2026-08-23T12:47:01Z

    public static final long CONTROLLER_PERFORMANCE_ALWAYS_LOG_THRESHOLD_MS_DEFAULT = 2000;
    public static final String CONTROLLER_PERFORMANCE_ALWAYS_LOG_THRESHOLD_MS_DOC = "We will log an error message about controller events that take longer than this threshold.";

+    public static final String CONTROLLER_MAX_PARTITIONS_PER_BATCH_CONFIG = "controller.max.partitions.per.batch";


I considered naming the config controller.max.partitions.per.create.topics.batch as well. As of now, this config is only used when handling CREATE_TOPICS requests so arguably that name may be clearer. controller.max.partitions.per.batch is a bit generic and could imply it is used for other RPCs, but maybe that's better if we want to reuse this config in the future, for example for CREATE_PARTITIONS?

Curious to hear other folks' opinions.

andrewgrantcflt · 2026-08-23T12:47:33Z

I put the config in KRaftConfigs since it's only to be used by the controller.

AndrewJSchofield

Thanks for the PR. I don't like adding the config without a KIP, but this is an internal-only config and it is making a hard-coded limit configurable. I reckon it sneaks in under the KIP bar :)

I do suggest making it a bit more general-purposes in the ReplicationControlManager. For example, deleteTopics and others limit the request size to MAX_RECORDS_PER_USER_OP, which is also 10000. Aren't these essentially the same thing, checking that the user operation is not too complex for the controller to handle? Let me know if that makes sense too.

AndrewJSchofield · 2026-08-24T12:05:58Z

+    public static final String CONTROLLER_MAX_PARTITIONS_PER_BATCH_CONFIG = "controller.max.partitions.per.batch";
+    public static final int CONTROLLER_MAX_PARTITIONS_PER_BATCH_DEFAULT = 10_000;
+    public static final String CONTROLLER_MAX_PARTITIONS_PER_BATCH_DOC = "The maximum number of partitions that the " +
+            "active controller will allow to be created by a single CreateTopics request. This limit protects the " +


Can we make this a bit less specific by removing the "CreateTopics" part of this?

Agreed. Updated, especially as now the config is used in other places where we had the 10,000 limit for the max number of metadata records we can add to a single batch.

andrewgrantcflt · 2026-08-24T14:41:18Z

Thanks for the PR. I don't like adding the config without a KIP, but this is an internal-only config and it is making a hard-coded limit configurable. I reckon it sneaks in under the KIP bar :)

I do suggest making it a bit more general-purposes in the ReplicationControlManager. For example, deleteTopics and others limit the request size to MAX_RECORDS_PER_USER_OP, which is also 10000. Aren't these essentially the same thing, checking that the user operation is not too complex for the controller to handle? Let me know if that makes sense too.

@AndrewJSchofield and I chatted. I am going to close this PR because it's a bit controversial adding a config without a KIP.

AndrewJSchofield · 2026-08-24T15:13:48Z

This area is ripe for a KIP as part of the ongoing work to harden the Kafka protocol.

andrewgrantcflt · 2026-08-24T16:31:30Z

Re-opening the PR. After chatting with @AndrewJSchofield, @jsancio and @rajinisivaram, it should be OK to add an internal config without a KIP if we include an integration test showing it in use.

AndrewJSchofield · 2026-08-25T10:59:05Z

@andrewgrantcflt Checkstyle failure

andrewgrantcflt · 2026-08-25T13:22:29Z

@andrewgrantcflt Checkstyle failure

Thanks, fixing it.

AndrewJSchofield

Thanks for the updated PR. It's good to see all of the related operations use the same config now. Just a few comments, but it's getting there.

AndrewJSchofield · 2026-08-25T15:10:41Z

          setQuorumFeatures(quorumFeatures).
          setDefaultReplicationFactor(config.defaultReplicationFactor.toShort).
          setDefaultNumPartitions(config.numPartitions.intValue()).
+          setMaxRecordsPerBatch(config.controllerMaxRecordsPerBatch).


nit: Any reason why this config is not added at the end, preserving the order after controllerPerformanceAlwaysLogThresholdMs?

Hmm no good reason. I will move it.

AndrewJSchofield · 2026-08-25T15:19:37Z

            }
-            if (totalPartitions > MAX_PARTITIONS_PER_BATCH) {
+            if (totalPartitions > maxRecordsPerBatch) {
                throw new PolicyViolationException("Excessively large number of partitions per request.");


This exception message is really not that elegant. It sounds like the configuration for the number of partitions per request is excessive, but actually it's just complaining about an individual request. How about something like "Too many partitions in request".

I do agree. Is it OK to change the error message? I know users probably shouldn't be depending on the message, but in practice it inevitably will happen.

Yes, you can change it.

AndrewJSchofield · 2026-08-25T15:21:07Z

    ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
        ElectionType electionType = electionType(request.electionType());
-        List<ApiMessageAndVersion> records = BoundedList.newArrayBacked(MAX_RECORDS_PER_USER_OP);
+        List<ApiMessageAndVersion> records = BoundedList.newArrayBacked(maxRecordsPerBatch);


Just an observation from reading an unfamiliar area of code. This is going to result in a BoundedListTooLongException if the list capacity is exceeded, I think. Any reason why validateTotalNumberOfPartitions is taking a different path?

I'm not too sure. I do see the following:
https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/errors/EventHandlerExceptionInfo.java#L83-L88

} else if (internal instanceof BoundedListTooLongException) { // The operation could not be performed because it would have created an overly large // batch. return new EventHandlerExceptionInfo(false, false, internal, new PolicyViolationException("Unable to perform excessively large batch " + "operation."));

I think BoundedListTooLongException does eventually get mapped to the same exception, albeit with a different error message. I probably would have made the code that throws BoundedListTooLongException catch it and re-throw a PolicyViolationException directly but that's probably out of scope for this PR.

KAFKA-20976: Make KRaft max partitions per request limit a config

636e84c

github-actions Bot added triage PRs from the community core Kafka Broker kraft small Small PRs labels Aug 23, 2026

andrewgrantcflt commented Aug 23, 2026

View reviewed changes

andrewgrantcflt added 2 commits August 23, 2026 09:07

define config in one place

0a59635

improve defalt setting

1e9a9f9

andrewgrantcflt changed the title ~~KAFKA-20976: Make KRaft max partitions per request limit a config~~ KAFKA-20976: Make max partitions per request configurable Aug 23, 2026

github-actions Bot removed the triage PRs from the community label Aug 24, 2026

AndrewJSchofield requested changes Aug 24, 2026

View reviewed changes

AndrewJSchofield added the ci-approved label Aug 24, 2026

andrewgrantcflt closed this Aug 24, 2026

andrewgrantcflt reopened this Aug 24, 2026

github-actions Bot added the triage PRs from the community label Aug 24, 2026

andrewgrantcflt added 2 commits August 24, 2026 12:56

add integration test

c0a4cdd

add test

a6914d0

github-actions Bot removed the small Small PRs label Aug 24, 2026

andrewgrantcflt added 2 commits August 24, 2026 16:18

consolidate config

5d780fd

change doc

630fdd1

github-actions Bot removed the triage PRs from the community label Aug 25, 2026

fix build

5b6c5c7

AndrewJSchofield requested changes Aug 25, 2026

View reviewed changes

Uh oh!

Conversation

andrewgrantcflt commented Aug 23, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewgrantcflt commented Aug 24, 2026

Uh oh!

AndrewJSchofield commented Aug 24, 2026

Uh oh!

andrewgrantcflt commented Aug 24, 2026

Uh oh!

AndrewJSchofield commented Aug 25, 2026

Uh oh!

andrewgrantcflt commented Aug 25, 2026

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewgrantcflt Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andrewgrantcflt commented Aug 23, 2026 •

edited by github-actions Bot

Loading

andrewgrantcflt Aug 25, 2026 •

edited

Loading