Surface queue rate limited state in TaskQueueStats#10944
Merged
rkannan82 merged 24 commits intoJul 8, 2026
Conversation
rkannan82
changed the title
rkannan82
force-pushed
the
kannan/wci-rate-limit-stats
branch
from
39ad6e8 to
3d53eb1
Compare
Add IsWholeQueueRateLimitingActive() to rateLimitManager and populate RateLimitingActive on stats entries in GetStatsByPriority(). MergeStats ORs the boolean across partitions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rkannan82
force-pushed
the
kannan/wci-rate-limit-stats
branch
from
97eef6c to
3156f86
Compare
Drop the whole-queue qualifier — the flag represents whether rate limiting is blocking dispatches, without exposing which limiter. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rkannan82
commented
Jul 7, 2026
ShahabT
reviewed
Jul 7, 2026
Replace instantaneous GCRA limiter check with a taskTracker that records rate-limit events over a 30s sliding window, consistent with the existing tasks_add_rate and tasks_dispatch_rate fields. The tracker is incremented when TrySyncMatch returns syncMatchRateLimited, which covers both whole-queue and per-key rate limits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consistent with tasksAdded/tasksDispatched naming convention. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mporal into kannan/wci-rate-limit-stats
ShahabT
reviewed
Jul 7, 2026
rkannan82
added a commit
to temporalio/api
that referenced
this pull request
Jul 7, 2026
## What Adds `bool rate_limiting_active` to `TaskQueueStats` proto message. ## Why Auto-scaling systems need to know when backlog growth is caused by rate limiting rather than insufficient worker capacity. When the whole-queue rate limit is blocking dispatches, scaling up workers doesn't help — the bottleneck is the rate limit, not worker count. [Server PR](temporalio/temporal#10944) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
temporal-cicd Bot
pushed a commit
to temporalio/api-go
that referenced
this pull request
Jul 7, 2026
## What Adds `bool rate_limiting_active` to `TaskQueueStats` proto message. ## Why Auto-scaling systems need to know when backlog growth is caused by rate limiting rather than insufficient worker capacity. When the whole-queue rate limit is blocking dispatches, scaling up workers doesn't help — the bottleneck is the rate limit, not worker count. [Server PR](temporalio/temporal#10944) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Move tasksRateLimited tracking into findAndWakeMatches so both sync match and backlog dispatch paths are covered. Update go.mod to point at merged api-go main. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers the backlog dispatch path where a poller arrives and finds a rate-limited backlog task, verifying that tasksRateLimited tracker records the event. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- require-error: swap args in ErrorIs to (actual, expected) - nil-compare: add nolint for intentional typed-nil check - expected-actual: swap Equal/NotEqual args for TaskId%4 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dnr
reviewed
Jul 7, 2026
ShahabT
approved these changes
Jul 8, 2026
matcherData now takes an onRateLimited func() callback instead of a *taskTracker. The caller (physicalTaskQueueManager) wraps its tracker in a closure, keeping matcherData decoupled from the tracker impl. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dnr
approved these changes
Jul 8, 2026
dnr
left a comment
dnr
left a comment
Contributor
There was a problem hiding this comment.
Thanks. I'll update my PR after this merges
Add doc comments explaining when onRateLimited is called. Acquire taskTrackerLock in the closure to be safe after taskTracker's internal lock is removed (temporalio#10964). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…it-stats # Conflicts: # service/matching/physical_task_queue_manager.go
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
v1.63.2 includes the rate_limiting_active field (temporalio/api#818). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
trangnguyenkp
pushed a commit
to trangnguyenkp/temporal
that referenced
this pull request
Jul 9, 2026
## What Populates the new `rate_limiting_active` field in `TaskQueueStats` so auto-scaling systems can detect when backlog growth is caused by rate limiting rather than insufficient worker capacity. Records rate-limit events via a callback from `matcherData.findAndWakeMatches` into a `taskTracker`. The field is true when any dispatch was rate-limited in the last 30 seconds. ## Why When a task queue has a whole-queue rate limit configured, WorkerControllerInterface (WCI) `PullStats` path sees backlog and scales up workers. More workers don't help — they poll but can't receive tasks faster than the rate limit. This field lets WCI suppress scaling in that case. ## How did you test it? - Unit tests for sync match and backlog dispatch paths (`TestSyncMatchRateLimitedIncrementsStats`, `TestBacklogRateLimitedIncrementsStats`) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Populates the new
rate_limiting_activefield inTaskQueueStatsso auto-scaling systems can detect when backlog growth is caused by rate limiting rather than insufficient worker capacity.Records rate-limit events via a callback from
matcherData.findAndWakeMatchesinto ataskTracker. The field is true when any dispatch was rate-limited in the last 30 seconds.Why
When a task queue has a whole-queue rate limit configured, WorkerControllerInterface (WCI)
PullStatspath sees backlog and scales up workers. More workers don't help — they poll but can't receive tasks faster than the rate limit. This field lets WCI suppress scaling in that case.How did you test it?
TestSyncMatchRateLimitedIncrementsStats,TestBacklogRateLimitedIncrementsStats)🤖 Generated with Claude Code