Skip to content

feat: Add support for MaxConcurrentActivityExecutionSize for activities#1056

Merged
Shaddoll merged 6 commits into
masterfrom
act
May 15, 2026
Merged

feat: Add support for MaxConcurrentActivityExecutionSize for activities#1056
Shaddoll merged 6 commits into
masterfrom
act

Conversation

@Shaddoll

@Shaddoll Shaddoll commented May 11, 2026

Copy link
Copy Markdown
Member

What changed?
pick temporalio/sdk-java#286
pick temporalio/sdk-java#291

Why?
Prevent polling an activity when the activity task executor pool is full. However, locally dispatched activity is not included in this PR because it requires big refactoring

How did you test it?
./gradlew test

Potential risks
Workflows can be stuck

Release notes

Documentation Changes

Comment thread src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java
@codecov

codecov Bot commented May 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.31298% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.67%. Comparing base (a582d59) to head (f90542d).

Files with missing lines Patch % Lines
...ternal/sync/LocalActivityExecutionContextImpl.java 0.00% 5 Missing ⚠️
.../main/java/com/uber/cadence/activity/Activity.java 70.00% 3 Missing ⚠️
...ce/internal/sync/ActivityCompletionClientImpl.java 85.71% 2 Missing ⚠️
.../cadence/internal/worker/ActivityPollTaskBase.java 92.30% 0 Missing and 2 partials ⚠️
...internal/sync/TestActivityEnvironmentInternal.java 66.66% 1 Missing ⚠️
...m/uber/cadence/internal/worker/ActivityWorker.java 96.00% 0 Missing and 1 partial ⚠️

❌ Your project check has failed because the head coverage (69.67%) is below the target coverage (85.00%). You can increase the head coverage or adjust the target coverage.

Files with missing lines Coverage Δ Complexity Δ
...ce/internal/sync/ActivityExecutionContextImpl.java 90.21% <100.00%> (+3.37%) 23.00 <6.00> (+5.00)
...m/uber/cadence/internal/sync/ActivityInternal.java 100.00% <ø> (+22.22%) 1.00 <0.00> (-4.00) ⬆️
...m/uber/cadence/internal/sync/ActivityTaskImpl.java 77.77% <100.00%> (+2.77%) 11.00 <2.00> (+1.00)
...internal/sync/CurrentActivityExecutionContext.java 50.00% <ø> (ø) 4.00 <0.00> (ø)
...cadence/internal/sync/POJOActivityTaskHandler.java 81.74% <100.00%> (+0.91%) 24.00 <3.00> (ø)
.../cadence/internal/sync/WorkflowClientInternal.java 71.21% <100.00%> (+0.21%) 51.00 <1.00> (+1.00)
...com/uber/cadence/internal/worker/ActivityTask.java 100.00% <100.00%> (ø) 3.00 <3.00> (?)
...r/cadence/internal/worker/ActivityTaskHandler.java 100.00% <100.00%> (ø) 0.00 <0.00> (ø)
...r/cadence/internal/worker/LocalActivityWorker.java 94.16% <100.00%> (+0.04%) 10.00 <0.00> (ø)
...ternal/worker/LocallyDispatchedActivityWorker.java 100.00% <ø> (ø) 3.00 <0.00> (ø)
... and 6 more

... and 9 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a582d59...f90542d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java
@Shaddoll Shaddoll force-pushed the act branch 2 times, most recently from eb474e7 to 228f18e Compare May 12, 2026 16:21
@Shaddoll Shaddoll force-pushed the act branch 6 times, most recently from a8345e0 to 4e4d5a1 Compare May 12, 2026 20:50
@Shaddoll Shaddoll changed the title Act feat: Add support for MaxConcurrentActivityExecutionSize for activities May 12, 2026
@Shaddoll Shaddoll force-pushed the act branch 2 times, most recently from 3b1cc60 to 6681553 Compare May 12, 2026 21:01
* prevents automatic activity completion on return and returns a client that can be used to
* complete the activity asynchronously.
*/
ActivityCompletionClient useLocalManualCompletion();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an entirely new feature, do we need it and is this the interface we want to expose?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use it in the new tests and Temporal also exposed it. I think it should be fine to expose it and we can achieve function parity with them.

result.getStartedTimestamp() - result.getScheduledTimestampOfThisAttempt()),
HistogramBuckets.HIGH_1MS_24H);
isSuccessful = true;
return new ActivityTask(result, pollSemaphore::release);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to prevent duplicate releases from users incorrectly using the useLocalManualCompletion. One way to do that would be to wrap pollSemaphore::release such that calling it a second time does nothing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


public ActivityPollTaskBase(SingleWorkerOptions options) {
this.options = options;
// TODO: we need to share this semaphore with the locally dispatched activity poll task

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do they currently share a thread pool?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I think they don't share the pool.


public static final int MAX_CONCURRENT_ACTIVITIES = 1;
public static final double TASK_LIST_ACTIVITIES_PER_SECOND =
100.0; // change this to 0.0 after we add semaphore limit for locally dispatched activities

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's a clever workaround lol.

@gitar-bot

gitar-bot Bot commented May 14, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 3 resolved / 3 findings

Implements MaxConcurrentActivityExecutionSize to prevent activity over-polling, resolving permit leaks and double-release issues found during review.

✅ 3 resolved
Bug: Semaphore not released on poll failure before metrics/tagging

📄 src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java:121-135
In ActivityWorker.handle(), if the code between line 122 and the try block (e.g., response.getActivityType().getName()) throws a NullPointerException (which is possible if the response has a null activity type), the exception propagates out of handle() without ever reaching the finally block that calls completionHandle.apply(). The semaphore permit acquired in ActivityPollTaskBase.poll() would be permanently leaked.

This could happen if the server returns a malformed response.

Bug: Semaphore permit leak when handle() throws an unexpected exception

📄 src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java:149 📄 src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java:192-194 📄 src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java:184-195 📄 src/main/java/com/uber/cadence/internal/worker/ActivityWorker.java:184-194
In ActivityWorker.TaskHandlerImpl.handle(), if an exception other than CancellationException is thrown (e.g., from handler.handle() throwing an Error, or from code before line 154), handlerResponse remains null. The finally block condition handlerResponse != null && !handlerResponse.isManualCompletion() evaluates to false, so task.getCompletionHandle().apply() (which releases the semaphore) is never called.

While POJOActivityTaskHandler.handle() catches most exceptions internally, unexpected errors (e.g., OutOfMemoryError, or a bug in metrics/tracing code before the handler call) would permanently leak a semaphore permit. Over time, this could starve the activity worker since no new polls can proceed once all permits are exhausted.

The fix should release the semaphore in the finally block whenever it's NOT a manual completion case, including when handlerResponse is null (which indicates an unexpected failure).

Edge Case: ActivityCompletionClientImpl allows double semaphore release

📄 src/main/java/com/uber/cadence/internal/sync/ActivityCompletionClientImpl.java:38-52
Each method in ActivityCompletionClientImpl (complete, completeExceptionally, reportCancellation) calls completionHandle.apply() in its finally block. If a user mistakenly calls multiple completion methods on the same client, the semaphore will be released multiple times, adding extra permits beyond the original count and undermining the prefetching limit. Consider using an AtomicBoolean guard to ensure the handle is invoked at most once.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@Shaddoll Shaddoll merged commit c9875af into master May 15, 2026
15 checks passed
@Shaddoll Shaddoll deleted the act branch May 15, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants