lock only domains that have finite limits for checked reservation

[sb-abhish3k]

Description

We are observing high failures from CheckedReservation timeouts and elsewhere seeing MySQLTransactionRollbackException: Lock wait timeout exceeded from a similar bottleneck with quota transaction timeouts.

During VM creation 5-7 CheckedReservation are constructed, which acquire a GlobalLock and go check resource limits and create/persist a reservation.

During the "check resource limits" portion, in ResourceLimitManager#checkResourceLimitWithTag within the context of a DB transaction which has InnoDB timeout implications, it attempts before proceeding and releasing the GlobalLock to:

1. lock account and owner domain rows
2. check account limits
3. check domain limits

The issue with step 3 is that it:
a. skips the ROOT domain explicitly after locking it in step 1 and
b. short circuits and doesn't even do the validation if the domain limits are unlimited (-1)

We are currently configuring limits at the account's domain level only, and have intermediate domain levels mapping to availability zones.

Effectively what the current unchanged code means for us is if we have thousands of concurrent requests to create a VM, even if they are segmented by accounts/domains in different zone domain hierarchies, they will all try and wait on other processes to lock on the same rows for the ROOT domain and the zone domains (configured with -1), never throwing an exception and generally bypassing the checks altogether.

It seems like an inefficiency to lock rows that will never be checked.

Fixing it should decrease the time it takes to provision a VM and reduce transaction load on the DB, especially during create surges. It seems like it could also apply to other users of CloudStack upstream.

Changes

This PR eliminates unnecessary CheckedReservation contention by removing quota limit checks for cases where a tagged resource limit is -1 (unlimited) on a parent domain.

Summary of changes:

• engine/schema/.../ResourceLimitDao.java — added listDomainIdsWithFiniteLimit interface method.
• engine/schema/.../ResourceLimitDaoImpl.java — added two named SearchBuilders (tag-EQ and tag-NULL variants, both with domainId IN + max NEQ) and the implementation, matching the existing constructor-built style.
• server/.../ResourceLimitManagerImpl.java — lockAccountAndOwnerDomainRows now delegates to a new protected listRowsToLockForLimitCheck helper that locks the account row + only ancestor-domain rows with a finite explicit limit, with
  fallbacks to listAllRowsToUpdate for finite global default and tagged-account-row create-on-miss.
• server/.../ResourceLimitManagerImplTest.java — 5 new tests covering the four scenarios from the plan plus the create-on-miss fallback.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[sb-abhish3k]

@blueorangutan package

[blueorangutan]

@sb-abhish3k a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19243

[codecov]

Codecov Report

❌ Patch coverage is 73.86364% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.39%. Comparing base (8eeccdb) to head (b310d92).

Files with missing lines	Patch %	Lines
.../cloud/configuration/dao/ResourceLimitDaoImpl.java	45.45%	12 Missing ⚠️
.../cloud/resourcelimit/ResourceLimitManagerImpl.java	83.33%	5 Missing and 6 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #14169      +/-   ##
============================================
+ Coverage     16.38%   16.39%   +0.01%     
- Complexity    13614    13635      +21     
============================================
  Files          5669     5669              
  Lines        501532   501619      +87     
  Branches      60922    60939      +17     
============================================
+ Hits          82153    82242      +89     
+ Misses       410172   410163       -9     
- Partials       9207     9214       +7     

Flag	Coverage Δ
uitests	4.16% <ø> (ø)
unittests	17.25% <73.86%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[alexandremattioli]

@sb-abhish3k coincidentally I hit exactly that bug today when deploying VMs with terraform. On 4.22.

"2026-09-15 14:02:10,066 ERROR [c.c.a.ApiServer] (qtp1816522350-51060:[ctx-a7d48c12, ctx-ca20d110, ctx-e49907f9]) (logid:a743aceb) unhandled exception executing api command: [Ljava.lang.String;@4f72b376 com.cloud.utils.exception.CloudRuntimeException: Unable to find by id on DB, due to: Deadlock found when trying to get lock; try restarting transaction
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1090)
at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:1062)
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:997)
at com.cloud.domain.dao.DomainDaoImpl.getDomainParentIds(DomainDaoImpl.java:290)
at jdk.internal.reflect.GeneratedMethodAccessor385.invoke(Unknown Source)"