Refactor many slow functional tests to be faster - #7968
Draft
gerrod3 wants to merge 28 commits into
Draft
Conversation
Contributor
Author
|
After comparing this to #7930, it seems the only thing that moved the needle was the Next I'm going to take a look and see if these tests should be marked parallel. There might be too many tests with many tasks causing congestion among the workers. Results:
|
Contributor
Author
|
Moving some parallel tests over to serial improved another ~30 secs or so. Still not happy that azure is close to 30 mins. Seems to be quite wild in its variance.
|
dralley
marked this pull request as draft
August 14, 2026 03:09
Contributor
Author
|
After-parallel Worker wait (
|
| Matrix | n | mean | p90 | p99 | max |
|---|---|---|---|---|---|
| pulp | 950 | 50ms | 80ms | 262ms | 471ms |
| s3 | 1000 | 40ms | 53ms | 85ms | 2.5s* |
| lowerbounds | 950 | 37ms | 49ms | 84ms | 143ms |
| azure (redis) | — | n/a | (no unblocked_at) |
*s3 max is a single synchronize outlier; p99 is still 85ms.
Verdict: worker starvation is not the bottleneck. With 4 workers and -n 8, unblocked tasks almost never wait. Occasional resource waits hit ~2–3s. Slow parallel tests are mostly runtime (and some lock contention), not queued behind busy workers — bumping PULP_WORKERS is unlikely to help much.
The functional test waits on the worker heartbeat cleanup interval, so keep it long_running for e2e/nightly and cover the cleanup path with a fast unit test for routine CI. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Use a smaller page size so fewer versions are needed, and create content units before modifying the repository so create tasks can overlap. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-create content units so create tasks can overlap before the serial modify and publish steps that assert distributed versions are retained. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Dispatch content create tasks before monitoring so they can overlap, instead of creating each unit serially through the factory. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-create content so create tasks can overlap, and wait for distinct checkpoint timestamps only when wall-clock has not already advanced a second instead of always sleeping for a full second. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace two 3-file fixture syncs with overlapping content creates and repository modifies, which is enough for the with_content filter checks. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Use on_demand sync so the test still covers PRNs without downloading fixture artifacts. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Dispatch content creates concurrently and stop attaching each unit to a bucket repository, cutting setup time for several repo-version tests. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Create two visible and two hidden distributions instead of five each; that still covers index visibility without ten serial create tasks. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the immediate fixture sync with overlapping content creates and repository modifies; the test only needs two versions and PULP_MANIFEST checks. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the three-file fixture sync with a single content create attached to the repository; keep a remote only for the mirror=True rejection. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Class-scoped setup builds five publications; under xdist each parallel test rebuilt it. The HTTP assertions are far cheaper than that setup. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Group consumers of import_export_repositories into a serial class with class-scoped fixtures so the two immediate syncs run once. Keep light import-check tests and self-contained cases parallel. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Use a serial class with class-scoped three-repo sync and exporter fixtures so xdist does not redo the expensive setup for each export consumer. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Expose started_at - unblocked_at summaries via pulpcore-manager and print them after CI functional tests to spot worker congestion. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Move task-queue-stats out of post_script so it runs before serial/cli and task purge can erase the parallel-phase congestion signal. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Use ContentFilesApi.upload instead of task-backed creates, and fetch all Content-Type headers concurrently on a shared aiohttp session. Assisted-By: Cursor Grok 4.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace task-backed ContentFilesApi.create in the remaining overlapping content-create tests so units appear without monitor_task. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Keep two checkpoints and retain_checkpoints=1, and skip creating an unused distribution. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
The functional test slept 60s and queued 25 tasks. Cover FETCH_TASK_LIMIT doubling in fetch_task without a live worker. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Fetch all rewritten list URLs concurrently with limit=1 instead of serial full-page GETs. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Populate upstreams with one uploaded file instead of 3-file fixture syncs, and use fewer distributions where label/policy coverage remains. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
Skip unused same-domain syncs and successful exports; keep one uploaded file where a version or last_export is needed for the 400 cases. Assisted-By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most of it is just being smart in how we populate repos with content: create content first in parallel, then do modify.
https://redhat.atlassian.net/browse/PULP-2238
📜 Checklist
See: Pull Request Walkthrough