Add lazy-loading for extras packages by sjmonson · Pull Request #641 · vllm-project/guidellm

sjmonson · 2026-03-17T21:56:09Z

TODO

~~Pending upstream change: Support lazy importing submodules in attach scientific-python/lazy-loader#168~~ upstream seems dead so the module has been pulled in-house.
~~Support for soft extras (E.g. perf group)~~ deferred
~~Lazy load in __main__.py to improve CLI responsiveness~~ deferred
Docs cleanup (howto create a lazy module, do's and don'ts)
Actually test vLLM python (loading of vLLM tested but not a full run)

Summary

Lazy loads extras submodules in order to defer import errors to the time of use.

Details

TODO

Test Plan

Without vLLM:

Run guidellm benchmark run --help and observe no errors
Run uv run guidellm benchmark run --backend vllm_python --model test and observe error with helpful message

With vLLM:

Run guidellm benchmark run --help and observe no errors
Run uv run guidellm benchmark run --backend vllm_python --model test ... and observe successful benchmark
Run tox -re test-e2e and observe that tests pass (Previously they would fail when vLLM was installed due to load times)

Related Issues

Replaces Add helper to defer ImportError on extras and support lazy loading #636

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes AI-assisted code completion
Includes code generated by an AI application
Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

git log

commit 19d8aad
Author: Samuel Monson smonson@redhat.com
Date: Tue May 19 17:49:25 2026 -0400

Pull in custom lazy_load package as submodule

Signed-off-by: Samuel Monson <smonson@redhat.com>

commit de50d31
Author: Samuel Monson smonson@redhat.com
Date: Mon Jun 1 19:50:10 2026 -0400

Minor type fix

Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 6066364
Author: Samuel Monson smonson@redhat.com
Date: Tue Mar 17 16:33:04 2026 -0400

Switch to lazy-loading for extras packages

Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 591a160
Author: Samuel Monson smonson@redhat.com
Date: Wed Mar 18 16:58:33 2026 -0400

Use AttributeError for failed optionals

`torch` seems to break when it encounters lazy ImportErrors

Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 79bcd7d
Author: Samuel Monson smonson@redhat.com
Date: Wed May 20 17:42:54 2026 -0400

LazyModule which loads module attributes individually

Generated-by: claude-code Sonnet 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 812acf6
Author: Samuel Monson smonson@redhat.com
Date: Wed May 20 17:50:37 2026 -0400

Fix unit tests

Generated-by: claude-code Sonnet 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 5a81cc2
Author: Samuel Monson smonson@redhat.com
Date: Thu May 21 10:50:04 2026 -0400

Revert "LazyModule which loads module attributes individually"

This reverts commit e31862e9415f2f5a176789fdb0bd6c3bd5088e33.

Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 8a00c01
Author: Samuel Monson smonson@redhat.com
Date: Mon Jun 1 20:05:32 2026 +0000

Split lazy loading from functionality

Generated-by: claude-code Sonnet 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit b309ac4
Author: Samuel Monson smonson@redhat.com
Date: Mon Jun 1 21:08:01 2026 +0000

Cleanup tests after split

Generated-by: claude-code Sonnet 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 7106bf8
Author: Samuel Monson smonson@redhat.com
Date: Mon Jun 1 22:35:47 2026 +0000

Implement nested lazy loading

Generated-by: claude-code Opus 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit fccdcd7
Author: Samuel Monson smonson@redhat.com
Date: Tue Jun 2 00:09:19 2026 +0000

Switch attribute interface to a named tuple

Generated-by: claude-code Opus 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

commit 7dea6a0
Author: Samuel Monson smonson@redhat.com
Date: Tue Jun 2 00:47:11 2026 +0000

Add type stubs

Generated-by: claude-code Opus 4.6
Signed-off-by: Samuel Monson <smonson@redhat.com>

Generated-by: claude-code Opus 4.6
Generated-by: claude-code Sonnet 4.6
Signed-off-by: Samuel Monson smonson@redhat.com

mergify · 2026-03-17T22:25:55Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sjmonson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

## Summary Clean up some `__init__` package code and move uvloop config to `__init__`. ## Details This clean up was originally a part of #641 but as that PR is blocked I decided to split it out. Removing the transformers logging config does not seem to have any real affect; I do not get logs either way. Importing any huggingface libraries incurs a significant time cost so this is a prereq to improving CLI responsiveness. Additionally uvloop should be configured as early as possible so moved the setup to `__init__`. --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [ ] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)

sjmonson · 2026-03-20T19:17:29Z

This PR is currently blocked on "Lazy load in __main__.py" as our click definitions depend on heavy imports. CLI refactoring will be the main focus of v0.7.0.

## Summary Fixes spawn and forkserver multi-process contexts. ## Details I was hoping that after #647 we could switch to `forkserver` by default. However it turns out that `forkserver` and `spawn` will import the calling processes entrypoint (E.g. `__main__.py`) so we run into the same blocker as #641. However, I was able to confirm that striping every heavy import out of `__main__.py` solves the issue. So we should be good to switch in v0.7.0. On my machine there is about a ~10s overhead for `forkserver` and slightly more for `spawn`, which is not the worst for a default. However, the overhead may be more on other systems: ### `time guidellm benchmark run --profile poisson --rate 5 --data prompt_tokens=128,output_tokens=128 --max-seconds 30 --outputs json` | Context | real | user | sys | | ---------- | --------- | --------- | -------- | | Fork | 0m37.874s | 0m17.356s | 0m1.883s | | Forkserver | 0m47.344s | 0m14.862s | 0m0.860s | | Spawn | 0m49.515s | 1m51.230s | 0m8.915s | ### `time guidellm benchmark run --profile concurrent --rate 400 --data prompt_tokens=128,output_tokens=128 --max-seconds 30 --outputs json` | Context | real | user | sys | | ---------- | --------- | --------- | --------- | | Fork | 0m39.324s | 0m37.602s | 0m5.623s | | Forkserver | 0m49.609s | 0m19.710s | 0m1.311s | | Spawn | 0m50.399s | 2m9.724s | 0m11.374s | ### `time guidellm benchmark run --profile concurrent --rate 400 --data prompt_tokens=128,output_tokens=128 --max-seconds 120 --outputs json` | Context | real | user | sys | | ---------- | --------- | --------- | --------- | | Fork | 2m15.309s | 1m42.911s | 0m15.957s | | Forkserver | 2m25.964s | 0m38.891s | 0m2.802s | | Spawn | 2m27.454s | 3m24.325s | 0m22.531s | ## Test Plan Set `GUIDELLM__MP_CONTEXT_TYPE=forkserver` and confirm benchmarks run. --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [x] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`)

sjmonson · 2026-05-11T15:13:02Z

This is now an official feature in Python 3.15! Unfortunately since its a new keyword we won't be able to adopt it until 3.15 becomes our minimal supported version in half-a-decade from now.

mergify · 2026-05-27T04:53:16Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sjmonson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

jaredoconnell

I tested it and it properly worked with vLLM installed with the vLLM backend.

Signed-off-by: Samuel Monson <smonson@redhat.com>

`torch` seems to break when it encounters lazy ImportErrors Signed-off-by: Samuel Monson <smonson@redhat.com>

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

This reverts commit e31862e. Signed-off-by: Samuel Monson <smonson@redhat.com>

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

sjmonson · 2026-06-03T19:56:35Z

@Mergifyio queue

mergify · 2026-06-03T19:56:42Z

## TODO - [ ] ~Pending upstream change: scientific-python/lazy-loader#168 upstream seems dead so the module has been pulled in-house. - [ ] ~Support for soft extras (E.g. `perf` group)~ deferred - [ ] ~Lazy load in `__main__.py` to improve CLI responsiveness~ deferred - [ ] Docs cleanup (howto create a lazy module, do's and don'ts) - [ ] Actually test vLLM python (loading of vLLM tested but not a full run) ## Summary Lazy loads extras submodules in order to defer import errors to the time of use. ## Details TODO ## Test Plan Without vLLM: 1. Run `guidellm benchmark run --help` and observe no errors 2. Run `uv run guidellm benchmark run --backend vllm_python --model test` and observe error with helpful message With vLLM: 1. Run `guidellm benchmark run --help` and observe no errors 2. Run `uv run guidellm benchmark run --backend vllm_python --model test ...` and observe successful benchmark 3. Run `tox -re test-e2e` and observe that tests pass (Previously they would fail when vLLM was installed due to load times) ## Related Issues - Replaces vllm-project#636 --- - [x] "I certify that all code in this PR is my own, except as noted below." ## Use of AI - [x] Includes AI-assisted code completion - [ ] Includes code generated by an AI application - [ ] Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes `## WRITTEN BY AI ##`) --- # git log commit 19d8aad Author: Samuel Monson <smonson@redhat.com> Date: Tue May 19 17:49:25 2026 -0400 Pull in custom lazy_load package as submodule Signed-off-by: Samuel Monson <smonson@redhat.com> commit de50d31 Author: Samuel Monson <smonson@redhat.com> Date: Mon Jun 1 19:50:10 2026 -0400 Minor type fix Signed-off-by: Samuel Monson <smonson@redhat.com> commit 6066364 Author: Samuel Monson <smonson@redhat.com> Date: Tue Mar 17 16:33:04 2026 -0400 Switch to lazy-loading for extras packages Signed-off-by: Samuel Monson <smonson@redhat.com> commit 591a160 Author: Samuel Monson <smonson@redhat.com> Date: Wed Mar 18 16:58:33 2026 -0400 Use AttributeError for failed optionals `torch` seems to break when it encounters lazy ImportErrors Signed-off-by: Samuel Monson <smonson@redhat.com> commit 79bcd7d Author: Samuel Monson <smonson@redhat.com> Date: Wed May 20 17:42:54 2026 -0400 LazyModule which loads module attributes individually Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit 812acf6 Author: Samuel Monson <smonson@redhat.com> Date: Wed May 20 17:50:37 2026 -0400 Fix unit tests Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit 5a81cc2 Author: Samuel Monson <smonson@redhat.com> Date: Thu May 21 10:50:04 2026 -0400 Revert "LazyModule which loads module attributes individually" This reverts commit e31862e. Signed-off-by: Samuel Monson <smonson@redhat.com> commit 8a00c01 Author: Samuel Monson <smonson@redhat.com> Date: Mon Jun 1 20:05:32 2026 +0000 Split lazy loading from functionality Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit b309ac4 Author: Samuel Monson <smonson@redhat.com> Date: Mon Jun 1 21:08:01 2026 +0000 Cleanup tests after split Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit 7106bf8 Author: Samuel Monson <smonson@redhat.com> Date: Mon Jun 1 22:35:47 2026 +0000 Implement nested lazy loading Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit fccdcd7 Author: Samuel Monson <smonson@redhat.com> Date: Tue Jun 2 00:09:19 2026 +0000 Switch attribute interface to a named tuple Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> commit 7dea6a0 Author: Samuel Monson <smonson@redhat.com> Date: Tue Jun 2 00:47:11 2026 +0000 Add type stubs Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> --------- Generated-by: claude-code Opus 4.6 Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com> Signed-off-by: SkiHatDuckie <SkiHatDuckie@gmail.com>

mergify Bot added the needs-rebase label Mar 17, 2026

sjmonson force-pushed the feat/lazy_loader branch from e1e9203 to 5e4ee86 Compare March 18, 2026 18:39

sjmonson changed the base branch from main to fix/split_utils March 18, 2026 18:40

Base automatically changed from fix/split_utils to main March 18, 2026 18:41

mergify Bot removed the needs-rebase label Mar 18, 2026

sjmonson mentioned this pull request Mar 19, 2026

Cleanup __init__ #647

Merged

4 tasks

sjmonson force-pushed the feat/lazy_loader branch from 876cc30 to 8156271 Compare March 19, 2026 18:21

sjmonson changed the base branch from main to fix/init_cleanup March 19, 2026 18:21

Base automatically changed from fix/init_cleanup to main March 19, 2026 20:00

sjmonson mentioned this pull request Mar 20, 2026

Pass mp context to strategy #651

Merged

4 tasks

dbutenhof added this to the v0.7.0 milestone Mar 20, 2026

sjmonson mentioned this pull request May 12, 2026

guidellm benchmark --help fails with vLLM >= 0.20.0 (and other CLI commands fail as well). #716

Open

sjmonson force-pushed the feat/lazy_loader branch from 8156271 to 765b158 Compare May 19, 2026 21:51

mergify Bot added the needs-rebase label May 27, 2026

sjmonson force-pushed the feat/lazy_loader branch from 2c06bd3 to c9a1a81 Compare June 2, 2026 00:37

mergify Bot removed the needs-rebase label Jun 2, 2026

sjmonson force-pushed the feat/lazy_loader branch 2 times, most recently from 8c858ee to 7048da2 Compare June 2, 2026 01:03

sjmonson marked this pull request as ready for review June 2, 2026 01:06

sjmonson requested review from dbutenhof and jaredoconnell and removed request for dbutenhof June 2, 2026 01:06

dbutenhof approved these changes Jun 2, 2026

View reviewed changes

dbutenhof added internal filed by core contributor or associate feature Represents a new user-visible feature labels Jun 2, 2026

jaredoconnell approved these changes Jun 3, 2026

View reviewed changes

sjmonson added 12 commits June 3, 2026 15:56

Pull in custom lazy_load package as submodule

19d8aad

Signed-off-by: Samuel Monson <smonson@redhat.com>

Minor type fix

de50d31

Signed-off-by: Samuel Monson <smonson@redhat.com>

Switch to lazy-loading for extras packages

6066364

Signed-off-by: Samuel Monson <smonson@redhat.com>

Use AttributeError for failed optionals

591a160

`torch` seems to break when it encounters lazy ImportErrors Signed-off-by: Samuel Monson <smonson@redhat.com>

LazyModule which loads module attributes individually

79bcd7d

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Fix unit tests

812acf6

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Revert "LazyModule which loads module attributes individually"

5a81cc2

This reverts commit e31862e. Signed-off-by: Samuel Monson <smonson@redhat.com>

Split lazy loading from functionality

8a00c01

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Cleanup tests after split

b309ac4

Generated-by: claude-code Sonnet 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Implement nested lazy loading

7106bf8

Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Switch attribute interface to a named tuple

fccdcd7

Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

Add type stubs

7dea6a0

Generated-by: claude-code Opus 4.6 Signed-off-by: Samuel Monson <smonson@redhat.com>

sjmonson force-pushed the feat/lazy_loader branch from 7048da2 to 7dea6a0 Compare June 3, 2026 19:56

mergify Bot added the queued label Jun 3, 2026

mergify Bot added a commit that referenced this pull request Jun 3, 2026

Merge of #641

56a1e39

mergify Bot mentioned this pull request Jun 3, 2026

merge queue: embarking main (129f873), #753 and #641 together #770

Closed

34 tasks

mergify Bot merged commit 82b310a into main Jun 3, 2026
12 of 13 checks passed

mergify Bot deleted the feat/lazy_loader branch June 3, 2026 20:02

mergify Bot removed the queued label Jun 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lazy-loading for extras packages#641

Add lazy-loading for extras packages#641
mergify[bot] merged 12 commits into
mainfrom
feat/lazy_loader

sjmonson commented Mar 17, 2026 •

edited by github-actions Bot

Loading

Uh oh!

mergify Bot commented Mar 17, 2026

Uh oh!

sjmonson commented Mar 20, 2026

Uh oh!

sjmonson commented May 11, 2026 •

edited

Loading

Uh oh!

mergify Bot commented May 27, 2026

Uh oh!

jaredoconnell left a comment

Uh oh!

sjmonson commented Jun 3, 2026

Uh oh!

mergify Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sjmonson commented Mar 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Summary

Details

Test Plan

Related Issues

Use of AI

git log

Uh oh!

mergify Bot commented Mar 17, 2026

Uh oh!

sjmonson commented Mar 20, 2026

Uh oh!

sjmonson commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented May 27, 2026

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

sjmonson commented Jun 3, 2026

Uh oh!

mergify Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sjmonson commented Mar 17, 2026 •

edited by github-actions Bot

Loading

sjmonson commented May 11, 2026 •

edited

Loading

mergify Bot commented Jun 3, 2026 •

edited

Loading