fix(container): add Muse Glimmer compliance baseline - #78
Draft
jamesdborin wants to merge 170 commits into
Draft
jamesdborin wants to merge 170 commits into
jamesdborin wants to merge 170 commits into
Conversation
Signed-off-by: aidynamo-release-bot[bot] <292185679+aidynamo-release-bot[bot]@users.noreply.github.com>
Signed-off-by: aidynamo-release-bot[bot] <292185679+aidynamo-release-bot[bot]@users.noreply.github.com>
…VEXT,ADMIN_API} to DYN_DISABLE_* (ai-dynamo#11123) Signed-off-by: nnshah1 <neelays@nvidia.com>
Signed-off-by: xianlubird <xianlubird@gmail.com> Signed-off-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Xianlu Bird <xianlubird@gmail.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
…namo#11204) Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com> Signed-off-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Keiven C <213854356+keivenchang@users.noreply.github.com>
…patibility (ai-dynamo#11215) Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
…amo#11242) Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…1247) Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ses (ai-dynamo#11265) Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…namo#11196) Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
…#11261) (ai-dynamo#11271) Signed-off-by: Hannah Zhang <hannahz@nvidia.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: Dan Gil <dagil@nvidia.com>
…ns to 1.16.4 (DYN-3335) (ai-dynamo#11273) Signed-off-by: Matej Kosec <mkosec@nvidia.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com> Signed-off-by: Dan Gil <dagil@nvidia.com> Co-authored-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
…i-dynamo#11295) Signed-off-by: Jie Hao <jihao@nvidia.com>
…#11299) Signed-off-by: Zhuangcheng(Jesse) Gu <zcgu@connect.hku.hk>
) (ai-dynamo#11339) Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com> Co-authored-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
…namo#11278) (ai-dynamo#11333) Signed-off-by: Karen Chung <karenc@nvidia.com>
…ery identities (DYN-3359) (ai-dynamo#11014) (ai-dynamo#11311) Signed-off-by: zhongdaor <zhongdaor@nvidia.com>
…ynamo#11324) Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…amo#11359) Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
…o#10773) (ai-dynamo#11344) Signed-off-by: cmdy <zhang_lin66@foxmail.com> Co-authored-by: cmdy <zhang_lin66@foxmail.com>
) (ai-dynamo#11361) Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
…11379) Signed-off-by: Daniel Gil <dagil@nvidia.com> Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com> Co-authored-by: Dan Gil <dagil@nvidia.com>
…i-dynamo#11340) Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com> Co-authored-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
ai-dynamo#11312) (ai-dynamo#11360) Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
feat(router): add KV affinity routing across token and text backends
#71) * fix(helm): give sub-10s PodMonitor endpoints an explicit scrapeTimeout The 2026-08-04 upstream sync brought in the GMS shadow-failover worker endpoints (system-0, system-1), the frontend-sidecar http endpoint, and the dynamo-epp PodMonitor, all hardcoded at interval: 5s with no scrapeTimeout. Grafana Alloy's default scrape timeout is 10s, and timeout > interval makes the scrape config invalid — Alloy then rejects the entire PodMonitor, not just the offending endpoint, so all worker engine metrics disappeared when 1.4.0 rolled out (and epp metrics never appeared). This is the same failure mode #60 fixed for the primary endpoints. Set scrapeTimeout: 5s on every 5s endpoint, keeping the 5s cadence the failover dashboard relies on. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com> * chore: catch up generated operator docs for the checkpoint job controls helm-docs and crd-ref-docs output for jobPriorityClassName and maxConcurrentJobs, which landed without the operator check running. Applied verbatim from the check's own regeneration diff so 'make check' sees a clean tree. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com> --------- Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…ht tracking Record every dispatched request in a per-endpoint registry keyed by worker instance for the lifetime of its response stream, carrying the request's scheduling priority (nvext.agent_hints / PreprocessedRequest.routing), admission order, and stream context. Covers all router modes through the single dispatch choke point, with RAII release on completion, error, and drop. Counts are exported as dynamo_frontend_worker_admission_inflight / _total gauges so the accounting can be validated against worker-reported load. Accounting only: no admission decision keys off it yet. Disable with DYN_ADMISSION_TRACKING=0. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…ax_num_seqs Read the scheduler's derived effective_max_running_requests_per_dp from /get_server_info internal_states (falling back to the configured max_running_requests server arg) and carry it on the worker's ModelRuntimeConfig. The value is per DP rank, matching the engine's own semantics. This gives the frontend a running-capacity number for openai_backend workers, which previously advertised only KV capacity. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…st chunk _handle_cancellation blocked on the request-id future, which was only resolved from the first engine chunk's meta_info. A cancelled request still sitting in the engine's waiting queue was therefore not aborted until it started generating. Every async_generate call already passes an explicit rid, and the tokenizer manager registers that rid at submission, so the stream processors now pre-resolve the future with the dispatch rid and the monitor arms immediately. The first-chunk path remains as the fallback when no rid is supplied. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…helpers Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
… frontend Bound each worker's engine queue length at the dispatch choke point, opt-in via DYN_ADMISSION_QUEUE_MARGIN — one global margin, no per-model tuning and no capacity knowledge: the engine's own scheduler bounds the running set, the frontend only keeps waiting work shallow. The queue signal is the worker's reported num_waiting_reqs (summed across dp ranks), pushed into the admission state by the worker monitor from load reports; between reports a burst can overshoot the margin by at most one report-interval's arrivals. Below the margin requests admit normally; in selection-free modes a full-queue worker retargets to one with queue headroom; at the margin everywhere, a strictly-lower-priority in-flight request — running or queued, from the worker's whole in-flight set — is evicted (lowest priority first, most-recently-admitted tie-break, which naturally picks engine-queued work) and its stream synthesizes a non-migratable ResourceExhausted overload frame; with no victim the request is rejected with a typed AdmissionRejection that the HTTP layer maps to the overload status with a retry hint. Response bodies stay free of scheduling internals: no priorities, queue depths, or margins — only the generic overload message and retry_after_ms. Victim details go to logs and per-worker eviction/ rejection counters. Mid-stream structured overload errors now surface with their real code instead of the generic internal shape. Workers that have never reported a queue depth are unenforced; everything is off by default and unset knobs are byte-identical to before. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…stry feat(runtime): frontend admission registry for per-worker in-flight tracking
feat(runtime): priority-aware engine-queue bound with eviction at the frontend
…-advertise feat(openai-backend): advertise the engine running-request limit as max_num_seqs
…t-chunk fix(sglang): arm the abort monitor with the dispatch rid, not the first chunk
The bridge only knew how to register Chat|Completions and only resolved an upstream path for `messages` or `prompt`, so a site that attaches engines through it could not offer an embedding model at all. `--embedding-worker` registers ModelType.Embedding and routes requests carrying `input` to the upstream's /v1/embeddings. That path skips the generation machinery outright: no SSE, no stream flags, no tool-call coalescing, no request id, and no abort scheduling, none of which mean anything for a single pooling forward pass. Two details the frontend imposes. The worker hop is always base64, because the frontend decodes base64 to float at the HTTP boundary but never encodes in the other direction — forwarding floats would hand a client that asked for base64 the wrong shape. And the response is deserialized straight into CreateEmbeddingResponse, whose fields are all required, so a missing key is named here rather than surfacing as an opaque Rust deserialization failure. Capacity probing and load reporting are skipped for embedding workers: both key off a KV-usage gauge that a pooling engine never exports. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
Both embedding branches passed the frontend's configured router mode straight into the PushRouter, and PushRouter::generate bails outright in KV mode. A frontend started with `--router-mode kv` therefore failed every /v1/embeddings request with a 500 — "KV routing should not call generate on PushRouter" — no matter which backend served the model. Nothing caught it because no embedding model had been registered against a KV frontend. Pooling models hold no KV cache, so there is no prefix overlap for the KV router to score; the branches already say as much in a comment without acting on it. Fall back to round-robin, which is what uniform-cost single-pass requests want anyway, and log the substitution so an operator who set kv deliberately can see it did not apply here. Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
…ghten comment Signed-off-by: fergus barratt <fergus.barratt00@gmail.com>
feat(openai-backend): serve embeddings from a pooling upstream
… can route to them A rank only became a selectable KV-routing candidate after its first ActiveLoad event arrived, but load events are non-durable and deduped at the worker, so a frontend that subscribed after a rank's bootstrap never learned the rank existed: it received no traffic, so its load never changed, so it never published again. In production this pinned 100% of a dp=8 worker's traffic to the single visible rank. Seed kv_used_blocks and num_waiting_reqs to zero for every configured rank when the runtime config is applied, so a configured rank with no report yet is treated as idle rather than nonexistent. or_insert keeps config re-application from clobbering live reported values. Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
The worker metrics publisher deduped unchanged per-rank load reports forever, so an idle rank published exactly once in its lifetime on a non-durable event plane. Any subscriber that missed that message (a redeployed frontend, or a bootstrap lost to the discovery window) could never converge on the full rank set. Keep the dedupe for change-driven publishing, and additionally re-broadcast the full rank set every DYN_WORKER_METRICS_HEARTBEAT_SECS (default 30s, 0 disables), so any subscriber converges within one heartbeat of joining. A report whose NATS publish failed is also retried on the next beat. Adds e2e coverage for both halves of the fix: a late-subscribing frontend must not pin traffic to the only rank it saw a report for, and the heartbeat must deliver pre-subscription loads. Signed-off-by: Rushil Bhat <bhatrushil@gmail.com>
fix(router): make every dp rank routable despite missed load reports
Signed-off-by: jamesdborin <james.dborin@gmail.com>
jamesdborin
temporarily deployed
to
external_collaborator
August 10, 2026 14:43 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
rst0git
force-pushed
the
main
branch
2 times, most recently
from
September 1, 2026 17:02
b74f1e6 to
cafd486
Compare
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lmsysorg/sglang:dev-muse-glimmerWhy
The Dynamo Muse Glimmer worker build references
sglang@951b7a14, but that baseline was missing from the Dynamo source checkout. The image build therefore failed withFileNotFoundErrorbefore license generation. Without baseline subtraction, the same opaque vendor image produced 183 policy violations because its preinstalled packages were incorrectly attributed to Dynamo.The baseline is pinned to immutable digest
sha256:951b7a14f2279fac122e7198a2eb64d793348ce29969ea0a64e7ea57501b5e33. The captured vendor-to-vendor delta is empty and passes policy.Validation
capture_baseline_sbom.pyforlinux/amd64python3 container/compliance/base_sboms/check_drift.pygit diff --checkPYTHONPATH=container uv run --no-project --with pyyaml python container/compliance/tests/_run_manual.py(14 passed)