feat(dev): add local model serving tool

[binaryaaron]

Summary

• Add a source-tree developer tool that compiles and manages local vLLM inference services from pinned TOML profiles.
• Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through the pinned external vLLM Factory integration.
• Preserve pure compilation, digest-protected plans, task-aware capability probes, PID-reuse protection, and bounded process-group cleanup.
• Ship seven profiles: NVIDIA GLiNER, GLiNER2, TinyLlama, GPT-OSS 20B and 120B, Qwen3 30B A3B Instruct, and NVIDIA Nemotron 3.5 Lightning.
• Add deployment guidance and a checked-in NVIDIA GPU container image that runs the same lifecycle inside the container boundary.

Scope and lifecycle

The tool owns one deployment domain: a managed local-process vLLM server.

• The v2 input schema has four sections: task, model, vllm, and local.
• Lifecycle commands are compile, launch, probe, status, and stop.
• Factory-backed detection and stock vLLM generation share one compiler and lifecycle.
• The supplied container runs that unchanged local-process lifecycle. It does not add a second compiler path.
• Docker-specific compiler branches, native Transformers GLiNER serving, and Hugging Face cache discovery remain outside the design because no shipped profile requires them.
• Plan SHA-256 digests detect accidental storage or transport changes. They do not authenticate plan writers or prove semantic validity.

The tool remains under tools/ and is not part of the distributable Anonymizer package. It does not attach to existing endpoints or manage remote compute.

Validation

Validation on rebased HEAD 836a13f:

• Focused inference-service and vLLM Factory tests: 42 passed.
• make check: passed, including formatting, lint, ty, lock consistency, and SPDX checks.
• make test: 1,307 passed and 3 expected vLLM Factory skips.
• make docs-build: strict build passed.
• Range-diff matched all 17 product commits exactly after rebasing onto current main.
• Every rebased product commit has a DCO trailer and a valid repository ED25519 signature.

Runtime evidence from the patch-equivalent pre-rebase tree:

• Built tools/inference_service.Dockerfile as a real GPU image. The packaged CLI exposed all lifecycle commands and the vLLM Factory compatibility doctor passed.
• Co-hosted pinned NVIDIA GLiNER and GPT-OSS 20B services on one A100 80GB. Capability probes passed for dynamic labels, offsets, scores, and chat completions.
• Processed all 25 records in docs/data/NVIDIA_synthetic_biographies.csv with zero workflow failures.
• Published a sealed PR-versus-main pair to the W&B scorecard. The PR candidate completed in 458.09 seconds versus 558.92 seconds for the August 14 main baseline.
• Both managed services stopped through their receipts with cleanup_complete=true; final A100 memory use was 0 MiB.

Fresh NVSkills validation is requested after the rebase so generated skill artifacts and signatures bind to the new commit chain.

Fixes #252

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Post-rebase A100 dogfood completed on HEAD 1dd931ea8b7cf558c22b3cd2c2916d746492ca19 using the managed compiled plans and local endpoints for nvidia/gliner-pii plus pinned nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (33268dc8a6da85a56be2b12241453e4e1237bbe1).

• Full benchmark: 50 records across biographies redact+evaluate and legal hash+evaluate
• Both cases completed on their first attempt
• Model requests: 200 total, 200 successful, 0 failed
• Pipeline errors: 0 stage, 0 NDD workflow, 0 model workflow
• Replacement completeness: 0 missing final entities and 0 missing final values
• Managed shutdown: both services terminated with cleanup_complete: true
• Final A100 state: 0 MiB used

The analyzer reproduced the seven previously verified measurement false positives (1 biographies, 6 legal). No leak-metric changes are included in this hoster PR.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added pinned GPT-OSS 120B, GPT-OSS 20B, and Qwen3 30B A3B Instruct profiles on rebased HEAD 9d9536b990738e409e2ea335dd7d4b051dcd9c74. GPT-OSS 120B loaded successfully on the A100 through vLLM 0.27.1; the managed capability probe passed after accounting for reasoning-token output, and cleanup returned GPU use to 0 MiB. Fresh validation: 1,260 local tests passed, make check passed, strict docs build passed, and GitHub tests pass on Python 3.11, 3.12, and 3.13. NVSkills live evaluation is running.

[binaryaaron]

Medium hoster simplification is now on ccfedc8a4aeeb353d90bfc7692b4cc10a2c2c887.

• Removed Docker, native Transformers GLiNER, and cache-discovery branches.
• Compacted profiles to intent schema v2 with task, model, vllm, and local sections.
• Kept compile/launch/probe/inspect/cancel, plan digests, receipts, secrets, PID identity checks, bounded cleanup, LoRA, vLLM controls, and Factory-backed detection.
• Net hoster reduction: 988 production lines. Dedicated tests: 1,705 to 852 lines.
• Local validation: 1,226 tests passed, 35 focused inference-host tests passed, make check passed, and strict docs passed.
• A100 dogfood: local GPT-OSS 20B plus NVIDIA GLiNER passed managed probes; a two-record Anonymizer run found and redacted four PII entities with zero failed records; both receipt-owned services canceled cleanly; final GPU use was 0 MiB.

The implementation began as an Engineer Arc candidate. Parent review rejected its over-aggressive test reduction, restored lifecycle-focused behavioral coverage, removed a stray compiler branch, and reran all validation before push.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added the local-model deployment docs and checked-in GPU container image in commit 0034b8f. Validation: full suite 1,226 passed; make check passed; strict docs build passed; Docker image build passed; packaged CLI and vLLM Factory compatibility doctor passed inside the image.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Modern Python boundary refactor is now on HEAD ff40e2b.

• Closed FactoryPlugin typing now flows through intent, compiler, runtime, Factory integration, and adapter boundaries.
• Exhaustive match/assert_never dispatch covers owned task, environment, and plugin unions.
• Internal multi-field products are frozen slotted dataclasses; Factory metadata is one immutable exhaustive specification.
• CommandSpec owns redacted rendering, secret-source inventory, and strict execution-time resolution.
• HTTP client ownership is context-managed, compiler argument construction is more declarative, and obsolete delegation/test-loader seams are removed.
• CLI translation remains narrow while preserving exit 125 for known filesystem, transport, integrity, runtime, validation, TOML, and empty-source-revision failures.

A Terra implementation and independent-review cycle found and resolved plugin-spec duplication, secret ownership, positional chunk tuples, and stale delegation seams. Primary-agent review then caught the empty-source-revision exit regression and the remaining manual HTTP ownership flag before push.

Validation on the exact signed commit chain:

• focused inference-service tests: 36 passed, 3 skipped
• make check: passed, including ty
• make test: 1,227 passed, 3 skipped
• all four commits have DCO signoff and valid repository ED25519 signatures

No bundled skill files changed, so no new /nvskills-ci request is needed.

[binaryaaron]

Semantic cleanup is now on HEAD 7cc0c58037e64ae31f8f8ccfdfb5e5be3f817d9a.

• Reframed the draft input as LocalInferenceServiceSpec; profiles now use inference-service.local-spec/v2 in place, with no compatibility layer or schema-v3 migration.
• Renamed endpoint and compatibility concepts to EndpointAddress and CompatibilityAssessment; lifecycle commands are now status and stop.
• Removed redundant ServiceCompilation, CommandArgument, Generation.chat, RunPlan.runtime, intent_digest, declared_capabilities, and duplicate readiness coordinates.
• Preserved the pure spec-to-plan compiler, plan digest, secret-safe command resolution, task-aware readiness probing, process-group cleanup, and separate parent/child runtime boundaries.
• Tightened process identity: launch now fails closed and cleans up if a PID start marker cannot be recorded.
• Updated the deployment docs, README, devnote, profiles, and bundled Anonymizer skill to the new vocabulary.

A Terra reviewed-engineering Arc performed implementation, independent review, bounded fix-up, and confirmation. Review initially blocked on the PID-marker fallback and stale shipped instructions; both were fixed, and confirmation accepted the exact final tree with no findings.

Validation on that tree:

• focused inference-service tests: 24 passed
• make test: 1,229 passed, 3 skipped
• make check: passed, including formatting, lint, ty, lock, and SPDX checks
• make docs-build: passed
• final commit has DCO signoff and a valid repository ED25519 signature

[binaryaaron]

/nvskills-ci

[binaryaaron]

Follow-up council warnings addressed in abf8a7e:

• Bound the missing-start-marker cleanup path across TERM and KILL waits; an unreapable child now returns the normal diagnostic with cleanup_complete=false instead of hanging.
• Made plan validity explicit: SHA-256 detects accidental storage/transport changes only. The developer tool trusts plan writers and does not authenticate, sign, or recompile plans for semantic proof.
• Replaced stale reconnectable/proves/intent wording in CLI help, runtime/compiler docstrings, README, and deployment docs.

Validation:

• focused inference-service tests: 25 passed
• make check: passed
• make docs-build: passed
• commit has ED25519 signature and Signed-off-by trailer

[binaryaaron]

Post-fix A100 dogfood passed on HEAD abf8a7e.

• Managed services: pinned NVIDIA GLiNER on 127.0.0.1:8001 and pinned GPT-OSS 20B on 127.0.0.1:8000; GPT-OSS used the documented co-hosting cap of 0.75 GPU memory utilization.
• Launch receipts recorded exact PID/start markers. GLiNER passed dynamic-labels, offsets, and scores; GPT-OSS passed chat-completions.
• Required preview: 2 records, 14 entities across 8 labels, 0 failed records.
• Full Anonymizer Replace + Substitute run: 2/2 records transformed, 0 failed records, and no detected source value remained verbatim in substituted text.
• Local serving handled 24 chat-completion requests with no ERROR, Traceback, or Exception entries in either service log.
• Receipt-based stop: both services terminated with cleanup_complete=true; both status receipts report stopped; final A100 usage is 0 MiB.

Run artifacts are retained locally under /tmp/anonymizer-abf8a7e-full-run-U7pB7p.

[binaryaaron]

Documented the existing NVIDIA W&B PR-vs-main scorecard workflow in tools/measurement/README.md (commit 6bff538). The new section records the exact eligibility and comparison identity contract, explains why native online runs do not enter the saved scorecard, and identifies the external snapshot-publisher boundary. Validation: focused pre-commit hooks passed, make docs-build passed, and git diff --check passed.

[binaryaaron]

Published a provenance-correct full-dataset PR-vs-main pair to the existing W&B scorecard.

• Scorecard: https://wandb.ai/nemo-llm-service/nemo-anonymizer-benchmarks/reports/PR-vs-main-scorecard--VmlldzoxNzY4NTMzMA
• PR candidate (abf8a7e): https://wandb.ai/nemo-llm-service/nemo-anonymizer-benchmarks/runs/5716ea98e1413d1409acde5782b72512
• Current-main baseline (3c074eb): https://wandb.ai/nemo-llm-service/nemo-anonymizer-benchmarks/runs/5df07a70395f9c12277998529b88c256
• Workload/config: all 25 rows of NVIDIA_synthetic_biographies.csv; substitute-local-models; pinned GPT-OSS 20B + NVIDIA GLiNER endpoints
• Result: 25/25 completed and 0 failed records on both sides
• Total elapsed: candidate 458.09s vs main 558.92s (candidate 18.0% lower)
• Detection: candidate 296.54s vs main 389.40s (candidate 23.8% lower)
• Replacement: candidate 161.45s vs main 169.42s (candidate 4.7% lower)

Both 31-record measurement files were sealed and strictly imported with matching scorecard identities. The candidate uses measurement schema v1 and current main uses v2, so renamed replacement/leak counters appear separately instead of being treated as identical metrics. The refreshed snapshot was verified to contain both run IDs. Managed services were stopped cleanly; final GPU usage was 0 MiB.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[greptile-apps]

Greptile Summary

The PR adds a source-tree tool for compiling, launching, probing, monitoring, and stopping local vLLM inference services from pinned profiles.

• Supports stock generation models and GLiNER-family detection through vLLM Factory.
• Adds digest-protected plans, process receipts, capability probes, lifecycle cleanup, seven model profiles, tests, container deployment, and documentation.
• The prior per-request fan-out issue is addressed by rejecting requests above 256 chunks and limiting each accepted request to eight concurrent pooling calls.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the eligible follow-up scope.

No blocking failure remains; the previously reported per-request pooling fan-out is now constrained by both a chunk admission limit and a bounded worker frontier.

Important Files Changed

Filename	Overview
tools/inference_service_compiler/vllm_factory_adapter.py	Adds the GLiNER chat-to-pooling adapter, request validation, chunk admission cap, bounded per-request workers, and entity normalization; the prior per-request fan-out finding is fixed.
tools/inference_service_compiler/compiler.py	Compiles validated local-service profiles into frozen run plans with commands, compatibility information, endpoints, capabilities, and integrity digests.
tools/inference_service_compiler/runtime.py	Implements task-aware endpoint probing and receipt-based local-process lifecycle handling.
tools/inference_service_compiler/vllm_runtime.py	Translates compiled settings into vLLM engine and frontend arguments and installs the Factory detector middleware.
tools/inference_service_compiler/models.py	Defines the typed profile, plan, probe, process-handle, and lifecycle-receipt contracts.
tests/tools/test_vllm_factory_adapter.py	Covers request parsing, the 256-chunk admission limit, the eight-call per-request concurrency frontier, ordering, and GLiNER response normalization.
tools/inference_service.Dockerfile	Adds the NVIDIA GPU container environment for running the same source-tree lifecycle inside a container boundary.

Sequence Diagram

sequenceDiagram
    actor Developer
    participant CLI as inference_service.py
    participant Compiler
    participant Plan as Checksummed plan
    participant Runtime
    participant vLLM
    participant Receipt as Launch receipt
    Developer->>CLI: compile(profile)
    CLI->>Compiler: validate and compile
    Compiler->>Plan: write plan and digest
    Developer->>CLI: launch(plan)
    CLI->>Plan: verify digest
    CLI->>Runtime: start process group
    Runtime->>vLLM: launch configured service
    Runtime->>vLLM: model and capability probes
    Runtime->>Receipt: record process identity and probe
    Developer->>CLI: status or stop(receipt)
    CLI->>Receipt: verify process identity
    CLI->>Runtime: inspect or terminate process group

Loading

_{Reviews (2): Last reviewed commit: "fix: bound vLLM pooling request fan-out" | Re-trigger Greptile}

[greptile-apps]

Unbounded pooling request fan-out

When a client submits a large detector request with chunk_length=1, overlap=0, and non-empty labels, this asyncio.gather eagerly schedules one pooling operation per character, exhausting serving-process memory and pooling capacity and disrupting other inference requests. Bound the number of chunks and concurrent pooling calls per request.