feat(supervisor): stage gateway configuration snapshot delivery - #3244
feat(supervisor): stage gateway configuration snapshot delivery#3244pimlock wants to merge 10 commits into
Conversation
|
🌿 Preview your docs: https://nvidia-preview-pr-3244.docs.buildwithfern.com/openshell |
|
/ok to test a9387c7 |
pimlock
left a comment
pimlock
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
This Stage 1 implementation is project-valid under #1731 and the initial code review found one blocking fanout-bounding defect. Polling remains authoritative, but the new delivery path can still create fleet-sized queued work in one mutation.
Action required: bound waiting per-sandbox delivery work and add the constrained fleet-fanout regression test described inline.
Blocking findings:
GATOR-d33c96a8-01: Configuration fanout eagerly creates unbounded per-sandbox/component worker tasks.
Carried findings:
- None
Gator metadata
- Validation: Project-valid Stage 1 of the review-ready #1731 plan; author has repository admin authority.
- Docs: Architecture, gateway reference, and troubleshooting guidance updated.
- Checks: Current-head Branch Checks and Helm Lint are pending; Trivy and DCO are green.
- E2E: Gateway/supervisor delivery changes require
test:e2e; dispatch follows after blocking review feedback is resolved. - Head SHA:
d33c96a87392b28484b7ee64cb8fa10ca63c016e - Base SHA:
25021ee31d917074dbfbba971235d85966886a91 - Merge base SHA:
25021ee31d917074dbfbba971235d85966886a91 - Patch ID:
279bc7ec2dd36e79708ea1167ab91ba4b6891e5a - Gator payload:
8 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
pimlock
added
gator:in-review
|
/ok to test e7729ec |
|
Label |
pimlock
left a comment
pimlock
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @pimlock. I checked your worker-bound update in 43f4c9448: fleet fanout now waits for bounded admission before a sandbox/component key enters the pending map, direct publications fail fast at capacity, and the 10,000-recipient regression test confirms pending delivery work stays bounded. The prior GATOR-d33c96a8-01 obligation is resolved, and the follow-up review found no new blocking findings.
Action required: a maintainer must open the current-head Branch E2E Checks run and click Re-run all jobs so the required E2E suite actually executes with test:e2e; gator remains blocked until that run is queued.
Blocking findings:
- No blocking findings remain
Carried findings:
- None
Gator metadata
- Validation: Project-valid Stage 1 of #1731; the implementation remains within the reviewed gateway/supervisor configuration-delivery scope.
- Docs: Gateway and sandbox architecture documentation updated for bounded delivery.
- Checks: Current-head Branch Checks are running; Helm Lint, Trivy, and DCO are green.
- E2E:
test:e2eis applied, but the label-help workflow requires Re-run all jobs on run34533796187; the required jobs have not been dispatched. - Head SHA:
e7729ecca7d22751fa759e737bce2b53e33aedb8 - Base SHA:
5643e1f905f36dbf7d1e922174aaf513919c2260 - Merge base SHA:
5643e1f905f36dbf7d1e922174aaf513919c2260 - Patch ID:
fa533c9c94b2e2738d497548d32c5ede36c12616 - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
d33c96a87392b28484b7ee64cb8fa10ca63c016e - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:blocked - Blocked reason:
test_dispatch_required
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
The startup repair that creates version-one policy history for legacy sandboxes propagated validation failures, so a single stored policy that no longer passes current validation rules prevented the gateway from starting. Skip such sandboxes with a warning and a completion summary so they keep the pre-repair behavior where only their own configuration reads report the failure. Store errors remain fatal. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Fleet-wide configuration changes spawned one snapshot build per connected sandbox and component with no concurrency limit, so a global setting or provider change issued every store query and credential-driver call at once. Gate builds behind a semaphore sized from the database pool and start the build deadline only once a permit is held. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Sandboxes keep their supervisor binary until they are recreated, so a gateway upgrade meets supervisors that predate the handshake and report revision zero. Rejecting them severs every running sandbox with no automatic recovery. Accept revision zero for one release, log a warning per session, and count them in openshell_supervisor_protocol_legacy_sessions_total. The supervisor mirrors the allowance for gateways that predate the handshake. Add a shared ConnectSupervisor test harness and handler-level tests for legacy acceptance and unknown-revision rejection. Move the skill troubleshooting paragraph out of the numbered deployment list so the list renders. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
e7729ec to
70773d3
Compare
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
1 participant
Summary
Add Stage 1 of gateway-pushed configuration over
ConnectSupervisor. The gateway sends complete sandbox configuration and provider-environment snapshots; supervisors receive them while polling remains authoritative.Related Issue
Part of #1731. Replaces #2967. Stage 2 will apply snapshots and acknowledge revisions; Stage 3 will add durable completion semantics and remove polling.
Changes
Testing
Postgres-specific concurrency coverage requires
OPENSHELL_TEST_POSTGRES_URLand was not run.Checklist