perf(isolation): add TCP and DNS benchmark harnesses by drew · Pull Request #3229 · NVIDIA/OpenShell

drew · 2026-09-08T22:33:17Z

Summary

Add benchmark harnesses for the completed RFC 0012 stack. The tests separate startup, DNS, new TCP connections, reused TCP streams, policy denial, and representative live-Internet workloads.

This is a measurement-only layer. General application UDP remains out of scope; DNS still supports its normal UDP and TCP behavior through supervisor mediation.

Related Issue

Part of #1737.

Changes

Add a focused native-versus-filtered TCP microbenchmark.
Measure both intercepted connection setup and the established-stream fast path.
Add live-Internet and end-to-end network scenarios.
Exercise the same authenticated Sandbox Protocol used by the driver stack below.
Keep benchmark code behind the perf-harness feature and outside production paths.

Testing

mise run pre-commit
Full benchmark runs delegated to remote CI or dedicated benchmark hosts
Driver E2E reruns delegated to remote CI

Checklist

Follows Conventional Commits
Authored commits are signed off (DCO)

Stack

feat(isolation): implement the RFC 0012 sandbox architecture #2942 — RFC 0012 core architecture
feat(vm): run the supervisor outside the guest workload #2945 — VM driver
feat(docker): isolate workloads behind the host supervisor #2965 — Docker driver
feat(kubernetes): add proxy-pod isolation topology #3144 — Kubernetes proxy-pod driver
feat(podman): isolate workloads behind the host supervisor #3230 — Podman driver
perf(isolation): add TCP and DNS benchmark harnesses #3229 — performance harnesses (this PR)

copy-pr-bot · 2026-09-08T22:33:20Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-09-08T22:36:02Z

Label test:e2e applied, but pull-request/3229 does not exist yet. A maintainer needs to comment /ok to test ef9f882d2b4cddc495f5489638ee2506c237cd38 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

github-actions · 2026-09-09T04:42:47Z

🌿 Preview your docs: https://nvidia-preview-pr-3229.docs.buildwithfern.com/openshell

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

johntmyers

gator-agent

PR Review Status

The incremental performance-harness layer fits RFC 0012 and stays outside production paths, but three concrete correctness issues need fixes before pipeline handoff.

Action required: @drew, please address the three blocking findings and push an updated head for a focused follow-up review.

Blocking findings:

GATOR-6ea5a915-01: Linux-only harness compilation breaks supported macOS check and lint workflows.
GATOR-6ea5a915-02: worker setup failures can deadlock the benchmark coordinator.
GATOR-6ea5a915-03: tcp-connect reports throughput for bytes it never transfers.

Carried findings:

None

Non-blocking suggestions:

None

Gator metadata

Validation: Maintainer-authored measurement-only layer for the linked RFC 0012 isolation stack and issue #1737.
Docs: Published Fern docs are not needed because this adds opt-in developer performance harnesses without changing product UX.
Checks: Current-head Branch Checks and E2E are failing; review blockers take precedence over CI triage.
E2E: test:e2e is applied and the current-head Branch E2E workflow completed with failures.
Head SHA: 6ea5a915b928fa8b4b62f4a626f7ea9f24dcd9a6
Base SHA: a83bc00027b57eaf852148331f3c92935a4836a5
Merge base SHA: a83bc00027b57eaf852148331f3c92935a4836a5
Patch ID: 093348b2e23d754650572ff00348337fcfce75da
Gator payload: 8
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

johntmyers · 2026-09-11T23:23:29Z

+
+use clap::builder::{PossibleValue, PossibleValuesParser};
+use clap::{Parser, Subcommand, ValueEnum as _};
+use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol};


gator-agent

Warning — GATOR-6ea5a915-01 · Linux-only harness breaks macOS checks

Summary: A contributor running the required Rust check, lint, pre-commit, or CI task on macOS now compiles this binary with perf-harness. This import is unconditional, while openshell_sandbox::perf exists only on Linux, so supported macOS workflows fail before completing their checks.

Fix: Gate the binary implementation for Linux and provide a small non-Linux main that reports the platform requirement, or restrict all perf-specific task invocations to Linux while preserving cross-platform feature checks.

Verify: On aarch64 macOS, run mise run rust:check; both Cargo invocations must complete without an unresolved openshell_sandbox::perf import.

Agent context

Ownership: This PR adds the Linux-gated module and the generic tasks that compile its binary.

Location: crates/openshell-sandbox/src/bin/seccomp-perf.rs:10

johntmyers · 2026-09-11T23:23:29Z

+        }));
+    }
+    drop(sender);
+    barrier.wait();


gator-agent

Warning — GATOR-6ea5a915-02 · Setup errors can deadlock the coordinator

Summary: If any benchmark worker cannot connect or encounters an I/O error during warmup, it returns before reaching this barrier. The coordinator still waits for the original participant count, so the command hangs indefinitely instead of returning the useful setup error.

Fix: Have workers report readiness or initialization failure before timed execution. Release them only after every participant is ready; on failure, cancel or release the remaining participants and return the error promptly.

Verify: Reserve a loopback address, drop its listener, and invoke run_worker(TcpStream, address, 1, 0, 1, 1) through a bounded channel; it must return connection failure before the deadline rather than remain blocked.

Agent context

Ownership: This PR puts fallible initialization and warmup before a non-cancellable barrier.

Location: crates/openshell-sandbox/src/perf.rs:217

johntmyers · 2026-09-11T23:23:29Z

+        capability_scope: protocol.capability_scope().to_string(),
+        elapsed_ms: seconds * 1_000.0,
+        operations_per_second: operations as f64 / seconds,
+        throughput_mbit_per_second: operations as f64 * payload_bytes as f64 * 8.0


gator-agent

Warning — GATOR-6ea5a915-03 · Connect throughput counts bytes never transferred

Summary: An operator running tcp-connect receives application-throughput output computed from payload_bytes, although that protocol only opens connections and never transfers the payload. Changing --payload-bytes therefore changes the reported throughput without changing the measured work, making comparisons misleading.

Fix: Compute transferred bytes per protocol: report zero or an explicit unavailable value for TcpConnect, and define whether TcpStream counts sent bytes or both request and echoed response bytes. Add a regression check for that contract.

Verify: Run native tcp-connect twice with identical iterations and concurrency but different payload sizes; the connect workload must no longer report payload-dependent data throughput.

Agent context

Ownership: This PR introduces the report schema and protocol-independent calculation.

Location: crates/openshell-sandbox/src/perf.rs:241

drew added the test:e2e Requires end-to-end coverage label Sep 8, 2026

drew marked this pull request as ready for review September 8, 2026 22:36

drew requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 8, 2026 22:36

drew force-pushed the codex/1737-rfc12-network-perf branch 2 times, most recently from 3ab20c9 to cbef8e4 Compare September 9, 2026 04:39

drew mentioned this pull request Sep 9, 2026

feat(podman): isolate workloads behind the host supervisor #3230

Open

5 tasks

drew removed this pull request from stack #3152 September 9, 2026 04:42

drew changed the base branch from codex/kubernetes-proxy-pod-isolation to codex/podman-host-supervisor September 9, 2026 04:42

drew added this pull request to stack #3231 September 9, 2026 04:42

drew force-pushed the codex/1737-rfc12-network-perf branch from cbef8e4 to 1f345c2 Compare September 9, 2026 04:53

drew force-pushed the codex/1737-rfc12-network-perf branch from 1f345c2 to 1a88b94 Compare September 9, 2026 05:03

drew force-pushed the codex/1737-rfc12-network-perf branch from 1a88b94 to 002d3e8 Compare September 9, 2026 05:14

drew force-pushed the codex/1737-rfc12-network-perf branch from 002d3e8 to be8526a Compare September 9, 2026 05:27

drew force-pushed the codex/1737-rfc12-network-perf branch 2 times, most recently from 378b896 to e5b5b72 Compare September 9, 2026 06:23

drew force-pushed the codex/1737-rfc12-network-perf branch from e5b5b72 to bdb0acc Compare September 9, 2026 06:37

drew force-pushed the codex/1737-rfc12-network-perf branch from b5f87be to 0cf8481 Compare September 11, 2026 05:57

drew force-pushed the codex/1737-rfc12-network-perf branch from 0cf8481 to 4152a4f Compare September 11, 2026 17:36

drew force-pushed the codex/1737-rfc12-network-perf branch 2 times, most recently from 3bf1b94 to 94073e7 Compare September 11, 2026 17:52

drew force-pushed the codex/1737-rfc12-network-perf branch from 94073e7 to 5d27965 Compare September 11, 2026 19:13

drew force-pushed the codex/1737-rfc12-network-perf branch from 5d27965 to 19abaf4 Compare September 11, 2026 19:48

drew force-pushed the codex/1737-rfc12-network-perf branch from 19abaf4 to 854860e Compare September 11, 2026 20:21

drew force-pushed the codex/1737-rfc12-network-perf branch from 854860e to 84d7d95 Compare September 11, 2026 20:48

drew removed this pull request from stack #3231 September 11, 2026 21:09

drew added this pull request to stack #3293 September 11, 2026 21:12

drew force-pushed the codex/1737-rfc12-network-perf branch from 84d7d95 to d70e714 Compare September 11, 2026 22:13

drew added 2 commits September 11, 2026 15:17

perf(isolation): add TCP and DNS benchmark harnesses

53fb03b

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

fix(perf): align benchmark timing and supported protocols

6ea5a91

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the codex/1737-rfc12-network-perf branch from d70e714 to 6ea5a91 Compare September 11, 2026 22:17

johntmyers reviewed Sep 11, 2026

View reviewed changes

johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Sep 11, 2026

Navigation Menu

Uh oh!

perf(isolation): add TCP and DNS benchmark harnesses - #3229

perf(isolation): add TCP and DNS benchmark harnesses#3229
drew wants to merge 2 commits into
codex/podman-host-supervisorfrom
codex/1737-rfc12-network-perf

drew commented Sep 8, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

johntmyers left a comment

Uh oh!

johntmyers Sep 11, 2026

Uh oh!

johntmyers Sep 11, 2026

Uh oh!

johntmyers Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

drew commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Stack

Uh oh!

copy-pr-bot Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

johntmyers Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

johntmyers Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drew commented Sep 8, 2026 •

edited

Loading