[DO NOT MERGE] Reproduce CI setup locally and Catch stalling

[xyao-nv]

Setup

1. Constrains CPUs
2. Optionally wipes Kit caches (shader/USD caches)
3. Phase 1 (in-process camera tests)
4. Phase 2a (subprocess bulk tests)
5. Phase 2b (subprocess isolated)

Steps to reproduce

1. With 4 cpu cores and no cache cleanup, running tests succeed.
   CPUS=0-3 bash scripts/repro_ci_subprocess_stall.sh
   repro_20260409_222116.log

2. With only 2 cpu cores (our runner setup), it reproduces the CI stalling at phase 2a.
   CPUS=0-1 SKIP_CACHE_WIPE=1 bash scripts/repro_ci_subprocess_stall.sh
   repro_20260409_233745.log

Explanations from Claude

Phase 1 (in-process): 95s total, left behind omni.telemetry.transmitter orphan (PID 6774).

Phase 2 (subprocess) — each test spawns Kit, and you can see the degradation:

Test Kit startup Scene creation Total
test_action_chunking_client 23:39:30 → 23:40:19 = 49s 0.66s ~66s
test_external_environment_franka_table 23:40:36 → 23:40:59 = 23s 0.46s ~30s
test_external_environment_franka_table_with_task 23:41:06 → never completes -- STALLED

The pattern is clear: on 2 CPUs, each subprocess Kit instance starts slower, and by the 3rd one it deadlocks entirely. The scene creation itself is fast, but Kit startup is the bottleneck. Each test also leaves behind additional child processes (telemetry transmitters), so by test #3, you have the pytest host + orphans all fighting for 2 CPU cores.

The root cause is confirmed: Kit startup contends for CPU during extension loading/shader compilation, and with only 2 cores + accumulated orphans, it eventually deadlocks.

[kellyguo11]

Review: Reproduction Script for CI Subprocess Stalling

Overall: Solid diagnostic script — well-structured, good logging, and the phased approach (bulk → isolated) is exactly right for root-causing CI stalls. Since this is [DO NOT MERGE], treating it as a debugging artifact rather than production code.

What Works Well

• Clean phased design: cache wipe → in-process → bulk subprocess → isolated fallback
• taskset -c $CPUS to faithfully simulate CI core constraints
• Tee-to-logfile with timestamps — good for offline analysis
• Orphan process detection between tests (show_orphans) is the key diagnostic
• --subprocess-only and --loop modes are thoughtful additions

Suggestions

1. set -euo pipefail + || true pattern — The script uses set -e but then appends || true on most pytest calls. This is fine for the in-process section, but note that in run_subprocess_bulk, there is no || true — if the bulk run fails, it correctly falls through to isolation. Consistent and intentional.

2. kill_orphans is aggressive — pkill -9 -f "isaac-sim/kit/python" could match unrelated processes if someone runs this outside a container. Since this is a repro script meant for containers, it is acceptable, but worth a comment.

3. Missing --loop cache wipe between iterations — In loop mode, wipe_kit_caches runs each iteration, which is correct. But kill_orphans is only called on bulk failure. If loop mode is meant to stress-test accumulation, consider adding an option to skip orphan cleanup between loops to truly simulate process buildup.

4. Hardcoded $PYTHON path — /isaac-sim/python.sh assumes the NVIDIA container layout. A fallback like PYTHON="${PYTHON:-/isaac-sim/python.sh}" would make this usable outside the standard container (already done for other env vars).

5. Log rotation — In --loop mode, each iteration gets the same logfile (since TIMESTAMP is set once). This means all loop iterations append to one file, which is probably fine but worth noting. If separate logs per iteration are desired, move TIMESTAMP inside run_once.

6. Phase 1 has commented-out sections — The Newton and PhysX-no-cameras sections are commented out. Makes sense for focused repro, but a --full-ci flag that enables all sections could be useful for complete CI simulation.

Root Cause Analysis (from the PR description)

The diagnosis is convincing: Kit startup CPU contention + orphaned telemetry processes = deadlock on 2 cores by the 3rd subprocess test. The script proves this clearly. The fix should likely be:

• Kill orphan processes between subprocess tests in CI
• Or increase CI runner core count
• Or add subprocess timeouts with proper cleanup

Nice debugging work. 👍