Skip to content

tests(perf[conftest]) Pin $SHELL=/bin/sh β€” 51% faster suite#1041

Merged
tony merged 2 commits intomasterfrom
tests-pin-shell-bin-sh
May 10, 2026
Merged

tests(perf[conftest]) Pin $SHELL=/bin/sh β€” 51% faster suite#1041
tony merged 2 commits intomasterfrom
tests-pin-shell-bin-sh

Conversation

@tony
Copy link
Copy Markdown
Member

@tony tony commented May 10, 2026

Summary

Add a 14-line autouse fixture to the root conftest.py that pins $SHELL=/bin/sh for every test. Drops uv run py.test wall time by ~51% on this machine β€” from ~76s to ~36s β€” by bypassing the contributor's interactive shell init (zsh + oh-my-zsh + plugins, etc.) on every tmux pane the suite spawns.

Why this works

tmux falls back to $SHELL for new panes when the default-shell option is unset. libtmux's bundled test .tmux.conf does not set default-shell, so every pane in tmuxp's test suite spawns the contributor's interactive $SHELL β€” typically /bin/zsh β€” and pays the full rcfile cost. With hundreds of pane spawns across the workspace/builder tests, that's the single largest line item in suite wall time.

monkeypatch.setenv("SHELL", "/bin/sh") in a function-scoped autouse fixture forces every test pane to spawn /bin/sh instead. Empty rcfiles, deterministic prompt, no plugin chains. monkeypatch restores the prior value at teardown so no global state leaks.

Measured timing (this machine, uv run py.test)

master                ~76-78s pytest, ~71s wall
this branch (run 1)    37.01s pytest, 35.08s wall
this branch (run 2)    36.24s pytest, 34.29s wall

Both runs: 797 passed, 2 skipped, zero reruns. Master had occasional reruns on capture-pane assertions that depended on prompt timing; pinning the shell removes that source of flake along with the perf cost.

Relationship to libtmux PR #662

This same fix exists upstream in tmux-python/libtmux#662 (open since 2026-04-29), which pins $SHELL=/bin/sh in libtmux's own pytest plugin and reports a similar "~70s β†’ ~32s" delta on tmuxp's suite. Once libtmux #662 merges and ships in a release tmuxp picks up, this commit becomes redundant and can be reverted cleanly (the autouse fixture would just be a no-op repeating what libtmux's plugin already does).

Landing this at the tmuxp conftest level lands the gain immediately rather than waiting on the upstream PR.

What's not in this PR (deliberately)

  • The USING_ZSH constant and zshrc autouse fixture become dead at runtime once $SHELL is always /bin/sh (no zsh first-run greeting to suppress, no rcfile chain to write a quieting .zshrc for). They're left in place to keep this PR's diff to the single load-bearing change. Cleanup belongs in a follow-up β€” easier to review and easier to revert if the perf claim ever needs to be questioned.

Test plan

  • uv run ruff check . --fix --show-fixes β€” clean
  • uv run ruff format . β€” clean
  • uv run mypy β€” clean (124 source files)
  • uv run py.test β€” 797 passed, 2 skipped, no reruns (both runs)
  • just build-docs β€” succeeds
  • Timing comparison vs master β€” see table above

why: Profiling tmuxp's test suite showed the dominant per-pane cost
was tmux spawning the contributor's interactive \$SHELL β€” typically
``/bin/zsh`` with a full rcfile chain (oh-my-zsh, plugins, prompt
themes, etc.) β€” for every pane created across the suite. Each
init costs ~60ms warm and up to ~400ms cold; multiplied across
hundreds of pane spawns in workspace/builder tests, that's the
single largest line item in suite wall time.

tmux falls back to ``\$SHELL`` for new panes when ``default-shell``
is unset (the default in libtmux's bundled test ``.tmux.conf``),
so monkey-patching \$SHELL=/bin/sh in an autouse function-scoped
fixture forces every test pane to spawn ``/bin/sh`` instead.
Empty rcfiles, deterministic prompt, no plugin chains.

A/B measured locally with the project's standard
``uv run py.test`` (defaults: ``--reruns=0 --doctest-modules``):

  master                ~76-78s pytest, ~71s wall
  this branch (1st run)  37.01s pytest, 35.08s wall
  this branch (2nd run)  36.24s pytest, 34.29s wall

That's a ~51% wall-time reduction, consistent with the upstream
libtmux PR (tmux-python/libtmux#662) that pins the same env var
in libtmux's pytest plugin and reports a similar "~70s -> ~32s"
delta. Applying the pin at the tmuxp conftest level lands the
gain immediately without waiting for the libtmux PR to merge and
release; once libtmux ships the upstream pin, this commit
becomes redundant and can be reverted cleanly.

what:
- conftest.py: add ``_pin_test_shell_env`` function-scoped autouse
  fixture that calls ``monkeypatch.setenv("SHELL", "/bin/sh")``.
  ``monkeypatch`` restores the prior value at teardown so no
  global state leaks across tests or out of pytest's process.
- Leading-underscore name signals "internal autouse plumbing" β€”
  no test should depend on it explicitly.
- USING_ZSH constant and ``zshrc`` autouse-on-USING_ZSH fixture
  retained for now; they become dead at runtime once \$SHELL is
  always ``/bin/sh`` (no zsh first-run greeting to suppress) but
  the cleanup is left as a follow-up to keep this commit's diff
  minimal and the perf claim independently revertible.
@codecov
Copy link
Copy Markdown

codecov Bot commented May 10, 2026

Codecov Report

βœ… All modified and coverable lines are covered by tests.
βœ… Project coverage is 81.98%. Comparing base (c675892) to head (a4b416b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1041      +/-   ##
==========================================
+ Coverage   81.96%   81.98%   +0.02%     
==========================================
  Files          28       28              
  Lines        2545     2548       +3     
  Branches      485      485              
==========================================
+ Hits         2086     2089       +3     
  Misses        328      328              
  Partials      131      131              

β˜” View full report in Codecov by Sentry.
πŸ“’ Have feedback on the report? Share it here.

πŸš€ New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

why: Document the contributor-side test-suite speedup landing on
the unreleased tmuxp 1.68.0 block so anyone skimming next-version
notes can confirm the change is dev-only β€” `tmuxp load`,
`tmuxp freeze`, and workspace YAML behaviour are unchanged.

what:
- Add a Development entry under the unreleased 1.68.0 block,
  placed after the existing Documentation section per the
  mandatory section order. Heading style mirrors tmuxp 1.66.0's
  `#### Structured logging with extra context` entry β€” name the
  product surface that changed (test panes), describe what's
  different (spawn /bin/sh instead of \$SHELL), and cap with a
  one-line non-user-impact disclaimer.
- Frame the speedup as "roughly halve... on systems with slow
  shell startup" rather than the precise 51% measured locally β€”
  the gain depends entirely on the contributor's shell init
  weight; quoting a single number would be brittle.
@tony
Copy link
Copy Markdown
Member Author

tony commented May 10, 2026

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Claude Code generated.

@tony tony merged commit 45c076b into master May 10, 2026
13 checks passed
@tony tony deleted the tests-pin-shell-bin-sh branch May 10, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant