Interaction-based idle culling for VS Code (code-server)

[tylerpotts]

Closes #208

Problem

An open VS Code tab holds a websocket whose keepalives flow through jupyter-server-proxy and count as Jupyter activity, so pods with an idle VS Code tab are never culled. The heartbeat itself is a local file touch; the proxied connection traffic is what defeats culling. The env var proposed in the issue (CODE_SERVER_IDLE_TIMEOUT_SECONDS) only fires after all browser connections close (code-server's idle timer gates on getConnections() > 0), so on its own it does not cover the open-tab case — and it doesn't exist in the previously pinned code-server 4.104.3 (added in 4.106.0 via coder/code-server#7539).

What this PR does

Two composing mechanisms:

1. Interaction-based activity (open-but-idle tabs). The vscode jupyter-server-proxy entry is now registered by this repo (the jupyter-vscode-proxy package is removed) with update_last_activity: False, so VS Code keepalives stop refreshing the in-pod api_last_activity. A bundled server-side VS Code extension (nebari-activity-reporter, plain JS, packaged to a vsix by a stdlib script at image build, installed per-user by a postStart hook) reports real interaction — typing, scrolling, terminal use, focus — and treats a running terminal command as busy (mirrors cullBusy: false). Idle-tab pods are then culled by the in-pod singleuserCuller.server.shutdownNoActivityTimeout (900s default). Note: the hub-level jupyterhub.cull culler still sees CHP-level websocket activity while a tab is connected, so the in-pod culler is the load-bearing mechanism — documented in docs/src/content/docs/configuration.md.
2. The issue's mechanism (disconnected sessions). code-server is bumped 4.104.3 → 4.133.0 (install.sh is byte-identical between tags; hash pin unchanged) and CODE_SERVER_IDLE_TIMEOUT_SECONDS is derived from jupyterhub.cull.timeout (skipped when culling is disabled or timeout ≤ 60, which code-server rejects), so lingering code-server processes exit on the culler's schedule after the last connection drops.

Escape hatch: vscodeActivity.enabled: false reverts to counting raw proxied traffic. The polarity is fail-safe: an image running without the new chart plumbing defaults to the old behavior (over-spending) rather than culling active users who have no reporter installed.

Testing

• 147 unit tests pass (19 new: spawner env wiring incl. boundary/non-numeric cases, chart-derived plumbing, vscode proxy registration, vsix packager incl. XML escaping, postStart hook shell semantics).
• 4 new e2e tests (tests/e2e/test_vscode_idle_culling.py): proxied /vscode/ traffic does not advance last_activity; a contents-API ping does; the extension is installed; the idle-timeout env matches cull.timeout.
• Follow-up commit incoming on this PR: the e2e job pulls the singleuser image tag pinned in values.yaml, so once CI publishes this branch's image I'll run scripts/bump_image_tags.py so the two image-dependent e2e tests exercise the new image.
• Outstanding manual check before merge: browser soak of extension activation (idle tab culls at shutdownNoActivityTimeout; active typing does not).

[github-actions]

Docs preview for fix/vscode-idle-culling:
https://fix-vscode-idle-culling.data-science-pack.pages.dev

[tylerpotts]

CI status note: all checks are green except integration / rbac-bootstrap, which fails identically on latest main (https://github.com/nebari-dev/data-science-pack/actions/runs/32128283826, base commit a1e49ab) with configuration validation failed: repository field is required — pre-existing breakage unrelated to this PR.

The four new e2e tests pass against this branch's built image (sha-7a07d8e): the activity-reporter extension installs via postStart, CODE_SERVER_IDLE_TIMEOUT_SECONDS reaches the pod, proxied /vscode/ traffic no longer advances last_activity, and a contents-API ping does.

[tylerpotts]

Culling behavior reference: what keeps a VS Code pod alive, and where it can cull against user intent

For reviewers (and future docs): the exhaustive list, derived from the implementation in this PR (extension.js + images/nebi/jupyter_server_config.py + singleuserCuller).

What keeps the pod alive (resets the in-pod idle clock)

The extension reports activity (throttled to one ping / 60s) on:

1. Typing / document edits (onDidChangeTextDocument)
2. Cursor or selection movement (onDidChangeTextEditorSelection)
3. Scrolling (onDidChangeTextEditorVisibleRanges)
4. Window focus changes — switching to or away from the tab (onDidChangeWindowState)
5. Opening or closing a terminal
6. Starting or finishing a terminal command (shell-integration execution events)
7. A foreground terminal command still running — while any shell execution is in flight, the extension pings every 60s ("busy = active", mirroring cullBusy: false for kernels). Terminal output is irrelevant either way: a silent 3-hour training run in a bash/zsh terminal keeps the pod alive; detection is via shell-integration execution events, not output (VS Code's output-data event is proposed-API and unavailable to stable extensions).
8. Opening/reconnecting VS Code (activation ping)

Independent of the extension, the pod also stays alive through the pre-existing Jupyter-side signals: a busy notebook kernel (never culled, cullBusy: false), any authenticated Jupyter API traffic (JupyterLab UI use, JupyterLab terminals), and traffic through the default /proxy/<port> routes (those still count as activity — only the /vscode/ route is opted out).

Where it can cull against user intent (false-idle cases)

1. Long-running command in a shell without VS Code shell integration — fish/xonsh/custom shells, or bash/zsh where injection failed or was disabled (terminal.integrated.shellIntegration.enabled: false). Execution events never fire, so the busy-hold never engages. This is the closest real case to the "silent long-running process" example — the trigger is missing shell integration, not missing output.
2. Backgrounded or detached work — ./job &, nohup/disown, or jobs inside tmux/screen after detaching. The foreground execution ends (or the tmux client exits), and the background process is invisible to the extension.
3. Long debug sessions — the extension does not subscribe to debug-session events. A program running under the debugger via the default internalConsole with no user interaction for 15 min gets culled. (A debuggee launched in the integrated terminal with shell integration IS covered via the busy-hold.)
4. Passive reading — staring at code for >15 min without a single scroll, cursor move, or focus change. Any input event resets the clock, so this requires literally zero interaction.
5. Interacting only with a port forwarded through VS Code's Ports tab — that traffic rides the /vscode/ proxy route, which no longer counts as activity. (Using the JupyterLab-side /proxy/<port> route instead does count. And if the server behind the port was started as a foreground terminal command, the busy-hold covers it anyway.)
6. Extension delivery failure — if the vsix install or activation silently breaks, the whole keep-alive channel disappears and active users cull at shutdownNoActivityTimeout (900s). Guards: the e2e install tripwire in this PR, activation logging, and the vscodeActivity.enabled: false escape hatch.

Knobs / workarounds

• Run long jobs in a foreground bash/zsh terminal (busy-hold) or a notebook kernel (cullBusy: false) rather than backgrounded.
• Deployers can raise singleuserCuller.server.shutdownNoActivityTimeout, or set vscodeActivity.enabled: false to restore the old any-open-tab-keeps-alive behavior.
• jupyterhub.cull.maxAge remains available as a hard cap in the other direction.

[viniciusdc]

Late to this and most of what I had is already in your reference post and the description, so just the bits I don't think are covered.

Your item 6, extension-delivery failure. The guards listed are the e2e install tripwire and activation logging, and I think both sit at the wrong layer for the failure that'll actually happen. The chart sets VSCODE_PROXY_UPDATE_LAST_ACTIVITY=false from a chart value, reliably, while the reporter goes in via timeout 60 code-server --install-extension ... || true. Separate failure domains, so a single user can land with proxy activity disabled and no reporter, and cull at 900s while actively working. Wedged extensions dir, stale vsix on the home PVC, the timeout tripping on a slow mount. The e2e test catches build regressions, the logging is after the fact, and vscodeActivity.enabled: false is a deploy-time switch for a per-pod runtime failure.

Could the two share a fate? postStart writing a marker on success and jupyter_server_config.py requiring both the env var and the marker would make a failed install degrade to over-spending, which is the polarity you argue for everywhere else.

CODE_SERVER_IDLE_TIMEOUT_SECONDS derives from cull.timeout and is gated on cull.enabled, but you also establish that cull is the culler CHP defeats. Should it key off singleuserCuller.server.shutdownNoActivityTimeout instead? As written, a deployer disabling the hub culler silently turns the code-server timer off too.

For the soak: Heart.isActive is getConnections() > 0 behind a 5s timeout that resolves false, with a comment above it saying "getConnections appears to not call the callback when there are no more connections. Feels like it must be a bug?"

https://redirect.github.com/coder/code-server/blob/v4.133.0/src/node/routes/index.ts#L35-L53

If that trips with a browser attached the state goes expired and code-server exits 1800s later under someone. heart.beat() on any HTTP request clears it so it needs a quiet-HTTP window, but worth watching rather than assuming.

Last one, minor: /api/contents/ works because _track_activity isn't set on that handler upstream, and that flag has trended toward covering more read-only endpoints. If contents ever joins them the ping stops silently. last_activity() folds in any settings key ending _last_activity and calls it a hook for extensions, so a small server extension setting vscode_last_activity would sit on the documented seam instead.

Hey @krassowski , pinging you here because you might have some idea around idle culling or code-server on the Jupyter side. Short version is that an open but idle VS Code tab keeps the pod alive, since jupyter-server-proxy counts the websocket keepalives as API activity, and this PR opts the /vscode/ route out of activity reporting and feeds real interaction back in from a small VS Code extension. Mostly curious in your perspectives around this... or similar type of problems (in case code-server was a problem in the past or not)

[tylerpotts]

@viniciusdc all four points were on target. What happened with each:

Extension-delivery shared fate: implemented (f656491). Went with a variant of your marker idea: instead of a postStart-written marker, jupyter_server_config.py gates the opt-out on the installed artifact itself, globbing the extensions dir for nebari.nebari-activity-reporter-* before setting update_last_activity: False. A failed install now degrades to over-spending, matching the polarity everywhere else. One caveat baked into the code comment: postStart runs concurrently with the entrypoint, so on a user's first-ever spawn the install can lose the race against config evaluation and that one session over-spends; the PVC-backed extensions dir makes every later spawn see the artifact. (CI's e2e proxy-traffic test implicitly exercises the gate and passed, meaning the install won the race there; if that test ever flakes red, this race is why, and the failure direction is the safe one.)

CODE_SERVER_IDLE_TIMEOUT_SECONDS keying: implemented (f656491). Now derives from singleuserCuller.server.shutdownNoActivityTimeout via the chart-derived plumbing, and the cull.enabled gate is gone, so disabling the hub culler no longer silently kills the code-server timer. 0 and anything <= 60 (code-server rejects those) skip the env var. The two schedules also no longer disagree by default (both 900 now, vs 1800/900 before).

Heart.isActive / getConnections: watched on a live cluster. Soak config drives the timer down to 120s, so one 7-minute session of websocket-only editor use covered multiple potential expiry cycles with a connected client. code-server logged Idle timeout set to 120 seconds at startup and showed zero exit/restart markers; last_activity advanced every 60-62s from the reporter the whole time. Not proof the callback bug can never trip, but it survived the tightest window we can give it. If it ever does trip in the field, the fix is dropping the env var (one line in the spawner); the in-pod culler is unaffected.

/api/contents/ vs the *_last_activity settings seam: agreed, deferred. Moving to a vscode_last_activity settings key means shipping a jupyter server extension plus retargeting the reporter's ping, which is real scope for a hardening concern. The e2e test pins the current contract (contents ping advances last_activity) and fails loudly on the image bump that breaks it. Will open a follow-up issue referencing your comment so it doesn't get lost.

Full soak on the review-fix image (sha-ce941be) also re-verified the headline behaviors: busy-hold kept a hands-off sleep 300 pod alive with pings every 60s, and the pod culled ~2 minutes after the sleep ended with the tab still open.