Skip to content

feature: top and stats, and the supervisor history behind them - #4

Merged
DamonOehlman merged 4 commits into
mainfrom
damon-tui
Nov 1, 2026
Merged

DamonOehlman merged 4 commits into
mainfrom
damon-tui

Conversation

@DamonOehlman

Copy link
Copy Markdown
Contributor

Summary

ps answers what is running now. This adds the two questions it cannot answer: what is happening second to second, and what happened while I was not looking.

  • top — ps kept live in the terminal: state, health, uptime, restarts, CPU, memory and ports per service, an events pane for what changed between polls, and the selected service's logs. Read-only — it takes no lock and starts or stops nothing, so it runs beside a supervisor.
  • stats — CPU, memory, network and disk per service over a window, summarised from a history the supervisor records.
  • The supervisor records that history — a sampler thread appends the runtime's raw counters every 30 s, kept for seven days.

ps also gains a fix along the way: observe no longer gates on container system status, so ps works over SSH where it previously refused (see Runtime below).

What's in it

Resource history — crates/supervisor

  • history.rs: Sample/Reading and the JSONL file at ~/Library/Application Support/container-compose/history/<project>.jsonl (CONTAINER_COMPOSE_HISTORY_DIR overrides it, which is how the tests stay out of a real history). Raw counters only; rates are derived when read, so the file stays a plain record of what the runtime said. Lines older than seven days are pruned at start and hourly.
  • sampler.rs: a thread, because container stats --no-stream blocks for two seconds and would otherwise sit in front of every restart and probe. It owns its state, shares only the interrupt counter, and reports distinct problems to the loop over a channel once each.
  • run.rs: starts the sampler, and publishes restarts, last_restart_at and gave_up into the health cache — history for a reader, deliberately separate from the tick's attempt counter, which still resets after 60 s of healthy uptime.

stats

container-compose stats [--since 1h] [--json]

A service missing from a sample means the runtime gave no reading — never zero. A restart (a changed container start time) breaks the series rather than producing a negative rate; a gap means no supervisor was running, and the header counts it. --json emits one typed document (samples, first_at, last_at, gaps, and a summary per service). The stack file only orders the rows; without one, services come from the history.

top

container-compose top [--interval 1s] [-n 200]

New crates/top, split so the logic is testable without a terminal: model is pure (a poll and the clock in, rows and events out), poll is the worker thread that owns every runtime call, then app and ui. It polls with container list alone rather than the full observation, so it does not gate on container system status either. It works without the stack file, from the project's containers, and refuses without a terminal, pointing at ps --json.

Dependency: ratatui, default-features = false, features = ["crossterm"] — no calendar/macros/layout-cache features nothing uses, and no separate crossterm pin, since ratatui::crossterm is the crossterm it was built against.

Runtime

  • observe::stats parses stats --no-stream --format json into ObservedStats; PortSpec::display renders the ps form.
  • observe treats container system status as advisory. It answers unregistered to any launchd session other than the one that started the runtime — an SSH login, typically — while list works from anywhere. Status now only supplies the actionable "run container system start" message when list also fails.
  • dev/fake-container grew a stats subcommand; scripts/capture-runtime-fixtures.sh now captures a real container-stats.json, which runtime/tests/captured.rs parses.

Docs

docs/03-cli.md gains top and stats sections, including the runtime quirk found on the midnight-lab node: a container whose PID 1 holds an auto-bound route netlink socket (libp2p's interface watcher does) takes port ID 1, which the VM's init needs for its own statistics socket — the runtime then silently omits that container from container stats. init = true moves the process off PID 1; apple/containerization#910 fixes the bind itself. docs/01-model.md covers the cache's new fields and why the history is not disposable. CLAUDE.md is a regenerated managed block, not hand-edited.

Example output

top (from docs/03-cli.md):

 midnight-lab   supervisor pid 96974  ·  updated 02:33:12Z
  SERVICE   STATE    HEALTH    UPTIME  RESTARTS  CPU   CPU 1h  MEMORY             MEM 1h  PORTS
▶ node      running  healthy   2m 00s  0         0.9%     ▁▂  312 MiB / 1.0 GiB     ▃▃  127.0.0.1:9944->9944
  indexer   running  no check  1m 54s  0         0.5%     ▁▂  158 MiB / 1.0 GiB     ▂▂  127.0.0.1:8088->8088

stats (from docs/03-cli.md):

midnight-lab · last 10m · 3 samples from 2026-10-26T02:31:51Z to 2026-10-26T02:32:56Z

SERVICE  READINGS  CPU AVG  CPU PEAK  MEM AVG  MEM PEAK  LIMIT    NET IN   NET OUT  DISK READ  DISK WRITE  RESTARTS
node     3         0.9%     1.0%      310 MiB  312 MiB   1.0 GiB  286 KiB  114 KiB  1 MiB      404 KiB     0

No screenshot attached: top needs a live supervisor and a terminal, so it was not captured from this branch.

Compatibility

  • The health cache gained three fields, all #[serde(default)]; a_cache_written_before_restart_history_still_reads pins that a cache from the previous version still parses.
  • top and stats are new commands; nothing existing changes shape. stats writes nothing — only a supervisor writes history.

Testing

cargo test --workspace: 195 passed, 0 failed, 1 ignored — the ignored one is a_real_stack_round_trips_through_up_exec_run_ps_probe_and_down, which creates real containers.

33 new tests:

area tests covers
crates/top 19 11 model rules (rows, events, history bars, health labelling), 5 app key paths, 3 drawing including a narrow terminal
supervisor::history 7 summaries, gaps, retention/pruning, a restart breaking the series
supervisor::sampler 2 sampling and warning-deduplication
cli/tests/up.rs 2 stats end to end through the real binary against the fake runtime
runtime/tests/captured.rs 1 the captured container-stats.json parses
supervisor::cache 1 an older cache document still parses
stack::ports 1 PortSpec::display

@DamonOehlman
DamonOehlman merged commit 337b647 into main Nov 1, 2026
4 checks passed
@DamonOehlman
DamonOehlman deleted the damon-tui branch November 1, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant