[03/08] replace graph walking and batching internals with gitoxide-backed history reads by mjc · Pull Request #97 · asinglebit/guitar

mjc · 2026-07-01T21:56:04Z

Stacked on mjc/gitoxide-spike-02-gix-oids.

Why this PR exists:

PR 1 introduced the gitoxide boundary and dependencies.
PR 2 moved commit identity storage onto gix-backed OID aliases.
This PR is the first layer where graph loading behavior changes.
It replaces the libgit2 revwalk-based history loader with a gitoxide-backed commit cursor that produces aliased walked commits, parent aliases, and commit-time metadata in batches.
It also updates the graph buffer/window machinery so the renderer can keep reconstructing visible graph windows while using the new walked-commit stream.

What changed:

Replaces the old git2::Revwalk batcher with a gix-backed lazy commit cursor.
Adds WalkedCommit, carrying:
- gix object ID
- compact alias
- up to two parent aliases
- commit timestamp
Uses commit-graph lookup when available.
Falls back to object-database reads for loose or uncached commits.
Tracks seen commits through commit-graph position bits and loose-OID sets.
Reuses scratch buffers for gix object reads.
Removes the old revwalk Mutex/lifetime workaround path.
Updates Walker to own a gix::Repository.
Enables the history object cache for graph reads.
Uses commit-count hints to reserve OID and buffer capacity.
Seeds the graph cursor from visible branch tips, tag roots, stash roots, and optional HEAD reflog roots.
Streams heatmap counts while commits are walked instead of rebuilding the heatmap from all aliases after traversal.
Updates stash, reflog, branch, tag, parent/child, and lane construction paths to use the new alias stream.
Reworks graph buffer/history storage:
- lane snapshots are smallvec-backed
- deltas are stored in chunked op arenas
- checkpoints are periodic vector entries
- delta ops use compact integer indexes
- buffer/OID storage can shrink after graph loading completes
Updates graph-service window construction around the new buffer/history representation.
Adds commit metadata caching for graph rows.
Adds a graph-service worktree update command/event path.
Moves branch/tag/stash discovery toward gix-backed queries.
Moves HEAD reflog reads to gix-backed reflog entries.
Moves file-history status lookup toward gix tree/diff reads with local caches.
Updates benches and tests for the new graph, batcher, buffer, render, file-history, reflog, and heatmap paths.

Algorithm difference:

History traversal no longer depends on libgit2 revwalk pagination.
The new cursor starts from visible graph roots and walks commits through gitoxide data structures.
Commit priority is timestamp-based through a pending queue, matching the previous graph ordering intent.
Parent aliases are produced during the walk, so lane construction no longer has to rediscover parents through libgit2 for each row.
Graph lane updates still preserve the existing UI model:
- one uncommitted pseudo-row above HEAD
- at most two modeled parents per commit
- stash rows placed near their base parent
- flattened lane behavior for lane caps
- reconstructable graph windows
Heatmap generation now receives commit times during traversal instead of doing a second full pass over sorted aliases.
File-history and reflog helpers now cross more of the same gix boundary used by the graph walker.

Performance story:

This is the first PR in the stack where graph loading performance should change materially.
Expected improvements come from:
- avoiding libgit2 revwalk batching
- using commit-graph data when available
- reusing gix object-read scratch buffers
- avoiding repeated parent rediscovery during lane construction
- streaming heatmap counts during traversal
- reducing graph history allocation through smallvec snapshots and chunked delta storage
- caching commit metadata and file-history status lookups
No whole-app speedup should be claimed without benchmark results.
Bench coverage is updated for batch walking, walker paging, graph projection, and related graph paths.

Benchmark Results for this PR

Command shape: hyperfine --warmup 5 './target/release/guitar /home/mjc/src/linux --exit-when-graph-complete'.

CPU: M1 MacBook Pro 16GB

Ref	Total	vs main	User	System	CPU
`03-graph-walker`	`5.816s`	`27.05x`	`1.82s`	`0.74s`	`43%`

CPU: 5950x Linux/ZFS

Ref	Total	vs main	User	System	CPU
`03-graph-walker`	`2.094s`	`47.39x`	`1.59s`	`0.55s`	`102%`

Memory: 5950x Linux/ZFS

Ref	Peak heap	Peak RSS
`03-graph-walker`	`455.13M`	`1.15G`

Top 10 heaptrack allocation sites for `03-graph-walker`

289528 calls, 19.55M peak - git__calloc
100783 calls, 1.14K peak - git_str_try_grow
89663 calls, 1.27M peak - CRYPTO_malloc
50812 calls, 5.63M peak - git__reallocarray
21137 calls, 518.16K peak - git_pool_malloc
17534 calls, 1.61M peak - <>::finish_grow
17061 calls, 878.28K peak - guitar::core::walker::Walker::new
17056 calls, 0B peak - gix_ref::raw::convert::<>::from
15510 calls, 0B peak - inflateInit2_
13003 calls, 0B peak - __alloc_dir

(cherry picked from commit 122fdf1)

Wire a no-TUI entry point that loads the repo, skips workdir status to avoid case-colliding paths on case-insensitive filesystems, waits for the graph walker to finish, prints the commit count, and shuts down background tasks.

(cherry picked from commit 4b30648)

Copilot

Pull request overview

This PR is a major internal refactor that replaces libgit2 revwalk pagination + graph-buffer internals with a gitoxide (gix) backed history cursor that emits aliased commits (with parent aliases and timestamps) in batches, while keeping the renderer’s “reconstructable window” model intact. It also migrates several git queries (tags, stashes, reflogs, file-history status, heatmap inputs) across the same gix boundary and updates tests/benches accordingly.

Changes:

Replaces revwalk batching with a gix-backed commit cursor (Batcher/WalkedCommit) and updates Walker/graph loading to consume aliased walked commits with parent aliases and commit-time metadata.
Reworks graph history/buffer storage (smallvec snapshots + chunked delta-op arenas + checkpoints) and adapts window construction/rendering to the new representation.
Migrates/supports additional gix-backed queries and tooling (reflog reads, stash/tag enumeration, heatmap streaming, file-history status lookup caching), plus adds test scaffolding and bench updates.

Reviewed changes

Copilot reviewed 53 out of 54 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/tests/helpers/localisation.rs	Switches temp language path creation to `tempfile` and adds a language guard to avoid cross-test global state leakage.
src/tests/helpers/keymap.rs	Adds language guard usage and removes manual language reset in a localisation-sensitive keymap test.
src/tests/helpers/heatmap.rs	Adds tests for new heatmap counting/building behavior and OID prefix lookup.
src/tests/git/queries/remotes.rs	Uses shared test-support repo initialization helper.
src/tests/git/queries/reflogs.rs	Updates reflog tests to use `gix` reflog reader and adds coverage for missing reflog and non-commit entries.
src/tests/git/queries/diffs.rs	Renames local init helper and centralizes repo seeding behavior.
src/tests/git/actions/reverting.rs	Uses shared test-support repo initialization helper.
src/tests/git/actions/rebasing.rs	Uses shared test-support repo initialization helper.
src/tests/git/actions/merging.rs	Uses shared test-support repo initialization helper.
src/tests/git/actions/cherrypicking.rs	Uses shared test-support repo initialization helper.
src/tests/git/actions/branching.rs	Uses shared test-support repo initialization helper.
src/tests/core/walker.rs	Refactors walker tests to use `gix`-backed walker behavior, adds new fixtures, and expands coverage for tags/worktrees/stashes/reflogs.
src/tests/core/renderers.rs	Updates renderer tests to match new graph-history representation and worktree symbol behavior.
src/tests/core/oids.rs	Adds targeted tests for aliasing/prefix lookup and OID round-tripping.
src/tests/core/graph_service.rs	Introduces a harness for graph-service tests; updates window/lookup/file-history tests and adds worktree update event coverage.
src/tests/core/buffer.rs	Replaces tests to validate new delta/log/checkpoint buffer design and memory characteristics.
src/tests/app/state/app.rs	Adds tests for `wait_until_graph_complete` behavior and adapts file-history row shape changes.
src/tests/app/input/navigation.rs	Updates tests for current branch naming and file-history row shape; adds language guard in settings language test.
src/tests/app/input/git.rs	Uses shared test-support repo initialization helper.
src/tests/app/draw/status.rs	Updates `GraphRow` construction to match new fields/defaults.
src/tests/app/draw/settings.rs	Adds language guard to avoid global localisation flakiness.
src/tests/app/draw/search.rs	Updates test row OIDs and removes reliance on a stored `short_oid` field.
src/tests/app/draw/graph.rs	Updates cached history construction to new `GraphHistory` type and removes `short_oid` usage.
src/main.rs	Adds `--exit-when-graph-complete` mode and improves CLI repo-path handling while reusing `App` bootstrapping.
src/lib.rs	Exposes new `git::gix` module and adds `git::test_support` behind `cfg(test)`.
src/helpers/time.rs	Splits timestamp formatting into git2/gix variants and adds UTC date-time formatting support.
src/helpers/heatmap.rs	Reworks heatmap computation to `gix` commit reads and introduces `HeatmapCounts` for streamed counting.
src/helpers/branch_visibility.rs	Adds refname parsing helper to extract local/remote branch names from ref bytes.
src/git/test_support.rs	Adds shared test utilities for repos/commits/worktrees/remotes and a language guard for localisation tests.
src/git/queries/reflogs.rs	Replaces libgit2 reflog reads with `gix`-based reflog iteration and commit filtering.
src/git/queries/file_history.rs	Moves file-history status lookup toward `gix` tree/diff reads with thread-local caches.
src/git/queries/commits.rs	Migrates tip/tag/stash enumeration to `gix`, returns visible walk roots, and updates sorted-oid paging to use aliased walked commits.
src/git/gix.rs	Adds shared `gix` helpers (object cache sizing, commit-graph access, branch tip enumeration, error mapping).
src/core/walker.rs	Updates the history walker to own a `gix::Repository`, stream heatmap counts during walking, and consume aliased walked commits with parent aliases.
src/core/renderers.rs	Adapts graph rendering to the new history representation and optimizes allocations/lifetimes around symbol rendering.
src/core/oids.rs	Replaces OID storage with `gix::ObjectId` + compact alias mapping and adds prefix lookup support.
src/core/layers.rs	Removes per-token `String` allocations by storing `&str` symbols and adjusts empty-token handling.
src/core/chunk.rs	Makes `Chunk` `Copy` to reduce cloning in hot paths.
src/core/buffer.rs	Replaces prior `im`-based snapshot/delta model with smallvec snapshots, chunked delta-op arenas, and checkpointed replay.
src/core/batcher.rs	Introduces `gix`-backed lazy commit cursor with commit-graph acceleration, seen tracking, and aliased `WalkedCommit` output.
src/app/input/worktrees.rs	Updates worktree selection logic to use the new OID alias API.
src/app/input/navigation.rs	Updates parent/child navigation and alias lookup to use new OID accessors and avoid borrowing old OID map internals.
src/app/input/modals.rs	Replaces SHA-prefix lookup scan with `Oids::get_alias_by_prefix`.
src/app/input/git.rs	Adjusts stash OID access to new `Oids::get_oid_by_alias` return type.
src/app/draw/stashes.rs	Fixes stash commit lookup to match new OID accessor type.
src/app/draw/search.rs	Renders SHA prefix directly from `oid` formatting instead of a stored `short_oid` field.
src/app/draw/graph.rs	Generalizes line alignment lifetime and adapts to updated `GraphHistory` handling.
src/app/app.rs	Adds background-task shutdown, adds `wait_until_graph_complete`, improves graph worker disconnect handling, and wires Worktrees update events.
Cargo.toml	Adds `gix`/worktree crates, `iddqd`, `smallvec`, `rustc-hash` and related dependencies for new internals.
benches/render_graph_projection.rs	Adds helper row constructor and updates benchmarks to match new `GraphRow` fields.
benches/fixtures.rs	Updates benchmark fixtures for new `GraphRow` fields/defaults.
benches/commit_batching.rs	Migrates batching/walker benches to `gix`-backed batcher APIs and adds scratch reuse.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mjc · 2026-07-01T22:04:00Z

+fn open_repo(repo: &Repository) -> Result<gix::Repository, git2::Error> {
+    let path = repo.workdir().unwrap_or(repo.path());
+    GIX_REPO_CACHE.with(|cache| {
+        if let Some(repo) = cache.borrow().get(path).cloned() {
+            return Ok(repo);
+        }
+
+        let repo = gix::open(path).map_err(gix_error)?;
+        cache.borrow_mut().insert(path.to_path_buf(), repo.clone());
+        Ok(repo)
+    })
+}


these are fair points, I can do a follow-up PR if desired

+    let ids = (tree_id, parent_tree_id);
+    COMMIT_TREE_CACHE.with(|cache| {
+        cache.borrow_mut().insert(oid, ids);
+    });
+    Ok(ids)


mjc added 23 commits July 1, 2026 12:35

Introduce gitoxide dependency boundary

0b92f7a

Refactor core git test fixtures

25eaac1

(cherry picked from commit 122fdf1)

Keep test support module focused

7fcd544

Explain rustc-hash setup dependency

7aca72a

Harden shared test fixtures

f935bdd

Remove workdir status skip from gix boundary branch

10ac91d

Use gix object ids for commit aliases

da37164

Refactor PR 7 OID and heatmap helpers

61d8599

Refine prefix matcher iterator shape

3e97777

Clarify heatmap grid placement

0fcdc2b

Fix gitoxide oid boundary compilation

ce4f01c

Clarify git hex prefix validation

2339dfe

Rename git oid alias helper

09538cb

Keep heatmap on existing repository

5058ac1

Replace graph walking and batching internals

2bc6792

Refactor core git test fixtures

646e28f

Extract heatmap timestamp counting

3618d12

Refactor graph row payloads

742bfed

Refactor PR #7 heatmap and OID helpers

3a43367

(cherry picked from commit 4b30648)

Refactor graph walker internals

4e0b43f

Refactor buffer and graph service internals

735654a

Refactor graph pane window builders

5d5990a

Copilot AI review requested due to automatic review settings July 1, 2026 21:56

Copilot started reviewing on behalf of mjc July 1, 2026 21:56 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[03/08] replace graph walking and batching internals with gitoxide-backed history reads#97

[03/08] replace graph walking and batching internals with gitoxide-backed history reads#97
mjc wants to merge 23 commits into
asinglebit:mainfrom
mjc:mjc/gitoxide-spike-03-graph-walker

mjc commented Jul 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mjc Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mjc commented Jul 1, 2026

Benchmark Results for this PR

CPU: M1 MacBook Pro 16GB

CPU: 5950x Linux/ZFS

Memory: 5950x Linux/ZFS

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mjc Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants