Skip to content

Build the graph in Rust - #338

Draft
Peter554 wants to merge 9 commits into
python-grimp:mainfrom
Peter554:build-graph-in-rust
Draft

Peter554 wants to merge 9 commits into
python-grimp:mainfrom
Peter554:build-graph-in-rust

Conversation

@Peter554

@Peter554 Peter554 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

build_graph now builds the graph entirely in Rust: finding the modules, reading and writing the cache, scanning the imports and assembling the graph. Previously only the import scanning was in Rust, and everything else went through Python objects.

This reworks the demo in #264, rebased onto current main. This version is smaller, because it reuses the Rust that already exists (scan_for_imports_no_py, the JSON cache (de)serialisation, the encoding-aware file reading) rather than reimplementing it. It also removes the Python implementation, and everything that only it used.

Compatibility

  • Same graphs. Before the swap, the functional tests (including namespace packages, multiple roots, encodings, lazy imports, syntax errors and the real-package snapshots) were run against both implementations. I also compared the graphs from both on the Kraken codebase, with and without include_external_packages: modules, squashed modules and every import detail were identical.
  • Same cache. The file names, JSON format and schema version are unchanged, so existing caches keep working. (serde_json's float_roundtrip feature is needed so that mtimes compare equal after being read back.)

Performance

Measured on the Kraken codebase, with a release build.

before after change
uncached 9.11 s 6.04 s -34%
cache cold (writes the cache) 10.34 s 6.59 s -36%
cache warm (all hits) 7.01 s 3.65 s -48%

With follow-ups (see "Potential follow-ups" below) the performance gains could likely be even more. This change opens the door to further optimisation. Rough experimentation on a separate branch suggests it's possible to drop another second or so with parallel package discovery, for example.

Removed adapters and ports

This removes most of the adapters and ports that build_graph was built from: the module finder, the cache and the file system (grimp.adaptors.{modulefinder,caching,filesystem} and grimp.application.ports.{modulefinder,caching,filesystem}), along with grimp.application.scanning and the settings that wired them up. The package finder adapter remains.

This is a significant change to the architecture, so it requires justification. The reason is performance. Those ports sat on the boundary between Python and Rust, so everything crossing them had to be converted to and from Python objects: every module found, every cached import and every scanned import. The graph was then assembled one add_import call at a time. Keeping the ports would have meant having Rust implement them one call at a time, which keeps the conversion cost that this PR removes.

The main thing lost is the ability to substitute a fake file system in unit tests, which is why the tests have been ported to use real files (see below).

Tests

The unit tests of the Python implementation (module finding, import scanning, the cache and the arguments of build_graph) used an in-memory file system. The Rust implementation reads the file system directly. The tests have been ported to (python) functional tests that write real files to a temporary directory. They exercise the code path that users actually run.

The tests are still fast. On my machine the ported tests take under a second, and the whole suite still takes around 2 seconds, as before.

Notes

  • New dependencies: blake2 (cache file names, to match the existing blake2b hashes) and unicode-ident (to match str.isidentifier when skipping directories).
  • Logging. Rust can't use Python's logging, so rust.build_graph returns the messages to log along with the graph, and build_graph logs them. The loggers and messages are the same as before (grimp.adaptors.modulefinder for the warning about skipping a module with dots in its name, and grimp.adaptors.caching for the cache messages). An alternative would be the pyo3-log crate, which forwards messages from Rust's log crate to Python's logging as they happen. I haven't used it because it would add two dependencies (log and pyo3-log), and mean taking the GIL from code that deliberately runs without it, for only a handful of messages per build. It also caches the levels of the loggers, which can go stale if logging is configured after import. Returning the messages keeps Python's logging out of the Rust code, and is simple to test.

Potential follow-ups

I haven't profiled where the remaining time goes, so these are hypotheses until measured:

  • Parallel module discovery. The directory walk is sequential, and it stats every Python file for its mtime. It could be parallelised, for example with rayon or the ignore crate's parallel walker.
  • A faster cache format. The cache is JSON, which is comparatively slow to parse and write. A binary format (as Demo: Build graph rust (Sequential) #264 used) would be faster. Changing it means bumping the schema version, so existing caches would be rebuilt once. For this PR we've kept full cache compatibility.
  • Simplify the file system code. FileSystem now has a single implementation, and module finding bypasses it to use std::fs directly. It could be replaced by a concrete type.
  • grimp.DirectImport. It's still exported, but nothing in Grimp produces or consumes it any more (only tests do). Worth deciding whether to keep it as public API.
  • Remove the last adapter. The package finder is now the only adapter. It has a single implementation, settings exists only to hold it, and nothing substitutes a fake for it. It could be replaced by a plain function, which would also mean dropping the adaptors layer from the Import Linter contract. I've left that for a follow-up so that it can be discussed separately.

@codspeed

codspeed Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.8

⚡ 5 improved benchmarks
✅ 21 untouched benchmarks
⏩ 23 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_build_django_from_cache_a_few_misses[15] 169 ms 57.4 ms ×2.9
⚡ test_build_django_from_cache_a_few_misses[2] 162.2 ms 55.4 ms ×2.9
⚡ test_build_django_from_cache_no_misses 158.1 ms 54.3 ms ×2.9
⚡ test_build_django_from_cache_a_few_misses[350] 297.3 ms 103.6 ms ×2.9
⚡ test_build_django_uncached 150.8 ms 59.5 ms ×2.5

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing Peter554:build-graph-in-rust (b971ea9) with main (dde21ee)

Open in CodSpeed

Footnotes

  1. 23 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@Peter554
Peter554 force-pushed the build-graph-in-rust branch from 657c163 to ac54edb Compare November 19, 2026 07:09
@Peter554 Peter554 changed the title Add build_graph_rust, building the graph in Rust Build the graph in Rust Nov 19, 2026
@Peter554
Peter554 force-pushed the build-graph-in-rust branch from 5ffa3be to c342c11 Compare November 19, 2026 07:37
This is consistent with FileSystem::read, and means writes can be used from
Rust code that doesn't hold the GIL. I/O errors are surfaced as OSError.

Assistant-model: Claude Code
Converts a GrimpError into a Python exception, including translating
syntax errors into SourceSyntaxError. This will be reused when building
graphs in Rust.

Assistant-model: Claude Code
Builds an import graph entirely in Rust: finding modules, reading and
writing the cache, scanning imports and assembling the graph. This avoids
the cost of converting everything to and from Python objects.

- find_package is a port of the Python ModuleFinder, including namespace
  packages. It also returns file mtimes, in the same form as os.path.getmtime.
- Cache reads and writes the same files as the Python cache, so the two are
  interchangeable. serde_json's float_roundtrip feature is needed for mtimes
  to compare equal after being read back.
- Import scanning reuses the existing scan_for_imports_no_py.

Assistant-model: Claude Code
A faster alternative to build_graph, which builds the graph in Rust. It
takes the same arguments, and shares its cache with build_graph.

The functional tests now run against both implementations, via a
build_graph fixture. The test that checks the cache logs a message only
asserts that for the Python implementation, as the Rust one doesn't log.

Assistant-model: Claude Code
@Peter554
Peter554 force-pushed the build-graph-in-rust branch from c342c11 to ff5cfe5 Compare November 19, 2026 07:49
The unit tests of module finding, scanning, caching and build_graph's
arguments use an in-memory file system, so they only cover the Python
implementation. The Rust implementation that is about to replace it reads the
file system directly, so there is nothing to substitute. Keeping the
in-memory file system for the tests alone would mean keeping production code
that only they use: the Python-facing scanning and cache functions, and the
fake file system itself.

Port them instead to functional tests that write real files to a temporary
directory. These test the code path that users actually run. They run against
both implementations (via the build_graph fixture), ahead of replacing the
Python implementation, which shows that the Rust one behaves the same.

The tests are still fast: the ported tests take under a second, and the whole
suite still takes around 2 seconds.

Behaviours that are only observable through log messages are not ported, as
the Rust implementation doesn't log.

A write_files fixture is added, which writes files to a temporary directory
on sys.path.

Assistant-model: Claude Code
build_graph now builds the graph in Rust, and build_graph_rust is removed.

Remove everything that only the Python implementation used:

- Python: the module finder, cache and file system adaptors and ports, the
  scanning module, and the Settings entries and copy method they needed. The
  package finder now uses os.path rather than a file system port.
- Rust: the Python-facing scan_for_imports and cache functions, the
  conversions to and from Python objects, the in-memory file system and the
  Python wrappers around the real one, and the cache exceptions, which were
  never raised to Python by the new implementation. Rust only parses the
  cache to decide whether it can be used, so parse_json_to_map now returns an
  Option.
- Tests: the unit tests of the removed code (ported to functional tests in
  the previous commit), their fakes, and the build_graph fixture which ran
  the functional tests against both implementations.

Assistant-model: Claude Code
@Peter554
Peter554 force-pushed the build-graph-in-rust branch 2 times, most recently from 3a03850 to b2711dd Compare November 19, 2026 08:16
@Peter554 Peter554 self-assigned this Nov 19, 2026
Rust can't use Python's logging directly, so rust.build_graph now returns
the messages that should be logged, along with the graph, and build_graph
logs them. The loggers and messages are the same as before the module finder
and the cache were ported to Rust, so that nothing changes for anyone who
configures them:

- grimp.adaptors.modulefinder: the warning about skipping a module with dots
  in its name.
- grimp.adaptors.caching: whether each cache file was used, missing, corrupt
  or written by a different version of Grimp, and the files written.

This restores the assertions on the log messages that couldn't be ported to
the functional tests, and adds tests of the cache messages.

An alternative would be the pyo3-log crate, which forwards messages from
Rust's log crate to Python's logging as they happen. It isn't used because it
would add two dependencies (log and pyo3-log), and mean taking the GIL from
code that deliberately runs without it, for what is only a handful of
messages per build. It also caches the levels of the loggers, which can go
stale if logging is configured after import. Returning the messages keeps
Python's logging out of the Rust code, and is simple to test.

Assistant-model: Claude Code
These were already unused before build_graph was replaced: nothing read
settings.TIMER, and no Rust code used serde or serde_yaml.

Assistant-model: Claude Code
@Peter554
Peter554 force-pushed the build-graph-in-rust branch from b2711dd to b971ea9 Compare November 19, 2026 08:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant