Repository navigation
Conversation
Merging this PR will improve performance by ×2.8
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Peter554
force-pushed
the
build-graph-in-rust
branch
from
November 19, 2026 07:09
657c163 to
ac54edb
Compare
Peter554
force-pushed
the
build-graph-in-rust
branch
from
November 19, 2026 07:37
5ffa3be to
c342c11
Compare
This is consistent with FileSystem::read, and means writes can be used from Rust code that doesn't hold the GIL. I/O errors are surfaced as OSError. Assistant-model: Claude Code
Converts a GrimpError into a Python exception, including translating syntax errors into SourceSyntaxError. This will be reused when building graphs in Rust. Assistant-model: Claude Code
Builds an import graph entirely in Rust: finding modules, reading and writing the cache, scanning imports and assembling the graph. This avoids the cost of converting everything to and from Python objects. - find_package is a port of the Python ModuleFinder, including namespace packages. It also returns file mtimes, in the same form as os.path.getmtime. - Cache reads and writes the same files as the Python cache, so the two are interchangeable. serde_json's float_roundtrip feature is needed for mtimes to compare equal after being read back. - Import scanning reuses the existing scan_for_imports_no_py. Assistant-model: Claude Code
A faster alternative to build_graph, which builds the graph in Rust. It takes the same arguments, and shares its cache with build_graph. The functional tests now run against both implementations, via a build_graph fixture. The test that checks the cache logs a message only asserts that for the Python implementation, as the Rust one doesn't log. Assistant-model: Claude Code
Assistant-model: Claude Code
Peter554
force-pushed
the
build-graph-in-rust
branch
from
November 19, 2026 07:49
c342c11 to
ff5cfe5
Compare
The unit tests of module finding, scanning, caching and build_graph's arguments use an in-memory file system, so they only cover the Python implementation. The Rust implementation that is about to replace it reads the file system directly, so there is nothing to substitute. Keeping the in-memory file system for the tests alone would mean keeping production code that only they use: the Python-facing scanning and cache functions, and the fake file system itself. Port them instead to functional tests that write real files to a temporary directory. These test the code path that users actually run. They run against both implementations (via the build_graph fixture), ahead of replacing the Python implementation, which shows that the Rust one behaves the same. The tests are still fast: the ported tests take under a second, and the whole suite still takes around 2 seconds. Behaviours that are only observable through log messages are not ported, as the Rust implementation doesn't log. A write_files fixture is added, which writes files to a temporary directory on sys.path. Assistant-model: Claude Code
build_graph now builds the graph in Rust, and build_graph_rust is removed. Remove everything that only the Python implementation used: - Python: the module finder, cache and file system adaptors and ports, the scanning module, and the Settings entries and copy method they needed. The package finder now uses os.path rather than a file system port. - Rust: the Python-facing scan_for_imports and cache functions, the conversions to and from Python objects, the in-memory file system and the Python wrappers around the real one, and the cache exceptions, which were never raised to Python by the new implementation. Rust only parses the cache to decide whether it can be used, so parse_json_to_map now returns an Option. - Tests: the unit tests of the removed code (ported to functional tests in the previous commit), their fakes, and the build_graph fixture which ran the functional tests against both implementations. Assistant-model: Claude Code
Peter554
force-pushed
the
build-graph-in-rust
branch
2 times, most recently
from
November 19, 2026 08:16
3a03850 to
b2711dd
Compare
Rust can't use Python's logging directly, so rust.build_graph now returns the messages that should be logged, along with the graph, and build_graph logs them. The loggers and messages are the same as before the module finder and the cache were ported to Rust, so that nothing changes for anyone who configures them: - grimp.adaptors.modulefinder: the warning about skipping a module with dots in its name. - grimp.adaptors.caching: whether each cache file was used, missing, corrupt or written by a different version of Grimp, and the files written. This restores the assertions on the log messages that couldn't be ported to the functional tests, and adds tests of the cache messages. An alternative would be the pyo3-log crate, which forwards messages from Rust's log crate to Python's logging as they happen. It isn't used because it would add two dependencies (log and pyo3-log), and mean taking the GIL from code that deliberately runs without it, for what is only a handful of messages per build. It also caches the levels of the loggers, which can go stale if logging is configured after import. Returning the messages keeps Python's logging out of the Rust code, and is simple to test. Assistant-model: Claude Code
These were already unused before build_graph was replaced: nothing read settings.TIMER, and no Rust code used serde or serde_yaml. Assistant-model: Claude Code
Peter554
force-pushed
the
build-graph-in-rust
branch
from
November 19, 2026 08:18
b2711dd to
b971ea9
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
build_graphnow builds the graph entirely in Rust: finding the modules, reading and writing the cache, scanning the imports and assembling the graph. Previously only the import scanning was in Rust, and everything else went through Python objects.This reworks the demo in #264, rebased onto current
main. This version is smaller, because it reuses the Rust that already exists (scan_for_imports_no_py, the JSON cache (de)serialisation, the encoding-aware file reading) rather than reimplementing it. It also removes the Python implementation, and everything that only it used.Compatibility
include_external_packages: modules, squashed modules and every import detail were identical.serde_json'sfloat_roundtripfeature is needed so that mtimes compare equal after being read back.)Performance
Measured on the Kraken codebase, with a release build.
With follow-ups (see "Potential follow-ups" below) the performance gains could likely be even more. This change opens the door to further optimisation. Rough experimentation on a separate branch suggests it's possible to drop another second or so with parallel package discovery, for example.
Removed adapters and ports
This removes most of the adapters and ports that
build_graphwas built from: the module finder, the cache and the file system (grimp.adaptors.{modulefinder,caching,filesystem}andgrimp.application.ports.{modulefinder,caching,filesystem}), along withgrimp.application.scanningand the settings that wired them up. The package finder adapter remains.This is a significant change to the architecture, so it requires justification. The reason is performance. Those ports sat on the boundary between Python and Rust, so everything crossing them had to be converted to and from Python objects: every module found, every cached import and every scanned import. The graph was then assembled one
add_importcall at a time. Keeping the ports would have meant having Rust implement them one call at a time, which keeps the conversion cost that this PR removes.The main thing lost is the ability to substitute a fake file system in unit tests, which is why the tests have been ported to use real files (see below).
Tests
The unit tests of the Python implementation (module finding, import scanning, the cache and the arguments of
build_graph) used an in-memory file system. The Rust implementation reads the file system directly. The tests have been ported to (python) functional tests that write real files to a temporary directory. They exercise the code path that users actually run.The tests are still fast. On my machine the ported tests take under a second, and the whole suite still takes around 2 seconds, as before.
Notes
blake2(cache file names, to match the existingblake2bhashes) andunicode-ident(to matchstr.isidentifierwhen skipping directories).logging, sorust.build_graphreturns the messages to log along with the graph, andbuild_graphlogs them. The loggers and messages are the same as before (grimp.adaptors.modulefinderfor the warning about skipping a module with dots in its name, andgrimp.adaptors.cachingfor the cache messages). An alternative would be thepyo3-logcrate, which forwards messages from Rust'slogcrate to Python'sloggingas they happen. I haven't used it because it would add two dependencies (logandpyo3-log), and mean taking the GIL from code that deliberately runs without it, for only a handful of messages per build. It also caches the levels of the loggers, which can go stale if logging is configured after import. Returning the messages keeps Python's logging out of the Rust code, and is simple to test.Potential follow-ups
I haven't profiled where the remaining time goes, so these are hypotheses until measured:
rayonor theignorecrate's parallel walker.FileSystemnow has a single implementation, and module finding bypasses it to usestd::fsdirectly. It could be replaced by a concrete type.grimp.DirectImport. It's still exported, but nothing in Grimp produces or consumes it any more (only tests do). Worth deciding whether to keep it as public API.settingsexists only to hold it, and nothing substitutes a fake for it. It could be replaced by a plain function, which would also mean dropping theadaptorslayer from the Import Linter contract. I've left that for a follow-up so that it can be discussed separately.