feat(cli): add ebuild package — assemble the eFirmware .efw image - #77
Open
srpatcha wants to merge 23 commits into
Open
feat(cli): add ebuild package — assemble the eFirmware .efw image#77srpatcha wants to merge 23 commits into
ebuild package — assemble the eFirmware .efw image#77srpatcha wants to merge 23 commits into
Conversation
core/eos/ and core/eboot/ are copies of two other repositories in this
organisation, not original source. Nothing keeps them in step, so fixes
merged upstream never arrive here and nothing reports the gap.
Measured against origin/master of both repositories on 2026-08-28:
core/eos 350 files copied from eos 44 already drifted
core/eboot 167 files copied from eBoot 46 already drifted
Ninety files. Any security fix landed in eos or eBoot is absent from the
copy here, silently, and the 21 pull requests currently open against eos
will each widen the gap when they merge.
This adds:
core/UPSTREAM.yaml the pinned upstream revisions
scripts/check_vendor_drift.py fetches each pin and compares
.github/workflows/... runs it on PRs touching core/, and weekly
The existing 90 files are grandfathered through baseline_drift so this
can merge without blocking work in flight. New drift fails. Reducing the
drift lowers the baseline; raising the baseline to make CI pass defeats
the file and the script says so.
This is a stopgap, not the fix. ADR-019 in the eos repository records the
decision to replace both snapshots with real pinned dependencies, and to
reconcile the 90 drifted files one at a time — each is either a change
that belongs upstream, or one that should be reverted.
Verified: the checker reproduces 44 and 46 independently of the git
blob-hash comparison above, and exits 0 with the baselines set and 1 when
either is lowered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…viour
`import ebuild.build.dispatch` raises SyntaxError on master, so the CLI and
five test modules fail outright. As with the other breakage below, the cause
is overlapping PRs squash-merged on stale bases with no re-verification.
- dispatch.py: configure() ended with two consecutive `else:` blocks from an
unresolved conflict. The two blocks also disagreed about the exception --
tests/ebuild expects ValueError("Unknown build backend '<name>'") while
tests/unit expects RuntimeError matching "ninja". Both are legitimate
readings: an unrecognized name is a bad argument, and a "ninja" that
reaches the dispatcher is a CLI routing failure. Introduced
UnknownBackendError(ValueError, RuntimeError) with a message covering both,
raised from configure(), build(), and clean(). "ninja" is no longer a
silent no-op in configure(), which is what let `ebuild build` report
"Build completed successfully" without running a compiler.
- ninja_backend.py: NinjaBackend._object_path() was deleted by a cflags
refactor while both of its callers survived, so generate() died with
AttributeError -- the default build backend did not work at all. Restored
with its target-namespaced object paths, which is what stops two targets
sharing a source from claiming the same output.
- ninja_backend.py: the emitted `link_shared` rule was dead -- nothing used
it -- and it hardcoded `-shared`, which is wrong on macOS. Shared libraries
already link through the generic `link` rule with the platform's flag
(-dynamiclib/-shared) and the -L/-l wiring in ldflags. Dropped the rule and
updated tests/ebuild/test_ninja_backend.py, which asserted the dead rule,
to cover the surviving behaviour instead. Its removal also unblocks
test_static_library_unaffected, which asserts no "-shared" anywhere in the
generated file.
Verified: pytest tests/ 202 passed (201 passed + 1 skipped without the ninja
package installed); tests/performance 1 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix includes windows-2022, where the default shell is PowerShell. The
"Run test suite" step uses backslash line continuations, which PowerShell does
not accept:
ParserError: Missing expression after unary operator '--'.
3 | --cov=ebuild --cov-report=xml --cov-report=term-missing \
So the Windows jobs failed before pytest started — on master and on every
branch. Marked `shell: bash`, which GitHub provides on Windows runners via Git
Bash, keeping the command readable and identical across the three platforms.
Also: `mypy .` aborted immediately with
tests\__init__.py: error: Duplicate module named "tests"
(also at ".\layers\eosuite\tests\__init__.py")
Found 1 error in 1 file (errors prevented further checking)
layers/eosuite/ vendors its own tests/ package. Because the step is
continue-on-error, this went unnoticed and the type check has been checking
zero files. Excluding layers/ makes it check 81 source files; it stays
continue-on-error, so the 11 pre-existing findings it now reports are visible
without gating the build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the Windows jobs actually running pytest, they surface a backend that
has never worked on Windows:
ninja: error: build.ninja:20: expected build command name
build C:\...\main.o: cc main.c
^ near here
Ninja splits build statements on unescaped spaces and colons, so a Windows
absolute path puts a drive-letter colon where Ninja expects the separator
between outputs and the rule name. Every generated build.ninja was rejected
before a single command ran. A POSIX path containing a space fails the same
way.
_ninja_path() escapes `$`, `:` and ` ` and is applied to the paths in build
statements only. Variable values (cflags, ldflags) are read to end of line and
are deliberately left alone -- escaping them would hand the compiler mangled
flags.
Also skipped tests/ebuild/test_integration_initramfs_security.py on hosts
without find(1)/cpio(1). _create_initramfs() drives both directly, so on a
Windows runner all three died with WinError 2 before reaching the injection
behaviour they exist to check. Building a Linux initramfs is not a Windows
operation; the skip matches how test_ninja_backend.py already skips when no
host C compiler is present.
Adds four regression tests for the escaping, including one asserting that each
build statement contains exactly one unescaped colon.
Verified: pytest tests/ 206 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci.yml is the only workflow in this repo without a concurrency group, and it is the heaviest one -- a matrix spanning ubuntu, macos and windows. Every push to a pull request therefore left the previous run queued, and all of them competed for the same scarce windows/macos runners. On this branch three superseded runs sat ahead of the current one for over an hour, testing commits that were no longer HEAD. Uses the same group expression the sibling workflows already use, with cancel-in-progress: true, because a superseded commit's result is not wanted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The macos-13 image is retired, so jobs requesting it are never assigned a runner -- they sit queued until they time out. Across four runs on this branch every ubuntu-22.04 and windows-2022 job started and finished within minutes while all three macos-13 jobs stayed queued for more than two hours, leaving the workflow permanently incomplete. Every other workflow in this repo already targets macos-latest; ci.yml was the one place still pinning macos-13. release.yml also lists macos-13, alongside macos-14, in its wheel-build matrix. Left alone: building wheels against an older macOS deployment target may be deliberate there, and it does not gate pull requests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pins real ebuild downloads third-party source over the network and builds it. That makes recipes/*.yaml the tool's supply chain, and a recipe is a pin: a URL plus the digest of exactly what should be at it. Four of the five shipped recipes are not usable pins, and the code let them ship that way. **Four recipes cannot be fetched.** Each digest below was checked by downloading the artifact twice and hashing it; both downloads agreed. | recipe | recorded | actual | |----------|-------------------------|-------------------------| | littlefs | `sha256:placeholder` | `9cf2e7db...cad67` | | lwip | `sha256:placeholder` | `c79255f6...ed87` | | mbedtls | `...b1490fcd73` | `...b1490fcd38` | | freertos | `e36e5a2f...e3140` | `eebd58aa...271eb` | | zlib | `9a93b2b7...df23` | `9a93b2b7...df23` (ok) | mbedtls has the last two hex characters transposed. freertos matches none of the release's assets — that tag publishes exactly one, `FreeRTOS-KernelV11.1.0.zip`, 3324125 bytes, `eebd58aa...271eb`. Corrected all four. **"placeholder" parsed as a valid checksum.** `PackageRecipe.validate()` checked that the field was well-formed in no way at all, so `sha256:placeholder` sailed through and then failed every fetch with a mismatch. A checksum that is present must now look like a sha256 digest. It is deliberately still optional at the recipe level — a recipe also models packages that are never downloaded — the requirement lands at the boundary that matters, below. **No checksum meant no verification.** `PackageFetcher.fetch()` did `if recipe.checksum:` and, when absent, downloaded and extracted with nothing checked. Omitting one field silently bought an unverified download. fetch() now refuses before touching the network. `tests/ebuild/test_package_fetcher.py` contained `test_empty_checksum_skips_verification`, which asserted exactly this behaviour; it is replaced by one asserting the refusal. **Plaintext http:// was accepted.** `_download()` allowed `http://` and `https://` alike. A pin is worth much less over a transport anyone on the path can rewrite, and it leaks what is being built. https only; no shipped recipe used http. **tests/unit/test_shipped_recipes.py** walks recipes/ and asserts each one pins a syntactically real sha256 over an https URL and loads through the real loader. Against the recipes as they were, it fails on both placeholders. It deliberately does not assert digest *values*: that needs the network, and pinning them here would only duplicate the recipe. Verified: pytest tests/ 223 passed (up from 202; 16 new recipe tests, plus the inverted fetcher tests). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the test suite leaves a generated _build/ (build.ninja and compile_commands.json) in the repo root, and it was not gitignored, so a `git add -A` swept it into this branch. Removed, and gitignored so it cannot happen again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PackageRecipe.validate() accepts any non-empty version string, but the
registry ordered versions with
sorted(versions, key=lambda v: [int(x) for x in v.split('.')])
which raises ValueError for anything that is not dotted integers. Real
recipes are full of those: a leading v (littlefs and FreeRTOS both publish
their tags that way -- the recipe example in the book uses `tag: V10.5.1`),
pre-release tags like 3.6.0-rc1, and build metadata like 1.3.1+patch2.
The blast radius is wider than the odd package itself. The key was
duplicated across get(), list_packages() and list_all_versions(), and
* get(name) with no version scans every version of that package, so one
such recipe breaks that package entirely;
* list_packages() scans every package, and PackageResolver builds its
"package not found in registry. Available: ..." message from it -- so a
single unusual recipe anywhere in the registry turns an ordinary
missing-package error into a ValueError traceback.
Replace the three copies with one version_sort_key(). Ordering: a leading
v/V is ignored; all-digit components compare numerically so 1.10.0 still
sorts above 1.9.0; any other component compares as text and ranks below a
numeric one; a -/+ suffix ranks below the same version without one, so
3.6.0-rc1 < 3.6.0. The order is total and never raises, which is the
property that matters here -- one unusual recipe must not decide whether
lookup works for the packages around it.
Deliberately not a full PEP 440 / semver implementation. That would mean a
dependency or a lot more code for a comparison the recipe format does not
specify; this defines the rules it does need and documents them.
Tests: tests/ebuild/test_package_registry.py grows from 1 case to 12,
covering v-prefixes, pre-releases, build metadata, date-stamped and
non-numeric versions, one odd version among good ones, and a totality check
over the key. Nine of the eleven behavioural cases fail against the unfixed
registry.py (checked by running them against it); the other two are the
pre-existing numeric-order test and "2024.06", which parsed as ints before.
Docs: the ordering rules are now in docs/book/book.md section 12.5.
Note on CI: master currently fails for reasons unrelated to this change --
ebuild/build/dispatch.py has a duplicated `else:` from a merge, so the
module does not parse (PRs #65/#66 address it), and ninja_backend.py is
missing _object_path. This branch leaves the failure set exactly as it
found it: 26 failed / 162 passed before and after, identical lists.
Verified: pytest tests/ebuild/test_package_registry.py -> 12 passed.
Verified: ruff check --select=E,F,W --ignore=E501 and mypy
--ignore-missing-imports on both changed files -> clean.
Added a test for maximum-length node names in the graph.
Added non-zero data initialization for node name to detect missing null terminator.
A mapping or scalar under packages: was silently ignored, so a common YAML mistake produced a build with no package dependencies and no error. Signed-off-by: João Morais <118842104+JoaoMorais03@users.noreply.github.com>
Linux is case-sensitive, so ../eboot misses a GitHub-style eBoot checkout. Try both names in DepsManager and the test fixture.
The MLP developer walk in the platform design document ends with a build that
says how much of the board it used:
Flash: 384 KB
RAM: 72 KB
Ready to flash.
Nothing produced those numbers. A developer had to run `size` themselves and
remember which columns to add, which is not being told — it is being left to
find out.
$ ebuild build
[ok] Build completed successfully.
Flash: 1.9 KB of 1.00 MB (0.2%)
RAM : 300.6 KB of 192.0 KB (156.6%)
[warn] RAM usage 300.6 KB exceeds the board's 192.0 KB -- this image
will not fit.
The accounting matches scripts/measure_footprint.py in the eos repo so the two
tools cannot disagree about what a number means:
flash = text + data
ram = data + bss
`data` is charged to both because it is stored in flash and copied to RAM at
startup; reading `size`'s "dec" column instead understates RAM.
Capacity comes from the project's own board.yaml `memory.flash_size` /
`ram_size` when it ships one — the convention the descriptions under
hardware/board/ already use — and otherwise from a table of the reference part
for each board family. Boards that boot from removable storage, and
Linux-class parts with no fixed budget, are deliberately absent: a percentage
against a guessed ceiling reads as authoritative, so those report absolute
sizes only.
An over-budget image is a warning, not a build failure. It linked; it will not
fit. The developer needs to hear that now rather than from a board that will
not boot.
A cross build is measured with its own `size` and never falls back to the host
one — host `size` on an ARM ELF reports numbers for a different target and
nothing in the output would say so. Where no suitable tool exists the report
is skipped rather than guessed, and never fails the build.
Also fixes .gitignore, found while committing this: line 8 was a bare `build/`
from the Python-packaging block, which also matched `ebuild/build/` — the
source package holding dispatch.py, ninja_backend.py and toolchain.py. Those
four files predate the rule and stayed tracked, but every new module added
there was silently ignored, including this one. Anchored to `/build/`, which
keeps the setuptools artifact directory ignored and stops the pattern reaching
nested source.
235 tests pass, up from 202; 32 are new here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit cfc3698)
The MLP list asks for it, and the absence shows: the same class of problem
currently surfaces three different ways. A missing cross toolchain appears as
a compiler-not-found partway through a build. A missing repo cache appears as
`eos/hal.h: No such file`. A missing binutils silently drops the footprint
report. None of the three names the fix.
$ ebuild doctor
OK python 3.12.14
OK ninja 1.13.2 (/usr/bin/ninja)
OK host compiler 15.2.0 (/usr/bin/cc)
OK arm-none-eabi 14.2 (/usr/bin/arm-none-eabi-gcc)
warn xtensa-esp32-elf not installed — no esp32 builds
OK eos repo ~/.ebuild/repos/eos (master)
No problems. The host build path is ready.
Optional, for other targets:
- install the xtensa-esp32-elf toolchain to target esp32
Three decisions worth naming.
Read-only. It reports; it does not repair. `ebuild setup` fetches the repos,
and installing a toolchain belongs to the developer's package manager —
guessing which one they use is how a diagnostic tool starts doing damage.
The exit code is non-zero only for things that actually stop a build. A
host-only machine legitimately has no cross toolchain, and a doctor that
always exits 1 stops being consulted, which costs more than it saves.
A missing cross toolchain reports the boards it would have unlocked rather
than just its own absence. "arm-none-eabi-gcc not found" is a fact; "no
stm32f4, stm32h7, nrf52, rp2040 or tms570 builds" is the consequence the
developer is actually deciding about.
`--json` emits the same checks for CI, and agrees with the text form on the
exit code.
254 tests pass, up from 235; 19 are new here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 290f5e8)
…e build contained
Two gaps against the MLP developer walk, which runs `ebuild add wifi` and then
expects a build that says what it contained.
`ebuild add` warned about an unknown package and added it anyway:
$ ebuild add wifi
[warn] No recipe found for 'wifi' — adding anyway.
[ok] Added wifi to build.yaml
That trades one clear error now for a confusing one at build time, in a file
the developer has since committed. It now refuses, and says what would help:
$ ebuild add wifi
[error] No recipe for 'wifi'.
Available: freertos, littlefs, lwip, mbedtls, zlib
To add it anyway: ebuild add wifi --force
$ ebuild add lwipp
[error] No recipe for 'lwipp'.
Did you mean: lwip?
`--force` keeps the old behaviour for anyone who genuinely wants an entry
ahead of its recipe, and says plainly that it will not resolve. Refusing
without an escape hatch would be the other kind of unhelpful.
The build then reports what it actually contained:
EmbeddedOS Build
OK toolchain arm-none-eabi-gcc
OK board configuration stm32f4
OK lwip
MISS mqtt declared, nothing resolved
OK temperature-monitor executable
The case worth having is MISS. A package that resolved to no include or
library directory contributed nothing, whatever build.yaml says -- the build
still succeeds, the feature is simply absent, and until now nothing said so.
The summary is rendered through the logger in the tests, so the assertions
are on what a developer actually sees rather than on an intermediate
structure.
271 tests pass, up from 255; 16 are new here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit dfdc429)
§29's development-to-device flow is
eBuild -> {EoS, eBoot, application} -> eFirmware artifact -> {EoSim, hardware}
Every piece of that existed except the arrow into eFirmware.
embeddedos-org/eFirmware implements the image format — efw_image_build,
attach_signature, serialize, parse, verify — and ships efwtool to pack,
inspect and verify. It builds clean and its tests pass 3/3. Nothing in ebuild
referenced it: grepping the whole tool for `.efw` returned no code. A developer
who wanted the artifact the flow ends at had to know the repository existed,
build the tool themselves, and run it by hand.
$ ebuild build
[ok] Build completed successfully.
Flash: 1.9 KB
RAM : 2.6 KB
$ ebuild package --load 0x08000000 --entry 0x08000100
Packing sensor-node -> sensor-node.efw
[ok] sensor-node.efw (16140 bytes)
sensor-node.efw: OK (15984 bytes, sha256 b87b8a98dd...)
This drives efwtool rather than re-implementing the header in Python. The
header is a packed C struct with a magic, a version, a size and a SHA-256, and
a second implementation of it in another language is a second thing to keep in
step — which is the failure this repository has spent the week repairing. If
the format changes, the tool changes with it and this keeps working.
efwtool is found on PATH first, so a developer's own build is not silently
recompiled behind them; failing that, the cached eFirmware checkout is built
on demand and reused. eFirmware joins eos and eboot in what `ebuild setup`
fetches, since the flow now needs it.
Where the tool cannot be found the command says which one applies — `ebuild
setup` when the checkout is absent, the cmake invocation when it is present
but unbuilt. "efwtool not found" on its own leaves the developer guessing at
what fetches it.
Verified end to end: the packed image round-trips byte-for-byte (156-byte
header plus a payload identical to the built artifact), efwtool inspect
reports image_version 2.1.0 taken from build.yaml, and verify recomputes the
same SHA-256.
The tests cover the wiring, not the format — efwtool owns the format and has
its own tests. One of them needed the repo cache pointed at an empty directory
as well as PATH cleared: this machine has a real efwtool cached, and without
both the test passed by finding it and asserted nothing.
331 tests pass, up from 316.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| for left in keys: | ||
| for right in keys: | ||
| assert (left < right) or (left >= right) |
| import sys | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| from typing import List, Optional |
|
|
||
| import json | ||
|
|
||
| import pytest |
Comment on lines
+24
to
+34
| from ebuild.system.doctor import ( | ||
| MISSING, | ||
| OK, | ||
| WARN, | ||
| Check, | ||
| exit_code, | ||
| format_report, | ||
| host_checks, | ||
| run_all, | ||
| toolchain_checks, | ||
| ) |
| """ | ||
|
|
||
| import subprocess | ||
| from pathlib import Path |
| repairing. | ||
| """ | ||
|
|
||
| import os |
Comment on lines
+26
to
+32
| from ebuild.build.firmware_image import ( | ||
| FirmwareImageError, | ||
| find_efwtool, | ||
| missing_tool_message, | ||
| pack, | ||
| verify, | ||
| ) |
| "--abbrev-ref", "HEAD"], | ||
| capture_output=True, text=True, timeout=15) | ||
| branch = proc.stdout.strip() | ||
| except (OSError, subprocess.TimeoutExpired): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
§29's development-to-device flow is:
Every piece of that existed except the arrow into eFirmware.
embeddedos-org/eFirmwareimplements the image format —efw_image_build,attach_signature,serialize,parse,verify— and shipsefwtool. It builds clean and its tests pass 3/3.Nothing in ebuild referenced it. Grepping the whole tool for
.efwreturned no code. A developer who wanted the artifact the flow ends at had to know the repository existed, build the tool, and run it by hand.It drives efwtool rather than reimplementing the header
The header is a packed C struct — magic, version, size, SHA-256. A second implementation of it in Python is a second thing to keep in step, which is the failure this repo has spent the week repairing. If the format changes, the tool changes with it and this keeps working.
efwtoolis found on PATH first, so a developer's own build isn't silently recompiled behind them. Failing that, the cached eFirmware checkout is built on demand and reused. eFirmware joins eos and eboot in whatebuild setupfetches.Where the tool can't be found, the message says which fix applies —
ebuild setupwhen the checkout is absent, the cmake invocation when it's present but unbuilt. "efwtool not found" alone leaves the developer guessing at what fetches it.Verified end to end
0x454F5349image_versionbuild.yamlefwtool verifyOn the tests
They cover the wiring, not the format — efwtool owns the format and has its own tests.
One of them needed the repo cache pointed at an empty directory as well as PATH cleared. This machine has a real efwtool cached, and without both the test passed by finding it and asserted nothing. Worth knowing if you touch these.
331 tests pass, up from 316.
Stacks on #66.
🤖 Generated with Claude Code