Skip to content

fix(mvp): repair the first-run developer path - #64

Merged
srpatcha merged 3 commits into
masterfrom
fix/mvp-first-run-path
Aug 30, 2026
Merged

fix(mvp): repair the first-run developer path#64
srpatcha merged 3 commits into
masterfrom
fix/mvp-first-run-path

Conversation

@srpatcha

Copy link
Copy Markdown
Member

Measured against the MVP developer walk. On origin/master a new developer cannot reach a built binary, and pytest tests/unit cannot even be collected.

$ ebuild setup
[error] fatal: Remote branch main not found in upstream origin

$ pytest tests/unit
E   SyntaxError: invalid syntax
!!!! Interrupted: 1 error during collection !!!!

Six defects, each found by running the tool rather than reading it.

# Defect
1 ebuild setup failed for everyone, on the first command. DEFAULT_CONFIG pinned branch main for both cached repos; eos and eBoot both default to master.
2 ebuild build could not run at all. dispatch.py had two consecutive else: blocks — a SyntaxError. Python does not parse a module until something imports it, so this sat on master reachable only by the command that touches it. It also made tests/unit uncollectable.
3 NinjaBackend._object_path was called twice and defined nowhere. AttributeError on every ninja build.
4 ninja was invoked as python -m ninja, so a machine with ninja properly installed still failed with No module named ninja.
5 test_ninja_backend.py asserted "-shared" was absent from a static-library file whose rule preamble always contains it. This test had never run — defect 2 made the module uncollectable.
6 The tool scaffolded a project the tool could not build. All six templates include <eos/hal.h> with no path to the repo ebuild setup had just cloned.

On defect 2

The two branches were duplicates. The RuntimeError one is kept, because tests/unit/test_dispatch.py documents the intent: an unhandled backend must fail loudly rather than silently do nothing and let the caller report a false "Build completed successfully". I initially deleted the wrong one; the existing tests caught it.

On defect 6

Templates now declare uses: [eos], resolved at build time against ~/.ebuild/repos — not written into build.yaml, because the path is a fact about one machine and build.yaml is committed.

before   6/6 stopped at "fatal error: eos/hal.h"
after    3/6 compile and reach the linker (bare-metal, linux-app, rtos-app)
         2/6 fail on template source calling APIs eos does not have
         1/6 needs arm-none-eabi, absent from this machine

Regression guard

tests/unit/test_sources_are_importable.py — every shipped .py is parsed with ast, the CLI and its lazily imported modules are imported, --help is run for each MVP command, and the cached-repo default branch is checked against the real remote with git ls-remote. A hardcoded branch name is a fact about a remote, and facts about remotes go stale.

Verification

tests/unit: 111 passed. On origin/master the same command aborts during collection. From a clean ~/.ebuild, ebuild setup and ebuild new succeed and ebuild build drives gcc.

Deliberately not fixed — each needs a decision, not a guess

  • Linking: ebuild must build eos and expose its archives before any template produces a binary.
  • ble-sensor and secure-boot reference APIs that do not exist (eos_ble_send, eos_ota_status_t).
  • ebuild monitor, ebuild test, ebuild simulate are named in the MVP walk and do not exist as commands.
  • pyproject.toml declares requires-python >=3.8 while flake8>=6.0 needs >=3.8.1, so pip install -e .[dev] cannot resolve.

srpatcha and others added 2 commits August 28, 2026 02:36
Measured against the MVP developer walk. On origin/master a new developer
cannot reach a built binary, and `pytest tests/unit` cannot even be
collected. Five defects, each verified by running the tool.

1. `ebuild setup` failed for everyone, on the first command.
   DEFAULT_CONFIG pinned branch "main" for both cached repos while eos and
   eBoot both default to "master":
     fatal: Remote branch main not found in upstream origin

2. `ebuild build` could not run at all. ebuild/build/dispatch.py carried
   two consecutive `else:` blocks — a SyntaxError. Python does not parse a
   module until something imports it, so this sat on master reachable only
   by the command that touches it. It also made tests/unit uncollectable.

   The two branches were duplicates of each other. The RuntimeError one is
   kept: tests/unit/test_dispatch.py documents the intent, that an
   unhandled backend must fail loudly rather than silently do nothing and
   let the caller report a false "Build completed successfully". "ninja" is
   therefore no longer swallowed by the make/kbuild no-op branch, and
   build() raises for the same reason.

3. NinjaBackend._object_path was called twice and defined nowhere:
     AttributeError: 'NinjaBackend' object has no attribute '_object_path'
   Implemented. Objects are namespaced per target, since ninja rejects two
   edges writing one output and two targets may use different cflags. The
   source path is flattened into the filename rather than mirrored, so a
   source from outside the project cannot place its object outside the
   build directory.

4. ninja was invoked as `sys.executable -m ninja`, so a machine with ninja
   properly installed still failed with "No module named ninja". Now
   prefers the binary on PATH and falls back to the wheel.

5. tests/unit/test_ninja_backend.py asserted the string "-shared" was
   absent from a static-library build file, but the link_shared *rule* is
   always declared in the preamble. The assertion now checks that no build
   *edge* uses link_shared, which is what it meant. This test had never
   run — defect 2 made the module uncollectable.

Added tests/unit/test_sources_are_importable.py so this class of bug
cannot merge again: every shipped .py is parsed with ast, the CLI and its
lazily imported modules are imported, `--help` is run for each MVP
command, and the cached-repo default branch is checked against the actual
remote with git ls-remote.

Verified:
  - tests/unit: 111 passed. On origin/master the same command exits with
    "SyntaxError ... Interrupted: 1 error during collection".
  - From a clean ~/.ebuild: `ebuild setup` and `ebuild new hello` both
    succeed, and `ebuild build` now generates build.ninja and
    compile_commands.json and drives gcc.

Still failing, deliberately not fixed here: the scaffold emits
#include <eos/hal.h> with no include path to the eos repo that setup just
cloned, so all six templates stop at "fatal error: eos/hal.h". That needs
a decision about how an app declares a dependency on eos, not a guess.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of the six templates includes <eos/hal.h>, and the generated
build.yaml carried no path to it, so `ebuild new` followed by
`ebuild build` failed for all of them:

    src/main.c:9:10: fatal error: eos/hal.h: No such file or directory

The tool scaffolded a project that the tool could not build.

The templates now declare `uses: [eos]`, and _workspace_repo_paths()
resolves that against the repos `ebuild setup` already cloned into
~/.ebuild/repos. Paths are resolved at build time rather than written
into build.yaml, because the path is a fact about one machine and
build.yaml is a file the developer commits.

Headers sit at two depths — kernel/include and services/crypto/include —
so both are globbed; <eos/crypto.h> and <eos/ota.h> live only in the
deeper set. Declared packages still take precedence over the workspace
cache, so an explicit dependency always wins.

When the cache is absent the mapping is empty, leaving the developer with
the missing-header error and `ebuild setup` as the fix, rather than a
stack trace.

Measured across all six templates, from `ebuild new` to `ebuild build`:

  before   6/6 stopped at "fatal error: eos/hal.h"
  after    3/6 compile and reach the linker (bare-metal, linux-app,
               rtos-app), needing only the eos libraries
           2/6 fail on template source that calls APIs eos does not
               have — eos_ble_send, eos_ota_status_t
           1/6 needs arm-none-eabi, absent from this machine

Not fixed here, and each needs a decision rather than a guess:
  - linking: ebuild must build eos and expose its archives before a
    template can produce a binary
  - ble-sensor and secure-boot reference APIs that do not exist
  - `ebuild monitor`, `ebuild test` and `ebuild simulate` are named in
    the MVP walk and do not exist as commands

tests/unit: 111 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Walking the eight-command MVP path on this branch stopped at step five:
`build` reached the linker and failed with undefined references, and two of
the eight commands did not exist at all.

  before   setup ok · new ok · configure --board: no such option
           build: undefined reference to eos_hal_init · test: no command
           monitor: no command
  after    7 of 8 steps pass; step 7 (flash) needs attached hardware

Link step. `uses: [eos]` resolved include dirs but returned lib_dirs=[] and
libraries=[], so every scaffolded project compiled and then failed to link.
The cached repo is a CMake project with no install() rules, so there is
nothing to point -L at until it has been built once; _cached_repo_libraries
builds it on demand and caches the tree, degrading to ([], []) when cmake is
absent so the developer still sees a link error naming the symbol.

configure/build divergence. _configure_ninja_backend did not merge
_workspace_repo_paths(), so `configure` and `build` wrote different
build.ninja files to the same path and a developer who ran `configure` and
then invoked ninja directly built without the EoS paths.

New commands. `test` runs the project's own runner (ctest, cargo, meson,
make test) and, for a scaffolded project that has none, builds and runs its
`test` targets directly. `monitor` opens the serial device, and refuses to
guess when more than one is attached. `configure --board` records the board
in eos.yaml, since the next step is a bare `build` that has to read it back.

Templates now ship tests/test_main.c and a matching test target, so step six
passes on a fresh checkout instead of reporting that the project has no
tests. The test deliberately does not assert eos_hal_init() == 0: no HAL
backend is registered on a host build, so it returns -1 by design.

Also in this commit, found while running the above:

- ninja_backend emitted shared libraries through the generic `link` rule
  with -shared pushed into ldflags, leaving the declared link_shared rule
  dead and breaking test_shared_library_uses_shared_link_rule. Restored the
  rule and kept the -L/-l wiring this branch added; the platform's
  shared-object flag now lives in the rule, so darwin's -dynamiclib is
  chosen there rather than at the call site.
- BackendDispatcher raised RuntimeError from configure() and build() but
  ValueError from clean(), so one `except` could not guard all three.
  clean() now raises RuntimeError, and the two test files this branch added
  no longer disagree about which type to expect.
- _build/ is ebuild's default build dir and was not in .gitignore.

288 tests pass, up from 98 on master and 256/3-failed on this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@srpatcha

Copy link
Copy Markdown
Member Author

Review — measured by walking the path, not reading the diff

Verified the six claimed defects against current master first. Defects 2–5 no longer reproducedispatch.py parses, _object_path is gone, ninja is invoked directly, and the -shared assertion was corrected by #34. Defects 1 and 6 are still live on master, and this branch is the only thing fixing them:

$ git ls-remote --heads .../eos.git main     # empty — branch does not exist
$ git clone --branch main .../eos.git
fatal: Remote branch main not found in upstream origin

So ebuild setup, step one, fails for every user on master. Confirmed.

Three failures on this branch

pytest on the branch as pushed: 256 passed, 3 failed.

1. Shared libraries regressed. test_shared_library_uses_shared_link_rule passes on master and fails here. The branch routes shared_library through the generic link rule with -shared pushed into ldflags, leaving the declared link_shared rule dead:

master   build libfoo.so: link_shared foo.o          # correct rule, but no ldflags/libs line at all
branch   build libfoo.so: link foo.o                 # ldflags = -shared ...  → rule declared and never used

Both were wrong in different ways — master silently dropped the target's ldflags and every -L/-l from uses. Fixed by keeping this branch's -L/-l wiring on the link_shared rule, and moving the platform flag into the rule preamble so darwin's -dynamiclib is decided in one place.

2. The two new test files disagree with each other. This branch adds both tests/ebuild/test_dispatch.py (expects ValueError) and tests/unit/test_dispatch.py (expects RuntimeError) for the same condition. The implementation is itself inconsistent — configure() and build() raise RuntimeError, clean() raises ValueError — so no single except guards the dispatcher. Standardised on RuntimeError, which is the type tests/unit/test_dispatch.py documents the intent for.

Step five still did not produce a binary

The PR body is accurate that 3/6 templates "reach the linker" — but reaching the linker is not the MVP. uses: [eos] resolved include dirs and returned lib_dirs=[], libraries=[]:

undefined reference to `eos_hal_init'    (and 7 more)

The cached repo is a CMake project with no install() rules, so there is nothing to point -L at until it has been built once. Added on-demand build + caching, degrading to ([], []) when cmake is absent so the developer still gets a link error naming the symbol rather than an ebuild stack trace.

Also found while there: _configure_ninja_backend did not merge _workspace_repo_paths(), so configure and build wrote different build.ninja files to the same path. Anyone running configure then ninja directly built without the EoS paths.

Two of the eight commands did not exist

ebuild test (step 6) and ebuild monitor (step 8) were not registered at all. Added both, plus configure --board (step 4). Templates now ship tests/test_main.c and a test target so step six passes on a fresh checkout.

One finding I did not fix — it belongs to eos, not here

eos_hal_linux_register() is defined in hal/src/hal_linux.c but is never called and never declared in a public header. So on a host build no backend registers, eos_hal_init() returns -1, and every subsequent HAL call silently no-ops. The scaffolded main.c ignores that return, which is why hello-eos runs and prints nothing.

hal.h documents the function as "Initialize the HAL subsystem for the current platform", so the contract and the behaviour disagree. Fixing it means either auto-registering the compiled-in default inside eos_hal_init() or exporting the register call — an eos decision, so I left the generated smoke test asserting only what is true today and said so in its comment.

Result

                         before        after
pytest                   256 / 3 fail  288 pass
golden path              3 of 8        7 of 8   (flash needs hardware)

Pushed as 13fa683. Not merging — that is your call.

@srpatcha

Copy link
Copy Markdown
Member Author

Still needed after the whole backlog merges — and setup passing is an illusion

I merged every other open PR onto master in sequence (#66, #67, #68, #69, #73, #76, #63247 passing) and then walked the eight-command golden path against that stack:

OK    ebuild setup
OK    ebuild new gp-app
FAIL  ebuild configure --board     no such option
FAIL  ebuild build                 fatal error: eos/hal.h: No such file or directory
FAIL  ebuild test                  no such command

So three of the eight steps still fail with the entire rest of the backlog merged. --board, test and monitor are added by this PR, and nothing else resolves the EoS headers.

ebuild setup only passed because my cache was warm

That OK is not real. ebuild/deps/__init__.py on the merged stack still reads:

"branch": "main",

and neither repo has that branch:

$ git ls-remote --heads .../eos.git main     → empty
$ git ls-remote --heads .../eBoot.git main   → empty
$ git clone --branch main .../eos.git
fatal: Remote branch main not found in upstream origin

It reported success because ~/.ebuild/repos/ was already populated from an earlier run. On a developer's first-ever ebuild setup, step one of the golden path fails. That is the defect this PR opened with, it is unfixed by everything merged since, and it is invisible to anyone whose cache is warm — which is everyone who has run the tool before.

What has changed under it

dispatch.py and ninja_backend.py here are superseded by #66, which fixes the same SyntaxError and does it first. Those hunks should be dropped in favour of #66's on rebase; what remains unique and needed is:

  • ebuild/deps/__init__.py — the mainmaster branch fix
  • templates/*uses: [eos] plus the smoke-test template
  • ebuild/cli/commands.py_workspace_repo_paths library resolution, and the configure --board, test and monitor commands

One thing to watch on the rebase

This branch carries 1,849 CRLF lines in commands.py; the file has mixed endings, and a whole-file rewrite normalises them silently. Most of the seven conflict hunks in commands.py are that, not real disagreement. I hit the same trap on #71 — its diff read as 1,903 changed lines when the real change was 89 — and fixed it by replaying the edits onto the original bytes rather than checking out the rewritten blob.

Happy to do that rebase if you would rather I took it.

@srpatcha

Copy link
Copy Markdown
Member Author

Filed #81 to track the root cause behind this, with the reproduction that was missing.

Confirming the branch default part of this PR is load-bearing, not cosmetic. The
shipped default requests a branch neither remote has:

eos    default branch: master     has 'main'? 404 Branch not found
eBoot  default branch: master     has 'main'? 404 Branch not found

$ git clone --depth 1 --branch main https://github.com/embeddedos-org/eos.git
fatal: Remote branch main not found in upstream origin
exit=128

_git_clone raises on that, cmd_setup exits 1, and steps 2-8 of the golden
path are unreachable. This is the first command a new user runs.

Worth recording why it stayed hidden: it only reproduces with no
~/.ebuild/config.yaml. Any machine that has one — mine says branch: master
— clones fine from cache and prints [ok] Setup complete. So it is invisible
to everyone who has ever had a working setup, which is everyone who would test
it. I was myself convinced this was already fixed until I ran the clone
directly.

No change requested here — master is right today and this should go in as is.

The one thing I would not want lost: the name is still a hardcoded guess in a
constant that nothing checks. Renaming either repo's default to main later
reintroduces the identical failure with the identical invisibility. #81 carries
the follow-up — drop --branch when none is configured and let the remote's
HEAD decide, which is correct by definition and survives a rename. Deliberately
kept out of this PR to avoid widening it; an explicitly configured branch would
still be honoured.

This needs a rebase before it can merge.

@srpatcha
srpatcha merged commit 6a22e36 into master Aug 30, 2026
24 of 30 checks passed
@srpatcha
srpatcha deleted the fix/mvp-first-run-path branch August 30, 2026 22:33
srpatcha pushed a commit that referenced this pull request Sep 1, 2026
`import ebuild.cli.commands` raises NameError on master, so the `ebuild`
command does not start and pytest cannot collect nine test modules --
the suite reports "9 errors during collection" and runs zero tests.

Four PRs landed their tests while their implementation was discarded
during conflict resolution, and the merges that did it were not gated by
a run: f5209ac, current master, has no CI runs at all.

  #64 (6a22e36)  tests/unit/test_golden_path_commands.py landed;
                 _workspace_repo_paths, _cached_repo_libraries,
                 _record_board_selection, _EOS_PROJECT_CONFIG and the
                 `configure --board` option did not.
  #89 (32348f0)  commands.py gained five `re.` uses, no `import re`.
  #90 (d3958f7)  the initramfs test kept its assertions but lost its
                 `_newc_members` helper and its `os` / `stat` imports.
  #92            registry.version_key was renamed version_sort_key
                 without updating its caller in the tests.

Restoring them, taken verbatim from the branches that were merged:

* commands.py: `import re`; `_workspace_repo_paths`,
  `_cached_repo_libraries`, `_record_board_selection`,
  `_EOS_PROJECT_CONFIG`; the `--board` option on `configure` and the
  call that persists it; and the `{**_workspace_repo_paths(), **...}`
  merge at both `_install_packages` call sites.

  Without that last hunk a scaffolded project that `use`s eos or eboot
  gets no include or link paths from ~/.ebuild/repos, which is the
  "fatal error: eos/hal.h: No such file or directory" that #64 fixed.
  Without the option, the documented golden path -- `configure --board`
  then a bare `build` -- fails with "no such option: --board".

* the initramfs test: `os` / `stat` imports and the `_newc_members`
  newc parser the assertions call.

* test_package_registry.py: import the name the module now exports.

Collection goes from 9 errors / 0 tests to 446 passing. The 12 that
still fail are pre-existing and out of scope here: `doctor` (#72) and
`package` (#77) are two more features whose implementations the same
merges dropped and which should be restored by their authors, and the
rest are separate bugs -- see the PR description.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant