feat: implement package management system with recipe support, regist…

CHANGELOG.md

@@ -2,14 +2,27 @@
 
 ## [Unreleased]
 
+### Added
+- **Remote Package Index & Synchronization (`ebuild/packages/index_sync.py`).**
+  Downloads and validates central/mirror package repository indices into a local cache
+  (`~/.ebuild/index/`), caching full recipe definitions. Enforces HTTPS transport,
+  path-traversal sanitization (`^[a-zA-Z0-9_-]+$`), 10s socket timeouts, and 10MB response
+  size limits. Index synchronization supports air-gapped operation via `--offline` and `EBUILD_OFFLINE=1`; package archive fetching is not yet offline-gated.
+- **Package Discovery & Multi-Source Search (`ebuild search`, `ebuild/packages/repository.py`).**
+  Search across local project recipes, system-shipped recipes, and cached remote indices.
+  Supports `--all`, `--json`, `--build-system`, and `--license` filters.
+- **Index Synchronization Command (`ebuild update-index`).**
+  CLI command to refresh local package and recipe index caches from remote repositories.
+- **Source-Ranked Recipe Precedence (`ebuild/packages/registry.py`).**
+  Enforces strict 3-tier precedence hierarchy during package resolution: project-local recipes (`./recipes/`) > system-shipped recipes (`recipes/`) > cached remote index recipes (`~/.ebuild/index/recipes/`), guaranteeing reproducible builds (§9.2) and ensuring project-level pins override remote definitions.
+- **Stale Cached Recipe Pruning (`ebuild/packages/index_sync.py`, `ebuild update-index`).**
+  `ebuild update-index` automatically prunes stale cached `.yaml` and `.yml` recipes from `~/.ebuild/index/recipes/` that are absent from the newly synchronized remote package index. An index that yields no usable entries is treated as a delivery fault: nothing is pruned, the cached index is left in place, and the condition is reported. Surfaced the count of pruned recipes in CLI output and returned `SyncResult`.
+- **Expanded Shipped Recipes Catalog (`recipes/`).**
+  Added 5 verified recipes with HTTPS release pins and SHA-256 integrity digests:
+  `cjson` (v1.7.18), `nanopb` (v0.4.9.1), `lvgl` (v9.2.2), `tinyusb` (v0.18.0), and `unity` (v2.6.1).
+
 ### Fixed
-- **`ebuild flash --tool openocd` split an image path containing a space into
-  extra arguments.** The path was interpolated straight into the `-c`
-  argument's `program <path> <addr> verify reset exit` string; OpenOCD's own
-  Tcl interpreter re-parses that string and splits on unescaped whitespace, so
-  `ebuild flash "my firmware.bin"` sent OpenOCD `program my` as the filename.
-  The path is now wrapped in Tcl brace-grouping (`{...}`) so OpenOCD treats it
-  as one token regardless of embedded spaces (`ebuild/firmware/flash.py`).
+
 - **A path containing a space produced a silently wrong `build.ninja`.** Paths
   were written into build statements unescaped, but Ninja ends the output list
   at the first unescaped `:` and splits on unescaped spaces. A build directory

README.md

@@ -91,10 +91,10 @@ that each package's own build may already run parallel compile jobs, so a large
 [docs/dependency-management.md](docs/dependency-management.md#parallel-package-builds).
 
 Additional commands: `configure`, `install`, `add`, `list-packages`,
-`pipeline`, `system`, `firmware`, `flash`, `new`, `generate-project`,
-`generate-board`, `generate-boot`, `analyze`, `setup`, and the `repos` group
-(`status`, `update`, `set-url`, `set-branch`, `link`, `unlink`). Run
-`ebuild --help` for the full list.
+`search`, `update-index`, `pipeline`, `system`, `firmware`, `flash`, `new`,
+`generate-project`, `generate-board`, `generate-boot`, `analyze`, `setup`,
+and the `repos` group (`status`, `update`, `set-url`, `set-branch`, `link`,
+`unlink`). Run `ebuild --help` for the full list.
 
 
 `ebuild test` reports the counts the underlying runner printed, and reports none

TASKS.md

@@ -18,6 +18,7 @@ Status is one of: `todo`, `in-progress`, `blocked`, `review`, `done`.
 | ID | Task | Owner | Verified by | Evidence |
 |----|------|-------|-------------|----------|
 | T-001 | Make initramfs creation portable and self-contained | backend | independent reviewer | Focused archive tests: **5 passed, 1 skipped** (symlink creation unavailable on this Windows host). Independent `bsdtar` extraction validated hard-link identity and payload. Full Python suite: **288 passed, 2 skipped, 1 unrelated failure** in the pre-existing Windows Ninja path assertion, recorded as T-002. QEMU boot was not run on Windows. |
+| T-003 | Address PR #111 review findings 1, 2, 3, 6, 7, 8 | backend | reviewer | Unit tests in `tests/unit/test_index_sync.py` (22 passed) verify keep-set filename matching, empty index floor and fallback status, .yml preservation and pruning, lack-of-URL recipe preservation, CLI prune reporting, and trailing newline in `CHANGELOG.md`. Static analysis with `ruff` on all PR-touched files reports 0 errors; `mypy` on modified packages reports 0 errors without suppressions. Full test suite passes 605 tests on Linux. |
 
 ---
 

docs/architecture.md

@@ -7,74 +7,82 @@
 ```mermaid
 graph TD
     subgraph CLI["ebuild CLI (Python)"]
-        CMD[cli/commands.py<br>18 commands]
+        CMD["cli/commands.py<br/>CLI Commands"]
     end
 
     subgraph BUILD["Build Orchestrator"]
-        ORCH[build/orchestrator.py]
-        NINJA[build/ninja_backend.py]
-        TC[build/toolchain.py<br>5 predefined toolchains]
+        ORCH["build/dispatch.py<br/>Backend Dispatcher"]
+        NINJA["build/ninja_backend.py<br/>Ninja Backend"]
+        TC["build/toolchain.py<br/>5 Predefined Toolchains"]
     end
 
     subgraph PACKAGES["Package Pipeline"]
-        RECIPE[packages/recipe.py]
-        REG[packages/registry.py]
-        RESOLVE[packages/resolver.py]
-        FETCH[packages/fetcher.py]
-        BUILDER[packages/builder.py]
-        CACHE[packages/cache.py]
-        LOCK[packages/lockfile.py]
-        REPO[packages/repository.py<br>Remote index]
-        PROFILES[packages/profiles.py<br>Build profiles]
+        SYNC["packages/index_sync.py<br/>Remote Index Sync"]
+        REPO["packages/repository.py<br/>Package Discovery"]
+        RECIPE["packages/recipe.py<br/>Recipe Schema"]
+        REG["packages/registry.py<br/>Package Registry"]
+        RESOLVE["packages/resolver.py<br/>Dependency Resolver"]
+        FETCH["packages/fetcher.py<br/>Package Fetcher"]
+        BUILDER["packages/builder.py<br/>Package Builder"]
+        CACHE["packages/cache.py<br/>Build Cache"]
+        LOCK["packages/lockfile.py<br/>Lockfile"]
+        PROFILES["packages/profiles.py<br/>Build Profiles"]
     end
 
     subgraph HWAI["Hardware AI"]
-        ANALYZER[eos_ai/eos_hw_analyzer.py<br>MCU database + peripheral detection]
-        PROJGEN[eos_ai/eos_project_generator.py<br>Manifest + project scaffolding]
+        ANALYZER["eos_ai/eos_hw_analyzer.py<br/>MCU & Peripheral Analysis"]
+        PROJGEN["eos_ai/eos_project_generator.py<br/>Project & Config Generator"]
     end
 
-    subgraph CORE["Core Components (always built)"]
-        EOS[core/eos/<br>HAL, Kernel, Crypto, OTA, Drivers]
-        EBOOT[core/eboot/<br>Bootloader, 26 board ports]
+    subgraph CORE["Core Components (Native)"]
+        EOS["core/eos/<br/>HAL, Kernel, Crypto, Drivers"]
+        EBOOT["core/eboot/<br/>Bootloader, 26 Board Ports"]
     end
 
-    subgraph LAYERS["Optional Layers (--with flag)"]
-        EAI[layers/eai/<br>AI inference + LLM models]
-        ENI[layers/eni/<br>Neural interface]
-        EIPC[layers/eipc/<br>Secure IPC (Go + C)]
-        EOSUITE[layers/eosuite/<br>Dev tools + GUI apps]
+    subgraph LAYERS["Optional Platform Layers"]
+        EAI["layers/eai/<br/>AI & Embedded Inference"]
+        ENI["layers/eni/<br/>Neural Interface"]
+        EIPC["layers/eipc/<br/>Secure IPC SDK"]
+        EOSUITE["layers/eosuite/<br/>Developer Tools"]
     end
 
     subgraph HW["Hardware Intake"]
-        BOARD[hardware/board/<br>KiCad, Eagle, YAML, BOM]
-        SOC[hardware/soc/<br>Datasheets, TRMs]
-        BOOT[hardware/boot/<br>Image layout, boot flow]
-        SW[hardware/software/<br>Device trees, linker scripts]
+        BOARD["hardware/board/<br/>KiCad, Eagle, YAML, BOM"]
+        SOC["hardware/soc/<br/>Datasheets, TRMs"]
+        BOOT["hardware/boot/<br/>Image Layout, Boot Flow"]
+        SW["hardware/software/<br/>Device Trees, Linker Scripts"]
     end
 
     subgraph SDK_OUT["SDK Output"]
-        SDKGEN[sdk_generator.py]
-        SDKAPI[sdk/include/<br>Header-only API]
+        SDKGEN["sdk_generator.py"]
+        SDKAPI["sdk/include/<br/>Header-only API"]
     end
 
     subgraph TEMPLATES["Project Templates"]
-        T1[bare-metal]
-        T2[rtos-app]
-        T3[linux-app]
-        T4[safety-critical]
-        T5[secure-boot]
-        T6[ble-sensor]
+        T1["bare-metal"]
+        T2["rtos-app"]
+        T3["linux-app"]
+        T4["safety-critical"]
+        T5["secure-boot"]
+        T6["ble-sensor"]
     end
 
     CMD --> ORCH
     CMD --> SDKGEN
     CMD --> ANALYZER
     CMD --> PROJGEN
+    CMD --> SYNC
+    CMD --> REPO
+
+    SYNC --> REPO
+    REPO --> REG
 
     ORCH --> NINJA
     ORCH --> TC
-    ORCH --> CORE
-    ORCH --> LAYERS
+    ORCH --> EOS
+    ORCH --> EBOOT
+    ORCH --> EAI
+    ORCH --> EIPC
 
     ORCH --> RECIPE
     RECIPE --> REG
@@ -83,16 +91,24 @@ graph TD
     FETCH --> BUILDER
     BUILDER --> CACHE
     RESOLVE --> LOCK
-    REG --> REPO
 
-    HW --> ANALYZER
+    BOARD --> ANALYZER
+    SOC --> ANALYZER
+    BOOT --> ANALYZER
+    SW --> ANALYZER
+
     ANALYZER --> PROJGEN
-    PROJGEN --> TEMPLATES
-    PROJGEN --> CORE
+    PROJGEN --> T1
+    PROJGEN --> T2
+    PROJGEN --> T3
+    PROJGEN --> EOS
     PROJGEN --> EBOOT
 
-    TC -->|cmake| CORE
-    TC -->|cmake| LAYERS
+    TC -->|cmake| EOS
+    TC -->|cmake| EAI
+    TC -->|cmake| EIPC
+
+    SDKGEN --> SDKAPI
 ```
 
 ## Subsystem Details
@@ -157,14 +173,15 @@ recipe.yaml → Registry → Resolver → Fetcher → Builder → Cache
 ```
 
 1. **Recipe** (`recipe.py`) — YAML schema defining package name, version, URL, checksum, build system, dependencies
-2. **Registry** (`registry.py`) — scans recipe directories and indexes available packages
-3. **Repository** (`repository.py`) — remote package index for discovery and search
-4. **Resolver** (`resolver.py`) — dependency resolution with version constraint solving
-5. **Fetcher** (`fetcher.py`) — downloads and verifies source archives
-6. **Builder** (`builder.py`) — builds packages using the specified build system
-7. **Cache** (`cache.py`) — caches built artifacts to avoid rebuilding
-8. **Lockfile** (`lockfile.py`) — records exact resolved versions for reproducibility
-9. **Profiles** (`profiles.py`) — composable build profiles (minimal, standard, full, custom)
+2. **Registry** (`registry.py`) — scans recipe directories (local, system, and cached remote) and indexes available packages
+3. **Index Sync** (`index_sync.py`) — downloads, validates, and manages remote package repository indices and recipe caches (`~/.ebuild/index/`) with offline fallback
+4. **Repository** (`repository.py`) — unified package discovery and multi-source search engine (`ebuild search`)
+5. **Resolver** (`resolver.py`) — dependency resolution with version constraint solving
+6. **Fetcher** (`fetcher.py`) — downloads and verifies source archives against SHA-256 integrity pins
+7. **Builder** (`builder.py`) — builds packages using the specified build system (CMake, Make, Meson, autoconf)
+8. **Cache** (`cache.py`) — caches built artifacts to avoid rebuilding
+9. **Lockfile** (`lockfile.py`) — records exact resolved versions for reproducibility
+10. **Profiles** (`profiles.py`) — composable build profiles (minimal, standard, full, custom)
 
 ### Hardware AI (`ebuild/eos_ai/`)
 

docs/dependency-management.md

@@ -272,3 +272,100 @@ poles without thrashing; going much higher mostly adds memory pressure.
 Parallel builds also interleave package log lines. Output from each package is
 written atomically, but the packages themselves are no longer contiguous —
 use `-j 1` when reading a build log closely.
+
+---
+
+## Remote Package Index & Discovery
+
+ebuild provides index-based package discovery and remote recipe synchronization, allowing embedded projects to discover, query, and install external libraries seamlessly.
+
+### Discovering Packages (`ebuild search`)
+
+Search across local project recipes (`./recipes/`), bundled system recipes, and cached remote repository indices:
+
+```bash
+ebuild search json                 # Search by keyword in name, description, or license
+ebuild search --all                # List all available packages across all sources
+ebuild search --json               # Output machine-readable JSON array
+ebuild search --build-system cmake # Filter by build system (cmake, make, meson)
+ebuild search --license MIT        # Filter by license type
+```
+
+Example output:
+
+```
+=== ebuild - Package Search ===
+[info] Found 10 package(s):
+   cjson v1.7.18 [cmake] (MIT) - Ultralightweight JSON parser in ANSI C
+   freertos v11.1.0 [cmake] (MIT) - Real-time operating system kernel for embedded devices
+   littlefs v2.9.3 [make] (BSD-3-Clause) - Little fail-safe filesystem designed for microcontrollers
+   lvgl v9.2.2 [cmake] (MIT) - Light and Versatile Embedded Graphics Library
+   lwip v2.2.0 [cmake] (BSD-3-Clause) - Lightweight TCP/IP stack for embedded systems
+   mbedtls v3.6.0 [cmake] (Apache-2.0) - Lightweight TLS/SSL library for embedded systems
+   nanopb v0.4.9.1 [cmake] (zlib) - Protocol Buffers with small code size for microcontrollers
+   tinyusb v0.18.0 [cmake] (MIT) - Open-source cross-platform USB host/device stack for embedded system
+   unity v2.6.1 [cmake] (MIT) - Simple Unit Testing for C
+   zlib v1.3.1 [cmake] (Zlib) - General-purpose lossless data compression library
+```
+
+### Synchronizing Remote Index (`ebuild update-index`)
+
+Refresh the local package index and cached recipe definitions from a remote repository mirror:
+
+```bash
+ebuild update-index --url https://example.com/recipes/index.json  # Remote repository URL (HTTPS required)
+ebuild update-index --url https://example.com/recipes/index.json --force  # Bypass 24h cache TTL and re-download
+ebuild update-index --offline                   # Use local cached index without network
+```
+
+> [!WARNING]
+> **Index Authenticity & Provenance Notice (Unauthenticated Index)**:
+> Remote package index synchronization validates transport encryption (HTTPS only) and verifies individual package archive bytes against stated SHA-256 checksums. However, the index document itself is currently **unauthenticated** (detached cryptographic signature verification and package provenance proof are not yet implemented).
+>
+> In accordance with reproducible build guarantees (§9.2), recipe discovery enforces a strict 3-tier source-ranked precedence hierarchy:
+> 1. **Project-Local Recipes** (`./recipes/`): Absolute highest priority; project pins always override upstream.
+> 2. **Shipped Catalog Recipes** (`<ebuild>/recipes/`): Built-in verified recipes shipped with ebuild.
+> 3. **Cached Remote Index** (`~/.ebuild/index/recipes/`): Definitions synchronized from remote repositories.
+>
+> An index update will never override pinned URLs, build systems, or checksums defined in your project repository.
+>
+> Furthermore, `ebuild update-index` automatically prunes stale cached `.yaml` and `.yml` recipes from `~/.ebuild/index/recipes/` that are absent from the updated remote index, preventing retired packages from lingering in the local cache. An index that yields no usable entries is treated as a delivery fault: nothing is pruned, the cached index is left in place, and the condition is reported.
+>
+> To support integrity verification, the SHA-256 digest of the downloaded index is recorded in `~/.ebuild/index/packages.json.sha256`.
+
+### Component Contract (§10.1) Field Coverage
+
+eBuild recipes currently carry a subset of the Master Design §10.1 component contract:
+- **Identity**: Package name, version, and optional description.
+- **Dependencies**: List of external library package dependencies.
+- **Compliance**: Package open-source license identifier.
+- **Integrity**: Transport SHA-256 checksum pin for downloaded tarballs.
+
+*Deferred Contract Fields*: Compatibility constraints (EmbeddedOS ABI/API version, architecture, and SoC targets) and Resource constraints (`flash_max`, `ram_max`) are not yet evaluated by the registry layer and must be validated at project build configuration time.
+
+### Shipped Embedded Library Catalog
+
+ebuild includes a curated suite of pre-packaged recipes under `recipes/`:
+
+| Package | Version | Build System | License | Description |
+|:---|:---|:---|:---|:---|
+| **`cjson`** | 1.7.18 | CMake | MIT | Ultralightweight JSON parser in ANSI C |
+| **`freertos`** | 11.1.0 | CMake | MIT | Real-time operating system kernel for embedded devices |
+| **`littlefs`** | 2.9.3 | Make | BSD-3-Clause | Fail-safe power-resilient filesystem for microcontrollers |
+| **`lvgl`** | 9.2.2 | CMake | MIT | Light and Versatile Embedded Graphics Library |
+| **`lwip`** | 2.2.0 | CMake | BSD-3-Clause | Lightweight TCP/IP stack for embedded targets |
+| **`mbedtls`** | 3.6.0 | CMake | Apache-2.0 | Cryptographic primitives, TLS/SSL stack |
+| **`nanopb`** | 0.4.9.1 | CMake | zlib | Memory-efficient Protocol Buffers implementation |
+| **`tinyusb`** | 0.18.0 | CMake | MIT | Cross-platform USB host/device stack |
+| **`unity`** | 2.6.1 | CMake | MIT | Standard embedded C unit testing framework |
+| **`zlib`** | 1.3.1 | CMake | Zlib | General-purpose lossless data compression |
+
+### Offline & Air-Gapped Operation
+
+For isolated CI/CD pipelines and field deployments:
+- Set environment variable `EBUILD_OFFLINE=1` or pass `--offline` to `ebuild update-index` to use cached remote indices without attempting network synchronization.
+- Note: Package archive source fetching (`ebuild build`) currently verifies archive integrity against pinned SHA-256 checksums but is not yet gated by the `--offline` flag.
+- ebuild automatically searches local directories first and gracefully falls back to cached indices in `~/.ebuild/index/` if the network is unreachable.
+- All downloads are validated against SHA-256 integrity pins before extraction.
+
+