Add rich embedded skills with full test coverage

.agents/skills/obol-stack-dev/SKILL.md

@@ -108,20 +108,68 @@ go test ./internal/openclaw/                                    # Unit tests
 go test -tags integration -v -timeout 10m ./internal/openclaw/  # Integration tests
 ```
 
+## OpenClaw Skills System
+
+Skills are SKILL.md files (with optional scripts and references) that give the agent domain-specific capabilities. Delivered via host-path PVC injection to `/data/.openclaw/skills/` inside the pod.
+
+### Default Embedded Skills
+
+| Skill | Contents | Purpose |
+|-------|----------|---------|
+| `hello` | `SKILL.md` | Smoke test |
+| `obol-blockchain` | `SKILL.md`, `scripts/rpc.py`, `references/` | Ethereum JSON-RPC, ERC-20, ENS via eRPC |
+| `obol-k8s` | `SKILL.md`, `scripts/kube.py` | K8s cluster diagnostics via ServiceAccount API |
+| `obol-dvt` | `SKILL.md`, `references/api-examples.md` | DVT monitoring via Obol API |
+
+### Skills CLI
+
+```bash
+obol openclaw skills list                   # list installed skills
+obol openclaw skills sync                   # re-inject embedded defaults
+obol openclaw skills sync --from ./custom   # push custom skills
+obol openclaw skills add <package>          # add via openclaw CLI in pod
+obol openclaw skills remove <name>          # remove skill from pod
+```
+
+### Skills Delivery Flow
+
+1. `stageDefaultSkills(deploymentDir)` — copies embedded skills to deployment dir
+2. `injectSkillsToVolume(cfg, id, deploymentDir)` — copies to host PVC path (`$DATA_DIR/openclaw-<id>/openclaw-data/.openclaw/skills/`)
+3. `doSync()` — helmfile sync; OpenClaw file watcher discovers skills on startup
+
+### Skills Testing
+
+```bash
+# Unit tests (embedding + injection)
+go test -v -run TestGetEmbeddedSkillNames ./internal/embed/
+go test -v -run TestInjectSkillsToVolume ./internal/openclaw/
+
+# Integration tests (requires running cluster)
+go test -tags integration -v -run TestIntegration_Skills -timeout 10m ./internal/openclaw/
+
+# In-pod smoke tests (piped via kubectl exec)
+obol kubectl exec -i -n openclaw-<id> deploy/openclaw -c openclaw -- python3 - < tests/skills_smoke_test.py
+```
+
 ## Key Source Files
 
 | File | Purpose |
 |------|---------|
-| `internal/openclaw/openclaw.go` | `Onboard()`, `Sync()`, `Delete()`, `buildLLMSpyRoutedOverlay()`, `generateOverlayValues()` |
+| `internal/openclaw/openclaw.go` | `Onboard()`, `Sync()`, `Delete()`, `buildLLMSpyRoutedOverlay()`, `generateOverlayValues()`, `stageDefaultSkills()`, `injectSkillsToVolume()` |
 | `internal/openclaw/import.go` | `DetectExistingConfig()`, `TranslateToOverlayYAML()` |
 | `internal/openclaw/overlay_test.go` | Unit tests for overlay generation |
-| `internal/openclaw/integration_test.go` | Full-cluster integration tests (build tag: `integration`) |
+| `internal/openclaw/skills_injection_test.go` | Unit tests for skill staging and volume injection |
+| `internal/openclaw/integration_test.go` | Full-cluster integration tests (build tag: `integration`) — includes skills + inference tests |
 | `internal/model/model.go` | `ConfigureLLMSpy()` — patches llmspy Secret + ConfigMap + restart |
 | `cmd/obol/model.go` | `obol model setup` CLI command |
-| `cmd/obol/openclaw.go` | `obol openclaw` CLI commands |
+| `cmd/obol/openclaw.go` | `obol openclaw` CLI commands (including `skills` subcommands) |
 | `internal/embed/infrastructure/base/templates/llm.yaml` | llmspy Kubernetes resources |
+| `internal/embed/skills/` | Embedded default skills (hello, obol-blockchain, obol-k8s, obol-dvt) |
+| `internal/embed/embed.go` | `CopySkills()`, `GetEmbeddedSkillNames()` |
+| `internal/embed/embed_skills_test.go` | Unit tests for skill embedding |
 | `internal/openclaw/chart/values.yaml` | Default per-instance model config |
 | `internal/openclaw/chart/templates/_helpers.tpl` | Renders model providers into OpenClaw JSON config |
+| `tests/skills_smoke_test.py` | In-pod Python smoke tests for all rich skills |
 
 ## Constraints
 

.gitignore

@@ -39,6 +39,10 @@ build/
 !.env.example
 .envrc.local
 
+# Python
+__pycache__/
+*.pyc
+
 # Temporary files
 *.tmp
 tmp/

CLAUDE.md

@@ -770,34 +770,38 @@ models:
 
 ### Overview
 
-OpenClaw skills are SKILL.md files that give the AI agent domain-specific capabilities. The Obol Stack ships default skills embedded in the `obol` binary and supports runtime skill management via the openclaw CLI inside the pod.
+OpenClaw skills are SKILL.md files (with optional scripts and references) that give the AI agent domain-specific capabilities. The Obol Stack ships default skills embedded in the `obol` binary and supports runtime skill management via the CLI.
 
-### Two Delivery Channels
+### Delivery Mechanism: Host-Path PVC Injection
 
-**Compile-time (default skills)**: SKILL.md files embedded in `internal/embed/skills/` are staged to the deployment config directory and pushed as a ConfigMap during `doSync()`.
+Skills are delivered by writing directly to the host filesystem at `$DATA_DIR/openclaw-<id>/openclaw-data/.openclaw/skills/`, which maps to `/data/.openclaw/skills/` inside the OpenClaw container via k3d volume mounts and local-path-provisioner.
 
-**Runtime**: `obol openclaw skills add/remove/list` runs the native openclaw CLI inside the pod via `kubectl exec -c openclaw`. Skills are persisted on the pod's PVC.
+**Advantages over ConfigMap approach**: No 1MB size limit, works before pod readiness, survives pod restarts, supports binary files and scripts.
 
-### Default Skills (MVP)
+### Default Skills
 
-| Skill | Purpose |
-|-------|---------|
-| `hello` | Smoke test — confirms skills are loaded |
-| `ethereum` | Ethereum JSON-RPC queries via the eRPC gateway (`/rpc/<network>`) |
+| Skill | Contents | Purpose |
+|-------|----------|---------|
+| `hello` | `SKILL.md` | Smoke test — confirms skills pipeline works |
+| `obol-blockchain` | `SKILL.md`, `scripts/rpc.py`, `references/erc20-methods.md`, `references/common-contracts.md` | Ethereum JSON-RPC queries, ERC-20 token ops, ENS resolution, gas estimation via the eRPC gateway |
+| `obol-k8s` | `SKILL.md`, `scripts/kube.py` | Kubernetes cluster diagnostics — pods, logs, events, deployments via ServiceAccount API |
+| `obol-dvt` | `SKILL.md`, `references/api-examples.md` | Obol DVT cluster monitoring, operator audit, exit coordination via Obol API |
 
 ### Skill Delivery Flow
 
 ```
 Onboard / Sync:
   1. stageDefaultSkills(deploymentDir)
-     → writes embedded skills to $CONFIG_DIR/applications/openclaw/<id>/skills/
-     → skips if skills/ directory already exists
+     → copies embedded skills from internal/embed/skills/ to deploymentDir/skills/
+     → skips if skills/ directory already exists (preserves user customizations)
 
-  2. doSync() → helmfile sync (creates namespace, chart, pod)
+  2. injectSkillsToVolume(cfg, id, deploymentDir)
+     → copies skills/ from deployment dir to host PVC path:
+       $DATA_DIR/openclaw-<id>/openclaw-data/.openclaw/skills/
+     → this path is volume-mounted into the pod at /data/.openclaw/skills/
 
-  3. syncStagedSkills(cfg, id, deploymentDir)
-     → calls SkillsSync() which packages skills/ into ConfigMap openclaw-<id>-skills
-     → chart mounts ConfigMap into pod, extract-skills init container unpacks it
+  3. doSync() → helmfile sync (creates namespace, chart, pod)
+     → OpenClaw file watcher auto-discovers skills on startup
 ```
 
 ### Instance Resolution
@@ -810,21 +814,25 @@ All `obol openclaw` subcommands (except `onboard` and `list`) use `ResolveInstan
 ### CLI Commands
 
 ```bash
-obol openclaw skills list              # list installed skills (auto-resolves instance)
-obol openclaw skills add <package>     # add via openclaw CLI in pod
-obol openclaw skills remove <name>     # remove via openclaw CLI in pod
-obol openclaw skills sync --from <dir> # push local dir as ConfigMap (legacy)
+obol openclaw skills list                   # list installed skills (auto-resolves instance)
+obol openclaw skills add <package>          # add via openclaw CLI in pod
+obol openclaw skills remove <name>          # remove via openclaw CLI in pod
+obol openclaw skills sync                   # re-inject embedded defaults to volume
+obol openclaw skills sync --from <dir>      # push custom skills from local directory
 ```
 
 ### Key Source Files
 
 | File | Role |
 |------|------|
-| `internal/embed/skills/` | Embedded default SKILL.md files |
+| `internal/embed/skills/` | Embedded default SKILL.md files + scripts + references |
 | `internal/embed/embed.go` | `CopySkills()`, `GetEmbeddedSkillNames()` |
+| `internal/embed/embed_skills_test.go` | Unit tests for skill embedding and copying |
 | `internal/openclaw/resolve.go` | `ResolveInstance()`, `ListInstanceIDs()` |
-| `internal/openclaw/openclaw.go` | `stageDefaultSkills()`, `syncStagedSkills()`, `SkillAdd/Remove/List()` |
+| `internal/openclaw/openclaw.go` | `stageDefaultSkills()`, `injectSkillsToVolume()`, `skillsVolumePath()`, `SkillAdd/Remove/List/Sync()` |
+| `internal/openclaw/skills_injection_test.go` | Unit tests for staging and volume injection |
 | `cmd/obol/openclaw.go` | CLI wiring for `obol openclaw skills` subcommands |
+| `tests/skills_smoke_test.py` | In-pod Python smoke tests for all 3 rich skills |
 
 ## Network Install Implementation Details
 
@@ -1081,12 +1089,17 @@ obol network delete ethereum-<generated-name> --force
   - `aztec/helmfile.yaml.gotmpl`
 - `internal/embed/defaults/` - Default stack resources
 - `internal/embed/infrastructure/` - Infrastructure resources (llmspy, Traefik)
-- `internal/embed/skills/` - Default OpenClaw skills (hello, ethereum) embedded in obol binary
+- `internal/embed/skills/` - Default OpenClaw skills (hello, obol-blockchain, obol-k8s, obol-dvt) embedded in obol binary
 
 **Skills system**:
 - `internal/openclaw/resolve.go` - Smart instance resolution (0/1/2+ instances)
 - `internal/embed/skills/hello/SKILL.md` - Hello world smoke-test skill
-- `internal/embed/skills/ethereum/SKILL.md` - Ethereum JSON-RPC via eRPC skill
+- `internal/embed/skills/obol-blockchain/` - Ethereum JSON-RPC, ERC-20, ENS via eRPC (SKILL.md + scripts/rpc.py + references/)
+- `internal/embed/skills/obol-k8s/` - Kubernetes cluster diagnostics (SKILL.md + scripts/kube.py)
+- `internal/embed/skills/obol-dvt/` - DVT cluster monitoring via Obol API (SKILL.md + references/api-examples.md)
+- `internal/embed/embed_skills_test.go` - Unit tests for skill embedding
+- `internal/openclaw/skills_injection_test.go` - Unit tests for skill staging and injection
+- `tests/skills_smoke_test.py` - In-pod Python smoke tests for all rich skills
 
 **Testing**:
 - `internal/openclaw/integration_test.go` - Full-cluster integration tests (Ollama, Anthropic, OpenAI inference through llmspy)

README.md

@@ -161,7 +161,28 @@ obol openclaw delete --force
 
 When only one OpenClaw instance is installed, the instance ID is optional — it is auto-selected. With multiple instances, specify the name: `obol openclaw setup prod`.
 
-Default Obol skills (`hello`, `ethereum`) are installed automatically on first deploy and provide the agent with eRPC JSON-RPC access and a basic smoke test.
+### Skills
+
+OpenClaw ships with four embedded skills that are installed automatically on first deploy:
+
+| Skill | Purpose |
+|-------|---------|
+| `hello` | Smoke test — confirms skills pipeline works |
+| `obol-blockchain` | Ethereum JSON-RPC queries, ERC-20 token ops, ENS resolution via the eRPC gateway |
+| `obol-k8s` | Kubernetes cluster diagnostics — pods, logs, events, deployments |
+| `obol-dvt` | Obol DVT cluster monitoring, operator audit, exit coordination |
+
+Manage skills at runtime:
+
+```bash
+obol openclaw skills list                   # list installed skills
+obol openclaw skills sync                   # re-inject embedded defaults
+obol openclaw skills sync --from ./my-skills  # push custom skills from local dir
+obol openclaw skills add <package>          # add via openclaw CLI in pod
+obol openclaw skills remove <name>          # remove via openclaw CLI in pod
+```
+
+Skills are delivered via host-path PVC injection — no ConfigMap size limits, works before pod readiness, and survives pod restarts.
 
 ## Public Access (Cloudflare Tunnel)
 

cmd/obol/model.go

@@ -23,7 +23,7 @@ func modelCommand(cfg *config.Config) *cli.Command {
 				Flags: []cli.Flag{
 					&cli.StringFlag{
 						Name:  "provider",
-						Usage: "Provider name (anthropic, openai)",
+						Usage: "Provider name (e.g. anthropic, openai, zai, deepseek)",
 					},
 					&cli.StringFlag{
 						Name:    "api-key",
@@ -38,7 +38,7 @@ func modelCommand(cfg *config.Config) *cli.Command {
 					// Interactive mode if flags not provided
 					if provider == "" || apiKey == "" {
 						var err error
-						provider, apiKey, err = promptModelConfig()
+						provider, apiKey, err = promptModelConfig(cfg)
 						if err != nil {
 							return err
 						}
@@ -64,7 +64,7 @@ func modelCommand(cfg *config.Config) *cli.Command {
 
 					fmt.Println("Global llmspy providers:")
 					fmt.Println()
-					fmt.Printf("  %-12s %-8s %-10s %s\n", "PROVIDER", "ENABLED", "API KEY", "ENV VAR")
+					fmt.Printf("  %-20s %-8s %-10s %s\n", "PROVIDER", "ENABLED", "API KEY", "ENV VAR")
 					for _, name := range providers {
 						s := status[name]
 						key := "n/a"
@@ -75,8 +75,12 @@ func modelCommand(cfg *config.Config) *cli.Command {
 								key = "missing"
 							}
 						}
-						fmt.Printf("  %-12s %-8t %-10s %s\n", name, s.Enabled, key, s.EnvVar)
+						fmt.Printf("  %-20s %-8t %-10s %s\n", name, s.Enabled, key, s.EnvVar)
 					}
+
+					// Show hint about available providers
+					fmt.Println()
+					fmt.Println("Run 'obol model setup' to configure a provider.")
 					return nil
 				},
 			},
@@ -85,38 +89,42 @@ func modelCommand(cfg *config.Config) *cli.Command {
 }
 
 // promptModelConfig interactively asks the user for provider and API key.
-func promptModelConfig() (string, string, error) {
+// It queries the running llmspy pod for available providers.
+func promptModelConfig(cfg *config.Config) (string, string, error) {
+	providers, err := model.GetAvailableProviders(cfg)
+	if err != nil {
+		return "", "", fmt.Errorf("failed to discover providers: %w", err)
+	}
+	if len(providers) == 0 {
+		return "", "", fmt.Errorf("no cloud providers found in llmspy")
+	}
+
 	reader := bufio.NewReader(os.Stdin)
 
-	fmt.Println("Select a provider:")
-	fmt.Println("  [1] Anthropic")
-	fmt.Println("  [2] OpenAI")
-	fmt.Print("\nChoice [1]: ")
+	fmt.Println("Available providers:")
+	for i, p := range providers {
+		fmt.Printf("  [%d] %s (%s)\n", i+1, p.Name, p.ID)
+	}
+	fmt.Printf("\nChoice [1]: ")
 
 	line, _ := reader.ReadString('\n')
 	choice := strings.TrimSpace(line)
 	if choice == "" {
 		choice = "1"
 	}
 
-	var provider, display string
-	switch choice {
-	case "1":
-		provider = "anthropic"
-		display = "Anthropic"
-	case "2":
-		provider = "openai"
-		display = "OpenAI"
-	default:
-		return "", "", fmt.Errorf("unknown choice: %s", choice)
+	idx := 0
+	if _, err := fmt.Sscanf(choice, "%d", &idx); err != nil || idx < 1 || idx > len(providers) {
+		return "", "", fmt.Errorf("invalid choice: %s", choice)
 	}
+	selected := providers[idx-1]
 
-	fmt.Printf("\n%s API key: ", display)
+	fmt.Printf("\n%s API key (%s): ", selected.Name, selected.EnvVar)
 	apiKey, _ := reader.ReadString('\n')
 	apiKey = strings.TrimSpace(apiKey)
 	if apiKey == "" {
 		return "", "", fmt.Errorf("API key is required")
 	}
 
-	return provider, apiKey, nil
+	return selected.ID, apiKey, nil
 }