fix: repo review cleanup — CI, eval harness, manifests, and docs

.agents/plugins/marketplace.json

@@ -0,0 +1,24 @@
+{
+  "name": "flyte-agent-plugins",
+  "interface": {
+    "displayName": "Flyte Agent Plugins"
+  },
+  "plugins": [
+    {
+      "name": "flyte",
+      "description": "Work with Flyte 2. Deploy Flyte on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers.",
+      "source": {
+        "source": "local",
+        "path": "./plugins/flyte"
+      },
+      "policy": {
+        "installation": "AVAILABLE",
+        "authentication": "ON_USE"
+      },
+      "category": "Developer Tools",
+      "interface": {
+        "displayName": "Flyte"
+      }
+    }
+  ]
+}

.claude-plugin/marketplace.json

@@ -4,12 +4,12 @@
     "name": "flyteorg",
     "url": "https://github.com/flyteorg/flyte-agent-plugins"
   },
-  "description": "Claude Code skills and MCP servers for working with Flyte.",
+  "description": "Agent skills and MCP servers for working with Flyte.",
   "plugins": [
     {
       "name": "flyte",
       "source": "./plugins/flyte",
-      "description": "Work with Flyte 2 — deploy Flyte v2 (flyte-binary) clusters on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers."
+      "description": "Work with Flyte 2. Deploy Flyte on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers."
     }
   ]
 }

.github/workflows/skill-evals.yml

@@ -8,6 +8,7 @@ on:
     paths:
       - "plugins/**"
       - "evals/**"
+      - ".github/workflows/skill-evals.yml"
   schedule:
     - cron: "0 7 * * *"   # nightly full matrix (incl. real tier)
   workflow_dispatch: {}
@@ -16,6 +17,17 @@ env:
   PYTHONPATH: ${{ github.workspace }}
 
 jobs:
+  # 0) Harness unit tests — cheap, no cluster, no secrets.
+  harness-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pytest pyyaml requests
+      - run: python -m pytest evals/tests -q
+
   # 1) Decide what to run from the diff.
   select:
     runs-on: ubuntu-latest
@@ -61,7 +73,7 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
-      - run: pip install flyte>=2.5.0 pyyaml requests
+      - run: pip install 'flyte>=2.5.0' pyyaml requests
       - name: Run evals on demo.hosted
         env:
           # The flyte v2 SDK authenticates to demo.hosted via an API key read from

.gitignore

@@ -3,3 +3,7 @@ __pycache__/
 .ruff_cache/
 .flyte/
 .DS_Store
+.venv/
+*.egg-info/
+*.zip
+__MACOSX/

README.md

@@ -1,7 +1,9 @@
 # Flyte Agent Plugins
 
-A [Claude Code](https://docs.claude.com/en/docs/claude-code) plugin marketplace for
-working with [Flyte](https://flyte.org).
+A plugin marketplace for working with [Flyte](https://flyte.org) — in
+[Claude Code](https://docs.claude.com/en/docs/claude-code),
+[OpenAI Codex](https://developers.openai.com/plugins), or any agent harness that
+supports [Agent Skills](https://agentskills.io).
 
 Everything ships in a single `flyte` plugin: 14 skills, plus two **MCP servers** that let
 Claude search Flyte docs and act on your own cluster — see
@@ -115,8 +117,8 @@ pi install git:github.com/flyteorg/flyte-agent-plugins@<tag>         # pinned to
 
 | Skill | Description |
 |-------|-------------|
-| [`flyte-deploy-aws`](plugins/flyte/skills/flyte-deploy-aws) | Deploy a Flyte v2 (`flyte-binary`) cluster on AWS from scratch — EKS + S3 + RDS PostgreSQL + AWS Load Balancer Controller + `helm`, with optional TLS (ACM, incl. cross-account DNS) and Okta/OIDC SSO. |
-| [`deploy-flyte-kind`](plugins/flyte/skills/deploy-flyte-kind) | Deploy a Flyte v2 (`flyte-binary`) cluster on `kind` — on your local machine or a cloud VM (DigitalOcean, AWS EC2, or GCP), backed by a hosted PostgreSQL (Supabase/external) and object store (S3/R2), with optional OIDC auth via Traefik + oauth2-proxy. |
+| [`flyte-deploy-aws`](plugins/flyte/skills/flyte-deploy-aws) | Deploy a Flyte 2 (`flyte-binary`) cluster on AWS from scratch — EKS + S3 + RDS PostgreSQL + AWS Load Balancer Controller + `helm`, with optional TLS (ACM, incl. cross-account DNS) and Okta/OIDC SSO. |
+| [`deploy-flyte-kind`](plugins/flyte/skills/deploy-flyte-kind) | Deploy a Flyte 2 (`flyte-binary`) cluster on `kind` — on your local machine or a DigitalOcean droplet (for AWS EC2 or GCP VMs, see `deploy-flyte-kind-vm`), backed by a hosted PostgreSQL (Supabase/external) and object store (S3/R2), with optional OIDC auth via Traefik + oauth2-proxy. |
 | [`deploy-flyte-kind-vm`](plugins/flyte/skills/deploy-flyte-kind-vm) | Provision a host (local or a fresh DigitalOcean / AWS EC2 / GCP VM), install the tooling, and run the kind Flyte deploy on it with access tunneled back to your machine. |
 | [`start-dex-local`](plugins/flyte/skills/start-dex-local) | Deploy Dex as an in-cluster OIDC provider for testing kind-based Flyte auth with no cloud account or real users. |
 
@@ -155,7 +157,7 @@ Example:
 /plugin install flyte@flyte-agent-plugins
 ```
 
-Then ask Claude to "deploy a Flyte v2 cluster on AWS", or invoke a skill directly with
+Then ask Claude to "deploy a Flyte 2 cluster on AWS", or invoke a skill directly with
 `/flyte:flyte-deploy-aws`.
 
 ## Bundled MCP servers

evals/harness/evaluate.py

@@ -170,6 +170,16 @@ def skipped_scenario(scenario: Scenario, harness: str) -> ScenarioResult:
     return res
 
 
+def errored_scenario(scenario: Scenario, harness: str | None, error: str) -> ScenarioResult:
+    """A ScenarioResult marked errored because the eval action died before
+    returning a verdict (e.g. a failed `flyte.map` element), so the scenario
+    stays on the scorecard instead of silently vanishing."""
+    res = ScenarioResult(scenario.id, scenario.skill, scenario.tier, harness=harness)
+    for arm in scenario.arms():
+        res.arms[arm] = ArmResult(arm=arm, error=error)
+    return res
+
+
 def evaluate_static(scenario: Scenario) -> ScenarioResult:
     skill_dir = REPO_ROOT / "plugins" / "flyte" / "skills" / scenario.skill
     results = lint_skill(skill_dir)
@@ -235,7 +245,7 @@ def _maybe_real_run(scenario: Scenario, sandbox) -> "checks_mod.CheckResult":
     except (subprocess.TimeoutExpired, FileNotFoundError) as e:
         return checks_mod.CheckResult("real_run", False, f"flyte run failed to start: {e}")
     out = proc.stdout + proc.stderr
-    ok = proc.returncode == 0 and (spec.expect_status.upper() in out.upper() or proc.returncode == 0)
+    ok = proc.returncode == 0 and spec.expect_status.upper() in out.upper()
     tail = " / ".join(out.strip().splitlines()[-3:])
     return checks_mod.CheckResult("real_run", ok, f"exit {proc.returncode} :: {tail}")
 

evals/harness/runners/base.py

@@ -18,7 +18,9 @@
 from ..spec import Scenario
 
 # Files we never treat as "agent-produced artifacts" when snapshotting a workspace.
-_IGNORE = {".stublog", ".opencode", ".pi", ".git"}
+# `.hermes` holds the installed skill (would leak SKILL.md into the judge prompt as
+# agent output); `opencode.json` holds the GLM API key.
+_IGNORE = {".stublog", ".opencode", ".pi", ".hermes", ".git", "opencode.json"}
 
 
 @dataclass

evals/select.py

@@ -1,7 +1,8 @@
 """Map changed files -> the subset of scenarios to run.
 
 Used by CI to run only what a PR affects:
-  - a changed `plugins/flyte/skills/<skill>/**` -> that skill's scenarios
+  - a changed `plugins/flyte/skills/<skill>/**` or `evals/scenarios/<skill>/**`
+    -> that skill's scenarios
   - a change to engine/shared-infra paths (manifest.shared_infra_globs) -> run ALL
   - flags whether the kind DinD smoke and the real tier are in scope
 
@@ -27,6 +28,12 @@
 from evals.harness.spec import Manifest, load_scenarios, scenarios_by_skill
 
 SKILL_PATH_RE = re.compile(r"plugins/flyte/skills/([^/]+)/")
+SCENARIO_PATH_RE = re.compile(r"evals/scenarios/([^/]+)/")
+
+
+def _skill_for(path: str) -> str | None:
+    m = SKILL_PATH_RE.search(path) or SCENARIO_PATH_RE.search(path)
+    return m.group(1) if m else None
 
 
 def changed_from_git(base: str, repo_root: pathlib.Path) -> list[str]:
@@ -54,9 +61,7 @@ def select(changed: list[str], manifest: Manifest, scenarios, force_all: bool =
     if run_all:
         chosen_skills = sorted(by_skill)
     else:
-        chosen_skills = sorted({
-            m.group(1) for p in changed if (m := SKILL_PATH_RE.search(p))
-        })
+        chosen_skills = sorted({s for p in changed if (s := _skill_for(p))})
 
     chosen = [sc for sk in chosen_skills for sc in by_skill.get(sk, [])]
     scenario_ids = sorted(sc.id for sc in chosen)

evals/tests/test_report.py

@@ -1,23 +1,46 @@
-from evals.report import to_html, to_markdown
+from evals.report import rating, to_html, to_markdown
 
 RESULTS = [
     {"scenario_id": "a", "skill": "flyte-sdk-author", "tier": "trajectory",
-     "harness": "opencode", "passed": True, "lift": 0.4,
-     "arms": {"treatment": {"score": 0.9, "checks": []}, "control": {"score": 0.5, "checks": []}}},
+     "harness": "opencode", "status": "scored", "passed": True, "score": 0.9,
+     "lift": 0.4, "is_regression": False,
+     "arms": {"treatment": {"score": 0.9, "checks": []},
+              "control": {"score": 0.5, "checks": []}}},
     {"scenario_id": "b", "skill": "deploy-flyte-kind", "tier": "static",
-     "harness": None, "passed": False, "lift": None,
+     "harness": None, "status": "scored", "passed": False, "score": 0.0,
+     "lift": None, "is_regression": True,
      "arms": {"treatment": {"score": 0.0, "error": "boom",
                             "checks": [{"kind": "frontmatter", "passed": False, "detail": "bad"}]}}},
+    {"scenario_id": "c", "skill": "flyte-sdk-run", "tier": "trajectory",
+     "harness": "pi", "status": "skipped", "passed": False, "score": None,
+     "lift": None, "is_regression": False,
+     "arms": {"treatment": {"unavailable": True, "error": "pi CLI not available",
+                            "checks": []}}},
+    {"scenario_id": "d", "skill": "flyte-sdk-run", "tier": "trajectory",
+     "harness": "opencode", "status": "error", "passed": False, "score": None,
+     "lift": None, "is_regression": False,
+     "arms": {"treatment": {"error": "harness crashed", "checks": []}}},
 ]
 
 
-def test_markdown_has_summary_and_failure_detail():
+def test_rating_aggregates():
+    m = rating(RESULTS)
+    assert m["total"] == 4 and m["scored"] == 2
+    assert m["skipped"] == 1 and m["errored"] == 1
+    assert m["regressions"] == 1
+    assert m["rating"] == 0.45  # mean treatment score over the 2 scored rows
+    assert m["mean_lift"] == 0.4
+
+
+def test_markdown_has_rating_glyphs_and_failure_detail():
     md = to_markdown(RESULTS)
-    assert "1/2 passing" in md
+    assert "2 scored, 1 skipped, 1 errored, 1 regressions" in md
     assert "flyte-sdk-author" in md and "+0.40" in md
-    assert "frontmatter" in md and "boom" in md
+    assert "✅" in md and "❌" in md and "⏭" in md and "⚠" in md
+    assert "frontmatter" in md and "bad" in md and "harness crashed" in md
 
 
 def test_html_renders():
     html = to_html(RESULTS)
-    assert "<table>" in html and "flyte-sdk-author" in html and "1/2 passing" in html
+    assert "<table>" in html and "flyte-sdk-author" in html
+    assert "rating 0.450" in html

evals/tests/test_select.py

@@ -34,6 +34,15 @@ def test_kind_skill_sets_run_kind():
     assert out2["run_kind"] is False
 
 
+def test_scenario_change_selects_that_skill():
+    m = _manifest()
+    scs = load_scenarios()
+    out = select(["evals/scenarios/flyte-sdk-author/static.yaml"], m, scs)
+    assert out["run_all"] is False
+    assert out["skills"] == ["flyte-sdk-author"]
+    assert "flyte-sdk-author-static" in out["scenario_ids"]
+
+
 def test_unrelated_change_selects_nothing():
     m = _manifest()
     scs = load_scenarios()

evals/workflows/eval_wf.py

@@ -88,7 +88,7 @@ def main(skills: list[str] | None = None,
          harnesses: list[str] | None = None,
          tiers: list[str] | None = None) -> dict:
     """Top-level workflow: build the matrix, fan out, aggregate."""
-    from evals.harness.evaluate import skipped_scenario
+    from evals.harness.evaluate import errored_scenario, skipped_scenario
     from evals.harness.runners import get_runner
     from evals.harness.spec import load_scenarios
 
@@ -124,7 +124,18 @@ def _available(h: str) -> bool:
         return {"total": 0, "scored": 0, "skipped": 0, "errored": 0,
                 "regressions": 0, "rating": None, "results": [], "markdown": "no units selected"}
 
-    results = [r for r in flyte.map(eval_unit, units) if isinstance(r, dict)] if units else []
+    # A failed map element comes back as a non-dict; keep it on the scorecard as
+    # an errored result instead of silently dropping the scenario.
+    results = []
+    for u, r in zip(units, flyte.map(eval_unit, units) if units else []):
+        if isinstance(r, dict):
+            results.append(r)
+        else:
+            print(f"ERROR {u['scenario_id']} [{u.get('harness') or '-'}]: "
+                  f"eval action failed: {r!r}", flush=True)
+            results.append(errored_scenario(
+                by_id[u["scenario_id"]], u.get("harness"),
+                f"eval action failed: {r!r}").to_dict())
     # Add skipped harnesses as synthetic results (no task spent) for visibility.
     results += [skipped_scenario(by_id[u["scenario_id"]], u["harness"]).to_dict()
                 for u in skipped]

package.json

@@ -1,6 +1,6 @@
 {
   "name": "flyte",
-  "version": "0.0.5",
+  "version": "0.3.0",
   "description": "Agent skills for working with Flyte.",
   "license": "Apache-2.0",
   "keywords": ["pi-package"],

plugins/flyte/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "flyte",
-  "description": "Work with Flyte 2 — deploy Flyte v2 (flyte-binary) clusters on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers.",
+  "description": "Work with Flyte 2. Deploy Flyte on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers.",
   "version": "0.3.0",
   "author": {
     "name": "flyteorg"

plugins/flyte/.codex-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "flyte",
   "version": "0.3.0",
-  "description": "Skills for working with Flyte 2 — deploy Flyte v2 (flyte-binary) clusters on AWS or kind, and author, run, and operate Flyte workflows, apps, agents, data, and ML workloads with the SDK.",
+  "description": "Work with Flyte 2. Deploy Flyte on AWS or kind, author and run workflows, apps, agents, data, and ML workloads with the SDK, and operate your cluster through bundled MCP servers.",
   "skills": "./skills/",
   "mcpServers": "./.mcp.json"
 }

plugins/flyte/README.md

@@ -1,14 +1,17 @@
 # flyte
 
-A single [Claude Code](https://docs.claude.com/en/docs/claude-code) plugin for
-[Flyte](https://flyte.org): cluster deployment and SDK / workflow authoring skills, plus
-two bundled MCP servers.
+A single plugin for [Flyte](https://flyte.org): cluster deployment and SDK / workflow
+authoring skills, plus two bundled MCP servers.
+[Claude Code](https://docs.claude.com/en/docs/claude-code) and
+[OpenAI Codex](https://developers.openai.com/plugins) install it as a full plugin — skills
+**and** MCP servers — and any harness that supports
+[Agent Skills](https://agentskills.io) (Hermes, opencode, pi) can install the skills.
 
 ## Skills
 
 ### Deployment
 
-- **`flyte-deploy-aws`** — deploy a Flyte v2 (`flyte-binary`) cluster on AWS from scratch:
+- **`flyte-deploy-aws`** — deploy a Flyte 2 (`flyte-binary`) cluster on AWS from scratch:
   EKS + S3 + RDS PostgreSQL + AWS Load Balancer Controller + `helm`, with optional TLS
   (ACM, incl. cross-account DNS) and Okta/OIDC SSO.
 - **`deploy-flyte-kind`** — deploy a complete Flyte stack on a [kind](https://kind.sigs.k8s.io/)
@@ -133,7 +136,7 @@ flyte-sdk repo, which stay current as the SDK changes.
 /plugin install flyte@flyte-agent-plugins
 ```
 
-Then ask Claude to, e.g., "deploy a Flyte v2 cluster on AWS", "deploy Flyte on kind", or
+Then ask Claude to, e.g., "deploy a Flyte 2 cluster on AWS", "deploy Flyte on kind", or
 "scaffold a Flyte workflow" — or invoke a skill directly, e.g.
 `/flyte:flyte-deploy-aws`.
 
@@ -148,17 +151,20 @@ and append `#<ref>` — a tag or branch name (not a bare commit SHA; tag the com
 (To change the pinned version later, `/plugin marketplace remove flyte-agent-plugins` and re-add
 with the new ref.)
 
-## Install (other agent harnesses)
+## Install (OpenAI Codex)
 
-The skills are standard [Agent Skills](https://agentskills.io) (`SKILL.md`), so they also
-work with:
-
-**OpenAI Codex CLI** — add the repo as a plugin marketplace, then install via `/plugins`:
+Add the repo as a plugin marketplace, then install via `/plugins` — the skills and both
+MCP servers come with it:
 
 ```
 codex plugin marketplace add flyteorg/flyte-agent-plugins    # or --ref <tag-or-branch> to pin
 ```
 
+## Install (other agent harnesses)
+
+The skills are standard [Agent Skills](https://agentskills.io) (`SKILL.md`), so they also
+work with:
+
 **Hermes** — install individual skills by repo path (default branch only):
 
 ```

plugins/flyte/skills/deploy-flyte-kind/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: deploy-flyte-kind
-description: Deploy a complete Flyte stack (flyte-binary + a hosted PostgreSQL + an object store) onto a kind cluster, running on the user's own machine or a DigitalOcean VM (droplet). PostgreSQL is hosted (Supabase or external); the object store is AWS S3 or Cloudflare R2. Use when the user wants to run Flyte on kind — either reusing an existing kind cluster or creating a new one. For evaluation only (no TLS/auth on the base deployment).
+description: Deploy a complete Flyte stack (flyte-binary + a hosted PostgreSQL + an object store) onto a kind cluster, running on the user's own machine or a DigitalOcean VM (droplet). PostgreSQL is hosted (Supabase or external); the object store is AWS S3 or Cloudflare R2. Use when the user wants to run Flyte on kind — either reusing an existing kind cluster or creating a new one. For evaluation only (no TLS/auth on the base deployment; optional OIDC auth via Traefik + oauth2-proxy is covered).
 ---
 
 # Deploy Flyte to a kind cluster

plugins/flyte/skills/flyte-sdk-types/SKILL.md

@@ -400,4 +400,4 @@ Default threshold is generous. For ML training outputs or large DataFrames, set
 2. **Passing `flyte.io.File` as a string** — always use `flyte.io.File(path=...)` objects. The path string alone won't serialize.
 3. **Using Pandas instead of Polars** — Flyte 2's native DataFrame is Polars. Use `df.to_polars()` to get the underlying DataFrame.
 4. **Not registering custom transformers** — if you register a custom type transformer, it must be registered before task execution.
-5. **Forgetting `.path` on flyte.io.File** — inside a task, `file` is a FlyteFile object, not a string. Use `file.path` for the local path.
+5. **Forgetting `.path` on flyte.io.File** — inside a task, `file` is a `flyte.io.File` object, not a string. Use `file.path` for the local path.