fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034) - #180
Closed
claude[bot] wants to merge 7 commits into
Closed
fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034)#180claude[bot] wants to merge 7 commits into
claude[bot] wants to merge 7 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
claude
Bot
force-pushed
the
fix/remove-eval-prompt-hints
branch
from
August 7, 2026 16:37
f2952bb to
ecfee71
Compare
The eval harness handed every agent a synthetic system prompt. For the three CLI harnesses that prompt described a tool surface they do not have (`bash`, `files_read`) and coached them on how to end a turn — both of which bias exactly what an eval is supposed to measure: out-of-the-box behaviour. The CLI engine now treats the system prompt as optional. When it is empty, nothing is staged and nothing is passed: - `engine` writes `$HOME/.eval/system-prompt.txt` only for a non-empty prompt, and leaves `RunnerExecArgs.systemPromptPath` undefined otherwise. - claude-code omits `--append-system-prompt-file` entirely. - codex and opencode, which have no system-prompt flag, stop prepending a block (and its blank-line separator) to the *user* prompt. `LocalStackSessionArgs` gains a required `agent: AgentHarnessId` so the sandbox layer can tell a CLI harness from the in-process ai-sdk one. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>
…listing (AI-1034) Skills were installed with a flagless `skills add`, which — finding no agent CLI installed yet — falls back to every one of the 71 agents the CLI knows. That scattered ~53 stray roots across the workspace (`.adal`, `.factory`, and non-dotted `data/` and `skills/` among them), and the workspace is exported into run artifacts and scored. It is also order-dependent: had an agent CLI been installed first, the fallback would have quietly stopped producing `.claude/skills` altogether. `skills add` now names the three CLI harnesses explicitly, which installs into exactly the two project scopes they discover natively: - `.claude/skills/` — Claude Code - `.agents/skills/` — Codex and OpenCode This is the actual fix for Codex, which does not read `.claude/skills` at all and therefore saw no skills in any eval. All three are installed unconditionally: the ids collapse to two directories, an unused copy costs a few kilobytes, and no agent id has to be threaded through `createAgentEnvironment` for correctness. Argument order matters — `--agent` is variadic, so the source directory must precede it and `--skill` terminates the list. `--copy` stays: symlink mode skips agents whose top-level directory does not already exist. With each CLI discovering, advertising and loading skills itself, `buildSkillsPrompt` becomes ai-sdk-only, like `buildToolSurfaceAddendum`. The block it rendered told agents to read `.claude/skills/<name>/SKILL.md` with `files_read` — a path Codex cannot see and a tool no CLI harness has. The post-install check now verifies every agent scope, so a skill missing from one of them fails loudly instead of leaving that harness silently skill-less. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>
… (AI-1034) Prompt assembly moves out of `run-eval.ts` (an entry script that runs `main()` on import, so it cannot be unit-tested) into `harness/system-prompt.ts`, keyed on `exp.agent.id`. Every block is now ai-sdk-only — the task framing, the tool-surface addendum, the skills listing — so a CLI harness assembles to `''` and the engine stages no system prompt file. The two "end your turn with a short summary" sentences are gone from both modes: stopping behaviour is part of what is measured. `runOne` now returns the exact assembled `systemPrompt`, so it lands in `results/<experiment>/<eval>.json` and what an agent was told is verifiable from the artifacts. It was previously unrecorded for every CLI harness. `export-results.ts` builds an explicit whitelist, so it does not reach the published web data. `apps/framework` gains a `test` script (`vitest run harness`), wired into `check`, so the prompt-assembly tests have a runner. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>
claude
Bot
force-pushed
the
fix/remove-eval-prompt-hints
branch
from
August 7, 2026 16:46
ecfee71 to
a2873a2
Compare
Brings in a week of upstream work, most notably #192 (eval execution moved from GitHub Actions to Vercel Sandboxes), #178 (claude-code MCP tool-name normalization), #168 (RLS docs guide eval) and several results refreshes. One conflict, in apps/framework/package.json: both sides extended the `check` script. Upstream added `test:vercel-runner` for the new Vercel runner tests; this branch added `test` to give the prompt-assembly tests a runner. Both are kept: check: pnpm typecheck && pnpm test && pnpm test:framework && pnpm test:vercel-runner The dependency block in that file merged cleanly (upstream re-sorted it and added @vercel/sandbox, p-limit, p-retry and zod). run-eval.ts, packages/core/src/agents/engine.ts and packages/core/src/index.ts merged without conflict and both intents are intact: - run-eval.ts now takes readFlag/positiveInteger from lib/cli-args.js (upstream hoisted the local readFlag out) while still assembling its prompt through buildSystemPrompt() and recording systemPrompt in the result JSON. - engine.ts still stages no system-prompt file when the assembled prompt is empty, and additionally passes upstream's mcpServerNames to parseTranscript. - index.ts keeps upstream's parser/transcript type changes alongside this branch's AgentRunArgs.systemPrompt and LocalStackSessionArgs.agent docs. No new CLI harness landed upstream (still claude-code, codex, opencode), so the `skills add --agent` list is unchanged. The new Vercel runner shells out to `pnpm eval`, so it inherits this branch's prompt behaviour with no change. apps/web/src/data/*eval-results.json and results/ are untouched by this branch and the merge takes main's versions wholesale. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>
Second resolution against upstream; the previous one merged main up to d5db72e. Five commits landed since: the SUPABASE_MCP_SERVER_PATH local MCP build support (#193), the AgentDojo-style prompt-injection scenarios (#202), and three eval-result refreshes (#208, #209, #213). Two conflicts, both from #193 threading a new `mounts` option through the sandbox layer alongside this branch's new `agent` option: - packages/sandbox/src/bare-sandbox.ts — union of both. This branch's named `BareSandboxOptions` (required, because `agent` is mandatory) gains upstream's `mounts?: readonly SandboxMount[]`, and the import list takes `AgentHarnessId` and `SandboxMount`. - apps/framework/harness/run-eval.ts — the `createBareSandbox` call passes both `agent: exp.agent.id` and `mounts: supabaseMcpServerMounts()`. No new harness landed, so the no-synthetic-prompt gating is unchanged. apps/web/src/data/ and results/ are taken from main verbatim. Co-Authored-By: Claude <noreply@anthropic.com>
…aller buildSystemPrompt gated its base framing on the agent but passed the addendum and skills blocks straight through. Both are ai-sdk-only today, but that's enforced by their producers across three files rather than by the assembler, and a block reaching a CLI harness fails silently: no error, no red test, just an eval measuring our prompt instead of the agent's own behaviour. An MCP server carrying a promptAddendum is the live path in. Only executorMcpServer has one, and only ai-sdk experiments use it, so nothing changes today — a new CLI-harness experiment paired with it would.
seanoliver
force-pushed
the
fix/remove-eval-prompt-hints
branch
from
August 24, 2026 22:50
0ef478c to
1120e9b
Compare
Contributor
|
Closing in favour of a two-PR stack:
Same change, split along package lines so each half can be reviewed on its own rationale. Every commit here is carried over, plus review fixes and verification runs. Nothing from this branch is lost. One correction worth recording, since this PR's description got it wrong: removing the injected skills listing would not have stripped Codex of skills. The old flagless |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The harness writes its own system prompt into every eval run, so evals measure our framing as much as the agent's out-of-the-box behaviour. Two strings do most of the damage:
You are an agent solving a Supabase eval task...andWhen you are done, end your turn with a short summary.Problem
bashandfiles_readtools that only the ai-sdk harness has.end your turn with a short summaryfeedsagentReport, which the LLM judge reads. Coaching stopping behaviour is a scoring change, not a formatting one..claude/skills. Codex reads.agents/skillsand cannot read.claude/skills, so the injected## Available skillslisting was the only reason skills ever worked there. The flaglessskills addfallback also created dozens of stray agent directories inside the scored workspace.Fix
apps/framework/harness/system-prompt.tsand gates on the harness. ai-sdk keeps a short factual prompt because it has none of its own. The three CLI harnesses get'', and the engine then stages no system-prompt file rather than an empty one.end your turn with a short summaryis gone for every harness, including ai-sdk.skills add <dir> --agent claude-code codex opencode, writing.claude/skillsand.agents/skillsrather than a directory per known agent. The injected listing is gone for CLI harnesses. The installer also drops askills-lock.jsonin the workspace root, which it did before this change too.buildSystemPromptdrops caller-supplied blocks for a CLI harness rather than trusting call sites to gate them. OnlyexecutorMcpServercarries apromptAddendumand only ai-sdk experiments use it, so nothing changes today. A future CLI experiment paired with it would have silently reintroduced injected text.systemPrompt, so this class of regression shows up inresults/*.jsonrather than only in code.export-results.tsdrops the field, so it does not reach the web export.Testing
0ef478c: framework 7/7, core 122/122, sandbox unit 49/49, sandbox docker 4/4, both typechecks exit 0,pnpm format:checkclean.claude-code-sonnet-5againstbuild-cli-001-bootstrap-app.systemPromptin the run artifact is"", and the eval passed.apps/frameworksmoke script fails at the first judge-backed scenario for lack of root.envcredentials. Byte-identical failure onmain.Comparability
No recorded numbers change here. The raw results gain a
systemPromptfield and the schema is loose, so old files still parse.gpt-5.6-solat medium effort. Baselines onmainwere refreshed after it, so both sides of this comparison are post-fix(core): default the judge to gpt-5.6-sol at medium reasoning effort #222.Open questions
Should #202's baselines be annotated at merge, or left to the nightly regression refresh to re-baseline?
AI-1034