Plumb effort through prompt config; bump reviewer to Opus 5 - #27
Merged
Conversation
- Prompt supports optional model.effort (low/medium/high/xhigh/max),
emitted as output_config: {effort: ...}; validated at config load.
- prompt_testing runner now calls build_api_payload directly instead of
mirroring its thinking/temperature logic - one source of truth, and
experiments exercise exactly what production sends.
- Reviewer default claude-opus-4-7 -> claude-opus-5 (same price, better
review). Thinking is now always explicit on the reviewer request: on
Opus 5 an omitted thinking field runs adaptive by default, which would
silently defeat --reviewer-thinking off. Thinking-enabled reviews get
max_tokens 4096 since thinking counts against the budget.
- Effort sweep results (21 cases, thinking disabled): latency and cost
flat across low/medium/high, accuracy best at high (18/21 vs 16/21).
Production therefore stays on the API default (high); the plumbing
matters for the useThinking path and future experiments. Variant YAMLs
kept for reproducibility; findings recorded in CLAUDE.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jJRqmqhE11VAc3biKbxUY
mattgodbolt
approved these changes
Jul 29, 2026
mattgodbolt-molty
added a commit
that referenced
this pull request
Jul 30, 2026
…tent (#29) Final check-up documentation pass. **Stacked on #28** (both touch CLAUDE.md); GitHub will retarget this to main when #28 merges — merge #28 first. - **Records the prompt-caching evaluation and rejection** with the data: ~104 fresh Claude calls/day against a 5-minute cache TTL and a prefix fragmented by language/arch/audience/type; generous math says ~$0.40/fortnight saved of ~$22 spend, and the restructuring needed to clear Sonnet 5's 1024-token minimum cacheable prefix isn't paid for. The note includes the revisit threshold (~50x traffic, or sustained >3 same-combo req/hour for the 1-hour TTL) and the CloudWatch query to rerun the analysis. - **Condenses the whole document** (137 -> 103 lines, nothing load-bearing lost): merges Project Structure and the workflow notes into Overview/Development Commands, adds the prompt-test workflow as a first-class step before prompt changes, drops historical narration (the 1536-token era), dedupes the thinking gotchas, and documents the `build_api_payload` single-source-of-truth rule established in #27. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012jJRqmqhE11VAc3biKbxUY --------- Co-authored-by: mattgodbolt-molty <mattgodbolt-molty@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check-up step two: the
effortexperiment plus the reviewer-model bump.Effort plumbing:
model.effortin a prompt YAML now flows through asoutput_config: {effort: ...}, validated at load. The prompt-test runner also now callsbuild_api_payloaddirectly rather than duplicating the thinking/temperature resolution, so experiments run exactly what production would send.Sweep verdict — production unchanged. Across all 21 cases with thinking disabled, effort made no measurable difference to latency (7.2 s p50 at every level) or cost (within 1.5%), and accuracy was best at the default
high(18/21 vs 16/21 at medium/low). Effort mostly modulates thinking depth, and with thinking off there is nothing to modulate. Soapp/prompt.yamldeliberately does not set it; the plumbing is for theuseThinkingpath and future experiments. Findings recorded in CLAUDE.md.Reviewer: default bumped from Opus 4.7 to Opus 5 (same $5/$25, better reviews). One behavioural fix that the bump forced: on Opus 5 an omitted
thinkingfield runs adaptive by default, which would have silently turned--reviewer-thinking offinto thinking-on. The reviewer now always sends an explicit thinking config, and thinking-enabled reviews getmax_tokens: 4096since thinking counts against the budget. Smoke-tested in both modes.🤖 Generated with Claude Code
https://claude.ai/code/session_012jJRqmqhE11VAc3biKbxUY