docs: document Factory Benchmarks (GROW-6129)

[rachaelrenk]

What this feature does

Benchmarks let Warp Factories teams compare model and runner configurations on representative tasks before changing production work. This update documents creating suites, inspecting results, and storing reusable suites in code. Shipped in v0.2026.09.02.08.27.stable_01 (2026-09-02).

Summary

Adds the public documentation for the Warp Factories Benchmarks launch in GROW-6129. The page follows the factory Early Access audience, covers the dashboard workflow, and links it to the Factory definition reference.

Changes

• Adds src/content/docs/factories/benchmarks.mdx with the feature model, suite and task workflow, launch guidance, result interpretation, the current third-party harness limitation, and related pages.
• Documents benchmarks/<suite-slug>/suite.yaml and task files in src/content/docs/factories/factory-as-code.mdx.
• Updates the factory overview, dashboard, measurement page, and sidebar so Benchmarks is discoverable without duplicating the workflow.

Content design plan

Audience and JTBD: A member of a team with Warp Factories Early Access who administers a factory and needs evidence from representative work before changing an agent's model or runner.

Problem: The factory dashboard exposes Benchmarks, but the docs did not explain how to build a suite, compare configurations, inspect results, or store a reusable suite in a factory definition.

Goals:

• Create and run a benchmark suite with representative tasks and explicit correctness criteria.
• Compare baseline and candidate model or runner configurations using quality, cost, and time results.
• Apply a reviewed configuration change and preserve a production baseline.

Purpose and value: A dedicated page documents a multi-step, cost-bearing workflow that the existing measurement overview could not cover without obscuring its Scorer and Self-improvement guidance.

Content type: Feature documentation (combined) — readers need both a concise model and the procedure for creating, launching, and reviewing a benchmark.

Skill and template: draft_feature_doc / .agents/templates/feature-doc.md

High-impact scenarios:

• Covers: dashboard-managed suites from completed runs or authored tasks; model and runner comparisons; Scorers, repetitions, results, and code-managed suite files.
• Excludes: universal model claims, WarpBench internal results, cross-harness comparisons, scheduled benchmark runs, and warehouse or API analytics. Third-party harness comparison is marked as coming soon in the released UI.

Unverified claims

None — all UI labels, workflow behavior, result states, cost semantics, and definition-file fields were verified against the released Factory source and public launch material.

Documentation risk

Risk: engineering-review-required
Rationale: This PR adds a new feature page and changes published factory configuration reference content.
Source files consulted: warp-server/client/packages/factory/src/pages/FactoryBenchmarks/SuiteEditor.tsx, TaskSheet.tsx, LaunchRunDialog.tsx, and RunDetail.tsx at 17ccf4e151c87d9bca07cd46e9dccab8510f1838; warp-server/logic/factoryfile/schema/v1alpha1/benchmark_suite.schema.json and benchmark_suite_task.schema.json at the same revision; warp-server/client/packages/factory/src/pages/FactoryBenchmarks/RunDetail.tsx, client/packages/factory/src/api/benchmarks.ts, and router/handlers/public_api/benchmarks.go at eb1e91339bcb26f0ceeaa0e27800348692f8d417 for Total cost coverage.
Requested engineering reviewers: none — source-owner resolution returned no match, and no reviewer is requested during the author’s interactive review.
Engineering review status: pending
Docs override: none

Validation

• python3 .agents/skills/check_for_broken_links/check_links.py --internal-only — passed; 4,064 internal links and anchors checked.
• npm run build — passed.
• python3 .agents/skills/style_lint/style_lint.py --changed — no blocking findings; remaining output is unrecognized-term suggestions on pre-existing UI labels and the existing Early Access note.
• npm run lint — not run successfully because Trunk is not installed in this environment.

Related: https://linear.app/warpdotdev/issue/GROW-6129/benchmark-launch-docs

Co-Authored-By: Warp agent@warp.dev

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 9, 2026 4:28pm UTC

[warp-for-oss]

@rachaelrenk

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[warp-for-oss]

Overview

This PR adds a Factory Benchmarks page, links it from the factory overview/dashboard/measurement pages, and documents benchmark suite files in the factory definition reference. I found one documentation-convention issue in the new page: the web-app product surface and URL are hardcoded instead of using the repository's centralized variable system.

Concerns

• The new benchmarks page hardcodes Warp Factories web app and https://platform.warp.dev instead of the existing VARS.FACTORY_WEB_APP and VARS.FACTORY_WEB_APP_URL variables used by neighboring factory docs.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• hardcoded-var (1)
• missing-source-citations (1)
• ui-copy-clarity (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "1cfe2f8cb5a59043f13b4452bac540ffca033fb4", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 1, "top_categories": ["hardcoded-var (1)", "missing-source-citations (1)", "ui-copy-clarity (1)"], "verdict": "Request changes"}

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• missing-source-citations (1)
• description-length (1)
• ui-copy-clarity (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "6bfea0667fcc91607320ed5129d94adcd2483785", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["missing-source-citations (1)", "description-length (1)", "ui-copy-clarity (1)"], "verdict": "Request changes"}

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• source-verification-gap (1)
• frontmatter-description-length (1)
• screenshot-width-precedent (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "faacecc7136bfd5b5b08bd8a0e59cbdae7bcf71e", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["source-verification-gap (1)", "frontmatter-description-length (1)", "screenshot-width-precedent (1)"], "verdict": "Request changes"}

[rachaelrenk]

[style-rule-gap]
Static UI labels such as “Tasks” should use quotation marks, while interactive controls such as Add task should use bold. The style guidance covers field labels, but an explicit section-label example would make this distinction easier to apply.

[rachaelrenk]

[skill-feedback]
For procedural UI docs, verify state transitions in the live product as well as control labels. The first Benchmarks draft missed that Add task saves a new benchmark before task setup opens, and it briefly documented the alternate empty-state button instead of the persistent New button.

[rachaelrenk]

General feedback (already implemented, saved for log and future improvements)
Tighten the Benchmarks page’s opening and procedure:
• Rewrite the frontmatter description in clearer, plain language; two sentences are fine.
• Move the Early Access note before the opening paragraph.
• Define both a benchmark suite and a trial. Replace “holds” with more direct language.
• Start the procedure with “To use Benchmarks, you need a factory with an agent to evaluate,” and use active voice for the completed-run guidance.
• Document the persistent New button as the primary creation path. The empty state also has New benchmark, but that is not the primary instruction.
• Simplify task creation. Don’t format the static “Tasks” section label as an interactive UI element, and don’t enumerate self-explanatory task fields.
• Explain that clicking Add task saves a new benchmark before task setup opens.
• Shorten the baseline, configuration, repetition, cost, and run-status guidance without dropping the important behavior.
• Add the cropped benchmark editor screenshot after the name and agent-selection step, with descriptive alt text and a short caption.

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• missing-source-citations (1)
• screenshot-width-precedent (1)
• frontmatter-description-length (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "725298d028abbd40f634cfa2b930367467b3fe83", "important": 2, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["missing-source-citations (1)", "screenshot-width-precedent (1)", "frontmatter-description-length (1)"], "verdict": "Request changes"}

[rachaelrenk]

/warp-agent-review

[warp-for-oss]

@rachaelrenk

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• inline-context (1)
• screenshot-coverage (1)

Verdict

Approve

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• technical-accuracy (1)

Verdict

Approve

Superseded by the current Agent docs review check.

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• procedural-step-clarity (1)

Verdict

Approve

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• src/content/docs/factories/benchmarks.mdx:93 (and :83) — [CRITICAL] benchmark-run-results-detail.png/benchmark-run-results-overview.png show a real employee's name ("Ran by: Suraj Gupta") and an internal codename task title ("Stop Routing Fable 5 Through Claude-On-Vertex") unredacted. Requested resolution: recapture with synthetic/sanitized data, consistent with the avatar-blurring precedent in factory-dashboard-activity.png.
• src/content/docs/factories/benchmarks.mdx:53 — [CRITICAL] benchmark-suite-editor.png's alt text/caption describe "a documentation-link review suite" and "a selected triage agent" but the image actually shows the real internal "Mini WarpBench 0.1" suite with agent "implementation" and real internal task titles. Requested resolution: recapture the screenshot to match the described sanitized example, or rewrite the alt text/caption to match sanitized content.
• src/content/docs/factories/benchmarks.mdx:67 (and :61) — [CRITICAL] benchmark-task-editor.png/benchmark-task-source-run.png expose the real warpdotdev/warp-server repo, real internal PR links, real work-item titles, and the internal factory implementation-agent system prompt verbatim. Requested resolution: recapture both with a fictitious repository, task, and prompt.
• src/content/docs/factories/benchmarks.mdx:19 — [IMPORTANT] factory-benchmarks-list.png shows real internal "WarpBench" suite names under a real team name ("wilson"), contradicting the PR's content-design plan, which excludes "WarpBench internal results." Requested resolution: recapture with a generic, non-WarpBench suite list.

Verdict

Request changes

[hongyi-chen]

This is a lot better, thank you!

Approving to unblock, but one change we should make is to move one of these videos (the more general, higher-level one) towards the top of the page. I think the video is a great resource we should highlight above the fold, and having two videos back to back isn't ideal, but otherwise the screenshots and other changes are great.

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• unrecognized-term glossary suggestions (35)
• technical-claim-consistency (1)

Verdict

Approve

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• missing-first-reference-context (1)

Verdict

Approve

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• No blocking findings.

Verdict

Approve