[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-07-13

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-06-23 → 2026-07-13)
Total PRs Analyzed: 1,000 Copilot coding-agent PRs (app/copilot-swe-agent)
Clusters Identified: 7 (K-means over TF-IDF, k chosen by silhouette)
Overall Merge Success Rate: 78% (778 merged · 212 closed · 10 open)

Seven coherent task families emerge. Two extremes stand out: linter-rule development merges at 93% with tiny diffs, while WIP/placeholder PRs — where the body is a progress stub rather than a real task description — merge at only 33%. Prompt specificity, not task size, is the dominant driver of outcome.

Success rate & effort by cluster

Cluster	Theme	Size	Success	Avg commits	Avg reviews	Avg files	Top terms
C6	Engine & compiler behavior	349 (35%)	78%	3.7	2.7	20	path, step, behavior, existing
C3	Agentic workflow authoring	176 (18%)	78%	2.2	1.0	22	workflow, agent, daily, prompt
C0	Docs, schema & config	168 (17%)	71%	3.9	2.6	17	docs, schema, field, reference
C1	PR Sous Chef feature work	158 (16%)	87%	6.6	4.4	20	sous chef, id, safe-output
C4	Linter rule development	71 (7%)	93%	3.1	3.1	6	rule, eslint, function, flagged
C2	Version bumps & smoke fixes	54 (5%)	70%	3.6	3.1	132	bump, version, release, smoke
C5	Incomplete / WIP PRs	24 (2%)	33%	1.5	0.4	3	date, progress, started, wip, plan

Cluster detail & representative PRs

C6 · Engine & compiler behavior — 349 PRs (35%), 78% success

The largest, broadest family: hardening and behavior-preserving changes to the compiler, engine startup, and job-dependency wiring. Prompts emphasize "preserve existing behavior" and reference concrete code paths/steps. Moderate iteration (3.7 commits, 2.7 reviews).

• Harden Claude startup failure handling for empty-log Avenger runs #44332 Harden Claude startup failure handling for empty-log Avenger runs
• fix: detection job missing engine.env job dependencies in needs list #44202 fix: detection job missing engine.env job dependencies in needs list
• fix(windows): remove compat import to prevent ConPTY startup crash #41235 fix(windows): remove compat import to prevent ConPTY startup crash

C3 · Agentic workflow authoring — 176 PRs (18%), 78% success

Authoring and tuning .github/workflows agentic definitions and MCP prompts. Lowest review load (1.0) — these are largely self-contained prompt/config edits.

• fix(pr-sous-chef): relax checks-pending guard — ignore checks running > 1h #43888 fix(pr-sous-chef): relax checks-pending guard
• Stop recommending get_me for workflow identity in GitHub MCP prompts #44936 Stop recommending get_me for workflow identity in GitHub MCP prompts
• fix(daily-code-metrics): raise max-file-size to 256KB and enforce 90-day history.jsonl pruning #43870 fix(daily-code-metrics): raise max-file-size to 256KB, enforce 90-day history

C0 · Docs, schema & config — 168 PRs (17%), 71% success

Documentation, JSON schema, and config-reference work. Lowest success outside the WIP cluster — schema/reference tasks are prone to review churn over naming and completeness.

• compiler: inject missing model pricing from models.dev at compile time #43003 compiler: inject missing model pricing from models.dev at compile time
• Add issue_intents runtime support for issue labels, type, and fields #41092 Add issue_intents runtime support for issue labels, type, and fields
• Improve assign-to-agent diagnostics for Copilot assignee alias lookup #41047 Improve assign-to-agent diagnostics for Copilot assignee alias lookup

C1 · PR Sous Chef feature work — 158 PRs (16%), 87% success

Feature work around the PR Sous Chef workflow and safe-outputs. Highest iteration in the set (6.6 commits, 4.4 review threads) yet strong 87% success — heavy review converges rather than kills these.

• Add disclosure-header message for AI authorship disclosure in safe-outputs #44497 Add disclosure-header message for AI authorship disclosure in safe-outputs
• feat: content redaction for public-facing safe-outputs #44501 feat: content redaction for public-facing safe-outputs (closed)
• Add declarative custom-engine frontmatter and migrate Crush/OpenCode to shared behavior definitions #44465 Add declarative custom-engine frontmatter; migrate Crush/OpenCode (open)

C4 · Linter rule development — 71 PRs (7%), 93% success ⭐

Custom Go/ESLint analyzer rules. Smallest diffs (6 files) and highest success. Tightly-scoped "detect pattern X, flag false-positive Y" prompts are the agent's strongest suit.

• httpnoctx: detect context-free NewRequest + Do paths #42536 httpnoctx: detect context-free NewRequest + Do paths
• fix(tolowerequalfold): reject case-mismatched literals and mixed ToLower/ToUpper comparisons #44888 fix(tolowerequalfold): reject case-mismatched literals and mixed ToLower/ToUpper
• Targeted custom-lint cleanup: defer-in-loop resource handling and continuation options struct #43414 Targeted custom-lint cleanup: defer-in-loop resource handling

C2 · Version bumps & smoke fixes — 54 PRs (5%), 70% success

Dependency/version bumps and smoke-test repair. Huge diffs (avg 132 files) from vendored/pinned artifact refreshes; 70% success reflects occasional schema-drift breakage.

• feat: add sandbox.agent.runtime: gvisor for gVisor container runtime support #44796 feat: add sandbox.agent.runtime: gvisor for gVisor container runtime
• Fix external detection job setup for codex and pi workflows #40954 Fix external detection job setup for codex and pi workflows
• Bump gh-aw-firewall to v0.27.31 and refresh AWF pin/schema artifacts #45176 Bump gh-aw-firewall to v0.27.31 and refresh AWF pin/schema artifacts (open)

C5 · Incomplete / WIP PRs — 24 PRs (2%), 33% success ⚠️

Not a task type but a failure mode: PR bodies are progress placeholders ("I will get started... keeping this description up to date", [WIP]) with no crisp task statement. Lowest commits (1.5), near-zero reviews (0.4), and a 33% merge rate — less than half the fleet average.

• [WIP] Fix failing GitHub Actions job: Integration Workflow Misc Part 2 #43733 [WIP] Fix failing GitHub Actions job: Integration Workflow Misc Part 2
• [WIP] Fix failing GitHub Actions job for parser location and validation #43306 [WIP] Fix failing GitHub Actions job for parser location and validation

Key findings

1. Scope tightness beats task size. C4 (linter rules) has the smallest diffs and the highest success (93%); C1 (PR Sous Chef) is the most-iterated and second-highest (87%). What separates winners from losers is a concrete, verifiable target — not how much code changes.
2. Placeholder prompts are a distinct failure mode. C5's stub bodies merge at 33% vs 78% overall. A missing task statement in the PR body correlates with a >2× drop in merge probability.
3. Docs/schema work is the soft spot among "real" tasks (71%). Reference/schema PRs churn on naming and completeness — the ambiguity is in "what counts as done," not in the change itself.
4. Heavy review is not a red flag. C1 averages 4.4 review threads yet succeeds 87% of the time; review iteration converges these rather than stalling them.
5. Bulk-artifact bumps carry drift risk. C2's 132-files-avg diffs merge at 70%, dragged down by schema/version mismatches surfacing at smoke-test time.

Recommendations

1. Require a task statement in every PR body. Gate or nudge Copilot PRs whose body is a WIP/progress stub — pushing C5's crisp-prompt rate up is the single highest-leverage fix (33% → fleet 78%).
2. Lean into tightly-scoped analyzer/lint tasks. These are the agent's best-performing category; when decomposing larger work, prefer "detect/flag pattern X" framings that give a verifiable target.
3. Add explicit done-criteria to docs/schema prompts. Enumerate the exact fields/sections expected so C0 reviews stop churning on completeness.
4. Split large version bumps from behavioral changes. Isolate vendored-artifact refreshes (C2) so smoke-test drift is diagnosed independently of logic changes.

Methodology & limitations

• Corpus: 1,000 app/copilot-swe-agent PRs, created 2026-06-23 → 2026-07-13, from github/gh-aw. Full data (comments, reviews, commits, files) available for all 1,000.
• Text: PR title + body; quoted issue blocks (>), fenced/inline code, URLs, and issue refs stripped before vectorization.
• Model: TF-IDF (1–2 grams, min_df=3, max_df=0.6, sublinear TF, 800 features) → K-means. k=7 selected by silhouette over k∈[4,8] (score 0.043 — low absolute value is expected for sparse short-text, but clusters are semantically coherent and stable).
• Effort proxies: commits/reviews/files/additions per PR stand in for iteration cost.
• Limitation — turn counts unavailable. Copilot coding-agent PRs are produced by GitHub's Copilot agent, not by gh-aw workflow runs; the aw_info.json logs present in this run cover gh-aw's own internal agentic workflows, so per-PR turn/token metrics could not be joined. Commit count is used as the iteration proxy instead.

References: §29243602880

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 152.4 AIC · ⌖ 19.1 AIC · ⊞ 13.2K · ◷

• expires on Jul 14, 2026, 3:06 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #45446.