[workflow-analysis] Weekly Workflow Analysis — 148 recent runs: 85% success, Smoke CI 100% failure, 0% actuation

[github-actions[bot]]

Overview

Analysis of the 148 most recent workflow runs in github/gh-aw. Because this repo runs agentic workflows at very high cadence (~148 runs in a 6.5h window ⇒ 500+ runs/day, several thousand per week), a literal full-week pull would span thousands of runs; this report analyzes the most recent representative slice (2026-07-13 03:41Z → 10:13Z). Failure and cost patterns were consistent across every 50-run page sampled, so the findings below generalize.

Metric	Value
Runs analyzed	148
Success rate	85.1% (126 ✓ / 22 ✗)
Total cost	~8,731 AIC
Compute	23.9h wall / 1,510 action-minutes
GitHub API calls	2,026
Wasted on failed runs	~1,383 AIC (16% of spend) + 185 min
Write-capable safe outputs emitted	0 / 148 (100% read-only)

Key issues (ranked)

1. 🔴 Smoke CI — 100% failure (5/5), all failing at the Execute GitHub Copilot CLI step (engine copilot, model claude-sonnet-4.6). A smoke canary failing every run means the core signal it exists to provide is dark.
2. 🔴 PR Sous Chef — 50% failure (3/6), all failing at the pre-agent shell step Fetch open non-draft PR queue. AIC=0 on these runs → the agent never starts; this is a setup/data-fetch bug (likely a gh pr list in the custom step), not a model problem.
3. 🟠 Smoke/engine-validation tests dominate failures — Smoke CI (5), Smoke Copilot Small (2/3), Smoke Copilot (1/3), Smoke OpenCode (1/1), Smoke Crush (1/1) = ~45% of all 22 failures. Copilot-engine smoke tests are the common thread.
4. 🟠 0% actuation — not one run emitted a write-capable safe output. For a fleet full of triage / fixer / reviewer / generator workflows, either everything is running staged/dry-run, or actuation is broken. Worth confirming it's intentional.
5. 🟡 Cost concentration & waste — top 5 runs = 31% of spend; the single costliest run (daily-experiment-report, 936 AIC / 29min) also failed, i.e. pure waste.

View Details — Failures

All 22 failures by workflow (fails/total):

Workflow	Fails/Total	Engine	Failing step
Smoke CI	5/5	copilot	Execute GitHub Copilot CLI
PR Sous Chef	3/6	pi	Fetch open non-draft PR queue (pre-agent)
Smoke Copilot Small	2/3	copilot	agent exec
Smoke Copilot	1/3	copilot	agent exec
Organization Health Report	1/1	—	early (no engine)
daily-experiment-report	1/1	copilot	after 936 AIC / 29min
Layout Specification Maintainer	1/1	copilot	after 23min
PR Triage Agent	1/1	copilot	after 181 AIC
Changeset Generator	1/2	codex	—
Smoke OpenCode / Smoke Crush	1/1 each	—	early (no engine)
PR Code Quality Reviewer	1/8	copilot	—
GitHub Remote MCP Auth Test	1/1	copilot	—
Documentation Noob Tester	1/1	copilot	—
Code Simplifier	1/1	copilot	after 19min

Reliability by engine:

Engine	Runs	Fails	AIC
claude	30	0	4,584
copilot	86	15	3,973
pi	16	3	123
codex	9	1	36
gemini	2	0	15
(none/setup-fail)	3	3	0

Copilot-engine runs carry the failure load (15 of 22); Claude-engine runs were 100% reliable but ~2× costlier per run.

Note: raw agent stderr could not be retrieved — api.github.com is firewall-blocked from the analysis sandbox and the agent stderr is not retained in the usage artifact. Step-level attribution above comes from job/step conclusion data in the run summaries.

View Details — Performance & Cost

Slowest runs:

Workflow	Duration	AIC	Result
Copilot Session Insights	38.3m	270	✓
Daily yamllint Fixer	35.3m	0	✓ (long run, 0 AIC — investigate idle/hang)
daily-experiment-report	29.2m	936	✗
Layout Specification Maintainer	23.3m	65	✗
Daily AgentRx Trace Optimizer	20.2m	394	✓
ESLint Refiner	19.6m	412	✓

Costliest runs (AIC): daily-experiment-report 936 (✗), Safe Output Health Monitor 490, Go Logger Enhancement 476, ESLint Refiner 412, Daily AgentRx Trace Optimizer 394, Sergo 369. Top-5 = 31% of total spend.

API-heaviest: AI Moderator (89, 85 calls — codex), Smoke CI (33/30/26), Issue Monster (28). AI Moderator's API volume (~85+ calls/run) stands out vs a fleet median near ~12.

Recommended next actions

1. Fix Smoke CI (P0). 100% failure at Execute GitHub Copilot CLI points to a Copilot-CLI/engine integration regression (or expired auth/quota for the copilot smoke path). Until fixed, the smoke signal is unreliable.
2. Fix PR Sous Chef's Fetch open non-draft PR queue step (P1). It fails before the agent runs (AIC=0). Add error handling / retry to that shell step and verify the gh query it issues.
3. Confirm the 0% actuation posture. If workflows are intentionally staged, document it; if not, actuation is silently broken across the fleet — high impact given how many fixer/triage workflows run.
4. Add cost guardrails on long/expensive runs. daily-experiment-report burned 936 AIC then failed. Add early-exit + a per-run AIC ceiling so failing runs abort before consuming the most expensive tail.
5. Investigate 0-AIC long runs (e.g. Daily yamllint Fixer, 35min/0 AIC) for hangs or wasted wall-clock.
6. Rebalance engine choice for cost-sensitive high-volume workflows — Claude runs were flawless but ~2× the per-run AIC of copilot; reserve them for tasks that need the reliability.

References:

• §29236751428 — Smoke CI failure (Execute GitHub Copilot CLI)
• §29241104755 — PR Sous Chef failure (Fetch open non-draft PR queue)
• §29236848215 — daily-experiment-report (936 AIC, failed)

Generated by 🔍 Weekly Workflow Analysis · 212.2 AIC · ⌖ 11.2 AIC · ⊞ 3.9K · ◷

• expires on Jul 14, 2026, 2:38 AM UTC-08:00