Skip to content

ci-cleaner: switch to Claude with max-turns, scope recompile, add exit guardrails#24623

Merged
pelikhan merged 5 commits intomainfrom
copilot/copilot-token-optimizer-deep-audit
Apr 5, 2026
Merged

ci-cleaner: switch to Claude with max-turns, scope recompile, add exit guardrails#24623
pelikhan merged 5 commits intomainfrom
copilot/copilot-token-optimizer-deep-audit

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 5, 2026

The hourly-ci-cleaner workflow had a ~43% failure rate, dominated by "no safe outputs" (44% of failures) and oversized PRs from unconditional make recompile runs (166 files in one case).

Engine

  • Switched from copilotclaude with max-turns: 20, providing the hard turn limit Copilot doesn't support

CI Status Check

  • Re-verify CI status at agent start via gh run list --limit=2 before doing any work — handles the case where CI self-healed between the check_ci_status job and agent execution
  • Check activation context first; only hit the API if status was failure

Recompile Guard (was causing E003 "PR too large")

Only run make recompile when .md workflow files actually changed:

git diff --name-only | grep '\.md$'
# empty → skip make recompile entirely

After recompile, abort with noop if >50 files changed (signals binary/template mismatch).

File-Count Guard

Before creating a PR, check total staged files — call noop instead if >80:

git add -A && git diff --cached --name-only | wc -l
# > 80 → noop with explanation

Exit Protocol

Added an ABSOLUTE FINAL RULE block (in both the workflow prompt and the agent file) with unconditional language: if the response is about to end without a safe-output tool call, call noop immediately.

Deps Warning

Moved the make deps-dev / make agent-finish prohibition to be immediately visible under "Your Task" — both commands take 10–15 min and deps are already installed by setup steps.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw cXU4ZOQ/cBxinx7a-1 (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw ase,codename,all-Dcodeql.chainer.v2=/tmp/codeql.8AmckaiT (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha get --global docker-buildx http.https://gitgit (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha --get-regexp --global rgo/bin/git (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .visibility (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .visibility --local committer.email r: $owner, name: $name) { hasDiscussionsEnabled } } (http block)
  • https://api.github.com/repos/githubnext/agentics/git/ref/tags/
    • Triggering command: /usr/bin/gh gh api /repos/githubnext/agentics/git/ref/tags/# --jq .object.sha (http block)
    • Triggering command: /usr/bin/gh gh api /repos/githubnext/agentics/git/ref/tags/# --jq .object.sha get --local $name) { hasDiscussionsEnabled } } user.email (http block)

If you need me to access, download, or install something from one of these locations, you can either:

…count check, stronger exit protocol

Addresses recommendations from the deep audit issue:
- Rec 1: Add ABSOLUTE FINAL RULE block to both prompt and agent file
- Rec 2: Scope make recompile to only run when .md files changed; abort if >50 files
- Rec 3: Switch engine from copilot to claude with max-turns: 20
- Rec 4: Add file-count guard (>80 staged files → noop instead of oversized PR)
- Rec 5: Secondary CI re-check at start of agent (handles self-healed CI)
- Rec 6: Explicit warning against make deps-dev / make agent-finish
- Regenerate hourly-ci-cleaner.lock.yml

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/caa147dd-be37-4692-b883-55a17607416b

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@github-actions github-actions bot mentioned this pull request Apr 5, 2026
Copilot AI changed the title [WIP] Fix recurring failures in CI Cleaner workflow ci-cleaner: switch to Claude with max-turns, scope recompile, add exit guardrails Apr 5, 2026
Copilot AI requested a review from lpcox April 5, 2026 01:04
Copilot finished work on behalf of lpcox April 5, 2026 01:04
@pelikhan pelikhan marked this pull request as ready for review April 5, 2026 01:40
Copilot AI review requested due to automatic review settings April 5, 2026 01:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the hourly-ci-cleaner automation to use the Claude engine with a hard turn limit, reduce oversized PRs from unnecessary workflow recompiles, and add explicit exit guardrails to avoid “no safe outputs” failures.

Changes:

  • Switch workflow engine from Copilot to Claude with max-turns: 20
  • Add CI self-heal verification, conditional make recompile, and staged-file count guardrails
  • Strengthen “must call safe-output tool” exit protocol in both workflow prompt and agent doc
Show a summary per file
File Description
.github/workflows/hourly-ci-cleaner.md Updates the workflow prompt to Claude/max-turns and adds CI verification + recompile/PR size guardrails.
.github/workflows/hourly-ci-cleaner.lock.yml Regenerates the locked workflow implementation to run Claude Code CLI and update engine/secrets/config.
.github/agents/ci-cleaner.agent.md Aligns agent instructions with conditional recompile, file-count guardrails, and stricter exit protocol.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (2)

.github/workflows/hourly-ci-cleaner.lock.yml:737

  • PROMPT_TEXT is a shell variable but is not exported before invoking awf/bash -c. As a result, $PROMPT_TEXT inside the container will be empty, so Claude will run with an empty prompt. Export the variable (or write the prompt to a file and reference it) before the awf invocation.
                    # Combine agent content with prompt
                    PROMPT_TEXT="$(printf '%s\n\n%s' "$AGENT_CONTENT" "$(cat /tmp/gh-aw/aw-prompts/prompt.txt)")"
          touch /tmp/gh-aw/agent-step-summary.md
          # shellcheck disable=SC1003
          sudo -E awf --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" --tty --env-all --exclude-env ANTHROPIC_API_KEY --exclude-env GITHUB_MCP_SERVER_TOKEN --exclude-env MCP_GATEWAY_API_KEY --mount /opt/hostedtoolcache/go:/opt/hostedtoolcache/go:ro --mount /usr/bin/go:/usr/bin/go:ro --mount /usr/bin/make:/usr/bin/make:ro --mount /usr/local/bin/node:/usr/local/bin/node:ro --mount /usr/local/bin/npm:/usr/local/bin/npm:ro --mount /usr/local/lib/node_modules:/usr/local/lib/node_modules:ro --allow-domains '*.githubusercontent.com,anthropic.com,api.anthropic.com,api.github.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,cdn.playwright.dev,codeload.github.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,files.pythonhosted.org,ghcr.io,github-cloud.githubusercontent.com,github-cloud.s3.amazonaws.com,github.com,go.dev,golang.org,goproxy.io,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,lfs.github.com,objects.githubusercontent.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,pkg.go.dev,playwright.download.prss.microsoft.com,ppa.launchpad.net,proxy.golang.org,pypi.org,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,sentry.io,statsig.anthropic.com,storage.googleapis.com,sum.golang.org,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com' --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --audit-dir /tmp/gh-aw/sandbox/firewall/audit --enable-host-access --image-tag 0.25.13 --skip-pull --enable-api-proxy \
            -- /bin/bash -c 'export PATH="$(find /opt/hostedtoolcache -maxdepth 4 -type d -name bin 2>/dev/null | tr '\''\n'\'' '\'':'\'')$PATH"; [ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH" || true && claude --print --disable-slash-commands --no-chrome --max-turns 20 --mcp-config /tmp/gh-aw/mcp-config/mcp-servers.json --allowed-tools Bash,BashOutput,Edit,ExitPlanMode,Glob,Grep,KillBash,LS,MultiEdit,NotebookEdit,NotebookRead,Read,Task,TodoWrite,Write,mcp__github__download_workflow_run_artifact,mcp__github__get_code_scanning_alert,mcp__github__get_commit,mcp__github__get_dependabot_alert,mcp__github__get_discussion,mcp__github__get_discussion_comments,mcp__github__get_file_contents,mcp__github__get_job_logs,mcp__github__get_label,mcp__github__get_latest_release,mcp__github__get_me,mcp__github__get_notification_details,mcp__github__get_pull_request,mcp__github__get_pull_request_comments,mcp__github__get_pull_request_diff,mcp__github__get_pull_request_files,mcp__github__get_pull_request_review_comments,mcp__github__get_pull_request_reviews,mcp__github__get_pull_request_status,mcp__github__get_release_by_tag,mcp__github__get_secret_scanning_alert,mcp__github__get_tag,mcp__github__get_workflow_run,mcp__github__get_workflow_run_logs,mcp__github__get_workflow_run_usage,mcp__github__issue_read,mcp__github__list_branches,mcp__github__list_code_scanning_alerts,mcp__github__list_commits,mcp__github__list_dependabot_alerts,mcp__github__list_discussion_categories,mcp__github__list_discussions,mcp__github__list_issue_types,mcp__github__list_issues,mcp__github__list_label,mcp__github__list_notifications,mcp__github__list_pull_requests,mcp__github__list_releases,mcp__github__list_secret_scanning_alerts,mcp__github__list_starred_repositories,mcp__github__list_tags,mcp__github__list_workflow_jobs,mcp__github__list_workflow_run_artifacts,mcp__github__list_workflow_runs,mcp__github__list_workflows,mcp__github__pull_request_read,mcp__github__search_code,mcp__github__search_issues,mcp__github__search_orgs,mcp__github__search_pull_requests,mcp__github__search_repositories,mcp__github__search_users --debug-file /tmp/gh-aw/agent-stdio.log --verbose --permission-mode bypassPermissions --output-format stream-json "$PROMPT_TEXT"${GH_AW_MODEL_AGENT_CLAUDE:+ --model "$GH_AW_MODEL_AGENT_CLAUDE"}' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log

.github/workflows/hourly-ci-cleaner.lock.yml:737

  • --debug-file /tmp/gh-aw/agent-stdio.log writes to the same file that stdout/stderr is being appended to via tee -a /tmp/gh-aw/agent-stdio.log. Concurrent writes will interleave and can break downstream log parsing. Use a separate debug file, or stop teeing into the same path.
            -- /bin/bash -c 'export PATH="$(find /opt/hostedtoolcache -maxdepth 4 -type d -name bin 2>/dev/null | tr '\''\n'\'' '\'':'\'')$PATH"; [ -n "$GOROOT" ] && export PATH="$GOROOT/bin:$PATH" || true && claude --print --disable-slash-commands --no-chrome --max-turns 20 --mcp-config /tmp/gh-aw/mcp-config/mcp-servers.json --allowed-tools Bash,BashOutput,Edit,ExitPlanMode,Glob,Grep,KillBash,LS,MultiEdit,NotebookEdit,NotebookRead,Read,Task,TodoWrite,Write,mcp__github__download_workflow_run_artifact,mcp__github__get_code_scanning_alert,mcp__github__get_commit,mcp__github__get_dependabot_alert,mcp__github__get_discussion,mcp__github__get_discussion_comments,mcp__github__get_file_contents,mcp__github__get_job_logs,mcp__github__get_label,mcp__github__get_latest_release,mcp__github__get_me,mcp__github__get_notification_details,mcp__github__get_pull_request,mcp__github__get_pull_request_comments,mcp__github__get_pull_request_diff,mcp__github__get_pull_request_files,mcp__github__get_pull_request_review_comments,mcp__github__get_pull_request_reviews,mcp__github__get_pull_request_status,mcp__github__get_release_by_tag,mcp__github__get_secret_scanning_alert,mcp__github__get_tag,mcp__github__get_workflow_run,mcp__github__get_workflow_run_logs,mcp__github__get_workflow_run_usage,mcp__github__issue_read,mcp__github__list_branches,mcp__github__list_code_scanning_alerts,mcp__github__list_commits,mcp__github__list_dependabot_alerts,mcp__github__list_discussion_categories,mcp__github__list_discussions,mcp__github__list_issue_types,mcp__github__list_issues,mcp__github__list_label,mcp__github__list_notifications,mcp__github__list_pull_requests,mcp__github__list_releases,mcp__github__list_secret_scanning_alerts,mcp__github__list_starred_repositories,mcp__github__list_tags,mcp__github__list_workflow_jobs,mcp__github__list_workflow_run_artifacts,mcp__github__list_workflow_runs,mcp__github__list_workflows,mcp__github__pull_request_read,mcp__github__search_code,mcp__github__search_issues,mcp__github__search_orgs,mcp__github__search_pull_requests,mcp__github__search_repositories,mcp__github__search_users --debug-file /tmp/gh-aw/agent-stdio.log --verbose --permission-mode bypassPermissions --output-format stream-json "$PROMPT_TEXT"${GH_AW_MODEL_AGENT_CLAUDE:+ --model "$GH_AW_MODEL_AGENT_CLAUDE"}' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log
  • Files reviewed: 3/3 changed files
  • Comments generated: 4

Comment on lines +169 to +172
git diff --name-only | grep '\.md$'
```
2. **If NO `.md` files changed** → **SKIP `make recompile` entirely**.
3. **If `.md` files changed** → Run `make recompile`, then immediately check:
Copy link

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is described as “workflow .md files”, but grep '\.md$' will match any Markdown file anywhere in the repo (docs, READMEs, etc.), causing unnecessary make recompile runs and the large-diff problem this guard is meant to prevent. Consider scoping the pattern to workflow markdown paths (e.g., .github/workflows/*.md).

Suggested change
git diff --name-only | grep '\.md$'
```
2. **If NO `.md` files changed****SKIP `make recompile` entirely**.
3. **If `.md` files changed** → Run `make recompile`, then immediately check:
git diff --name-only | grep '^\.github/workflows/.*\.md$'
  1. If NO workflow .md files changedSKIP make recompile entirely.
  2. If workflow .md files changed → Run make recompile, then immediately check:

Copilot uses AI. Check for mistakes.
Comment on lines +135 to +139
**Before running recompile**, check whether any workflow `.md` files were modified:

```bash
git diff --name-only | grep '\.md$'
```
Copy link

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intended to detect changes to workflow Markdown files, but git diff --name-only | grep '\.md$' will match any .md file in the repo (including unrelated docs). That can trigger unnecessary make recompile runs and recreate the “PR too large” failure mode. Narrow the match to workflow markdown paths (e.g., .github/workflows/*.md).

Copilot uses AI. Check for mistakes.
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented Apr 5, 2026

@copilot apply changes based on the comments in this thread

Previously grep '\.md$' matched any markdown file in the repo (docs,
READMEs, etc.), which could trigger unnecessary make recompile runs and
recreate the E003 "PR too large" failure. Narrowed to
'^\.github/workflows/.*\.md$' in both the workflow prompt and agent file.

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/31e0681e-fef5-4712-8a1a-d3dd8374f40a

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 5, 2026

@copilot apply changes based on the comments in this thread

Applied in d1a2694. The grep '\.md$' pattern in both hourly-ci-cleaner.md and ci-cleaner.agent.md has been narrowed to '^\.github/workflows/.*\.md$' so that docs, READMEs, and other markdown files outside the workflows directory no longer trigger make recompile.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw cXU4ZOQ/ls5ibYj1-1 (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .visibility (http block)
  • https://api.github.com/repos/githubnext/agentics/git/ref/tags/
    • Triggering command: /usr/bin/gh gh api /repos/githubnext/agentics/git/ref/tags/# --jq .object.sha (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of lpcox April 5, 2026 01:54
lpcox and others added 2 commits April 4, 2026 19:19
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@pelikhan pelikhan merged commit 245d168 into main Apr 5, 2026
56 checks passed
@pelikhan pelikhan deleted the copilot/copilot-token-optimizer-deep-audit branch April 5, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[copilot-token-optimizer] CI Cleaner deep audit: recurring failures, optimization recommendations

4 participants