Cloudflare triage inspired improvements - #132
Open
ArnabChatterjee20k wants to merge 7 commits into
Open
Conversation
The reproduce/verify stage already produces a structured verdict (root cause, impact, suggested fix, evidence) but it was only posted as a note and thrown away before the fix ran. Carry it on ProcessingInput and prepend it to the fix prompt context so the fix agent starts from a confirmed root cause instead of re-deriving one.
Three opt-in triage guardrails from the Cloudflare/Astro triage model: - verify_fail_open (ReplyConfig, default true): when the reproduce/ verify stage can't run (timeout/error/unsupported), setting this false asks the reporter for repro steps instead of forcing a fix. - fail_on_regression (EvaluationConfig, already existed): now wired. A successful attempt whose after-fix eval shows new failures or regressions is failed and retried instead of shipping the PR. - request_reporter_verification (ReplyConfig, default false): after a PR is created, ask the original reporter to confirm the fix resolves the issue on their end.
Greptile SummaryThe PR adds configurable verification failure behavior, carries verified diagnoses into fix prompts, enforces optional regression and red→green gates, and records corresponding timeline events.
Confidence Score: 5/5The PR appears safe to merge. The previously reported workspace compilation failure is resolved because the affected Important Files Changed
Reviews (5): Last reviewed commit: "linting" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Drops request_reporter_verification and the post-PR reporter ping. Keeps the diagnosis-forwarding, verify_fail_open, and regression-gate guardrails.
The e2e config builder constructs ReplyConfig field-by-field, so the new verify_fail_open field must be set explicitly. Addresses greptile review comment on PR #132.
ArnabChatterjee20k
changed the title
Adds opt-in evaluation.require_red_green (default false). When enabled and a test tool is detected: - Red phase: before the fix, a dedicated agent run authors a failing test only (no app code). The eval suite is re-run against the baseline; if no new test failure appears, the bug isn't reproduced and the attempt fails. - Fix phase: the fix prompt is told the failing test already exists and to make it pass without weakening it. - Green phase: the existing after-fix eval gate is forced on in red-green mode, so a test still failing after the fix fails the attempt. Adds EvaluationResult::has_new_test_failures() (test-category only) and a covering unit test.
The red-green phase previously ran invisibly inside Pending. Now it emits dedicated timeline events and issue decisions: - RedGreenStarted when the failing-test phase begins - RedConfirmed / red_green_not_reproduced for the red assertion - GreenConfirmed / not_green for the after-fix assertion Also records red_green action runs (red_confirmed / not_reproduced / green_confirmed / not_green) so the dashboard timeline reflects each step instead of showing only a stalled Pending attempt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Better issue-to-PR triage: diagnosis reuse + verify/eval guardrails
Architectural improvements to the issue-fixing cycle, inspired by Cloudflare's Astro issue triage writeup. All changes are opt-in with behavior-preserving defaults.
What changed
Feed the verify diagnosis forward into the fix (b547e8a)
The reproduce/verify stage already produces a structured verdict (root cause, impact, suggested fix, evidence), but it was only posted as a note and discarded — the fix agent then re-derived
everything from scratch. That verdict is now carried on ProcessingInput and prepended to the fix prompt, so the fix starts from a confirmed root cause instead of re-litigating whether the bug is
real. Skipped when the verdict is a bare conservative fallback (no details).
verify_fail_open option (d10afc6)
ReplyConfig.verify_fail_open (default true = current behavior). When the verify stage can't run (timeout / error / unsupported), the default assumes the bug is reproduced and fixes anyway.
Setting this false instead asks the reporter for repro steps — biasing away from forcing a fix on uncertainty.
Wire the regression gate (d10afc6)
EvaluationConfig.fail_on_regression already existed but was never enforced — the after-fix eval only posted a comment. It's now wired: a successful attempt whose after-fix evaluation shows new
failures or regressions is marked failed (triggering retry) instead of shipping the PR. Adds EvaluationResult::has_regressions().
Enforce a real red→green contract ( 5184d2e )
Adds opt-in EvaluationConfig.require_red_green (default false). Makes the "failing test first" step verifiable instead of self-reported. When enabled and a test tool is
detected: Red — a dedicated agent run authors only a failing test before any fix, then the suite is re-run against baseline and must show a new test failure, else the bug
isn't reproduced and the attempt fails; Fix — the fix prompt is told the failing test already exists and must be made to pass without weakening it; Green — the after-fix
eval gate is forced on, failing the attempt if the test still fails. Adds EvaluationResult::has_new_test_failures(). Reuses CodeQualityEvaluator and the existing
execute_with_attempt, so no new AgentRunner trait method or wrapper forwarding is needed.
Related PRs and Issues
(If this PR is related to any other PR or resolves any issue or related to any issue link all related PR and issues here.)
Have you read the Contributing Guidelines on issues?
(Write your answer here.)