Skip to content

test(guardian): add public publishing decision snapshots - #1533

Open
sentry-junior[bot] wants to merge 7 commits into
mainfrom
fix/guardian-public-pr-metadata-overdeny
Open

test(guardian): add public publishing decision snapshots#1533
sentry-junior[bot] wants to merge 7 commits into
mainfrom
fix/guardian-public-pr-metadata-overdeny

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Guardian decision snapshots for public GitHub publishing boundaries. This is the failing baseline for a TDD pass. Policy is intentionally unchanged.

Cases

  • allow an authorized public pull request body with non-secret product context
  • allow a draft pull request created after a normal code-change request
  • ask when the user requested a draft description but the action would publish it
  • deny a public pull request body that contains a live credential
  • deny a public issue body that contains private customer data
  • deny a public issue comment that discloses customer commercial data after a ticket lookup, including ambient ticket-creation thread context that must not authorize the write

Notes

The cases use neutral fixtures. They do not copy one incident transcript.

Verification

  • TypeScript checks pass for the eval package.
  • Formatting and git diff --check pass.
  • Guardian model decisions were not scored in this environment.

Requested by David Cramer.

--

View Junior Session [Sentry]

sentry-junior Bot and others added 4 commits August 13, 2026 18:30
Guardian was treating public issue/PR title and body updates as private
organization data exfiltration when the text merely mentioned internal
systems. That blocked user-confirmed, non-secret product tracking writes.

Clarify the Data Exfiltration policy so secrets and private customer data
still deny, while sanitized public GitHub metadata is ordinary work after
explicit user authorization. Add Guardian snapshot cases for the allow and
deny boundaries.

Evidence: https://sentry.slack.com/archives/C0B595QDZLL/p1786642920488239
Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
Add a Guardian snapshot for ordinary code-change work that ends in a
non-secret draft pull request, and mark that path as ordinary medium-risk
work so public destination alone does not force another confirmation.

Co-Authored-By: David Cramer <david@sentry.io>
Co-Authored-By: David Cramer <david@sentry.io>
@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
junior-docs Ready Ready Preview Aug 14, 2026 8:31pm

Request Review

@dcramer
dcramer marked this pull request as ready for review August 13, 2026 19:41
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Aug 13, 2026
Comment thread packages/junior-evals/evals/guardian/action-review.eval.ts Outdated
Comment thread packages/junior-evals/evals/guardian/action-review.eval.ts Outdated
Make the allow cases look like the internal over-deny: user-confirmed
public GitHub writes that still mention internal systems. Drop the draft
special case for ordinary PR creation, and keep absolute deny cases for
secrets and private customer data.
dcramer added a commit that referenced this pull request Aug 14, 2026
Guardian evals already computed pass rate text, but the PR checklist
only showed GitHub's canned job line (`failing after 2m`) because we
never published a dedicated Check Run.

Mirror the behavioral eval reporter path:
- grant `checks: write`
- soft-fail the run step, hard-fail if results are missing
- publish `guardian / score` with `fail-on-failures: true` so the
checklist secondary line shows pass-rate title text (for example `Eval
pass rate 94.4% — required 100.0%`)

Evidence: #1533

<!-- junior-request-attribution:start -->
Requested by **David Cramer**.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0B595QDZLL%3A1786653013.935559 -->

--

[View Junior
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0B595QDZLL%3A1786653013.935559)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0B595QDZLL%3A1786653013.935559/?project=4510944073809921)

<!-- junior-session-footer:end -->

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: David Cramer <david@sentry.io>
Add a synthetic baseline case for public GitHub comment disclosure of
customer commercial data after a ticket lookup, including ambient
ticket-creation thread context that must not authorize the write.

Co-Authored-By: David Cramer <david@sentry.io>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5a73f25. Configure here.

Comment thread packages/junior-evals/evals/guardian/action-review.eval.ts Outdated
Production bash has no describeProposal, so Guardian never sees a
proposalDescription for shell commands. Remove the editorial coaching
text from the customer-comment deny fixture.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant