Skip to content

Require user journey tests for behavior changes - #652

Open
rohita5l wants to merge 3 commits into
mainfrom
rohit/user-journey-test-gate
Open

rohita5l wants to merge 3 commits into
mainfrom
rohit/user-journey-test-gate

Conversation

@rohita5l

Copy link
Copy Markdown
Collaborator

Summary

  • add a trusted LLM-backed CI gate that detects new or materially changed ug user journeys and requires meaningful tests/integration coverage
  • judge bounded product/test diffs using the existing UCODE_TEST_WORKSPACE and DATABRICKS_BEARER secrets plus the trusted tests/AGENTS.md policy
  • fail closed with a direct CTA to add a complete user-journey test
  • allow only @rohita5l or @lilly-luo to bypass with the exact /skip-user-journey-test comment, mirrored to a visible PR label

Security

  • run the gate from the trusted default branch with pull_request_target
  • never check out or execute PR-head code
  • treat titles and patches as untrusted text and cap the submitted diff
  • reject malformed judge responses and gateway failures unless an authorized waiver exists

Testing

  • uv run pytest tests/test_user_journey_gate.py -q (8 passed)
  • Ruff check passed
  • Ruff format check passed
  • both workflow YAML files parse successfully
  • git diff --check passed

Required repository setup

  • set UG_CI_USER_JOURNEY_JUDGE_MODEL to a chat-capable system.ai model
  • add User Journey Test Required as a required branch-protection check after this merges

@bbqiu bbqiu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! you all can refine the workflow as time goes on

@rohita5l
rohita5l enabled auto-merge (squash) September 17, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants