FIX: Aligning Scenarios with Technique Registry#2186
Open
rlundeen2 wants to merge 11 commits into
Open
Conversation
Drop the redundant `core`-tag gates from RapidResponse and
AdversarialBenchmark so scenarios expose whatever techniques the active
initializer has registered, rather than re-narrowing to `core`.
- rapid_response: remove `available=TagQuery.all('core')` — the pool is
now every registered factory (matches the leakage pattern and the
published blog, which lists the extra-only `pair` technique).
- benchmark: drop the `'core' in technique_tags` clause but keep the
genuine capability filters (`uses_adversarial` and
`adversarial_chat is None`), so extra adversarial techniques such as
`pair`/`violent_durian` can participate in the model sweep.
- Update the benchmark test snapshot to mirror the production predicate.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
Auto-promote every catalog tag present in a scenario's technique pool into a selectable aggregate, so `--techniques <tag>` (e.g. `core`, or a custom `airt_internal`) expands to every technique carrying that tag. `all` and `default` stay reserved synthetic aggregates; a tag that collides with a technique name stays a concrete technique (name selection wins). - attack_technique_registry: derive aggregates from pool tags; make aggregate_tags optional (escape hatch for compound/renamed aggregates only). - Strip the now-redundant 1:1 aggregate_tags from all six call sites (rapid_response, cyber, text_adaptive, leakage, benchmark, doctor) and drop unused TagQuery imports. - cyber: drop the curation gate so ALL exposes the full registered pool while DEFAULT stays curated to red_teaming (aligns with rapid_response/benchmark). - Add a shared invariant that every catalog tag is a selectable aggregate. - Fix benchmark test mock factory (adversarial_chat=None) so it lands in the benchmarkable pool now that `light` is auto-derived rather than declared. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
ValbuenaVC
pushed a commit
to ValbuenaVC/PyRIT
that referenced
this pull request
Jul 14, 2026
The plug-in PR had RapidResponse filter its technique pool to the built-in 'core' tag via get_factories_for_scenario(base_query=TagQuery.all('core')), with a fallback whose return value was discarded. Restore the builder to the upstream form (aligning with microsoft#2186): read every registered factory via get_factories_or_raise and build the technique enum with no core filter.
This lets an initializer-registered private technique (any tag) surface through --techniques with no scenario-routing bridge, and removes a latent empty-enum path.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 153315ea-4360-489a-89c5-630a41cf3fd2
ValbuenaVC
reviewed
Jul 15, 2026
ValbuenaVC
left a comment
ValbuenaVC
left a comment
Contributor
There was a problem hiding this comment.
Great PR! One comment I think needs resolution before we can approve and merge it
… robust
Bump cyber, rapid_response, and adversarial benchmark to VERSION = 3 since
dropping the ``core`` pool gate broadened their selectable ``all`` pool;
``use_cached`` only matches prior runs at the current VERSION, so stale v2
results must not suppress v3 runs.
Make AdversarialBenchmark.__init__ fall back to the ``all`` aggregate when no
``light``-tagged factory survives the pool filter, instead of unconditionally
resolving ``technique_class("light")`` which raised for custom technique sets.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
…m membership Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
…ault set RapidResponse now defaults to the `light` aggregate (with an `all` fallback when no light-tagged factory survives the initializer's pool), matching AdversarialBenchmark. Removes the orphaned `default_technique_names` from both scenarios so `light` is the single, unambiguous curated default rather than a dead `default` aggregate nothing referenced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
Scenarios previously passed default_technique to the base Scenario constructor, duplicating the light->all fallback in rapid_response and adversarial benchmark __init__. Move that ownership onto the technique class: ScenarioTechnique.default() resolves the catalog default, the builder records it once (with the light->all fallback), and the base Scenario reads it via technique_class.default(). Static-enum scenarios override default() with their member. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
Drop two provably-dead parameters (no caller or test passed either) and reconcile the two default idioms into a clean either/or: - Remove aggregate_tags: catalog tags already auto-promote to selectable aggregates, so the explicit map was redundant. - Remove available: the caller already hands in factories and pre-filters the pool, so a second in-function filter just duplicated that. - Replace default (str) + default_technique_names (set) with mutually exclusive default_tags / default_names; both build the DEFAULT aggregate and raise if both are given. Also collapse the "no default -> ALL" fallback to a single owner: the builder now only records _default_technique_value when a DEFAULT aggregate was actually built, and ScenarioTechnique.default() owns the ALL fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
Update scenarios.instructions.md to reflect that the default technique is owned by the technique catalog (technique_class.default()) rather than passed to super().__init__(): remove the stale default_technique super-arg from the code examples and requirements, add a "Default Technique" subsection covering the default() override for static enums and default_tags/default_names for dynamically built enums, and fix the stale TagQuery reference to describe auto-promoted aggregates. Add TestBuildTechniqueClassFromFactories covering tag auto-promotion, default_tags/default_names building the DEFAULT aggregate, the single-source ALL fallback when no default is declared (or nothing matches), and the ValueError when both defaults are passed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
Resolve conflict in garak/encoding.py: main added default_technique=EncodingTechnique.DEFAULT to super().__init__() and bumped the encoding datasets to max_dataset_size=10. This branch removed the default_technique constructor arg (defaults are now catalog-owned), so preserve main's intent by overriding EncodingTechnique.default() to return DEFAULT and keeping the size-10 datasets. Also reconcile two sites the merge surfaced: - test_rapid_response.py: main removed the standalone ContextComplianceAttack class (context_compliance is now a simulated-conversation PromptSendingAttack). Drop the stale class assertion in test_default_technique_is_light; the has_simulated_conversation check already covers it. - doc examples (0_scenarios, 1_pyrit_scan .py + .ipynb): our refactor removed the default_technique super-arg but missed these tutorials. Remove it and show the default() override pattern instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
- attack_technique_registry: use the repo's ty-ignore syntax (ty:unresolved-attribute) for the dynamically-set _default_technique_value - encoding.py EncodingTechnique.default(): imperative-mood docstring + Returns section - ruff-format normalization of an assert-message wrap in the registry test Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
…ation (ruff UP037) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bc715e-cd3a-480c-afe8-5cac8339224f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aligns scenarios with PyRIT's intended design: the initializer is the single gate deciding which techniques are registered; scenarios expose whatever is registered rather than re-narrowing the pool themselves (see
doc/code/framework.md). Drops the redundantcore-tag gates, makes catalog tags synonymous with selectable aggregates, and moves the default technique into the catalog as a single source of truth.Changes
Drop the redundant narrowing gates.
rapid_responseandcybernow expose every registered factory (droppingavailable=TagQuery.all("core")/_CYBER_TECHNIQUE_NAMES);benchmark/adversarialdrops thecoreclause but keeps its real capability filters (uses_adversarial,adversarial_chat is None). This lets extra techniques (pair,violent_durian, …) participate and fixes arapid_responsecontradiction with the published blog.ALLis now decoupled from the default (the oldDEFAULT == ALLinvariant no longer holds for cyber).Tags are aggregates. Every catalog tag in a scenario's pool auto-promotes to a selectable aggregate, so
--techniques airt_internal(orlight,single_turn, …) resolves to every technique carrying that tag — previously it raised.allanddefaultstay reserved; a tag colliding with a technique name stays a concrete technique.Simplify the builder + centralize the default.
build_technique_class_from_factoriesis now four kwargs —class_name,factories, and mutually-exclusivedefault_tags/default_names(raises if both). Dropped the deadaggregate_tags/availableparams. The default technique is no longer aScenario.__init__arg: the base readstechnique_class.default()(static enums override it; dynamic enums declare it viadefault_tags/default_names), and the lone "select ALL when nothing is chosen" fallback now lives only inScenarioTechnique.default(). VERSION bumped to 3 forcyber,rapid_response,adversarial_benchmark.Testing
rapid_response,cyber,leakage,text_adaptive,doctor,benchmark/adversarial,registry,setup, backend) plus amainmerge; ruff clean.test_scenario_technique_invariants(every tag is an aggregate expanding to its tagged techniques) andTestBuildTechniqueClassFromFactories(tag promotion,default_tags/default_names, the ALL fallback, and the both-passedValueError).--list-scenariosand a realairt.jailbreakrun againstopenai_chatboth completed (exit 0).