Problem
#411 adds JSON Object mode to the packaged capability and escalation classifiers. Those classifiers always have a packaged verdict schema, so Switchyard can append it to the prompt and validate the returned object locally.
Custom classifiers differ: the user owns both the prompt and the inner response_schema. Exposing response_format_type = "json_object" for custom mode requires a clear contract for these cases:
response_schema is omitted.
- The prompt already contains a hard-coded copy of the schema or other JSON output instructions.
response_schema contains a provider response-format wrapper such as {"type": "json_object"} instead of an inner JSON Schema.
- The prompt and configured schema disagree.
Without an explicit policy, Switchyard could duplicate or contradict prompt instructions, or accept verdicts without the local validation used by the packaged classifiers.
Proposed solution
Define the custom-classifier contract before exposing JSON Object mode. A conservative design would:
- continue to require an inner
response_schema in both response modes;
- append that schema to the prompt and validate verdicts locally in
json_object mode;
- reject provider response-format wrappers where an inner JSON Schema is expected; and
- document that the configured schema, rather than a hard-coded prompt copy, is the source of truth.
The design should explicitly decide whether schema-free JSON Object output is unsupported or a separate, unvalidated mode.
Alternatives considered
- Request JSON Object output without a schema. This preserves JSON syntax but loses the contract needed for local validation and deterministic fallback.
- Treat a schema hard-coded in the prompt as authoritative. Switchyard cannot reliably extract or validate arbitrary prompt text, and it can drift from
response_schema.
Scope notes
ClassifierContract::from_inner_schema and CustomClassifierConfig in libsy.
- The server
llm_classifier custom-mode TOML surface.
- Python bindings if the response-format option is exposed there.
- Keep existing custom classifiers on JSON Schema by default.
Additional context
Problem
#411 adds JSON Object mode to the packaged capability and escalation classifiers. Those classifiers always have a packaged verdict schema, so Switchyard can append it to the prompt and validate the returned object locally.
Custom classifiers differ: the user owns both the prompt and the inner
response_schema. Exposingresponse_format_type = "json_object"for custom mode requires a clear contract for these cases:response_schemais omitted.response_schemacontains a provider response-format wrapper such as{"type": "json_object"}instead of an inner JSON Schema.Without an explicit policy, Switchyard could duplicate or contradict prompt instructions, or accept verdicts without the local validation used by the packaged classifiers.
Proposed solution
Define the custom-classifier contract before exposing JSON Object mode. A conservative design would:
response_schemain both response modes;json_objectmode;The design should explicitly decide whether schema-free JSON Object output is unsupported or a separate, unvalidated mode.
Alternatives considered
response_schema.Scope notes
ClassifierContract::from_inner_schemaandCustomClassifierConfigin libsy.llm_classifiercustom-mode TOML surface.Additional context