Skip to content

[feature] Define JSON Object contracts for custom classifiers #429

Description

@ting-hong-shieh

Problem

#411 adds JSON Object mode to the packaged capability and escalation classifiers. Those classifiers always have a packaged verdict schema, so Switchyard can append it to the prompt and validate the returned object locally.

Custom classifiers differ: the user owns both the prompt and the inner response_schema. Exposing response_format_type = "json_object" for custom mode requires a clear contract for these cases:

  • response_schema is omitted.
  • The prompt already contains a hard-coded copy of the schema or other JSON output instructions.
  • response_schema contains a provider response-format wrapper such as {"type": "json_object"} instead of an inner JSON Schema.
  • The prompt and configured schema disagree.

Without an explicit policy, Switchyard could duplicate or contradict prompt instructions, or accept verdicts without the local validation used by the packaged classifiers.

Proposed solution

Define the custom-classifier contract before exposing JSON Object mode. A conservative design would:

  • continue to require an inner response_schema in both response modes;
  • append that schema to the prompt and validate verdicts locally in json_object mode;
  • reject provider response-format wrappers where an inner JSON Schema is expected; and
  • document that the configured schema, rather than a hard-coded prompt copy, is the source of truth.

The design should explicitly decide whether schema-free JSON Object output is unsupported or a separate, unvalidated mode.

Alternatives considered

  • Request JSON Object output without a schema. This preserves JSON syntax but loses the contract needed for local validation and deterministic fallback.
  • Treat a schema hard-coded in the prompt as authoritative. Switchyard cannot reliably extract or validate arbitrary prompt text, and it can drift from response_schema.

Scope notes

  • ClassifierContract::from_inner_schema and CustomClassifierConfig in libsy.
  • The server llm_classifier custom-mode TOML surface.
  • Python bindings if the response-format option is exposed there.
  • Keep existing custom classifiers on JSON Schema by default.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions