[mcp:error-surface] tool failures are not flagged as errors and the error envelope differs per tool

[PYDuquesnoy]

What

Tool failures come back as successful MCP results, and the field that carries the
failure differs from tool to tool. A consumer (PostToolUse hook, telemetry, retry
logic, any automation) cannot tell success from failure generically — it has to
special-case every tool.

Observed over a full workshop day on one training VM: 733 tool calls, 38 genuine tool
failures, none of them flagged as an error.

Expected

A tool failure sets isError: true on the MCP CallToolResult, and the payload uses one
consistent shape across tools.

Actual

Every failure returns a normal result. The message lives in a different field per tool:

Tool	Failure shape	Message field	error_code?	Count
iris_query	{"error": …, "error_code": "SQL_ERROR"}	error	yes	16
iris_execute	{"error_code": "IRIS_RUNTIME_ERROR", "success": false, "output": …}	output	yes	12
iris_production	{"error": …, "error_code": "INTEROP_ERROR"}	error	yes	5
iris_doc (compile)	{"success": false, "compile_console": [...], "compile_errors": [...]}	compile_console	no	4
iris_test	{"error_code": "NO_TESTS_FOUND", …}	error	yes	1

Three consequences:

1. isError is never set — 38/38.
2. success: false is not usable as a substitute, because iris_test legitimately
   returns success: false for a red test, which is a valid outcome and not a tool
   failure. Anything keying on success alone will count red tests as tool errors.
3. iris_doc compile failures carry no error_code at all, so they can't even be
   classified by code.

Repro

Each of these returns a result with no error flag:

// 1. iris_query — message in `error`, no hint
iris_query(query: "SELECT COUNT(*) AS n FROM public.menus", namespace: "<NS>")
-> {"error": "ERROR #5540: SQLCODE: -1 Message: IDENTIFIER expected, reserved word PUBLIC found ^ SELECT COUNT ( * ) AS n FROM PUBLIC",
    "error_code": "SQL_ERROR"}          // no `hint`

// 2. iris_execute — message in `output`, NOT in `error`
iris_execute(code: "set bo=##class(<Pkg>.BO.<Name>).%New() …", namespace: "<NS>")
-> {"error_code": "IRIS_RUNTIME_ERROR", "success": false, "method": "http",
    "output": "ERROR: <INVALID OREF> 192 RunUser+2^IrisDevTmp.Run<id>.1 …"}

// 3. iris_doc compile failure — no `error_code`, message in `compile_console`
iris_doc(...)   // compiling a class with a syntax error
-> {"success": false,
    "compile_console": ["Compiling class <Pkg>.BS.<Name>",
                        "ERROR: <Pkg>.BS.<Name>.cls(OnProcessInput+5) #1012: Expected EOL or spaces …"],
    "compile_errors": [...]}

// 4. iris_production — no hint
iris_production(action: "start", namespace: "<NS>", production: "<Pkg>.Production")
-> {"error": "ERROR:ERROR <Ens>ErrProductionNotShutdownCleanly: Production '<Other>.Production' was not shutdown cleanly",
    "error_code": "INTEROP_ERROR"}      // no `hint`

Secondary: hint coverage is uneven

hint is the field that teaches the model how to recover, and it's the thing that
actually shortens the retry loop. 22 of 38 failures had no hint:

Code	with hint	without
iris_query:SQL_ERROR	10	6
iris_execute:IRIS_RUNTIME_ERROR	5	7
iris_production:INTEROP_ERROR	0	5
iris_doc (compile)	0	4

INTEROP_ERROR never carries one. The ErrProductionNotShutdownCleanly case above is a
good example: the fix is well-known and mechanical, but nothing in the response says so,
and the error names a different production than the one the caller asked to start,
which reads as a contradiction without a hint to explain it.

Impact

The model itself copes — it reads the message text and usually reasons its way out, so
this is not a runtime blocker. The cost lands on everything around the model:

• PostToolUse hooks and guards can't branch on "did this call fail".
• Retry/backoff logic has to hardcode per-tool field names.
• Any measurement of failure rates silently undercounts. (This surfaced while building
  telemetry over workshop transcripts: keying on the standard error flag reported 8
  failures where there were 46.)

Suggested fix

1. Set isError: true on the CallToolResult for genuine tool failures — explicitly
   not for a red iris_test run, which is a valid outcome.
2. Normalize one envelope across tools, e.g. {success, error_code, error, hint}, keeping
   output / compile_console / compile_errors as extra detail rather than as the
   only place the message appears.
3. Give iris_doc compile failures an error_code (e.g. COMPILE_ERROR).
4. Add hint for INTEROP_ERROR and for compile failures.

Environment

• MCP: 0.6.22 — the binary self-reports iris-agentic-dev 0.6.22, and check_config
  returns "mcp_version": "0.6.22". Built from tag v0.6.22-interop.
• Client: Claude Code + iris-interop-skills 1.5.9.
• Host: IRIS for Windows (x86-64) 2026.1 (Build 235U), MCP over http.
• Evidence: 9 sessions / 733 tool calls from one training VM (TR-26014 workshop).

Note on which code this is. 0.6.22 is the newest tagged code, despite the
repo's "latest release" being v0.7.0-interop: that tag points at commit 2081c4a
(2026-06-19), whose Cargo.toml reads version = "0.6.14", and v0.6.22-interop is
16 commits ahead of it and 0 behind. So this report is against the newest code, not
against something already superseded — but the release labelling is worth a look
separately.

[PYDuquesnoy]

Confirmed at cohort scale — 15,079 calls, 18 students

The original report measured one training VM: 733 calls, 38 silent failures. The full workshop has now finished and the numbers hold, an order of magnitude larger:

	day 1 (1 VM)	full course (18 students)
tool calls	733	15,079
genuine failures	38	980
flagged with the standard error signal	0	0
silent MCP-envelope failures	38	592 (60% of all failures)

The remaining 388 are real errors visible by other means (234), user rejections (120) and harness problems (26). So six out of ten failures in the entire course are invisible to anything keying on the standard flag.

Point 3 of the original report is the single most widespread failure of the workshop

"iris_doc compile failures carry no error_code at all, so they can't even be classified by code."

Ranked by how many students hit it, that is #1 of everything measured — 90 occurrences, 18 of 18 students. Every person who wrote a class hit it. Our own telemetry has to synthesise a placeholder code for it (we label it SIN_CODIGO) purely so the failures can be counted, which is a fair summary of the problem: the only way to classify these is to invent a code the server didn't send.

Distribution of the 592 silent failures by tool and code:

iris_execute:IRIS_RUNTIME_ERROR          127
iris_query:SQL_ERROR                     125
iris_production:INTEROP_ERROR            105
iris_doc:<no error_code at all>           90     <-- point 3
iris_test:NO_TESTS_FOUND                  44
iris_production_item:INVALID_PARAMS       17
iris_credential_list:INTEROP_ERROR        14
iris_production:NO_PRODUCTION             12
iris_production_item:INTEROP_ERROR        10
iris_compile:SIN_CODIGO / NOT_FOUND       20
iris_lookup_transfer:INTEROP_ERROR         7
… (rest below 7)

The hint gap also held up

INTEROP_ERROR still never carries one — 105 occurrences across iris_production alone, plus 31 more across iris_production_item / iris_credential_list / iris_lookup_transfer. Two concrete consequences filed separately, both of which a hint would have resolved on the first attempt rather than the fifth:

• [interop] omitting the optional namespace fails 95% of the time, with internal errors that never name the cause #5 — omitting the optional namespace fails 95% of the time (37/39) and reports <CLASS DOES NOT EXIST> Ens.Director or Table 'ENS_CONFIG.CREDENTIALS' not found, never the actual cause.
• [iris_lookup_transfer] action=import always fails with <SYNTAX> in the MCP's own generated routine (7/7) #6 — iris_lookup_transfer(action=import) fails 7/7 with a bare <SYNTAX> from the MCP's own generated routine.

Still not a runtime blocker

Same conclusion as the original report, now with more evidence behind it: the model copes. Across the cohort, TDD reached green in 1.51 attempts on average, 270 green runs against 58 red, and only one test class never went green. Compilation failed on 89 of 975 attempts (9%). The model reads the message text and reasons its way out.

The cost is still borne by everything around the model — hooks, retry logic, and any attempt to measure. It just turns out to be 592 events per cohort rather than 38 per VM.