Skip to content

Commit 028b369

Browse files
eb8680claude
andcommitted
Serialize code-derived values as raw strings (#775)
A synthesized function crossed the model boundary as source escaped inside a JSON object -- `{"code": ...}`, via the `EncodedFunction` model on the way out and the `SynthesizedFunction` family on the way in. Providers that decode structured output approximately (ds4) have to get that escaping right by hand, and the wrapper dragged a `$defs`/`$ref` into every schema mentioning a callable. Generalize what `synthesis.snippet` already does for `types.CodeType`: a bare `str`, with `BeforeValidator`/`PlainSerializer` doing the work. `EncodedFunction` becomes an `Annotated[str, Field(...)]`, and the `SynthesizedFunction`/`SynthesizedSkillBody`/`SynthesizedMethodSkillBody` hierarchy flattens into `_signature_str`, `_synthesized_source_schema` and `_checked_source`. The prose those classes carried was split across a model docstring and a field description, which the single string schema merges: the signature line and all five constraints survive, and constraints 1-3 -- until now repeated verbatim at three sites -- are stated once. The method body's `_extra_instructions` is dropped as the third statement of its `self` receiver, which the rendered `Callable[[self, ...], R]` and constraint 4 both still make. A prompt-spliced function now also reaches the model as unencoded source rather than escaped inside a string, which `to_content_blocks` gets for free once the encoding is a top-level string. The replay fixtures are migrated rather than re-recorded: `REBUILD_FIXTURES` cannot run, because `ReplayLiteLLMProvider._completion` forwards `fwd(*args, **kwargs)` without `model` and litellm rejects the call. That fails identically on the base branch, so it is a separate bug; only each fixture's JSON envelope is rewritten, and every recorded source string is preserved byte for byte. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent c863012 commit 028b369

24 files changed

Lines changed: 316 additions & 372 deletions

File tree

‎effectful/handlers/llm/harness/observability/rich.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -240,9 +240,9 @@ def _is_python(text: str, *, partial: bool = False) -> bool:
240240
"""Whether `text` looks like a Python source snippet worth highlighting.
241241
242242
Detects code by *content* rather than schema/field name, so it covers every
243-
`Encodable` type that serializes Python as a string -- the synthesis
244-
`SynthesizedFunction.code` field, `exec_code`'s `types.CodeType`
245-
argument, and any future code-carrying tool -- uniformly. Requires a
243+
`Encodable` type that serializes Python as a string -- a synthesized
244+
`Callable`, `exec_code`'s `types.CodeType` argument, and any future
245+
code-carrying tool -- uniformly. Requires a
246246
multi-line string that parses as a module with at least one real statement
247247
(not a lone expression), which excludes prose and JSON-as-string.
248248
@@ -261,8 +261,8 @@ def _is_python(text: str, *, partial: bool = False) -> bool:
261261
def _extract_code(args: typing.Any, *, partial: bool = False) -> str | None:
262262
"""Return an embedded Python source string from parsed tool-call arguments.
263263
264-
Walks nested dicts (a synthesized callable is ``{"implementation":
265-
{"code": ...}}``; `exec_code` is a flat ``{"code": ...}``) and returns
264+
Walks nested dicts (a synthesized callable is ``{"implementation": ...}``;
265+
`exec_code` is ``{"code": ...}``) and returns
266266
the first string value that :func:`_is_python` recognizes. ``partial`` is
267267
forwarded, and is set while streaming, where the payload is source cut
268268
mid-line and has to be judged as a prefix rather than as a module.

‎effectful/handlers/llm/harness/serialization.py‎

Lines changed: 21 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -712,40 +712,33 @@ def _pydantic_type_image(ty: type[Image.Image]):
712712
]
713713

714714

715-
# The *serialization* view of a synthesized callable: the shape the model reads
716-
# when a function is handed to it as a value (e.g. a tool's return) -- just the
717-
# source, with none of the synthesis instructions the `SynthesizedFunction` subtype
718-
# carries for the generation direction. Its JSON schema (docstring included, since
719-
# pydantic renders it as the schema `description`) is what every synthesized-callable
720-
# encoding serializes as, so keep the docstring model-facing.
721-
class EncodedFunction(pydantic.BaseModel):
722-
"""A function, encoded as a string of its complete Python source."""
723-
724-
code: str = pydantic.Field(..., description="Python source defining the function.")
725-
726-
727-
def _serialize_callable(value: collections.abc.Callable) -> EncodedFunction:
728-
"""Encode a callable back to its ``code`` form (source, or a stub).
729-
730-
Emits a plain `EncodedFunction` -- which is exactly what the serialization JSON
731-
schema declares -- rather than the `SynthesizedFunction` subclass that governs
732-
the *other* direction. The two directions carry different obligations, and
733-
conflating them was a real bug: `SynthesizedFunction`'s constraints ("the last
734-
statement must be a function definition", "every parameter must be annotated")
735-
are demands on code a model is *writing*, and a value being serialized is under
736-
no such obligation. It is any callable that reached this point -- a class handed
737-
back by a lexical-scope read, or an inner function a Skill *body* returned,
738-
which was never required to annotate anything -- and re-validating its recovered
739-
source rejected those with a `ValidationError` raised from inside pydantic's
740-
serializer, aborting the whole call rather than encoding the value.
715+
# The *serialization* view of a synthesized callable: the shape the model reads when
716+
# a function is handed to it as a value (e.g. a tool's return) -- the bare source,
717+
# with none of the synthesis instructions the validation direction carries.
718+
EncodedFunction = typing.Annotated[
719+
str,
720+
pydantic.Field(
721+
description="A function, as a string of its complete Python source."
722+
),
723+
]
724+
725+
726+
def _serialize_callable(value: collections.abc.Callable) -> str:
727+
"""Encode a callable as its source, or as a stub when there is none.
728+
729+
The synthesis constraints ("the last statement must be a function definition",
730+
"every parameter must be annotated") are demands on code a model is *writing*;
731+
a value being serialized is under no such obligation -- it may be a class from a
732+
lexical-scope read, or an inner function a Skill body returned -- so they are
733+
deliberately not re-applied here.
741734
"""
742735
try:
743736
source = inspect.getsource(value)
744737
except (OSError, TypeError):
745738
source = None
746739

747740
if source:
748-
return EncodedFunction(code=textwrap.dedent(source))
741+
return textwrap.dedent(source)
749742

750743
name = getattr(value, "__name__", None)
751744
docstring = inspect.getdoc(value)
@@ -759,11 +752,10 @@ def _serialize_callable(value: collections.abc.Callable) -> EncodedFunction:
759752
except (ValueError, TypeError):
760753
sig_str = "(...)"
761754

762-
stub_code = f'''def {name}{sig_str}:
755+
return f'''def {name}{sig_str}:
763756
"""{docstring}"""
764757
...
765758
'''
766-
return EncodedFunction(code=stub_code)
767759

768760

769761
@TypeToPydanticType.register(collections.abc.Callable)

‎effectful/handlers/llm/harness/synthesis/body.py‎

Lines changed: 36 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@
3838
import collections.abc
3939
import functools
4040
import inspect
41-
import textwrap
4241
import types
4342
import typing
4443
from collections.abc import Callable
@@ -69,9 +68,10 @@
6968
)
7069
from effectful.handlers.llm.harness.synthesis.function import (
7170
SplicedRegion,
72-
SynthesizedFunction,
71+
_checked_source,
7372
_def_nodes,
7473
_recover_skill_def,
74+
_synthesized_source_schema,
7575
)
7676
from effectful.handlers.llm.types import Encodable, Skill, Tool
7777
from effectful.ops.semantics import fwd, handler
@@ -177,40 +177,20 @@ def __class_getitem__(cls, item):
177177
return types.GenericAlias(cls, item)
178178

179179

180-
class SynthesizedSkillBody(SynthesizedFunction):
181-
"""Structured output for synthesizing a `Skill`'s body (`write_and_run_body`).
180+
# A Skill body is type-checked against the Skill's own (already-annotated) signature
181+
# by `_splice_body`, so the synthesized body's own annotations are optional.
182+
_SKILL_BODY_CONSTRAINTS: list[str] = [
183+
"Write the function with the Skill's signature; parameter and return "
184+
"annotations are optional.",
185+
"Do not include a docstring or doctests; the Skill's are supplied automatically.",
186+
]
182187

183-
Decoded through `_pydantic_skill_body`: the function is type-checked against
184-
the enclosing Skill's source and its doctests are run with self/recursive
185-
calls routed to the synthesized implementation.
186-
187-
Unlike `SynthesizedFunction`, the parameter and return *annotations* are not
188-
required: a Skill body is type-checked against the Skill's own signature
189-
(see `splice_skill_body`), so the model may omit or vary them -- in
190-
particular it need not annotate the ``self`` receiver of an instance-method
191-
Skill.
192-
"""
193-
194-
code: str = pydantic.Field(
195-
...,
196-
description=textwrap.dedent("""
197-
The complete Python source implementing the Skill shown in its spec.
198-
The code MUST satisfy the following constraints, or it will fail validation:
199-
200-
<constraints>
201-
1. The code MUST be one complete syntactically valid Python module.
202-
2. The code MUST NOT use star imports or ``__future__`` imports.
203-
3. The function definition MUST be the LAST statement - do not add any code after it.
204-
4. Write the function with the Skill's signature; parameter and return
205-
annotations are optional.
206-
5. Do not include a docstring or doctests; the Skill's are supplied automatically.
207-
</constraints>
208-
"""),
209-
)
210-
211-
# A Skill body is checked against the Skill's own (already-annotated)
212-
# signature, so the synthesized body's annotations are optional.
213-
_require_annotations: typing.ClassVar[bool] = False
188+
_METHOD_SKILL_BODY_CONSTRAINTS: list[str] = [
189+
"Write the function with the Skill's signature: its FIRST parameter is the "
190+
"instance receiver ``self`` (which you may leave unannotated); all other "
191+
"parameter and return annotations are optional too.",
192+
"Do not include a docstring or doctests; the Skill's are supplied automatically.",
193+
]
214194

215195

216196
@TypeToPydanticType.register(SkillBody)
@@ -223,30 +203,30 @@ def _pydantic_skill_body(ty: typing.Any) -> typing.Any:
223203
implementation, so a doctest that calls the Skill (including for recursion)
224204
exercises the freshly synthesized code rather than re-invoking the model.
225205
"""
226-
typed_enc = SynthesizedSkillBody._create_model_from_callable_type(
206+
schema = _synthesized_source_schema(
227207
ty if typing.get_args(ty) else Callable[..., typing.Any], # type: ignore[arg-type]
208+
"the Skill shown in its spec",
209+
_SKILL_BODY_CONSTRAINTS,
228210
)
229211

230212
def _validate(
231-
value: SynthesizedSkillBody | dict | str | Callable,
213+
value: str | Callable,
232214
info: pydantic.ValidationInfo,
233215
) -> Callable:
234-
if isinstance(value, str):
235-
value = typed_enc.model_validate({"code": value})
236-
elif isinstance(value, dict):
237-
value = typed_enc.model_validate(value)
238-
elif callable(value):
216+
value = _checked_source(value, require_annotations=False)
217+
if not isinstance(value, str):
239218
return typing.cast(Callable, value)
219+
240220
ctx = info.context or {}
241221
anchor = ctx.get(_TYPE_CHECK_ANCHOR_KEY)
242222
if anchor is not None:
243223
# skill bodies should not have access to call-local variables
244224
assert isinstance(anchor, Skill)
245225
ctx = anchor.__context__
246226

247-
filename = f"<synthesis:{id(value.code)}>"
227+
filename = f"<synthesis:{id(value)}>"
248228
module: ast.Module = effectful.handlers.llm.harness.execution.hooks.parse(
249-
value.code, filename
229+
value, filename
250230
)
251231

252232
# `None` means the Skill's source can't be recovered (REPL/exec/notebook
@@ -282,7 +262,7 @@ def _validate(
282262

283263
return typing.Annotated[
284264
pydantic.InstanceOf[ty_], # type: ignore
285-
pydantic.BeforeValidator(_validate, json_schema_input_type=typed_enc),
265+
pydantic.BeforeValidator(_validate, json_schema_input_type=schema),
286266
pydantic.PlainSerializer(_serialize_callable, return_type=EncodedFunction),
287267
]
288268

@@ -299,58 +279,6 @@ class MethodSkillBody(SkillBody):
299279
"""
300280

301281

302-
class SynthesizedMethodSkillBody(SynthesizedSkillBody):
303-
"""Structured output for synthesizing an *instance-method* `Skill`'s body.
304-
305-
Decoded through `_pydantic_skill_body`: the function is type-checked against
306-
the enclosing Skill's source and its doctests are run with self/recursive
307-
calls routed to the synthesized implementation.
308-
309-
Unlike `SynthesizedFunction`, the parameter and return *annotations* are not
310-
required: a Skill body is type-checked against the Skill's own signature
311-
(see `splice_skill_body`), so the model may omit or vary them -- in
312-
particular it need not annotate the ``self`` receiver of an instance-method
313-
Skill.
314-
"""
315-
316-
code: str = pydantic.Field(
317-
...,
318-
description=textwrap.dedent("""
319-
The complete Python source implementing the instance-method Skill shown in
320-
its spec. The code MUST satisfy the following constraints, or it will fail
321-
validation:
322-
323-
<constraints>
324-
1. The code MUST be one complete syntactically valid Python module.
325-
2. The code MUST NOT use star imports or ``__future__`` imports.
326-
3. The function definition MUST be the LAST statement - do not add any code after it.
327-
4. Write the function with the Skill's signature: its FIRST parameter is the
328-
instance receiver ``self`` (which you may leave unannotated); all other parameter
329-
and return annotations are optional too.
330-
5. Do not include a docstring or doctests; the Skill's are supplied automatically.
331-
</constraints>
332-
"""),
333-
)
334-
335-
@classmethod
336-
def _param_names(cls, param_types: typing.Iterable[typing.Any]) -> list[str]:
337-
# The method's callable type already carries the receiver as its first
338-
# parameter (with an uninformative Agent-class type); relabel it ``self`` so
339-
# the model reproduces it rather than inventing one -- do NOT prepend a receiver.
340-
names = super()._param_names(param_types)
341-
if names:
342-
names[0] = "self"
343-
return names
344-
345-
@classmethod
346-
def _extra_instructions(cls) -> str:
347-
return (
348-
"\n\nThis implements an instance method: the first parameter is the "
349-
"instance receiver `self`. Include it as the first parameter; you may "
350-
"leave it unannotated."
351-
)
352-
353-
354282
def _class_skill_of(op: typing.Any) -> typing.Any | None:
355283
"""The class-level `Skill` underlying an Agent-method Skill ``op``.
356284
@@ -382,30 +310,31 @@ def _pydantic_method_skill_body(ty: typing.Any) -> typing.Any:
382310
their own instances -- route ``agent.method(...)`` on *any* instance to the
383311
synthesized implementation.
384312
"""
385-
typed_enc = SynthesizedMethodSkillBody._create_model_from_callable_type(
313+
schema = _synthesized_source_schema(
386314
ty if typing.get_args(ty) else Callable[..., typing.Any], # type: ignore[arg-type]
315+
"the instance-method Skill shown in its spec",
316+
_METHOD_SKILL_BODY_CONSTRAINTS,
317+
receiver=True,
387318
)
388319

389320
def _validate(
390-
value: SynthesizedMethodSkillBody | dict | str | Callable,
321+
value: str | Callable,
391322
info: pydantic.ValidationInfo,
392323
) -> Callable:
393-
if isinstance(value, str):
394-
value = typed_enc.model_validate({"code": value})
395-
elif isinstance(value, dict):
396-
value = typed_enc.model_validate(value)
397-
elif callable(value):
324+
value = _checked_source(value, require_annotations=False)
325+
if not isinstance(value, str):
398326
return typing.cast(Callable, value)
327+
399328
ctx = info.context or {}
400329
anchor = ctx.get(_TYPE_CHECK_ANCHOR_KEY)
401330
if anchor is not None:
402331
# skill bodies should not have access to call-local variables
403332
assert isinstance(anchor, Skill)
404333
ctx = anchor.__context__
405334

406-
filename = f"<synthesis:{id(value.code)}>"
335+
filename = f"<synthesis:{id(value)}>"
407336
module: ast.Module = effectful.handlers.llm.harness.execution.hooks.parse(
408-
value.code, filename
337+
value, filename
409338
)
410339
anchor_asts = _recover_skill_def(anchor) if anchor is not None else None
411340
if anchor_asts is not None:
@@ -445,7 +374,7 @@ def _doctest_apply(op, *args, **kwargs):
445374

446375
return typing.Annotated[
447376
pydantic.InstanceOf[ty_], # type: ignore
448-
pydantic.BeforeValidator(_validate, json_schema_input_type=typed_enc),
377+
pydantic.BeforeValidator(_validate, json_schema_input_type=schema),
449378
pydantic.PlainSerializer(_serialize_callable, return_type=EncodedFunction),
450379
]
451380

0 commit comments

Comments
 (0)