diff --git a/CHANGELOG.md b/CHANGELOG.md index e49d53d..aba0b0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,25 @@ What changed in PACT, newest first. Each entry says what an author can now write, or what now holds that did not; the commits are listed under it for whoever wants the detail. +## 2026-11 — plugins under the eight laws (11 November) + +The plugins kind (D29), rebased onto the seven primitives, now holds the laws through the +interfaces a plugin implements (D29 consequence 7). + +- **`plugin.implements:`** names the interfaces (`interface: yes` tools, procedures or sets of + documents) whose code the plugin supplies. A name that is not an interface is refused + (`loader/plugin-implements-what-is-not-an-interface`); a plugin counts as an implementation, + so `loader/an-interface-nothing-implements` no longer fires for one, and it may be a choice of + `chosen-by: run-inputs.`. An agent using an interface only absent plugins implement is + warned (`loader/an-interface-only-an-absent-plugin-implements`). +- **Law 1:** `pact discover`'s `effect` and `moves-money` for an agent include every interface its + plugins implement. +- **Law 3:** an interface action a plugin implements that cannot be undone names + `same-request-key:` (`loader/a-plugin-action-with-no-claim`). +- **Law 8:** `Learner.cycle` holds a `plugins` proposal to `pact check` before holding it for a + person (`checking.check_agent_field`); `explode` keeps a plugin's own `config:` whole, so a + package's camel-case setting names no longer make the document unexplodable. + ## 2026-11 — a digest moves only when the tree does, and who answered a wait (13 November) Bud Agent Flow asked whether "a new version of PACT never moves the digest of a tree nobody @@ -810,6 +829,32 @@ which also lists what is still pending at run time. server changed, and a change an approval rests on always stops it (`mcp_bridge.when_it_changed`). `tool.unlisted-actions: each-call-asks-a-person` offers what the server adds behind a person. +### Plugins: code a runtime loads, named, versioned and governed (26 October) + +- **`plugins:` (D29, narrowing R17).** `plugins/.yaml` asks whatever runs these agents + to load a package: `package:` with its registry (`npm:@acme/crm-tools`, + `pypi:acme-crm-tools`), `version:` as a range in that registry's own grammar, the plugin's + own `config:`, `if-it-cannot-be-loaded:` (`refuse-to-start`, the default, or + `run-without-it`) and `applies-to:` (`the-agents-that-name-it` or `every-agent`); an agent + names the plugins it needs on `plugins:`. `S-EXEC`, so adding one needs a person (D23). + `bundle.brings:` and `needs-a-person-to-approve:` gain `plugins`. +- **`pact check` reads the request and never the package** (R5, D17; + `crates/pact-loader/src/plugins.rs`): `loader/plugin-package-has-no-registry`, + `loader/plugin-version-is-not-a-range` (npm ranges by `nodejs-semver`, pypi specifiers by + `pep440_rs`), `loader/plugin-secret-written-in-the-file` (a key naming a token, secret, + password or API key holds `{by-reference: …}`), and the note + `loader/plugin-runs-only-where-loaded` naming the agents a `refuse-to-start` plugin keeps + to the runtimes that can load it. A plugin nothing names is `loader/nothing-points-at-it`; + `plugins//plugin.yaml` is the folder form. `pact discover` lists each agent's plugins + with the default written out; a plugin moves the workspace digest and the doc-digest of a + workflow that reaches its agent, never the contract-digest. +- **Both ports refuse or report what they did not load.** A missing `refuse-to-start` + plugin ends the turn before the first model call with `halted: plugins-not-loaded` and + `turn.run.failed`; a missing `run-without-it` one is named on `unenforced`. A host that + loads plugins passes `run(loaded_plugins=)`. Every lattice gains `plugins`, `unsupported` + on every shipped transport, so the conformance report states it before execution. + `examples/plugin-desk/` is the worked example; `tests/trees/a-desk-that-needs-a-plugin/` + and `tests/trees/a-plugin-with-no-registry/` the refusal and the error. ### Tests over recorded runs (18 October) diff --git a/Cargo.lock b/Cargo.lock index 1ebfb2d..4b27e10 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -182,6 +182,22 @@ dependencies = [ "serde", ] +[[package]] +name = "nodejs-semver" +version = "7.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "938185814d480ac0eff0ac4676d27deef920cbd2b83df83603a873af5b7c8fc9" +dependencies = [ + "smallvec", + "thiserror", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + [[package]] name = "pact-cli" version = "0.1.0" @@ -236,10 +252,12 @@ dependencies = [ "camino", "indexmap", "minijinja", + "nodejs-semver", "pact-diag", "pact-doc", "pact-lang", "pact-schema", + "pep440_rs", "serde_json", "sha2", "unicode-normalization", @@ -256,6 +274,18 @@ dependencies = [ "serde_json", ] +[[package]] +name = "pep440_rs" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31095ca1f396e3de32745f42b20deef7bc09077f918b085307e8eab6ddd8fb9c" +dependencies = [ + "once_cell", + "serde", + "unicode-width", + "unscanny", +] + [[package]] name = "proc-macro2" version = "1.0.107" @@ -329,6 +359,12 @@ dependencies = [ "digest", ] +[[package]] +name = "smallvec" +version = "1.16.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8" + [[package]] name = "syn" version = "3.0.3" @@ -340,6 +376,26 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "thiserror" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "tinyvec" version = "1.12.0" @@ -382,6 +438,12 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +[[package]] +name = "unscanny" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9df2af067a7953e9c3831320f35c1cc0600c30d44d9f7a12b01db1cd88d6b47" + [[package]] name = "version_check" version = "0.9.5" diff --git a/Cargo.toml b/Cargo.toml index 8b696b7..ef6af46 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -48,6 +48,11 @@ serde_json = { version = "1", features = ["preserve_order"] } thiserror = "2" unicode-normalization = "0.1" sha2 = "0.10" +# D29: a plugin's `version:` is held to its own registry's grammar — npm ranges +# as node-semver reads them, pypi specifiers as PEP 440 writes them — rather than +# to a third grammar PACT would have to keep in step with both. +nodejs-semver = { version = "7", default-features = false } +pep440_rs = { version = "0.7", default-features = false } yaml-rust2 = "0.11" # 96 §C. The checker for `template-language: jinja`. Pinned EXACTLY, and with # `unstable_machinery`, because what `holes.rs` reads is the compiled diff --git a/README.md b/README.md index 7803257..64ad79b 100644 --- a/README.md +++ b/README.md @@ -650,8 +650,8 @@ never runs a line of yours. ## What you can express -**54 kinds, 384 fields.** 262 are `core` — what a non-technical author writes. -122 are `expert`, and none of them is required for any core capability. +**64 kinds, 545 fields.** 327 are `core` — what a non-technical author writes. +218 are `expert`, and none of them is required for any core capability. @@ -666,6 +666,7 @@ never runs a line of yours. +
CapabilityYou writeWhat it buys
Teamsteam:, teamwork:Delegation, quorum, races, budget shares — eight worked patterns ship
Memoryremembers:A variable a run reads into a call and writes back from one
Exact logicprograms/WebAssembly in a locked room, with fuel and a consent gate
Runtime pluginsplugins/*.yaml, plugins:Code your runtime loads, named with a version range — refused or reported where it cannot be loaded
How you knowevals/The correctness oracle for portability, model swaps and learning
Self-improvementlearning.yamlChanges arrive as reviewable diffs, class-gated, never silent
@@ -760,8 +761,9 @@ note: 'refund-policy' holds 5 written rules under `## Rules` — lines a person One folder, loaded by the Rust CLI, executes over **all seven targets** and produces *byte-identical traces, tool sequences and model-call counts*. The claim is bounded and the bound is written down: it covers `instructions:`, `tools:`, -`skills:`, `knowledge:`, `team:`, `answers-with:`, the `loop:` and the `limits:` -ceilings — and **not** the thirteen governance keys the Node port reports on +`skills:`, `knowledge:`, `team:`, `answers-with:`, the `loop:`, the `limits:` +ceilings and whether a `plugins:` line stops the agent before it starts — and +**not** the thirteen governance keys the Node port reports on `unenforced`, nor the documents under `knowledge/`, which nothing on that port can look anything up in and which it names on `unretrieved` for every set an answer did not come from. See @@ -773,7 +775,7 @@ inside the claim and reported anyway, since the names are offered to the model and asking one only comes back as an error there — and every `limits:` key that port does not read gets its own line under a `limits.` prefix. -The conformance suite compares 30 agents × 7 targets = **180 pairs, 0 +The conformance suite compares 31 agents × 7 targets = **186 pairs, 0 divergences**, with every exclusion declared rather than discovered. Each target takes the framework's **lowest** seam, so none of them owns the loop: @@ -865,8 +867,8 @@ for things a framework would have let you do. Declaring `spends-money: yes` obliges you to say what makes two calls the same call. That is the point, and it is still friction. -**There is a vocabulary to learn.** Not code, but 54 kinds and 384 fields. The -core is 185 of them and the diagnostics name the exact line to type, but "no +**There is a vocabulary to learn.** Not code, but 64 kinds and 545 fields. The +core is 327 of them and the diagnostics name the exact line to type, but "no code" is not "nothing to learn". **The eval suite is work, and it is not optional.** Every portability claim is @@ -996,13 +998,13 @@ framework is an extra. | | | |---|---| -| **Design** | Thesis, 28 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | +| **Design** | Thesis, 29 binding decisions, FRD (120 requirements), 60-row refusal ledger — complete | | **Research** | 14 source-grounded studies, ~15,750 lines, over 140 repos (~15 GB) + 57 papers | -| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **4969 tests (1648 Rust + 3321 adapter), clippy clean, TypeScript type-checked** | +| **Code** | Loader, diagnostics, schema engine, CLI, harness, resolver, evals, SLO — **5013 tests (1673 Rust + 3340 adapter), clippy clean, TypeScript type-checked** | | **Adapters** | **All 7 named targets**, proven against one shared conformance suite | ```bash -./scripts/test-all.sh # 4969 tests, Rust + 7 adapters, fully offline +./scripts/test-all.sh # 5013 tests, Rust + 7 adapters, fully offline ``` **What is still owed** is not hidden — it lives in @@ -1021,7 +1023,7 @@ SLOs **report**; they do not stop a run. That is deliberate and | | | |---|---| | [`docs/00-THESIS.md`](docs/00-THESIS.md) | The argument, goals, objectives, acceptance criteria | -| [`docs/01-DECISIONS.md`](docs/01-DECISIONS.md) | 28 binding decisions — **read this before proposing anything** | +| [`docs/01-DECISIONS.md`](docs/01-DECISIONS.md) | 29 binding decisions — **read this before proposing anything** | | [`docs/30-FRD.md`](docs/30-FRD.md) | 120 functional requirements, each traced to its justification | | [`docs/97-PRIMITIVES.md`](docs/97-PRIMITIVES.md) | The seven building blocks every keyword reduces to, and the eight laws | | [`docs/99-THE-PACT-LANGUAGE.md`](docs/99-THE-PACT-LANGUAGE.md) | The PACT language and reactions: the design, and where the build differs from it | diff --git a/adapters/out-of-tree/echo_adapter/transport.py b/adapters/out-of-tree/echo_adapter/transport.py index e821105..3a7738b 100644 --- a/adapters/out-of-tree/echo_adapter/transport.py +++ b/adapters/out-of-tree/echo_adapter/transport.py @@ -62,6 +62,7 @@ def lattice(self) -> dict[str, str]: # transport that takes the LOOP away — `a2a_transport.py` — is # `unsupported` there. "connected_tools": "emulated", + "plugins": "unsupported", } #: Whether ANYTHING can put a price on what a call here carried. Nothing diff --git a/adapters/python/src/pact_adapters/checking.py b/adapters/python/src/pact_adapters/checking.py index d010488..7557165 100644 --- a/adapters/python/src/pact_adapters/checking.py +++ b/adapters/python/src/pact_adapters/checking.py @@ -72,17 +72,23 @@ def _agent_key(document: Mapping[str, Any], agent: str) -> str: raise Uncheckable(f"the document has no agent {agent!r} to put the change in") -def check_instructions( - document: Mapping[str, Any], agent: str, text: str, source: "str | Path | None" = None +def check_agent_field( + document: Mapping[str, Any], + agent: str, + field: str, + value: Any, + source: "str | Path | None" = None, ) -> list[str]: - """`pact check`'s errors once `agent`'s `instructions:` become `text`. + """`pact check`'s errors once `agent`'s `field:` becomes `value`. Through the document when it can be written out; through a copy of the author's own tree when it carries files the document only references (a - skill's scripts, a program's body), which is the worked example's case.""" + skill's scripts, a program's body), which is the worked example's case. + `instructions` is prose and lands in the agent's `instructions.md`; every + other field is a setting and lands in its `agent.yaml`.""" agent = _agent_key(document, agent) try: - return check_document(with_instructions(document, agent, text)) + return check_document(with_field(document, agent, field, value)) except CarriesPayloads: pass binary = pact_binary() @@ -96,7 +102,10 @@ def check_instructions( with tempfile.TemporaryDirectory(prefix="pact-proposed-") as scratch: root = Path(scratch) / "tree" shutil.copytree(source, root) - _write_instructions(root, agent, text) + if field == "instructions": + _write_instructions(root, agent, value) + else: + _write_setting(root, agent, field, value) return _checked(binary, root) @@ -164,6 +173,22 @@ def as_a_folder(root: Path, binary: "Path | None" = None) -> bool: return True +def _write_setting(root: Path, agent: str, field: str, value: Any) -> None: + """Put `field: value` where `agent`'s settings live in a copied tree. + + A tree written in the PACT language is written out as its folder first, as + `_write_instructions` does, so the setting lands in the agent's own file.""" + if pact_binary() is not None: + as_a_folder(root) + folder = root / "agents" / agent + settings = folder / "agent.yaml" if folder.is_dir() else root / "agents" / f"{agent}.yaml" + body = yaml.safe_load(settings.read_text()) if settings.exists() else {} + body = body if isinstance(body, dict) else {} + body[field] = value + settings.parent.mkdir(parents=True, exist_ok=True) + settings.write_text(yaml.safe_dump(body, sort_keys=False, allow_unicode=True)) + + def _write_instructions(root: Path, agent: str, text: str) -> None: """Put `text` where `agent`'s instructions live in a copied tree. @@ -220,11 +245,11 @@ def _checked(binary: Path, root: Path) -> list[str]: ] -def with_instructions(document: Mapping[str, Any], agent: str, text: str) -> dict[str, Any]: - """A copy of `document` with one agent's `instructions:` replaced.""" +def with_field(document: Mapping[str, Any], agent: str, field: str, value: Any) -> dict[str, Any]: + """A copy of `document` with one agent's `field:` replaced.""" proposed = copy.deepcopy(dict(document)) agents = proposed.setdefault("agents", {}) entry = dict(agents.get(agent) or {}) - entry["instructions"] = text + entry[field] = value agents[agent] = entry return proposed diff --git a/adapters/python/src/pact_adapters/exploding.py b/adapters/python/src/pact_adapters/exploding.py index 4456bf3..e9212ff 100644 --- a/adapters/python/src/pact_adapters/exploding.py +++ b/adapters/python/src/pact_adapters/exploding.py @@ -63,6 +63,20 @@ STAYS_INLINE = frozenset({"description"}) +#: Settings PACT hands over as written and never reads as fields, by +#: `(collection, field)`: a plugin's own `config:` (`spec/schema.yaml` +#: `plugin.config`, `map of anything`). Its keys are the plugin's names and not +#: PACT's (`searchMaxResults` is a setting of the package), so the value stays +#: in the entry's settings file whole: never split into folders, and never held +#: to the portable key alphabet, since none of its keys becomes a file name. +OPAQUE: frozenset[tuple[str, str]] = frozenset({("plugins", "config")}) + + +def _opaque(at: tuple[str, ...], key: str) -> bool: + """Whether `key` under the path `at` is one of the `OPAQUE` values.""" + return len(at) >= 2 and (at[-2], key) in OPAQUE + + #: The key the loader puts on a payload folder — a directory it carried #: verbatim rather than read as fields. PAYLOAD = "$payload" @@ -148,7 +162,8 @@ def _unportable(node: Any, at: tuple[str, ...] = ()) -> dict[str, str]: why = why_not_portable(str(key)) if why: found[".".join((*at, str(key)))] = why - found.update(_unportable(value, (*at, str(key)))) + if not _opaque(at, str(key)): + found.update(_unportable(value, (*at, str(key)))) elif isinstance(node, list): for i, value in enumerate(node): found.update(_unportable(value, (*at, str(i)))) @@ -202,7 +217,9 @@ def explode(document: Mapping[str, Any], into: "str | Path") -> Explosion: return said -def _write_map(node: dict, where: Path, said: Explosion, *, top: bool) -> None: +def _write_map( + node: dict, where: Path, said: Explosion, *, top: bool, at: tuple[str, ...] = () +) -> None: """One mapping as a directory: prose to `.md`, the rest to a settings file. Nested maps become directories with their own settings file, which is the @@ -215,7 +232,7 @@ def _write_map(node: dict, where: Path, said: Explosion, *, top: bool) -> None: path.write_text(value if value.endswith("\n") else value + "\n") said.wrote.append(str(path)) continue - if isinstance(value, dict) and value and all( + if not _opaque(at, key) and isinstance(value, dict) and value and all( isinstance(v, dict) for v in value.values() ): # A map OF maps is a folder of things — `agents/`, `tools/`. A map of @@ -226,7 +243,7 @@ def _write_map(node: dict, where: Path, said: Explosion, *, top: bool) -> None: for name, inner in value.items(): child = folder / name child.mkdir(parents=True, exist_ok=True) - _write_map(dict(inner), child, said, top=False) + _write_map(dict(inner), child, said, top=False, at=(*at, key, name)) continue settings[key] = value diff --git a/adapters/python/src/pact_adapters/harness.py b/adapters/python/src/pact_adapters/harness.py index f6329e5..0916b99 100644 --- a/adapters/python/src/pact_adapters/harness.py +++ b/adapters/python/src/pact_adapters/harness.py @@ -62,7 +62,7 @@ from .facts import TURN_ENDS, Fact, Facts from .holes import fill from .interceptors import Chain -from .ir import AgentSpec, SkillSpec +from .ir import RUN_WITHOUT_IT, AgentSpec, PluginSpec, SkillSpec from .limits import RAN_OUT, Action, Ceiling, Limits, Meter, Reached, money, step_ceiling from .limits import _CEILING_FIELDS as CEILING_FIELDS from .loops import DONE, Does, Loop, LoopError, Phase @@ -635,6 +635,11 @@ async def _ask( # before the first call, because the failure otherwise is `error: no tool # named ...`, which reads as a mistake in the author's own file. "run_program", + # Which plugins the host loaded (D29). The DOCUMENT says which plugins an + # agent needs, which versions will do and what to do without one, and all + # of that reaches `AgentSpec.plugins`; loading a package is something only + # the process can do, and this harness loads none. + "loaded_plugins", # Which reactions are being carried out above this run (docs/99 §5): a # consulted agent's run is handed the names, and none of them fires again # beneath itself. `delegate_by_running` passes it down — a host passes @@ -643,6 +648,37 @@ async def _ask( }) +def _plugin_named(p: PluginSpec) -> str: + """`'web-search' (`npm:@acme/web-search`, versions `^0.2.0`)`.""" + return f"'{p.name}' (`{p.package}`, versions `{p.version}`)" + + +def _cannot_start_without(needed: "Sequence[PluginSpec]") -> str: + """Why the turn ended before it began, and the two ways out. + + Word for word the second port's `cannotStartWithout`; the conformance + driver compares the two. + """ + named = [_plugin_named(p) for p in needed] + listed = named[0] if len(named) == 1 else f"{', '.join(named[:-1])} and {named[-1]}" + registries = " or ".join(dict.fromkeys(f"`{p.registry}:`" for p in needed)) + return ( + f"This agent cannot start: it needs {listed}, and nothing running it loaded " + f"{'it' if len(needed) == 1 else 'them'}. fix: run it where a runtime loads " + f"{registries} packages, or write `if-it-cannot-be-loaded: run-without-it` in " + f"the plugin's file under `plugins/` if the agent can do its job without it." + ) + + +def _ran_without(p: PluginSpec) -> str: + """The `unenforced` line for a plugin the agent started without.""" + return ( + f"plugin {_plugin_named(p)} was not loaded by whatever runs this agent, so " + f"nothing it adds is here and the agent ran without it. fix: run it where a " + f"runtime loads `{p.registry}:` packages." + ) + + def _and_list(names: "Sequence[str]") -> str: """`a`, `a and b`, `a, b and c` — for a sentence a customer reads.""" quoted = [f"'{n}'" for n in names] @@ -687,6 +723,11 @@ async def run( #: `RunResult.unretrieved`. `None` means nothing retrieved anything, which is #: the honest state on a machine with no index. retrieved_by: "Mapping[str, Sequence[str]] | None" = None, + #: What the host loaded for this agent, as plugin name to the version it + #: loaded (D29). PACT loads no plugin itself; a host that does hands this + #: in, having held each package to its line's range. `None` means nothing + #: loaded anything, which is the honest state of every transport here. + loaded_plugins: "Mapping[str, str] | None" = None, approved: frozenset[str] = frozenset(), bus: Bus | None = None, chain: Chain | None = None, @@ -1640,6 +1681,23 @@ def after_a_yes() -> str: # by `ir.ToolSpec`, and a tool here answers in TEXT, so nothing on this # runtime holds a result to it. Said, for `bind:`'s reason one line up. result.unenforced = result.unenforced + _unchecked_answers(spec) + _unheld(spec) + # D29. A plugin is code the host loads and this harness never does, so on + # every shipped transport each one is missing — and that must not be the + # SILENT state. One the agent may run without is named on `unenforced`, in + # the same door as every other line this run did not carry out. One it may + # not run without ends the turn HERE, before a single model call: an agent + # written to rely on code it did not get is not the agent that was reviewed, + # and paying for its first answer would look like working. + missing = [p for p in spec.plugins if p.name not in (loaded_plugins or {})] + result.unenforced = result.unenforced + tuple( + _ran_without(p) for p in missing if p.if_cannot_load == RUN_WITHOUT_IT + ) + needed = [p for p in missing if p.if_cannot_load != RUN_WITHOUT_IT] + if needed: + result.halted = "plugins-not-loaded" + result.output = _cannot_start_without(needed) + bus.emit("turn.run.failed", at=(0,), reason="plugins-not-loaded") + return result # A7. PACT does not retrieve, so a set of documents this run never consulted # is the ordinary state on a machine with no index — and it must not be the # SILENT state, because the agent then answers from what the model already diff --git a/adapters/python/src/pact_adapters/ir.py b/adapters/python/src/pact_adapters/ir.py index f275a34..a9536ca 100644 --- a/adapters/python/src/pact_adapters/ir.py +++ b/adapters/python/src/pact_adapters/ir.py @@ -345,6 +345,42 @@ class KnowledgeSpec: reranked_by: str = "" +#: What `if-it-cannot-be-loaded:` means when the author left it out. +REFUSE_TO_START = "refuse-to-start" +#: The other answer: the agent starts without the plugin and every run says so. +#: Anything that is not this refuses, so a word the checker never let through +#: cannot start an agent the author meant to stop. +RUN_WITHOUT_IT = "run-without-it" + + +@dataclass(frozen=True) +class PluginSpec: + """One plugin this agent needs (D29): a request for code its runtime loads. + + PACT loads nothing, so what reaches the harness is the request and never the + code — which package, which versions will do, and what the agent does where + it was not loaded. Whether it WAS loaded is a fact about the process, handed + in by the host as `run(loaded_plugins=)`, the way `retrieved_by` is. + + The plugin's own `config:` is not carried: nothing in this harness loads a + plugin, so nothing here could hand the settings to one. A host that loads + plugins reads them off the document it was handed, beside this. + """ + + name: str + #: `:` — `npm:@acme/crm-tools`, `pypi:acme-crm`. + package: str + #: A range in that registry's own grammar; `pact check` holds it there. + version: str + #: `refuse-to-start` or `run-without-it`, the default written out. + if_cannot_load: str = REFUSE_TO_START + + @property + def registry(self) -> str: + """The word in front of the colon: which runtimes can load it at all.""" + return self.package.split(":", 1)[0].strip() + + @dataclass(frozen=True) class ProgramSpec: """One carried program, as the run is told about it (P6/P7). @@ -803,6 +839,11 @@ class AgentSpec: #: same way `skills` is — off the agent's own `uses:` line — so a corpus #: nobody named reaches no run, exactly like a skill nobody named. knowledge: tuple[KnowledgeSpec, ...] = () + #: The plugins this agent needs (D29): the ones its own `plugins:` line names + #: and every one whose `applies-to:` is `every-agent`, sorted like `tools`. + #: A run checks them against what its host loaded before the first model + #: call, and refuses or reports each one that is missing. + plugins: tuple[PluginSpec, ...] = () #: The workflows this agent may start as tools (N2): the names on its own #: `uses:` line that are workflows, sorted like `tools`. `pact check` lets a #: workflow be named there only when its `published-as:` lists `tool`, so @@ -980,6 +1021,7 @@ def from_document( tools=tools, skills=skills, knowledge=knowledge, + plugins=_plugins_needed_by(doc, a), workflows=tuple( n for n in sorted((doc.get("workflows") or {})) @@ -1095,6 +1137,27 @@ def available(agent: "AgentSpec", condition: str) -> bool: ) +def _plugins_needed_by(doc: dict[str, Any], agent: dict[str, Any]) -> tuple[PluginSpec, ...]: + """The plugins `agent` names, plus every one that applies to every agent. + + The same reading `pact discover` publishes (`pact_loader::plugins:: + needed_by`), so the inventory a runtime chose this agent by and the run it + then starts agree about what it needs. + """ + named = set(_as_list(agent.get("plugins"))) + return tuple( + PluginSpec( + name=n, + package=_text(p.get("package", "")), + version=_text(p.get("version", "")), + if_cannot_load=_text(p.get("if-it-cannot-be-loaded", "")) or REFUSE_TO_START, + ) + for n, p in sorted((doc.get("plugins") or {}).items()) + if isinstance(p, Mapping) + and (n in named or _text(p.get("applies-to", "")) == "every-agent") + ) + + def _questions_this_agent_puts(doc: dict[str, Any], agent: dict[str, Any]) -> set[str]: """Every question name this agent can reach, by any line that names one. diff --git a/adapters/python/src/pact_adapters/learning.py b/adapters/python/src/pact_adapters/learning.py index a04bdd6..f6366fa 100644 --- a/adapters/python/src/pact_adapters/learning.py +++ b/adapters/python/src/pact_adapters/learning.py @@ -32,7 +32,9 @@ from pathlib import Path from typing import Any, Callable, Mapping -from .checking import Uncheckable, check_instructions +import yaml + +from .checking import Uncheckable, check_agent_field from .evals import Case, CaseOutcome, Verdict, check, verdict from .harness import run from .ir import AgentSpec @@ -51,8 +53,26 @@ class Risk: HIGH_RISK_FIELDS = ( "uses", "tools", "team", "model", "policy", "needs-approval-before", "limits", "needs", "evals", "learning", "answers-with", "run-inputs", + "plugins", ) +def _names_in(text: str) -> list[str]: + """The names a list-valued proposal's `after` carries, written either as + YAML (`- web-search`, `[web-search]`) or as a comma list, the two ways a + proposal for a list field is written.""" + read = yaml.safe_load(text) if text.strip() else [] + if isinstance(read, list): + return [str(n).strip() for n in read if str(n).strip()] + return [n.strip() for n in str(read or "").split(",") if n.strip()] + + +#: Fields no cycle applies whose proposal is held to `pact check` before it is +#: held for a person (law 8), each with how its `after` text becomes the value +#: written into the document. `plugins` (D29): which code the agent's runtime +#: loads is a change to what it can do, and a name no plugin file defines is a +#: tree the checker refuses. +CHECKED_BEFORE_HELD: dict[str, Any] = {"plugins": _names_in} + #: What each of `may-improve-on-its-own:`'s four choices names, as a field of a #: spec document. The schema's words are written for whoever reads the file; the #: classifier works on field names, and one mapping in one place is what keeps @@ -1496,6 +1516,16 @@ def _decide(self, proposal: Proposal, transport_for, score=None) -> Outcome: # reviewer and, for `uses:`, simply false: it is the skills the agent # may read. What is true is narrower, and is said below. if cls.needs_a_person: + # LAW 8 for a field no cycle applies: a person is asked to + # approve a change only once it passes the checker the author's + # own tree is held to, so nobody is asked to say yes to a tree + # that does not load. + if proposal.field in CHECKED_BEFORE_HELD: + refused = self._refused_by_the_checker( + proposal, CHECKED_BEFORE_HELD[proposal.field](proposal.after), cls + ) + if refused is not None: + return refused return Outcome( False, f"held for review: {cls.reason}", cls, unmeasured=self._unmeasured(), held=True, @@ -1682,28 +1712,10 @@ def _decide(self, proposal: Proposal, transport_for, score=None) -> Outcome: # paid one, so a change that breaks a # rule the author's own tree is held to never takes effect — whoever, or # whatever, proposed it. - if self.document is not None and proposal.field == "instructions": - try: - refused = check_instructions( - self.document, - self.agent_key or self.spec.key or self.spec.name, - proposal.after, - source=self.spec.workspace or None, - ) - except Uncheckable as why: - # Not a refusal: said on every outcome, as a ceiling nobody - # could hold is (`_unmeasured`). - refused = [] - self.unchecked = f"law 8: this change was not held to `pact check`: {why}" - if refused: - self.rejected.append(proposal) - return Outcome( - False, - "the proposed change does not pass `pact check`, the checker the " - "author's own tree is held to: " + "; ".join(refused[:3]), - cls, - unmeasured=self._unmeasured(), - ) + if proposal.field == "instructions": + refused = self._refused_by_the_checker(proposal, proposal.after, cls) + if refused is not None: + return refused if score is None: before_v = self._score(self.spec, self.holdout, transport_for) @@ -1811,6 +1823,37 @@ def _refuse( "if-unsure": "if_unsure", } + def _refused_by_the_checker( + self, proposal: Proposal, value: Any, cls: Classification + ) -> "Outcome | None": + """The refusal when `pact check` refuses the document with `proposal` + written into it (law 8, docs/98 WP-15); `None` when it passes, or when + there is no document or checker to hold it to — which is said on every + outcome through `unchecked`, as a ceiling nobody could hold is.""" + if self.document is None: + return None + try: + refused = check_agent_field( + self.document, + self.agent_key or self.spec.key or self.spec.name, + proposal.field, + value, + source=self.spec.workspace or None, + ) + except Uncheckable as why: + refused = [] + self.unchecked = f"law 8: this change was not held to `pact check`: {why}" + if not refused: + return None + self.rejected.append(proposal) + return Outcome( + False, + "the proposed change does not pass `pact check`, the checker the " + "author's own tree is held to: " + "; ".join(refused[:3]), + cls, + unmeasured=self._unmeasured(), + ) + def _with(self, p: Proposal) -> AgentSpec: """The candidate this proposal describes — the spec that gets scored. diff --git a/adapters/python/src/pact_adapters/pydantic_ai_registry.yaml b/adapters/python/src/pact_adapters/pydantic_ai_registry.yaml index a8700a6..80fdbef 100644 --- a/adapters/python/src/pact_adapters/pydantic_ai_registry.yaml +++ b/adapters/python/src/pact_adapters/pydantic_ai_registry.yaml @@ -182,6 +182,11 @@ fields: spec-file: >- no field for rules that hide, stop or redirect. Pydantic AI's nearest shape is a capability, which is a Python object and not a sentence + agent.plugins: + construct: "nothing: the host installs and loads each package a plugin names, attaches what it provides, and hands the harness what it loaded (run(loaded_plugins=)); a plugin it did not load refuses the run or is named on unenforced" + outcome: loss + rows: [X11, T10] + spec-file: "no field for a package the runtime has to load, nor for the versions that will do" agent.reactions: construct: "nothing: Pydantic AI has no moment-and-handler shape; the PACT harness stops at six moments of its own loop and carries each reaction out there (reactions.Reactor)" diff --git a/adapters/python/src/pact_adapters/transports/a2a_transport.py b/adapters/python/src/pact_adapters/transports/a2a_transport.py index 230f554..167a623 100644 --- a/adapters/python/src/pact_adapters/transports/a2a_transport.py +++ b/adapters/python/src/pact_adapters/transports/a2a_transport.py @@ -150,6 +150,7 @@ def lattice(self) -> dict[str, str]: "streaming": "unsupported", "durable_resume": "unsupported", "connected_tools": "unsupported", + "plugins": "unsupported", } def unenforced(self) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/anthropic_transport.py b/adapters/python/src/pact_adapters/transports/anthropic_transport.py index 6b04cb9..5527844 100644 --- a/adapters/python/src/pact_adapters/transports/anthropic_transport.py +++ b/adapters/python/src/pact_adapters/transports/anthropic_transport.py @@ -161,6 +161,7 @@ def lattice(self) -> dict[str, str]: # PACT's own client above the seam, which is `emulated`; `mock.py` # states the rule once for every target that shares it. "connected_tools": "emulated", + "plugins": "unsupported", } def _message(self, history: list[dict[str, Any]], system: str = "") -> Message: diff --git a/adapters/python/src/pact_adapters/transports/autogen_transport.py b/adapters/python/src/pact_adapters/transports/autogen_transport.py index e725b91..2e1f019 100644 --- a/adapters/python/src/pact_adapters/transports/autogen_transport.py +++ b/adapters/python/src/pact_adapters/transports/autogen_transport.py @@ -243,6 +243,7 @@ def lattice(self) -> dict[str, str]: # server through PACT's own client above the framework, which is # what `emulated` means here; `mock.py` states the rule once. "connected_tools": "emulated", + "plugins": "unsupported", } def apply_settings(self, settings: dict[str, Any]) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/langchain_transport.py b/adapters/python/src/pact_adapters/transports/langchain_transport.py index d0aacf6..7433e4d 100644 --- a/adapters/python/src/pact_adapters/transports/langchain_transport.py +++ b/adapters/python/src/pact_adapters/transports/langchain_transport.py @@ -194,6 +194,7 @@ def lattice(self) -> dict[str, str]: # framework. `mock.py` states the rule once for every target that # shares it. "connected_tools": "emulated", + "plugins": "unsupported", } def apply_settings(self, settings: dict[str, Any]) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/langgraph_transport.py b/adapters/python/src/pact_adapters/transports/langgraph_transport.py index fcff66a..ae9e784 100644 --- a/adapters/python/src/pact_adapters/transports/langgraph_transport.py +++ b/adapters/python/src/pact_adapters/transports/langgraph_transport.py @@ -221,6 +221,7 @@ def lattice(self) -> dict[str, str]: # framework, which is what `emulated` means here; `mock.py` states # the rule once for every target that shares it. "connected_tools": "emulated", + "plugins": "unsupported", } def apply_settings(self, settings: dict[str, Any]) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/mock.py b/adapters/python/src/pact_adapters/transports/mock.py index c23f67d..5a6d6e8 100644 --- a/adapters/python/src/pact_adapters/transports/mock.py +++ b/adapters/python/src/pact_adapters/transports/mock.py @@ -71,6 +71,13 @@ def lattice(self) -> dict[str, str]: # to somebody else, and the TypeScript port has no PACT client at # all — which is what keeps this column from being decoration. "connected_tools": "emulated", + # Whether a `plugins:` line can be honoured here (D29): `native` where + # the runtime loads packages from a registry a plugin names, and says + # so through `run(loaded_plugins=)`. `unsupported` on every transport + # in this package, because the harness loads nothing — so a plugin an + # agent cannot start without halts it `plugins-not-loaded` before the + # first call, and one it can is named on `unenforced`. + "plugins": "unsupported", } async def model_call( diff --git a/adapters/python/src/pact_adapters/transports/ollama_transport.py b/adapters/python/src/pact_adapters/transports/ollama_transport.py index 6efcabf..fb9fcd2 100644 --- a/adapters/python/src/pact_adapters/transports/ollama_transport.py +++ b/adapters/python/src/pact_adapters/transports/ollama_transport.py @@ -240,6 +240,7 @@ def lattice(self) -> dict[str, str]: # runs the same document as a connected one. `mock.py` states the # rule once for every target that shares it. "connected_tools": "emulated", + "plugins": "unsupported", } def apply_settings(self, settings: dict[str, Any]) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/openai_agents_transport.py b/adapters/python/src/pact_adapters/transports/openai_agents_transport.py index 95ca28d..25f23d8 100644 --- a/adapters/python/src/pact_adapters/transports/openai_agents_transport.py +++ b/adapters/python/src/pact_adapters/transports/openai_agents_transport.py @@ -199,6 +199,7 @@ def lattice(self) -> dict[str, str]: # the framework, which is what `emulated` means; `mock.py` states # the rule once for every target that shares it. "connected_tools": "emulated", + "plugins": "unsupported", } def apply_settings(self, settings: dict[str, Any]) -> tuple[str, ...]: diff --git a/adapters/python/src/pact_adapters/transports/pydantic_ai_transport.py b/adapters/python/src/pact_adapters/transports/pydantic_ai_transport.py index 5a705c0..37a5dac 100644 --- a/adapters/python/src/pact_adapters/transports/pydantic_ai_transport.py +++ b/adapters/python/src/pact_adapters/transports/pydantic_ai_transport.py @@ -161,6 +161,7 @@ def lattice(self) -> dict[str, str]: # changed with the installed extras would make the published matrix # a property of one laptop instead of a property of the runtime. "connected_tools": "native", + "plugins": "unsupported", } def _respond(self, messages: list[Any], info: AgentInfo) -> ModelResponse: diff --git a/adapters/python/tests/test_a_document_explodes_back_into_its_tree.py b/adapters/python/tests/test_a_document_explodes_back_into_its_tree.py index 058b75b..235afd4 100644 --- a/adapters/python/tests/test_a_document_explodes_back_into_its_tree.py +++ b/adapters/python/tests/test_a_document_explodes_back_into_its_tree.py @@ -40,6 +40,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) from pact_adapters.exploding import ( # noqa: E402 + OPAQUE, CarriesPayloads, ExplodeRefused, explode, @@ -120,6 +121,48 @@ def test_every_payload_free_example_survives_the_round_trip( ) +def test_a_plugins_own_settings_survive_the_round_trip_whole(tmp_path: Path) -> None: + """A plugin's `config:` keys are the package's names (`searchMaxResults`), + not PACT field keys, so they are neither refused nor split into folders — + which is what lets law 8 hold a proposal about the plugin-desk to `pact + check` at all.""" + import copy + + original = load(EXAMPLES / "plugin-desk") + nested = copy.deepcopy(original) + nested["plugins"]["web-search"]["config"] = { + "searchMaxResults": 5, + "fetch": {"timeoutMs": 3000}, + "Ranking": {"boostRecent": True}, + } + for document in (original, nested): + root = tmp_path / f"exploded-{len(str(document))}" + explode(document, root) + assert not (root / "plugins" / "web-search" / "config").exists() + accepted = subprocess.run( + [str(PACT_BIN), "check", str(root)], capture_output=True, text=True + ) + assert accepted.returncode == 0, f"{accepted.stdout}{accepted.stderr}" + assert _settled(load(root)) == _settled(document) + + +def test_every_opaque_value_is_one_the_schema_leaves_open() -> None: + """`OPAQUE` names only fields `spec/schema.yaml` types `map of anything`, + under the collection whose entries are that kind.""" + import yaml + + schema = yaml.safe_load((REPO / "spec" / "schema.yaml").read_text()) + groups = schema["groups"] + collections = { + field: definition["type"].split("group:", 1)[1] + for field, definition in groups["workspace"]["fields"].items() + if str(definition.get("type", "")).startswith("map of group:") + } + for collection, field in OPAQUE: + kind = collections[collection] + assert groups[kind]["fields"][field]["type"] == "map of anything", (collection, field) + + def test_the_residual_is_exactly_trailing_whitespace_on_prose( tmp_path: Path, ) -> None: diff --git a/adapters/python/tests/test_a_plugin_a_cycle_proposes_is_checked_then_held.py b/adapters/python/tests/test_a_plugin_a_cycle_proposes_is_checked_then_held.py new file mode 100644 index 0000000..1e67703 --- /dev/null +++ b/adapters/python/tests/test_a_plugin_a_cycle_proposes_is_checked_then_held.py @@ -0,0 +1,78 @@ +"""Law 8 for a plugin (D29 under docs/97-PRIMITIVES.md §4): a learning cycle +that proposes changing which plugins an agent is loaded with is held to the +checker the author's own tree is held to, and only a change that passes it is +held for a person. Nobody is asked to approve a tree that does not load.""" + +from __future__ import annotations + +import copy +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from pact_adapters.checking import check_agent_field # noqa: E402 +from pact_adapters.ir import AgentSpec # noqa: E402 +from pact_adapters.learning import Learner, Proposal # noqa: E402 +from trees import shown # noqa: E402 + +REPO = Path(__file__).resolve().parents[3] +DESK = REPO / "examples" / "plugin-desk" +AGENT = "research-desk" + +#: A second plugin the workspace defines, so a proposal has one to add. +CRM = { + "description": "The customer-records client the desk's runtime loads.", + "package": "npm:@acme/crm-tools", + "version": "^1.4.0", + "if-it-cannot-be-loaded": "run-without-it", +} + + +def _document() -> dict: + doc = copy.deepcopy(shown(DESK)) + doc.setdefault("plugins", {})["crm-tools"] = CRM + return doc + + +def _learner(doc: dict) -> Learner: + spec = AgentSpec.from_document(doc, AGENT, source=DESK) + return Learner.from_document(doc, spec, tools={}) + + +def _never_scored(spec): # a scoring run would reach here; it must not + raise AssertionError("scored a change no cycle can apply") + + +def test_the_field_is_written_into_the_document_and_checked() -> None: + doc = _document() + assert check_agent_field(doc, AGENT, "plugins", ["web-search", "crm-tools"]) == [] + refused = check_agent_field(doc, AGENT, "plugins", ["web-search", "no-such-plugin"]) + assert refused and any("no-such-plugin" in r for r in refused), refused + + +def test_a_plugin_the_workspace_defines_passes_the_checker_and_is_held_for_a_person() -> None: + learner = _learner(_document()) + out = learner.cycle(Proposal("plugins", "- web-search", "- web-search\n- crm-tools"), _never_scored) + assert not out.applied + assert out.held, out.reason + assert out.reason.startswith("held for review"), out.reason + assert learner.unchecked is None or "plugins" not in learner.unchecked + + +def test_a_plugin_nothing_defines_is_refused_in_the_checkers_words_and_never_held() -> None: + learner = _learner(_document()) + out = learner.cycle(Proposal("plugins", "- web-search", "- web-search\n- no-such-plugin"), _never_scored) + assert not out.applied + assert not out.held, out.reason + assert "does not pass `pact check`" in out.reason, out.reason + assert "no-such-plugin" in out.reason, out.reason + + +def test_a_comma_list_is_the_same_proposal() -> None: + learner = _learner(_document()) + out = learner.cycle(Proposal("plugins", "web-search", "web-search, no-such-plugin"), _never_scored) + assert not out.held and "no-such-plugin" in out.reason, out.reason + out = learner.cycle(Proposal("plugins", "web-search", "web-search, crm-tools"), _never_scored) + assert out.held, out.reason diff --git a/adapters/python/tests/test_a_plugin_nobody_loaded_stops_the_run_before_it_starts.py b/adapters/python/tests/test_a_plugin_nobody_loaded_stops_the_run_before_it_starts.py new file mode 100644 index 0000000..85c149a --- /dev/null +++ b/adapters/python/tests/test_a_plugin_nobody_loaded_stops_the_run_before_it_starts.py @@ -0,0 +1,130 @@ +"""D29: a plugin an agent may not start without, on a runtime that did not load it. + +`plugins/.yaml` asks whatever runs the agent to load a package for it. The +harness loads nothing, so on every transport this repository ships the plugin is +missing; the host that does load one says so through `run(loaded_plugins=)`. +What a missing `refuse-to-start` plugin must NOT do is let the run go ahead: an +agent written to rely on code it did not get is not the agent somebody reviewed, +and the first answer it gives is paid for and wrong. + +So the turn ends before a single model call, `halted` says +`plugins-not-loaded`, the sentence names the plugin, its package and its range +and the two ways out, and `turn.run.failed` carries the reason — on every Python +target, and on the second port in the same words. The tree is +`tests/trees/a-desk-that-needs-a-plugin/`, read through the real loader. +""" + +from __future__ import annotations + +import asyncio +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from pact_adapters.conformance import TARGETS, report # noqa: E402 +from pact_adapters.events import Bus # noqa: E402 +from pact_adapters.harness import run # noqa: E402 +from pact_adapters.ir import AgentSpec # noqa: E402 +from pact_adapters.script import Script, Turn # noqa: E402 +from pact_adapters.transports.mock import ReferenceTransport # noqa: E402 +from test_the_golden_set_runs_everywhere import TRANSPORTS, _there # noqa: E402 + +REPO = Path(__file__).resolve().parents[3] +TREE = REPO / "tests" / "trees" / "a-desk-that-needs-a-plugin" +EXAMPLE = REPO / "examples" / "plugin-desk" +PACT = REPO / "target" / "debug" / "pact" + +REFUSED = ( + "This agent cannot start: it needs 'crm-tools' (`pypi:acme-crm-tools`, versions " + "`>=1.4,<2`), and nothing running it loaded it. fix: run it where a runtime loads " + "`pypi:` packages, or write `if-it-cannot-be-loaded: run-without-it` in the " + "plugin's file under `plugins/` if the agent can do its job without it." +) + +pytestmark = pytest.mark.skipif(not PACT.exists(), reason="build the CLI first: cargo build -p pact-cli") + + +def spec() -> AgentSpec: + out = subprocess.run([str(PACT), "show", str(TREE)], capture_output=True, text=True) + assert out.returncode == 0, out.stderr + return AgentSpec.from_document(json.loads(out.stdout), "desk", source=TREE) + + +def running(transport_cls=ReferenceTransport, **kw): + script = Script([Turn("Your account is in good standing.")]) + bus = Bus() + result = asyncio.run( + run(spec(), transport_cls(script), "Is my account in good standing?", {}, bus=bus, **kw) + ) + return result, script, bus + + +def test_the_plugin_reaches_the_spec_with_its_default_written_out() -> None: + (plugin,) = spec().plugins + assert (plugin.name, plugin.package, plugin.version, plugin.if_cannot_load) == ( + "crm-tools", "pypi:acme-crm-tools", ">=1.4,<2", "refuse-to-start" + ) + assert plugin.registry == "pypi" + + +def test_a_plugin_nobody_loaded_ends_the_turn_before_a_model_is_called() -> None: + result, script, bus = running() + assert result.halted == "plugins-not-loaded", result.halted + assert result.output == REFUSED + assert script.calls == 0, "nothing may be paid for before the plugin is known to be missing" + assert result.trace() == [] + (failed,) = list(bus.seen("turn.run.failed")) + assert failed.payload == {"reason": "plugins-not-loaded"} + + +def test_every_python_target_refuses_the_same_way() -> None: + """The refusal is the harness's, before any transport is asked anything, so a + target that went ahead would be a target that owns part of the loop.""" + for name, cls in TRANSPORTS.items(): + result, script, _ = running(cls) + assert (result.halted, result.output, script.calls) == ( + "plugins-not-loaded", REFUSED, 0 + ), name + + +def test_a_host_that_loaded_it_runs_the_agent_and_says_nothing_about_it() -> None: + result, script, _ = running(loaded_plugins={"crm-tools": "1.4.2"}) + assert result.halted == "final", result.halted + assert script.calls == 1 + assert not [u for u in result.unenforced if "crm-tools" in u], result.unenforced + + +def test_loading_some_other_plugin_is_not_loading_this_one() -> None: + result, _, _ = running(loaded_plugins={"web-search": "0.2.1"}) + assert result.halted == "plugins-not-loaded" + + +def test_the_second_port_refuses_in_the_same_words() -> None: + there = _there(spec()) + if there is None: + pytest.skip("node or the AI SDK is unavailable") + assert (there["halted"], there["output"]) == ("plugins-not-loaded", REFUSED) + assert there["told"] == [], "the second port called a model it had no business calling" + assert there["trace"] == [] + + +def test_every_target_says_before_execution_that_it_loads_no_plugin() -> None: + """The lattice column a reader checks before choosing where to run a tree with + plugins in it. Every transport here publishes `unsupported`, because the + harness loads nothing — and the conformance report states it ahead of the + results, which is the AC-2.2 obligation.""" + said = report([EXAMPLE], PACT) + declared = said["declared-before-execution"] + assert set(declared) == set(TARGETS) + assert {name: lattice["plugins"] for name, lattice in declared.items()} == { + name: "unsupported" for name in TARGETS + } + there = _there(spec()) + if there is not None: + assert there["lattice"]["plugins"] == "unsupported" diff --git a/adapters/python/tests/test_a_plugin_the_agent_can_do_without_is_named_on_unenforced.py b/adapters/python/tests/test_a_plugin_the_agent_can_do_without_is_named_on_unenforced.py new file mode 100644 index 0000000..364c436 --- /dev/null +++ b/adapters/python/tests/test_a_plugin_the_agent_can_do_without_is_named_on_unenforced.py @@ -0,0 +1,97 @@ +"""D29: a plugin an agent may start without, on a runtime that did not load it. + +`if-it-cannot-be-loaded: run-without-it` says the agent can still do its job +without the package — and the cost of that line is that every run has to SAY the +plugin was not there. A run that started without it in silence would answer from +whatever the model already knew while the author, reading their own file, +believed a search had been made. That is the T7 breach, and `unenforced` is the +door every other line this runtime did not carry out leaves by. + +The tree is `examples/plugin-desk/`, read through the real loader, so this also +holds the shipped example to doing what its own comments say. +""" + +from __future__ import annotations + +import asyncio +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from pact_adapters.harness import run # noqa: E402 +from pact_adapters.ir import AgentSpec # noqa: E402 +from pact_adapters.script import Script, Turn # noqa: E402 +from pact_adapters.transports.mock import ReferenceTransport # noqa: E402 +from test_the_golden_set_runs_everywhere import _there # noqa: E402 + +REPO = Path(__file__).resolve().parents[3] +EXAMPLE = REPO / "examples" / "plugin-desk" +PACT = REPO / "target" / "debug" / "pact" + +WITHOUT = ( + "plugin 'web-search' (`npm:@budmh/tool-web`, versions `^0.2.1-rc.3`) was not loaded " + "by whatever runs this agent, so nothing it adds is here and the agent ran without " + "it. fix: run it where a runtime loads `npm:` packages." +) + +pytestmark = pytest.mark.skipif(not PACT.exists(), reason="build the CLI first: cargo build -p pact-cli") + + +def document() -> dict: + out = subprocess.run([str(PACT), "show", str(EXAMPLE)], capture_output=True, text=True) + assert out.returncode == 0, out.stderr + return json.loads(out.stdout) + + +def spec(doc: dict | None = None) -> AgentSpec: + return AgentSpec.from_document(doc or document(), "research-desk", source=EXAMPLE) + + +def answering(s: AgentSpec, **kw): + script = Script([Turn("Nothing on that has been published.")]) + result = asyncio.run(run(s, ReferenceTransport(script), "What changed this week?", {}, **kw)) + return result, script + + +def test_the_agent_starts_without_it_and_every_run_says_so() -> None: + result, script = answering(spec()) + assert result.halted == "final", result.halted + assert script.calls == 1 + assert result.unenforced.count(WITHOUT) == 1, result.unenforced + + +def test_a_host_that_loaded_it_has_nothing_to_report() -> None: + result, _ = answering(spec(), loaded_plugins={"web-search": "0.2.1-rc.3"}) + assert result.halted == "final" + assert WITHOUT not in result.unenforced + assert not [u for u in result.unenforced if "web-search" in u], result.unenforced + + +def test_the_second_port_names_it_in_the_same_words() -> None: + there = _there(spec()) + if there is None: + pytest.skip("node or the AI SDK is unavailable") + here, _ = answering(spec()) + assert there["halted"] == here.halted == "final" + assert there["unenforced"].count(WITHOUT) == 1, there["unenforced"] + + +def test_a_plugin_for_every_agent_reaches_an_agent_that_does_not_name_it() -> None: + """`applies-to: every-agent` is written on the plugin so that an agent cannot + opt out by leaving it off its own line — the polarity `interceptor.applies-to` + and `policy.applies-to` already record. The spec, and so the run, has to see + it on an agent that names nothing.""" + doc = document() + del doc["agents"]["research-desk"]["plugins"] + assert spec(doc).plugins == () + doc["plugins"]["web-search"]["applies-to"] = "every-agent" + (plugin,) = spec(doc).plugins + assert plugin.name == "web-search" + result, _ = answering(spec(doc)) + assert WITHOUT in result.unenforced diff --git a/adapters/python/tests/test_a_proposed_change_passes_the_same_checker.py b/adapters/python/tests/test_a_proposed_change_passes_the_same_checker.py index e4f3462..a317ed0 100644 --- a/adapters/python/tests/test_a_proposed_change_passes_the_same_checker.py +++ b/adapters/python/tests/test_a_proposed_change_passes_the_same_checker.py @@ -11,7 +11,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) sys.path.insert(0, str(Path(__file__).resolve().parent)) -from pact_adapters.checking import check_instructions # noqa: E402 +from pact_adapters.checking import check_agent_field # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.learning import Learner, Proposal # noqa: E402 from trees import shown # noqa: E402 @@ -25,19 +25,19 @@ def test_a_sound_change_to_a_tree_that_carries_files_passes() -> None: doc = shown(DESK) - assert check_instructions(doc, "refund-desk", "Decide the refund. Be brief.", source=DESK) == [] + assert check_agent_field(doc, "refund-desk", "instructions", "Decide the refund. Be brief.", source=DESK) == [] def test_a_change_that_breaks_a_rule_is_refused_in_the_checkers_own_words() -> None: doc = shown(DESK) - refused = check_instructions(doc, "refund-desk", BROKEN, source=DESK) + refused = check_agent_field(doc, "refund-desk", "instructions", BROKEN, source=DESK) assert refused and any("nobody-declared-this" in r for r in refused), refused def test_a_document_with_no_files_is_checked_through_the_document_itself() -> None: doc = shown(PHASES) - assert check_instructions(doc, "builder", "Carry out the plan, in order.") == [] - assert check_instructions(doc, "builder", BROKEN), "no source tree needed to refuse it" + assert check_agent_field(doc, "builder", "instructions", "Carry out the plan, in order.") == [] + assert check_agent_field(doc, "builder", "instructions", BROKEN), "no source tree needed to refuse it" def test_a_learning_cycle_refuses_a_proposal_the_checker_refuses_before_scoring_it() -> None: diff --git a/adapters/python/tests/test_a_workspace_in_the_language_learns_as_a_folder_does.py b/adapters/python/tests/test_a_workspace_in_the_language_learns_as_a_folder_does.py index ae56836..735db5b 100644 --- a/adapters/python/tests/test_a_workspace_in_the_language_learns_as_a_folder_does.py +++ b/adapters/python/tests/test_a_workspace_in_the_language_learns_as_a_folder_does.py @@ -21,7 +21,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) sys.path.insert(0, str(Path(__file__).resolve().parent)) -from pact_adapters.checking import check_instructions # noqa: E402 +from pact_adapters.checking import check_agent_field # noqa: E402 from pact_adapters.exploding import CarriesPayloads, explode # noqa: E402 from pact_adapters.ir import AgentSpec # noqa: E402 from pact_adapters.learning import Learner # noqa: E402 @@ -48,17 +48,17 @@ def test_the_tree_this_is_about_carries_files_the_document_only_points_at(tmp_pa def test_a_sound_change_passes() -> None: - assert check_instructions(shown(STAFF), "helpdesk", SOUND, source=STAFF) == [] + assert check_agent_field(shown(STAFF), "helpdesk", "instructions", SOUND, source=STAFF) == [] def test_a_change_that_breaks_a_rule_is_refused_in_the_checkers_own_words() -> None: - refused = check_instructions(shown(STAFF), "helpdesk", BROKEN, source=STAFF) + refused = check_agent_field(shown(STAFF), "helpdesk", "instructions", BROKEN, source=STAFF) assert refused and any("nobody-declared-this" in r for r in refused), refused def test_the_authors_own_files_are_not_touched() -> None: before = {p: p.read_bytes() for p in STAFF.rglob("*") if p.is_file()} - check_instructions(shown(STAFF), "helpdesk", BROKEN, source=STAFF) + check_agent_field(shown(STAFF), "helpdesk", "instructions", BROKEN, source=STAFF) assert {p: p.read_bytes() for p in STAFF.rglob("*") if p.is_file()} == before @@ -94,8 +94,8 @@ def test_a_folder_with_a_runtime_area_is_not_taken_for_one_written_in_the_langua (copy / ".pact" / "learning").mkdir(parents=True) (copy / ".pact" / "learning" / "spend.jsonl").write_text("") doc = shown(copy) - assert check_instructions(doc, "refund-desk", "Decide the refund. Be brief.", source=copy) == [] - refused = check_instructions(doc, "refund-desk", BROKEN, source=copy) + assert check_agent_field(doc, "refund-desk", "instructions", "Decide the refund. Be brief.", source=copy) == [] + refused = check_agent_field(doc, "refund-desk", "instructions", BROKEN, source=copy) assert refused and any("nobody-declared-this" in r for r in refused), refused @@ -103,8 +103,8 @@ def test_a_workspace_in_several_files_is_held_the_same_way() -> None: # No files to carry, so this one goes through the document — and an agent # made from a pattern (`based on desk with …`) takes the change as its own. doc = shown(HELP) - assert check_instructions(doc, "billing-desk", "Answer billing questions, briefly.", source=HELP) == [] - refused = check_instructions(doc, "billing-desk", BROKEN, source=HELP) + assert check_agent_field(doc, "billing-desk", "instructions", "Answer billing questions, briefly.", source=HELP) == [] + refused = check_agent_field(doc, "billing-desk", "instructions", BROKEN, source=HELP) assert refused and any("nobody-declared-this" in r for r in refused), refused diff --git a/adapters/python/tests/test_the_golden_set_runs_everywhere.py b/adapters/python/tests/test_the_golden_set_runs_everywhere.py index 82d90f0..13c8fbb 100644 --- a/adapters/python/tests/test_the_golden_set_runs_everywhere.py +++ b/adapters/python/tests/test_the_golden_set_runs_everywhere.py @@ -219,6 +219,16 @@ def _there(spec: AgentSpec) -> dict[str, Any] | None: } for k in spec.knowledge ], + # D29, for the same reason: a plugin left off the wire would let Node + # answer where the reference port halted `plugins-not-loaded`, or say + # nothing where it named the plugin on `unenforced`. + "plugins": [ + { + "name": p.name, "package": p.package, "version": p.version, + "if-it-cannot-be-loaded": p.if_cannot_load, + } + for p in spec.plugins + ], "team": dict(spec.team), "loop": spec.loop.name if spec.loop else "", "loops": _loops_of(spec), diff --git a/adapters/python/tests/test_the_loader_to_adapter_boundary_is_total.py b/adapters/python/tests/test_the_loader_to_adapter_boundary_is_total.py index 47e376f..5542673 100644 --- a/adapters/python/tests/test_the_loader_to_adapter_boundary_is_total.py +++ b/adapters/python/tests/test_the_loader_to_adapter_boundary_is_total.py @@ -318,6 +318,12 @@ def test_every_spec_field_that_holds_authored_state_is_filled_from_the_document( "unbounded", # an owner's, and this example keeps its step limit "detectors", # authored, and this example declares none "rests", # authored, and this example rests no tool + # Authored (D29), and this example asks its runtime to load nothing: it + # demonstrates the no-code ceiling, and a plugin is code. What proves + # the line reaches the spec, refuses a run or is named on `unenforced` + # is `test_a_plugin_the_agent_can_do_without_is_named_on_unenforced.py` + # over `examples/plugin-desk/`, the example that exists for it. + "plugins", } empty: list[str] = [] for f in dataclass_fields(spec): diff --git a/adapters/typescript/src/harness.ts b/adapters/typescript/src/harness.ts index d69fd4b..e6cecb0 100644 --- a/adapters/typescript/src/harness.ts +++ b/adapters/typescript/src/harness.ts @@ -98,6 +98,17 @@ export type KnowledgeSpec = { "split-by"?: string; }; +// One plugin the agent needs (D29), in the author's own field names like +// `SkillSpec`: the request for code a runtime loads, and never the code. This +// port loads no plugin, so each one is missing here — refused before a model +// call, or named on `unenforced` — in the reference port's own words. +export type PluginSpec = { + name: string; + package: string; + version: string; + "if-it-cannot-be-loaded": string; +}; + export type AgentSpec = { name: string; instructions: string; @@ -151,6 +162,9 @@ export type AgentSpec = { // driver exists to catch: the Python run refused a `must-cite:` corpus it // could not read, and this one answered. knowledge?: KnowledgeSpec[]; + // The plugins it needs: the ones it names and every one that applies to + // every agent, as `ir.AgentSpec.plugins` carries them. + plugins?: PluginSpec[]; // What the agent hands back when it is done, as the author wrote it. Read by // nothing in EITHER port for a round, while `tier: core` and present in the // worked example — so every scripted answer in the eval suite hand-writes @@ -218,7 +232,7 @@ export const AGENT_SPEC_FIELDS = [ "name", "instructions", "tools", "skills", "team", "maxSteps", "limits", "loop", "loops", "interceptors", "contextPolicy", "policy", "teamwork", "settings", "slo", "model", "watches", "answersWith", "answersWithMode", - "knowledge", "boards", "reactions", + "knowledge", "boards", "reactions", "plugins", ] as const; //: The tool an agent says its own topic with. The same name `reactions.py` uses. @@ -498,6 +512,50 @@ export function neverLookedIn(spec: AgentSpec): string[] { ); } +//: The one `if-it-cannot-be-loaded:` answer that lets the agent start without +//: the plugin — `ir.RUN_WITHOUT_IT`. Anything else refuses, `refuse-to-start` +//: and an absent line alike, as in the reference port. +export const RUN_WITHOUT_IT = "run-without-it"; + +function pluginNamed(p: PluginSpec): string { + return `'${p.name}' (\`${p.package}\`, versions \`${p.version}\`)`; +} + +function registryOf(p: PluginSpec): string { + return p.package.split(":", 1)[0].trim(); +} + +function refusesToStart(p: PluginSpec): boolean { + return p["if-it-cannot-be-loaded"] !== RUN_WITHOUT_IT; +} + +//: Why the turn ended before it began. Word for word `harness._cannot_start_without` +//: — the fix is true here too, because this port loads nothing and so runs no +//: plugin anywhere. +export function cannotStartWithout(needed: PluginSpec[]): string { + const named = needed.map(pluginNamed); + const listed = named.length === 1 + ? named[0] + : `${named.slice(0, -1).join(", ")} and ${named[named.length - 1]}`; + const registries = [...new Set(needed.map((p) => `\`${registryOf(p)}:\``))].join(" or "); + return ( + `This agent cannot start: it needs ${listed}, and nothing running it loaded ` + + `${needed.length === 1 ? "it" : "them"}. fix: run it where a runtime loads ` + + `${registries} packages, or write \`if-it-cannot-be-loaded: run-without-it\` in ` + + `the plugin's file under \`plugins/\` if the agent can do its job without it.` + ); +} + +//: The `unenforced` line for a plugin the agent started without — word for word +//: `harness._ran_without`. +export function ranWithout(p: PluginSpec): string { + return ( + `plugin ${pluginNamed(p)} was not loaded by whatever runs this agent, so ` + + `nothing it adds is here and the agent ran without it. fix: run it where a ` + + `runtime loads \`${registryOf(p)}:\` packages.` + ); +} + export async function run( spec: AgentSpec, transport: Transport, @@ -520,6 +578,25 @@ export async function run( `neither honoured nor reported.`, ); } + // D29. No plugin is loaded here, so every one the agent needs is missing: one + // it may run without is named on `unenforced`, and one it may not ends the + // turn before a model call. Decided first, in the same place and the same + // order as the reference port, so a document needing both a plugin and a + // `must-cite:` corpus ends the same way on both. + const withoutPlugins = (spec.plugins ?? []).filter((p) => !refusesToStart(p)).map(ranWithout); + const needed = (spec.plugins ?? []).filter(refusesToStart); + if (needed.length > 0) { + return { + output: cannotStartWithout(needed), + steps: [], + halted: "plugins-not-loaded", + stoppedBy: null, + unmetered: [], + unenforced: withoutPlugins, + unretrieved: [], + phases: [], + }; + } // A7. Every declared set, named before anything else happens. PACT retrieves // nothing and this port has nowhere to be handed a retrieval either, so a set // of documents nobody looked in is the ordinary state — and it must not be the @@ -711,7 +788,7 @@ export async function run( // Named one field at a time, with the sentence a person can act on, because // "this runtime is smaller" is not something a reader can check against their // own file. - const unenforced = notDoneHere(spec); + const unenforced = [...notDoneHere(spec), ...withoutPlugins]; // The one line `notDoneHere` cannot see, appended where it can be: a stage's // `asks:`. It names WHICH question a person is put, which is a governance line // and not a wording choice — and it was parsed by `loops.ts` into `Phase.asks` diff --git a/adapters/typescript/src/vercel-transport.ts b/adapters/typescript/src/vercel-transport.ts index c38b3af..d2067e9 100644 --- a/adapters/typescript/src/vercel-transport.ts +++ b/adapters/typescript/src/vercel-transport.ts @@ -114,6 +114,7 @@ export class VercelAITransport implements Transport { // or the two ports cannot be compared (P-2), which is what // `test_all_seven_targets_publish_the_same_lattice_features` holds. connected_tools: "unsupported", + plugins: "unsupported", }; } diff --git a/crates/pact-cli/src/digests.rs b/crates/pact-cli/src/digests.rs index c889cd5..ca331c6 100644 --- a/crates/pact-cli/src/digests.rs +++ b/crates/pact-cli/src/digests.rs @@ -17,13 +17,13 @@ //! cases, and the `evals:` line of a workflow or an agent), which grade a //! run and govern none, so a new eval case keeps the version (02W §7.2 //! classes it additive). Everything else stays in: -//! its own stages, questions and policy, the tools, agents and programs of -//! the workspace (a program's body is in the document as its file's digest), -//! the workspace's settings. That is wider than 02W's list (its local -//! questions, its named policy, the actions it calls), and on purpose: a -//! policy that `applies-to: every-agent` or a redaction rule governs the -//! run without the workflow naming it, so leaving out what it does not name -//! would let a change the run depends on through it. +//! its own stages, questions and policy, the tools, agents, plugins and +//! programs of the workspace (a program's body is in the document as its +//! file's digest), the workspace's settings. That is wider than 02W's list +//! (its local questions, its named policy, the actions it calls), and on +//! purpose: a policy that `applies-to: every-agent` or a redaction rule +//! governs the run without the workflow naming it, so leaving out what it +//! does not name would let a change the run depends on through it. //! * **`contract-digest`** — what its callers rely on: `accepts:`, //! `answers-with:`, `published-as:`, and each port that answers it with its //! key (`same-conversation-when:`) and its `if-still-running:`. The derived diff --git a/crates/pact-cli/src/discover.rs b/crates/pact-cli/src/discover.rs index c5980d5..7854259 100644 --- a/crates/pact-cli/src/discover.rs +++ b/crates/pact-cli/src/discover.rs @@ -204,6 +204,10 @@ pub fn inventory(found: &Found) -> J { let effect = pact_loader::effects::of_agent(doc, a, &mut std::collections::BTreeSet::new()); o.insert("effect".into(), J::String(effect.word().into())); o.insert("moves-money".into(), J::Bool(effect.moves_money)); + // D29. What this agent asks its runtime to load, and what happens where + // that cannot be done — the fact a runtime has to decide on BEFORE it + // takes the agent, so it is here rather than only in `pact show`. + o.insert("plugins".into(), plugins(doc, a)); let runnable = a.get("instructions").is_some() || pact_loader::deciding::decides_alone(a, &decision); o.insert("runnable".into(), J::Bool(runnable)); @@ -368,6 +372,30 @@ fn governance(a: &Node) -> J { J::Object(o) } +/// The plugins an agent needs — the ones it names and the ones that apply to +/// every agent — each with its package, its range and what happens on a +/// runtime that cannot load it, the default written out. +/// +/// `pact_loader::plugins::needed_by` decides which, so the inventory and the +/// note `pact check` prints about the same agent cannot disagree. +fn plugins(doc: &Node, a: &Node) -> J { + let list = pact_loader::plugins::needed_by(doc, a) + .into_iter() + .map(|(name, p)| { + let mut o = Map::new(); + o.insert("name".into(), J::String(name.to_string())); + o.insert("package".into(), text(p, "package").unwrap_or(J::Null)); + o.insert("version".into(), text(p, "version").unwrap_or(J::Null)); + o.insert( + "if-it-cannot-be-loaded".into(), + J::String(pact_loader::plugins::if_it_cannot_be_loaded(p).to_string()), + ); + J::Object(o) + }) + .collect(); + J::Array(list) +} + /// The latency half of `limits:`, and nothing else. /// /// `finishes-within` is deliberately in BOTH: the author's own file says it is diff --git a/crates/pact-cli/src/main.rs b/crates/pact-cli/src/main.rs index 9d7a48d..38362b2 100644 --- a/crates/pact-cli/src/main.rs +++ b/crates/pact-cli/src/main.rs @@ -1836,6 +1836,12 @@ fn validate( // now stop this run — with nothing in this workspace changed and nobody // here having approved it. pact_loader::bundles::a_bundle_brings_only_what_it_said(root, &schema, &mut diags); + // D29. A plugin line asks a runtime to load code, and each half of the + // request has a grammar that is not PACT's — a registry word, a version + // range written that registry's way — which a runtime applies later, + // somewhere the author is not watching. Held here, where they are, and + // without resolving, downloading or opening the package (R5, D17). + pact_loader::plugins::check(root, &mut diags); // And the seventh, which was decided only by RUNNING the agent. // `spends-money:`'s own help promises the call is routed through the // author's approval policy; whether it is, is a fact about a different diff --git a/crates/pact-cli/tests/a_name_the_workspace_already_has.rs b/crates/pact-cli/tests/a_name_the_workspace_already_has.rs index 2a03a10..b5fd5a7 100644 --- a/crates/pact-cli/tests/a_name_the_workspace_already_has.rs +++ b/crates/pact-cli/tests/a_name_the_workspace_already_has.rs @@ -230,14 +230,15 @@ fn the_line_that_may_name_a_kind_is_read_from_the_specification_and_not_from_the // about the specification and may honestly be "none". let root = workspace( "derived", - // All three, because "nothing in this specification names a skill" has to - // be true of the whole file — `variant.may-use:` names one too now, and - // leaving it would have the checker honestly offer a line that this - // rewritten specification really does have. + // All four, because "nothing in this specification names a skill" has to + // be true of the whole file — `variant.may-use:` and `plugin.implements:` + // name one too now, and leaving either would have the checker honestly + // offer a line that this rewritten specification really does have. &[ ("agent", "uses", "[tools]"), ("stage", "may-use", "[tools, agents]"), ("variant", "may-use", "[tools]"), + ("plugin", "implements", "[tools, knowledge]"), ], &[NARROWS_TO_THE_SKILL], ); diff --git a/crates/pact-cli/tests/a_plugin_is_held_to_the_laws_through_its_interfaces.rs b/crates/pact-cli/tests/a_plugin_is_held_to_the_laws_through_its_interfaces.rs new file mode 100644 index 0000000..3d5a99c --- /dev/null +++ b/crates/pact-cli/tests/a_plugin_is_held_to_the_laws_through_its_interfaces.rs @@ -0,0 +1,276 @@ +//! D29 under the eight laws (`docs/97-PRIMITIVES.md` §4): a plugin is code no +//! checker reads, so the laws hold for it through the interfaces it says it +//! implements. Every rule is driven through the real binary on a copy of +//! `examples/plugin-desk` that adds a customer-records interface and a plugin +//! whose code stands behind it; the unbroken copy is held to saying nothing. +//! +//! Law 8 for a plugin is a learning proposal held to `pact check` before it is +//! held for a person, which is the Python reference's to do and is driven by +//! `adapters/python/tests/test_a_plugin_a_cycle_proposes_is_checked_then_held.py`. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +fn pact() -> Command { + Command::new(env!("CARGO_BIN_EXE_pact")) +} + +fn example() -> PathBuf { + PathBuf::from(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../examples/plugin-desk" + )) +} + +/// The interface: one action that looks things up, and one that cannot be +/// undone and so names the argument that makes two calls the same call. +const CRM: (&str, &str) = ( + "tools/crm.yaml", + "description: The customer records the desk reads and closes.\n\ + interface: yes\n\ + actions:\n \ + look-up:\n \ + description: Finds a customer by email.\n \ + takes:\n email: text\n \ + reads-only: yes\n \ + close-account:\n \ + description: Closes a customer's account.\n \ + takes:\n customer: text\n \ + same-request-key: customer\n", +); + +/// The plugin whose code stands behind it. +const PLUGIN: (&str, &str) = ( + "plugins/crm-tools.yaml", + "description: The customer-records client the desk's runtime loads.\n\ + package: npm:@acme/crm-tools\n\ + version: ^1.4.0\n\ + if-it-cannot-be-loaded: run-without-it\n\ + implements: [crm]\n", +); + +const AGENT: &str = "agents/research-desk/agent.yaml"; +const LOADED: &str = "plugins:\n - web-search\n - crm-tools\nuses: [crm]\n"; + +fn copy(src: &Path, dst: &Path) { + std::fs::create_dir_all(dst).unwrap(); + for e in std::fs::read_dir(src).unwrap().flatten() { + let (s, d) = (e.path(), dst.join(e.file_name())); + if s.is_dir() { + copy(&s, &d) + } else { + std::fs::copy(&s, &d).map(|_| ()).unwrap() + } + } +} + +/// The example with the interface, the plugin and the agent that uses both, +/// then each `(file, from, to)` edit made. +fn desk(name: &str, edits: &[(&str, &str, &str)]) -> PathBuf { + let dst = std::env::temp_dir().join(format!("pact-plugin-laws-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dst); + copy(&example(), &dst); + for (file, text) in [CRM, PLUGIN] { + let p = dst.join(file); + std::fs::create_dir_all(p.parent().unwrap()).unwrap(); + std::fs::write(&p, text).unwrap(); + } + let agent = dst.join(AGENT); + let text = std::fs::read_to_string(&agent).unwrap(); + assert!(text.contains("plugins:\n - web-search\n"), "fixture drifted: {AGENT}"); + std::fs::write(&agent, text.replacen("plugins:\n - web-search\n", LOADED, 1)).unwrap(); + for (file, from, to) in edits { + let p = dst.join(file); + let text = std::fs::read_to_string(&p).unwrap_or_default(); + assert!(text.contains(from), "fixture drifted: {from:?} not in {file}"); + std::fs::write(&p, text.replacen(from, to, 1)).unwrap(); + } + dst +} + +/// `pact check --deny-warnings` on `root`: whether it passed, every +/// `(severity, rule)` it printed, and the whole text. +fn check(root: &Path) -> (bool, Vec<(String, String)>, String) { + let out = pact() + .args(["check", root.to_str().unwrap(), "--deny-warnings"]) + .output() + .expect("runs"); + let text = format!( + "{}{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ); + let mut told = Vec::new(); + let mut severity = String::new(); + for line in text.lines() { + for word in ["error", "warning", "note"] { + if line.starts_with(&format!("{word}:")) { + severity = word.to_string(); + } + } + if let Some(rule) = line.trim().strip_prefix("rule: ") { + told.push((severity.clone(), rule.to_string())); + } + } + (out.status.success(), told, text) +} + +/// The rules `edits` makes `pact check` print, the tree removed afterwards. +fn checked(name: &str, edits: &[(&str, &str, &str)]) -> (bool, Vec<(String, String)>, String) { + let root = desk(name, edits); + let out = check(&root); + let _ = std::fs::remove_dir_all(&root); + out +} + +fn has(told: &[(String, String)], severity: &str, rule: &str) -> bool { + told.iter().any(|(s, r)| s == severity && r == rule) +} + +/// What `pact discover` says the research desk does to the world. +fn effect_of_the_desk(name: &str, edits: &[(&str, &str, &str)]) -> (String, bool) { + let root = desk(name, edits); + let out = pact() + .args(["discover", root.to_str().unwrap()]) + .output() + .expect("runs"); + let _ = std::fs::remove_dir_all(&root); + assert!(out.status.success(), "{}", String::from_utf8_lossy(&out.stderr)); + let inv: serde_json::Value = serde_json::from_slice(&out.stdout).expect("JSON"); + let desk = inv[0]["agents"] + .as_array() + .expect("agents") + .iter() + .find(|a| a["id"] == "pact:research-desk") + .cloned() + .expect("the research desk"); + ( + desk["effect"].as_str().expect("effect").to_string(), + desk["moves-money"].as_bool().expect("moves-money"), + ) +} + +#[test] +fn a_plugin_behind_an_interface_says_nothing_at_all() { + let (ok, told, text) = checked("clean", &[]); + assert!(ok, "{text}"); + assert!(told.is_empty(), "{text}"); +} + +#[test] +fn law_4_a_plugin_implements_only_an_interface() { + let (ok, told, text) = checked("not-an-interface", &[("tools/crm.yaml", "interface: yes\n", "")]); + assert!(!ok, "{text}"); + assert!( + has(&told, "error", "loader/plugin-implements-what-is-not-an-interface"), + "{text}" + ); + assert!(text.contains("'crm-tools' says its code implements 'crm'"), "{text}"); +} + +#[test] +fn law_4_an_interface_a_plugin_implements_is_implemented() { + // Without the plugin's `implements:` the interface has nothing behind it. + let (_, told, text) = checked("no-implements", &[("plugins/crm-tools.yaml", "implements: [crm]\n", "")]); + assert!(has(&told, "warning", "loader/an-interface-nothing-implements"), "{text}"); +} + +#[test] +fn law_4_an_agent_using_an_interface_only_an_absent_plugin_implements_is_warned() { + let (ok, told, text) = checked( + "absent", + &[(AGENT, " - crm-tools\n", "")], + ); + assert!(!ok, "{text}"); + assert!( + has(&told, "warning", "loader/an-interface-only-an-absent-plugin-implements"), + "{text}" + ); + assert!(text.contains("Add 'crm-tools' to `plugins:` in 'research-desk'"), "{text}"); + // Loaded for every agent, the plugin is behind the interface again. + let (_, told, text) = checked( + "every-agent", + &[ + (AGENT, " - crm-tools\n", ""), + ("plugins/crm-tools.yaml", "implements: [crm]\n", "implements: [crm]\napplies-to: every-agent\n"), + ], + ); + assert!( + !has(&told, "warning", "loader/an-interface-only-an-absent-plugin-implements"), + "{text}" + ); +} + +#[test] +fn law_4_a_run_input_may_choose_the_plugin() { + let (ok, told, text) = checked( + "chosen", + &[ + ("tools/crm.yaml", "interface: yes\n", "interface: yes\nchosen-by: run-inputs.records\n"), + (AGENT, "uses: [crm]\n", "uses: [crm]\nrun-inputs:\n records: one of crm-tools\n"), + ], + ); + assert!(ok, "{text}"); + assert!(told.is_empty(), "{text}"); + let (_, told, text) = checked( + "chosen-wrong", + &[ + ("tools/crm.yaml", "interface: yes\n", "interface: yes\nchosen-by: run-inputs.records\n"), + (AGENT, "uses: [crm]\n", "uses: [crm]\nrun-inputs:\n records: one of web-search\n"), + ], + ); + assert!(has(&told, "error", "loader/a-choice-that-implements-nothing"), "{text}"); +} + +#[test] +fn law_3_an_action_a_plugin_cannot_undo_names_its_key() { + let (ok, told, text) = checked( + "no-key", + &[("tools/crm.yaml", " same-request-key: customer\n", "")], + ); + assert!(!ok, "{text}"); + assert!(has(&told, "error", "loader/a-plugin-action-with-no-claim"), "{text}"); + assert!(text.contains("'crm/close-account' cannot be undone"), "{text}"); + // An action that looks things up needs no claim, and neither does one the + // tree can take back. + let (ok, told, text) = checked( + "undone", + &[( + "tools/crm.yaml", + " same-request-key: customer\n", + " undone-by: crm/reopen-account\n reopen-account:\n description: Reopens it.\n takes:\n customer: text\n same-request-key: customer\n", + )], + ); + assert!(ok, "{text}"); + assert!(!has(&told, "error", "loader/a-plugin-action-with-no-claim"), "{text}"); +} + +#[test] +fn law_1_an_agent_loaded_with_a_plugin_reports_what_its_interface_does() { + // The desk names the interface: it can close an account. + assert_eq!(effect_of_the_desk("uses", &[]), ("cannot-be-undone".into(), false)); + // It no longer names it and is still loaded with the code that does it. + assert_eq!( + effect_of_the_desk("loaded", &[(AGENT, "uses: [crm]\n", "")]), + ("cannot-be-undone".into(), false) + ); + // A plugin that implements nothing gives the agent nothing to report. + assert_eq!( + effect_of_the_desk( + "nothing", + &[(AGENT, "uses: [crm]\n", ""), ("plugins/crm-tools.yaml", "implements: [crm]\n", "")] + ), + ("nothing".into(), false) + ); + // Money is carried the same way. + assert_eq!( + effect_of_the_desk( + "money", + &[ + (AGENT, "uses: [crm]\n", ""), + ("tools/crm.yaml", " same-request-key: customer\n", " same-request-key: customer\n spends-money: yes\n needs-a-person: yes\n"), + ] + ), + ("cannot-be-undone".into(), true) + ); +} diff --git a/crates/pact-cli/tests/a_plugin_is_named_and_never_loaded.rs b/crates/pact-cli/tests/a_plugin_is_named_and_never_loaded.rs new file mode 100644 index 0000000..4c89a12 --- /dev/null +++ b/crates/pact-cli/tests/a_plugin_is_named_and_never_loaded.rs @@ -0,0 +1,357 @@ +//! D29: a plugin is a request for code the runtime loads, and `pact check` +//! holds the request without ever touching the code. Every rule is driven here +//! through the real binary on a broken copy of `examples/plugin-desk`, and the +//! unbroken example is held to saying nothing at all; `pact discover` publishes +//! what each agent needs; and an edit to a plugin moves the digests a run pins +//! exactly as an edit to anything else the run relies on does. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +fn pact() -> Command { + Command::new(env!("CARGO_BIN_EXE_pact")) +} + +fn example() -> PathBuf { + PathBuf::from(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../examples/plugin-desk" + )) +} + +const PLUGIN: &str = "plugins/web-search.yaml"; +const AGENT: &str = "agents/research-desk/agent.yaml"; + +/// A workflow that calls the research desk, and the two lines the desk needs to +/// be called with a question and to hand an answer back, written above its +/// `plugins:` line. +const WORKFLOW: (&str, &str) = ( + "workflows/ask-the-desk.yaml", + "description: Asks the research desk one question and hands back its answer.\n\ + accepts:\n question: text\nanswers-with:\n answer: text\nstarts-at: ask\nsteps:\n \ + ask:\n does: call\n call: research-desk\n bind: { question: input.question }\n \ + then: { answered: reply }\n reply:\n does: answer\n bind: { answer: steps.ask.answer }\n", +); +const CALLABLE: &str = "accepts:\n question: text\nanswers-with:\n answer: text\nplugins:"; + +fn copy(src: &Path, dst: &Path) { + std::fs::create_dir_all(dst).unwrap(); + for e in std::fs::read_dir(src).unwrap().flatten() { + let (s, d) = (e.path(), dst.join(e.file_name())); + if s.is_dir() { + copy(&s, &d) + } else { + std::fs::copy(&s, &d).map(|_| ()).unwrap() + } + } +} + +/// A copy of the example with each `(file, from, to)` edit made. A file that is +/// not there is written, from "". +fn desk(name: &str, edits: &[(&str, &str, &str)]) -> PathBuf { + let dst = std::env::temp_dir().join(format!("pact-plugin-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dst); + copy(&example(), &dst); + for (file, from, to) in edits { + let p = dst.join(file); + std::fs::create_dir_all(p.parent().unwrap()).unwrap(); + let text = std::fs::read_to_string(&p).unwrap_or_default(); + assert!( + text.contains(from), + "fixture drifted: {from:?} not in {file}" + ); + std::fs::write(&p, text.replacen(from, to, 1)).unwrap(); + } + dst +} + +/// `pact check --deny-warnings` on `root`: whether it passed, every +/// `(severity, rule)` it printed, and the whole text. +fn check(root: &Path) -> (bool, Vec<(String, String)>, String) { + let out = pact() + .args(["check", root.to_str().unwrap(), "--deny-warnings"]) + .output() + .expect("runs"); + let text = format!( + "{}{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ); + let mut told = Vec::new(); + let mut severity = String::new(); + for line in text.lines() { + for word in ["error", "warning", "note"] { + if line.starts_with(&format!("{word}:")) { + severity = word.to_string(); + } + } + if let Some(rule) = line.trim().strip_prefix("rule: ") { + told.push((severity.clone(), rule.to_string())); + } + } + (out.status.success(), told, text) +} + +/// The one rule a broken copy is refused by, at the severity given. +fn refused_by(name: &str, edits: &[(&str, &str, &str)], severity: &str, rule: &str) -> String { + let root = desk(name, edits); + let (ok, told, text) = check(&root); + let _ = std::fs::remove_dir_all(&root); + assert_eq!(told, [(severity.to_string(), rule.to_string())], "{text}"); + assert_eq!(ok, severity == "note", "{text}"); + text +} + +fn discover(root: &Path) -> serde_json::Value { + let out = pact() + .args(["discover", root.to_str().unwrap()]) + .output() + .expect("runs"); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let inv: serde_json::Value = serde_json::from_slice(&out.stdout).expect("JSON"); + inv.as_array() + .and_then(|a| a.first()) + .cloned() + .expect("one workspace") +} + +#[test] +fn the_example_says_nothing_at_all() { + let (ok, told, text) = check(&example()); + assert!(ok, "{text}"); + assert!(told.is_empty(), "{text}"); +} + +#[test] +fn a_package_that_names_no_registry_or_one_nobody_loads_from_is_refused() { + let text = refused_by( + "no-registry", + &[(PLUGIN, "npm:@budmh/tool-web", "\"@budmh/tool-web\"")], + "error", + "loader/plugin-package-has-no-registry", + ); + assert!( + text.contains("`npm:@budmh/tool-web`"), + "the fix spells the line: {text}" + ); + refused_by( + "unknown-registry", + &[(PLUGIN, "npm:@budmh/tool-web", "cargo:tool-web")], + "error", + "loader/plugin-package-has-no-registry", + ); +} + +#[test] +fn a_range_its_registry_would_not_read_is_refused() { + let text = refused_by( + "pypi-range-under-npm", + &[(PLUGIN, "^0.2.1-rc.3", "\">=0.2,<1\"")], + "error", + "loader/plugin-version-is-not-a-range", + ); + assert!( + text.contains("^1.4.0"), + "the fix shows an npm range: {text}" + ); + refused_by( + "npm-range-under-pypi", + &[(PLUGIN, "npm:@budmh/tool-web", "pypi:budmh-tool-web")], + "error", + "loader/plugin-version-is-not-a-range", + ); +} + +#[test] +fn a_secret_in_the_plugins_settings_is_refused_and_a_reference_is_not() { + refused_by( + "secret", + &[( + PLUGIN, + " searchMaxResults: 5\n", + " searchMaxResults: 5\n apiKey: sk-live-2f9c41\n", + )], + "error", + "loader/plugin-secret-written-in-the-file", + ); + let root = desk( + "reference", + &[( + PLUGIN, + " searchMaxResults: 5\n", + " searchMaxResults: 5\n apiKey: { by-reference: host/search-key }\n", + )], + ); + let (ok, told, text) = check(&root); + let _ = std::fs::remove_dir_all(&root); + assert!(ok && told.is_empty(), "{text}"); +} + +#[test] +fn a_plugin_the_agent_cannot_start_without_is_a_note_and_not_a_failure() { + let text = refused_by( + "required", + &[(PLUGIN, "if-it-cannot-be-loaded: run-without-it\n", "")], + "note", + "loader/plugin-runs-only-where-loaded", + ); + assert!( + text.contains("'research-desk' runs only where `npm:@budmh/tool-web` can be loaded"), + "{text}" + ); +} + +#[test] +fn a_plugin_name_nothing_defines_is_refused_where_it_is_written() { + // And the plugin the author meant is then named by nobody, which is the + // second thing they are told. + let root = desk("misspelt", &[(AGENT, " - web-search", " - web-serch")]); + let (ok, told, text) = check(&root); + let _ = std::fs::remove_dir_all(&root); + assert!(!ok, "{text}"); + assert_eq!( + told, + [ + ("error".to_string(), "schema/no-such-name".to_string()), + ( + "warning".to_string(), + "loader/nothing-points-at-it".to_string() + ), + ], + "{text}" + ); + assert!(text.contains("Change it to one of: web-search"), "{text}"); +} + +#[test] +fn a_plugin_nothing_names_is_loaded_for_nobody_and_says_so() { + let text = refused_by( + "unnamed", + &[(AGENT, "plugins:\n - web-search\n", "")], + "warning", + "loader/nothing-points-at-it", + ); + assert!(text.contains("applies-to: every-agent"), "{text}"); + let root = desk( + "every-agent", + &[ + (AGENT, "plugins:\n - web-search\n", ""), + (PLUGIN, "config:", "applies-to: every-agent\nconfig:"), + ], + ); + let (ok, told, text) = check(&root); + assert!(ok && told.is_empty(), "{text}"); + let inv = discover(&root); + let _ = std::fs::remove_dir_all(&root); + assert_eq!( + inv["agents"][0]["plugins"][0]["name"], "web-search", + "a plugin for every agent is listed for an agent that does not name it" + ); +} + +#[test] +fn the_folder_form_is_the_same_plugin() { + let flat = discover(&example()); + let root = desk("folder", &[]); + let file = root.join(PLUGIN); + let folder = root.join("plugins/web-search"); + std::fs::create_dir_all(&folder).unwrap(); + std::fs::rename(&file, folder.join("plugin.yaml")).unwrap(); + let (ok, told, text) = check(&root); + let folded = discover(&root); + let _ = std::fs::remove_dir_all(&root); + assert!(ok && told.is_empty(), "{text}"); + assert_eq!(folded["digest"], flat["digest"]); +} + +#[test] +fn discover_says_what_each_agent_asks_its_runtime_to_load() { + let inv = discover(&example()); + assert_eq!( + inv["agents"][0]["plugins"], + serde_json::json!([{ + "name": "web-search", + "package": "npm:@budmh/tool-web", + "version": "^0.2.1-rc.3", + "if-it-cannot-be-loaded": "run-without-it", + }]) + ); + let root = desk( + "default", + &[(PLUGIN, "if-it-cannot-be-loaded: run-without-it\n", "")], + ); + let inv = discover(&root); + let _ = std::fs::remove_dir_all(&root); + assert_eq!( + inv["agents"][0]["plugins"][0]["if-it-cannot-be-loaded"], "refuse-to-start", + "the default is written out, so a runtime never has to know it" + ); +} + +/// The workspace digest and `ask-the-desk`'s two digests, with the workflow and +/// the desk's two lines added, plus `edits`. +fn digests(name: &str, edits: &[(&str, &str, &str)]) -> (String, String, String) { + let mut all = vec![(WORKFLOW.0, "", WORKFLOW.1), (AGENT, "plugins:", CALLABLE)]; + all.extend_from_slice(edits); + let root = desk(name, &all); + let inv = discover(&root); + let _ = std::fs::remove_dir_all(&root); + let flow = &inv["workflows"][0]; + assert_eq!(flow["name"], "ask-the-desk"); + let text = |v: &serde_json::Value| v.as_str().expect("a digest").to_string(); + ( + text(&inv["digest"]), + text(&flow["doc-digest"]), + text(&flow["contract-digest"]), + ) +} + +#[test] +fn a_settings_edit_moves_what_a_run_relies_on_and_not_what_its_callers_do() { + let (workspace, document, contract) = digests("before", &[]); + let after = digests( + "after", + &[(PLUGIN, "searchMaxResults: 5", "searchMaxResults: 8")], + ); + assert_ne!(workspace, after.0, "the workspace digest moves"); + assert_ne!( + document, after.1, + "the doc-digest of a workflow that reaches the agent moves" + ); + assert_eq!( + contract, after.2, + "the contract-digest does not: a caller hands the same question and gets the same shape back" + ); +} + +fn tree(name: &str) -> PathBuf { + PathBuf::from(concat!(env!("CARGO_MANIFEST_DIR"), "/../../tests/trees")).join(name) +} + +#[test] +fn the_two_trees_say_what_their_readmes_say() { + let (ok, told, text) = check(&tree("a-desk-that-needs-a-plugin")); + assert!(ok, "{text}"); + assert_eq!( + told, + [( + "note".to_string(), + "loader/plugin-runs-only-where-loaded".to_string() + )], + "{text}" + ); + let (ok, told, text) = check(&tree("a-plugin-with-no-registry")); + assert!(!ok, "{text}"); + assert_eq!( + told, + [( + "error".to_string(), + "loader/plugin-package-has-no-registry".to_string() + )], + "the note waits until the registry is known: {text}" + ); +} diff --git a/crates/pact-cli/tests/the_subset_the_second_port_runs.rs b/crates/pact-cli/tests/the_subset_the_second_port_runs.rs index e5eae41..be63636 100644 --- a/crates/pact-cli/tests/the_subset_the_second_port_runs.rs +++ b/crates/pact-cli/tests/the_subset_the_second_port_runs.rs @@ -109,6 +109,10 @@ const COVERED: &[(&str, &str)] = &[ // and the driver only noticed once the payload started carrying the key, // which is the other half of what this file is for. ("knowledge", "knowledge:"), + // D29. In list A for the reason `knowledge:` is: no port loads a plugin, so + // both refuse a `refuse-to-start` one before a model call and name a + // `run-without-it` one on `unenforced`, in the same words. + ("plugins", "plugins:"), ]; /// Declared, and no row anywhere, because nothing about the run turns on it. diff --git a/crates/pact-lang/src/lib.rs b/crates/pact-lang/src/lib.rs index a447b6e..bfa1a97 100644 --- a/crates/pact-lang/src/lib.rs +++ b/crates/pact-lang/src/lib.rs @@ -73,6 +73,7 @@ pub const COLLECTIONS: &[(&str, &str)] = &[ ("topic", "topics"), ("event", "events"), ("reaction", "reactions"), + ("plugin", "plugins"), ]; /// The workspace fields that are one document each, written as a header with diff --git a/crates/pact-loader/Cargo.toml b/crates/pact-loader/Cargo.toml index 1b47006..4837c21 100644 --- a/crates/pact-loader/Cargo.toml +++ b/crates/pact-loader/Cargo.toml @@ -22,6 +22,8 @@ indexmap = { workspace = true } serde_json = { workspace = true } unicode-normalization = { workspace = true } minijinja = { workspace = true } +nodejs-semver = { workspace = true } +pep440_rs = { workspace = true } [dev-dependencies] serde_json = { workspace = true } diff --git a/crates/pact-loader/src/effects.rs b/crates/pact-loader/src/effects.rs index 02afcfd..a89d8ac 100644 --- a/crates/pact-loader/src/effects.rs +++ b/crates/pact-loader/src/effects.rs @@ -138,7 +138,10 @@ pub fn of(document: &Node, target: &str, seen: &mut BTreeSet) -> Effect Effect::NOTHING } -/// An agent: everything on `uses:` and everyone on `team:`. +/// An agent: everything on `uses:`, everyone on `team:`, and every interface +/// a plugin it is loaded with implements. The last is law 1 for code PACT +/// never reads: the plugin is in the agent's runtime whether or not the agent +/// names the interface, so the agent is never reported lighter than it. #[must_use] pub fn of_agent(document: &Node, agent: &Node, seen: &mut BTreeSet) -> Effect { let used: Vec = agent @@ -155,6 +158,11 @@ pub fn of_agent(document: &Node, agent: &Node, seen: &mut BTreeSet) -> E .into_iter() .flat_map(|m| m.keys().cloned()), ) + .chain( + crate::plugins::interfaces_loaded_for(document, agent) + .into_iter() + .map(str::to_string), + ) .collect(); used.iter() .fold(Effect::NOTHING, |e, n| e.and(of(document, n, seen))) diff --git a/crates/pact-loader/src/interfaces.rs b/crates/pact-loader/src/interfaces.rs index 38dacfe..b64e0e5 100644 --- a/crates/pact-loader/src/interfaces.rs +++ b/crates/pact-loader/src/interfaces.rs @@ -30,6 +30,11 @@ //! | `loader/passes-an-argument-the-interface-does-not-take` | a `passes:` line names an argument the interface's action does not take | //! | `loader/an-interface-nothing-implements` (warning) | an interface with no implementation, so a run that reaches it has nothing to use | //! | `loader/the-interface-and-one-of-its-own` (warning) | an agent `uses:` both the interface and one of its implementations | +//! +//! A plugin whose `implements:` names an interface is one of its +//! implementations too (`plugins.rs`): it is counted as one, and may be a +//! choice of `chosen-by: run-inputs.`. Its actions are the interface's +//! by construction, since the plugin writes none. use pact_diag::{Diagnostic, Diagnostics, Span}; use pact_doc::Node; @@ -154,7 +159,9 @@ pub fn check(document: &Node, diags: &mut Diagnostics) { .get(name.as_str()) .cloned() .unwrap_or_default(); - if found.is_empty() { + let by_plugins = crate::plugins::implementing(document, name); + let every: Vec<&str> = found.iter().chain(by_plugins.iter()).copied().collect(); + if every.is_empty() { diags.push(Diagnostic::warning( "loader/an-interface-nothing-implements", entry.key_span.clone(), @@ -163,12 +170,13 @@ pub fn check(document: &Node, diags: &mut Diagnostics) { reaches it has no {noun} to use." ), format!( - "Add a {noun} that says `implements: {name}`. A template may leave \ - this to the copies made from it." + "Add a {noun} that says `implements: {name}`, or name '{name}' on the \ + `implements:` line of the plugin whose code does it. A template may \ + leave this to the copies made from it." ), )); } - chosen_by(document, collection, name, &entry.node, &found, diags); + chosen_by(document, collection, name, &entry.node, &every, diags); if collection == "tools" { actions(entries, name, &entry.node, &entry.key_span, &found, diags); } diff --git a/crates/pact-loader/src/lib.rs b/crates/pact-loader/src/lib.rs index 8f4aa04..d2c096f 100644 --- a/crates/pact-loader/src/lib.rs +++ b/crates/pact-loader/src/lib.rs @@ -163,6 +163,7 @@ pub mod money; pub mod origins; pub mod phrases; pub mod picked; +pub mod plugins; pub mod policy; pub mod ports; pub mod pots; diff --git a/crates/pact-loader/src/plugins.rs b/crates/pact-loader/src/plugins.rs new file mode 100644 index 0000000..50386d4 --- /dev/null +++ b/crates/pact-loader/src/plugins.rs @@ -0,0 +1,683 @@ +//! **A plugin is named and never loaded** (D29). +//! +//! `plugins/.yaml` asks whatever runs these agents to load a package for +//! them. PACT reads the request and nothing else: this pass never resolves, +//! downloads or opens a package (R5), and needs no network to say what it says +//! (D17). What it can hold is the request itself, because each half of it has a +//! grammar that is not PACT's and that a runtime will apply later, somewhere the +//! author is not watching: +//! +//! * **`package:` names its registry.** `npm:@acme/crm-tools`, `pypi:acme-crm`. +//! The word in front is the portability fact — which runtimes can load the +//! line at all — so a package with no registry, or with one no runtime here +//! knows how to load from, is refused rather than guessed at. +//! * **`version:` is a range in that registry's own grammar.** npm ranges as +//! `node-semver` reads them, pypi specifiers as PEP 440 writes them. A range +//! that only the other registry understands is the commonest slip, and a +//! runtime meeting it would refuse to install with the author long gone. +//! * **No secret is written in `config:`.** The settings are the plugin's own +//! and the schema leaves them open, so this is the one rule held inside them: +//! a key that names a token, a secret, a password or an API key carries +//! `{by-reference: }`, never the value, as `resource.auth` does. +//! +//! And one note, because a plugin that cannot be loaded and must be there is a +//! portability fact a reviewer should see before a run finds it: every agent +//! such a plugin applies to runs only where its package can be loaded. +//! +//! **The laws hold for a plugin through the interfaces it implements** +//! (`docs/97-PRIMITIVES.md` §4). A plugin is code PACT never reads, so nothing +//! about what it does can be computed from the code. What can be held is the +//! contract it stands behind: `implements:` names interfaces the tree +//! declares, and those carry every law a checker can see. +//! +//! | Law | Rule | Fires when | +//! |---|---|---| +//! | 4 closed world | `loader/plugin-implements-what-is-not-an-interface` | `implements:` names a tool, procedure or set of documents that does not say `interface: yes` | +//! | 4 closed world | `loader/an-interface-only-an-absent-plugin-implements` (warning) | an agent uses an interface that nothing in the tree implements and that only plugins the agent is not loaded with implement | +//! | 3 exactly once | `loader/a-plugin-action-with-no-claim` | an action of an interface a plugin implements cannot be undone and names no `same-request-key:` | +//! | 1 closure | (`effects::of_agent`) | an agent's computed effect includes the worst action of every interface its plugins implement, so loading a plugin never leaves an agent reporting a lighter effect | +//! | 8 one checker | (`adapters/python` `checking.check_agent_field`) | a learning proposal that changes `plugins:` is held to `pact check` before it is held for a person | + +use pact_diag::{Diagnostic, Diagnostics, Span}; +use pact_doc::{Entry, Map, Node, Value}; +use std::str::FromStr; + +/// The registries a `package:` may name: the two any shipped runtime has a +/// loader for. A third is a spec change, not an author's choice. +pub const REGISTRIES: [&str; 2] = ["npm", "pypi"]; + +/// What `if-it-cannot-be-loaded:` means when it is left out. +pub const REFUSE_TO_START: &str = "refuse-to-start"; + +/// The words a `config:` key is a secret by. Compared against the key's words, +/// so `max-tokens` is a count and `access-token` is a secret. +const SECRET_WORDS: [&str; 3] = ["token", "secret", "password"]; + +/// The plugins `agent` needs, by name, in name order: the ones its own +/// `plugins:` line names and every one whose `applies-to:` is `every-agent`. +/// +/// The one reading of `applies-to:` in the checker, shared with `pact +/// discover`, so the note below and the inventory a runtime indexes can never +/// disagree about which agent a plugin is loaded for. +pub fn needed_by<'a>(document: &'a Node, agent: &Node) -> Vec<(&'a str, &'a Node)> { + let Some(plugins) = document.get("plugins").and_then(Node::as_map) else { + return Vec::new(); + }; + let named = names_on(agent); + let mut out: Vec<(&str, &Node)> = plugins + .iter() + .filter(|(name, entry)| named.contains(&name.as_str()) || for_every_agent(&entry.node)) + .map(|(name, entry)| (name.as_str(), &entry.node)) + .collect(); + out.sort_by_key(|(name, _)| *name); + out +} + +/// The interfaces `plugin` says its code stands behind, as written on +/// `implements:`, in the order written. +pub fn implements(plugin: &Node) -> Vec<&str> { + match plugin.get("implements").map(|p| &p.value) { + Some(Value::List(items)) => items + .iter() + .filter_map(Node::as_str) + .map(str::trim) + .collect(), + Some(Value::Str(s)) => vec![s.trim()], + _ => Vec::new(), + } +} + +/// The plugins whose `implements:` names `interface`, in name order: the +/// implementations of it that are a package rather than an entry of the tree. +pub fn implementing<'a>(document: &'a Node, interface: &str) -> Vec<&'a str> { + document + .get("plugins") + .and_then(Node::as_map) + .into_iter() + .flatten() + .filter(|(_, entry)| implements(&entry.node).contains(&interface)) + .map(|(name, _)| name.as_str()) + .collect() +} + +/// The interfaces the plugins `agent` needs implement, deduplicated, in name +/// order. What `effects::of_agent` adds to the agent's effect (law 1): the +/// code is loaded into the runtime that runs the agent, so whatever the +/// interface can do the agent's runtime can do. +pub fn interfaces_loaded_for<'a>(document: &'a Node, agent: &Node) -> Vec<&'a str> { + let mut out: Vec<&str> = needed_by(document, agent) + .into_iter() + .flat_map(|(_, plugin)| implements(plugin)) + .collect(); + out.sort_unstable(); + out.dedup(); + out +} + +/// The collection an interface name is an entry of, and the entry: tools +/// first, then procedures, then sets of documents, the order `names:` lists. +fn entry_named<'a>(document: &'a Node, name: &str) -> Option<(&'static str, &'a Entry)> { + ["tools", "skills", "knowledge"].into_iter().find_map(|collection| { + document + .get(collection) + .and_then(Node::as_map) + .and_then(|m| m.get(name)) + .map(|entry| (collection, entry)) + }) +} + +/// `if-it-cannot-be-loaded:` as written, or the default when it is not. +pub fn if_it_cannot_be_loaded(plugin: &Node) -> &str { + plugin + .get("if-it-cannot-be-loaded") + .and_then(Node::as_str) + .map_or(REFUSE_TO_START, str::trim) +} + +/// Hold every plugin line to the grammar a runtime will apply to it, and say +/// which agents run only where a required plugin can be loaded. +pub fn check(document: &Node, diags: &mut Diagnostics) { + let Some(plugins) = document.get("plugins").and_then(Node::as_map) else { + return; + }; + for (name, entry) in plugins { + let Some(fields) = entry.node.as_map() else { + continue; + }; + if let Some(config) = fields.get("config") { + no_secret_in(name, &config.node, diags); + } + // A package already refused for its registry has no portability fact to + // note: the error says what to fix first. + if let Some(registry) = registry_of(name, fields, diags) { + version_is_a_range(name, registry, fields, diags); + runs_only_where_loaded(document, name, entry, diags); + } + implements_interfaces(document, name, &entry.node, diags); + } + interfaces_only_absent_plugins_implement(document, diags); +} + +/// Law 4 and law 3 for one plugin: every name on `implements:` is an +/// interface, and every action of one that cannot be undone names its key. +fn implements_interfaces(document: &Node, plugin: &str, node: &Node, diags: &mut Diagnostics) { + let Some(line) = node.as_map().and_then(|m| m.get("implements")) else { + return; + }; + for interface in implements(node) { + // A name that is nowhere: `schema/no-such-name` is already saying so. + let Some((collection, target)) = entry_named(document, interface) else { + continue; + }; + if !target.node.get("interface").is_some_and(crate::workflows::said_yes) { + diags.push(Diagnostic::error( + "loader/plugin-implements-what-is-not-an-interface", + at(line), + format!( + "'{plugin}' says its code implements '{interface}', and '{interface}' is \ + an entry of `{collection}` in its own right, not an interface — so an \ + agent asking '{interface}' is asking the tree's own entry, and nothing \ + the plugin's code does is held to a contract written here." + ), + format!( + "Write `interface: yes` in '{interface}' and move what does the work into \ + the plugin, or take '{interface}' off `implements:`." + ), + )); + continue; + } + if collection == "tools" { + actions_claim_once(plugin, interface, &target.node, diags); + } + } +} + +/// Law 3: the plugin's code performs the interface's actions where no checker +/// sees, so an action it cannot take back carries its claim on the interface, +/// the one place a runtime retrying the call can read it. +fn actions_claim_once(plugin: &str, interface: &str, tool: &Node, diags: &mut Diagnostics) { + let Some(actions) = tool.get("actions").and_then(Node::as_map) else { + return; + }; + for (action, entry) in actions { + if crate::effects::of_action(&entry.node).level != crate::effects::Level::CannotBeUndone + || entry.node.get("same-request-key").is_some() + { + continue; + } + diags.push(Diagnostic::error( + "loader/a-plugin-action-with-no-claim", + entry.key_span.clone(), + format!( + "'{interface}/{action}' cannot be undone and names no same-request key, and \ + '{plugin}' implements it in code PACT never reads — so a runtime that \ + retries the call, or a model that asks twice, does it twice." + ), + format!( + "Write `same-request-key: ` on '{interface}/{action}', naming the \ + argument that makes two calls the same call, or `undone-by:` naming the \ + action that takes it back." + ), + )); + } +} + +/// Law 4 at the agent: an interface the agent uses whose every implementation +/// is a plugin the agent is not loaded with leaves a run of it nothing to use. +fn interfaces_only_absent_plugins_implement(document: &Node, diags: &mut Diagnostics) { + let Some(agents) = document.get("agents").and_then(Node::as_map) else { + return; + }; + for (key, agent) in agents { + if agent.node.get("base").is_some_and(crate::workflows::said_yes) { + continue; + } + let loaded: Vec<&str> = needed_by(document, &agent.node) + .into_iter() + .map(|(name, _)| name) + .collect(); + let used = agent + .node + .get("uses") + .map(crate::workflows::list) + .unwrap_or_default(); + for used in used.into_iter().filter_map(Node::as_str).map(str::trim) { + let by_plugins = implementing(document, used); + if by_plugins.is_empty() + || by_plugins.iter().any(|p| loaded.contains(p)) + || implemented_in_the_tree(document, used) + { + continue; + } + diags.push(Diagnostic::warning( + "loader/an-interface-only-an-absent-plugin-implements", + agent.key_span.clone(), + format!( + "'{key}' uses '{used}', which only {} implement{} — and '{key}' is not \ + loaded with {}, so a run of it has nothing behind '{used}'.", + quoted(&by_plugins), + if by_plugins.len() == 1 { "s" } else { "" }, + if by_plugins.len() == 1 { "it" } else { "any of them" }, + ), + format!( + "Add {} to `plugins:` in '{key}'.", + by_plugins + .iter() + .map(|p| format!("'{p}'")) + .collect::>() + .join(" or ") + ), + )); + } + } +} + +/// Whether an entry of the tree says `implements: `. +fn implemented_in_the_tree(document: &Node, interface: &str) -> bool { + ["tools", "skills", "knowledge"].into_iter().any(|collection| { + document + .get(collection) + .and_then(Node::as_map) + .into_iter() + .flatten() + .any(|(_, e)| e.node.get("implements").and_then(Node::as_str).map(str::trim) == Some(interface)) + }) +} + +/// The registry `package:` names, or `None` after saying why it names none. +fn registry_of<'a>(name: &str, fields: &'a Map, diags: &mut Diagnostics) -> Option<&'a str> { + let package = fields.get("package")?; + let written = package.node.as_str()?.trim(); + let registry = written.split_once(':').map(|(r, _)| r.trim()); + if let Some(r) = registry.filter(|r| REGISTRIES.contains(r)) { + return Some(r); + } + let what = match registry { + Some(r) if !r.is_empty() => { + format!("names the registry '{r}', which no runtime loads from") + } + _ => "names no registry".to_string(), + }; + diags.push(Diagnostic::error( + "loader/plugin-package-has-no-registry", + at(package), + format!( + "'{name}' asks for the package '{written}', which {what} — so no runtime can \ + say whether it is one it can load." + ), + format!( + "Write one of the registries a runtime loads from — {} — in front of the name: \ + `npm:{bare}` for an npm package, `pypi:{bare}` for a Python one.", + REGISTRIES + .iter() + .map(|r| format!("`{r}`")) + .collect::>() + .join(" or "), + bare = written.rsplit_once(':').map_or(written, |(_, n)| n).trim(), + ), + )); + None +} + +/// Refuse a `version:` its registry would not read as a range. +fn version_is_a_range(name: &str, registry: &str, fields: &Map, diags: &mut Diagnostics) { + let Some(version) = fields.get("version") else { + return; + }; + let written = match &version.node.value { + Value::Str(s) => s.trim().to_string(), + // `version: 1.4` arrives as a number; the schema reports the type, and + // this pass has nothing further to say about a line that is not text. + _ => return, + }; + let (reads, example) = match registry { + "npm" => ( + nodejs_semver::Range::parse(&written).is_ok(), + "`^1.4.0` (1.4.0 or any later 1.x), `~1.4.0` (any 1.4.x) or `>=1.4.0 <2.0.0`", + ), + _ => ( + pep440_rs::VersionSpecifiers::from_str(&written).is_ok(), + "`>=1.4,<2` (1.4 or any later 1.x), `~=1.4` or `==1.4.2`", + ), + }; + if reads { + return; + } + diags.push(Diagnostic::error( + "loader/plugin-version-is-not-a-range", + at(version), + format!( + "'{name}' asks for versions '{written}', which {registry} does not read as a \ + range of versions — so a runtime cannot tell which versions will do." + ), + format!("Write the range the way {registry} does, for example {example}."), + )); +} + +/// Refuse a secret written into a plugin's own settings, at any depth. +fn no_secret_in(name: &str, node: &Node, diags: &mut Diagnostics) { + match &node.value { + Value::Map(map) => { + for (key, entry) in map { + if names_a_secret(key) && !by_reference(&entry.node) { + diags.push(Diagnostic::error( + "loader/plugin-secret-written-in-the-file", + entry.key_span.clone().merge(&entry.node.span), + format!( + "'{name}' writes its `{key}` into the file, and a file in a \ + workspace is read by everyone who reviews, copies or checks it out." + ), + format!( + "Replace the value with where it is kept: `{key}: {{by-reference: \ + }}`." + ), + )); + continue; + } + no_secret_in(name, &entry.node, diags); + } + } + Value::List(items) => items.iter().for_each(|i| no_secret_in(name, i, diags)), + _ => {} + } +} + +/// Whether a settings key names a secret: one of its words is `token`, +/// `secret` or `password`, or two of them are `api` then `key`. +/// +/// Words and not substrings, because a plugin's settings are full of counts +/// named after tokens (`max-tokens`), and a rule that refused those would be +/// switched off by the first author it annoyed. +fn names_a_secret(key: &str) -> bool { + let words = words_of(key); + words.iter().any(|w| SECRET_WORDS.contains(&w.as_str())) + || words.windows(2).any(|p| p[0] == "api" && p[1] == "key") + || words.iter().any(|w| w == "apikey") +} + +/// `apiKey`, `api_key`, `API-Key` and `api key` as the same four words. +fn words_of(key: &str) -> Vec { + let mut words = Vec::new(); + let mut word = String::new(); + let mut previous_lower = false; + for c in key.chars() { + if !c.is_alphanumeric() { + if !word.is_empty() { + words.push(std::mem::take(&mut word)); + } + previous_lower = false; + continue; + } + if c.is_uppercase() && previous_lower && !word.is_empty() { + words.push(std::mem::take(&mut word)); + } + previous_lower = c.is_lowercase(); + word.extend(c.to_lowercase()); + } + if !word.is_empty() { + words.push(word); + } + words +} + +/// `{by-reference: }` and nothing else: the one shape a secret may take. +fn by_reference(node: &Node) -> bool { + node.as_map().is_some_and(|m| { + m.len() == 1 + && m.get("by-reference") + .and_then(|r| r.node.as_str()) + .is_some_and(|r| !r.trim().is_empty()) + }) +} + +/// Say which agents a `refuse-to-start` plugin keeps to the runtimes that can +/// load its package. A note, so `--deny-warnings` stays usable on a shipped tree +/// that means it. +fn runs_only_where_loaded(document: &Node, name: &str, entry: &Entry, diags: &mut Diagnostics) { + if if_it_cannot_be_loaded(&entry.node) != REFUSE_TO_START { + return; + } + let Some(package) = entry.node.get("package").and_then(Node::as_str) else { + return; + }; + let agents: Vec<&str> = document + .get("agents") + .and_then(Node::as_map) + .into_iter() + .flatten() + .filter(|(_, a)| !a.node.get("base").is_some_and(crate::workflows::said_yes)) + .filter(|(_, a)| needed_by(document, &a.node).iter().any(|(n, _)| *n == name)) + .map(|(key, _)| key.as_str()) + .collect(); + if agents.is_empty() { + return; + } + diags.push(Diagnostic::note( + "loader/plugin-runs-only-where-loaded", + entry.key_span.clone(), + format!( + "{} only where `{}` can be loaded: '{name}' says `refuse-to-start`, so on a \ + runtime that cannot load it {} before the first step.", + if agents.len() == 1 { + format!("{} runs", quoted(&agents)) + } else { + format!("{} run", quoted(&agents)) + }, + package.trim(), + if agents.len() == 1 { + "the agent stops" + } else { + "each of them stops" + }, + ), + format!( + "Nothing to do if that is what you mean. Write `if-it-cannot-be-loaded: \ + run-without-it` in `plugins/{name}.yaml` for an agent that can do its job \ + without it — every run then says the plugin was not there." + ), + )); +} + +/// The names an agent's own `plugins:` line writes. +fn names_on(agent: &Node) -> Vec<&str> { + match agent.get("plugins").map(|p| &p.value) { + Some(Value::List(items)) => items + .iter() + .filter_map(Node::as_str) + .map(str::trim) + .collect(), + Some(Value::Str(s)) => vec![s.trim()], + _ => Vec::new(), + } +} + +/// Whether a plugin says, on its own line, that it is loaded for every agent. +fn for_every_agent(plugin: &Node) -> bool { + plugin + .get("applies-to") + .and_then(Node::as_str) + .map(str::trim) + == Some("every-agent") +} + +/// The line the author typed: the key and its value. +fn at(entry: &Entry) -> Span { + entry.key_span.clone().merge(&entry.node.span) +} + +/// `'a'`, `'a' and 'b'`, `'a', 'b' and 'c'`. +fn quoted(names: &[&str]) -> String { + let q: Vec = names.iter().map(|n| format!("'{n}'")).collect(); + match q.split_last() { + Some((last, rest)) if !rest.is_empty() => format!("{} and {last}", rest.join(", ")), + Some((last, _)) => last.clone(), + None => String::new(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use pact_doc::parse_yaml; + + const WORKSPACE: &str = " +agents: + desk: + description: Answers. + instructions: Answer. + plugins: [web-search] + clerk: + description: Files. + instructions: File. +plugins: + web-search: + description: Searches the web. + package: npm:@acme/web-search + version: ^0.2.0 + if-it-cannot-be-loaded: run-without-it + config: + results-at-most: 5 + max-tokens: 400 + api-key: { by-reference: host/search-key } +"; + + fn check_text(text: &str) -> Diagnostics { + let node = parse_yaml(text, camino::Utf8Path::new("workspace.yaml")).expect("parses"); + let mut d = Diagnostics::new(); + d.add_source("workspace.yaml", text); + check(&node, &mut d); + d + } + + fn rules(d: &Diagnostics) -> Vec { + d.items().iter().map(|x| x.rule.to_string()).collect() + } + + #[test] + fn a_plugin_written_the_way_its_registry_reads_it_is_left_alone() { + let d = check_text(WORKSPACE); + assert!(d.is_empty(), "{}", d.render()); + } + + #[test] + fn a_package_with_no_registry_or_an_unknown_one_is_refused() { + for (written, says) in [ + ("\"@acme/web-search\"", "names no registry"), + ("cargo:web-search", "names the registry 'cargo'"), + ] { + let d = check_text(&WORKSPACE.replace("npm:@acme/web-search", written)); + assert_eq!( + rules(&d), + ["loader/plugin-package-has-no-registry"], + "{}", + d.render() + ); + let hit = &d.items()[0]; + assert!(hit.message.contains(says), "{}", hit.message); + assert!( + hit.fix.contains("`npm:") && hit.fix.contains("`pypi:"), + "{}", + hit.fix + ); + } + } + + #[test] + fn a_range_only_the_other_registry_reads_is_refused() { + let d = check_text(&WORKSPACE.replace("^0.2.0", "\">=0.2,<1\"")); + assert_eq!( + rules(&d), + ["loader/plugin-version-is-not-a-range"], + "{}", + d.render() + ); + let pypi = WORKSPACE.replace("npm:@acme/web-search", "pypi:acme-web-search"); + let d = check_text(&pypi); + assert_eq!( + rules(&d), + ["loader/plugin-version-is-not-a-range"], + "{}", + d.render() + ); + assert!( + d.items()[0].fix.contains(">=1.4,<2"), + "{}", + d.items()[0].fix + ); + let d = check_text(&pypi.replace("^0.2.0", "\">=0.2,<1\"")); + assert!(d.is_empty(), "{}", d.render()); + } + + #[test] + fn a_secret_written_into_the_settings_is_refused_wherever_it_sits() { + for (key, value) in [ + ("api-key", "sk-live-2f9c"), + ("apiKey", "sk-live-2f9c"), + ("access-token", "abc"), + ("client_secret", "abc"), + ("password", "{ value: hunter2 }"), + ] { + let text = WORKSPACE.replace( + " api-key: { by-reference: host/search-key }", + &format!(" nested:\n {key}: {value}"), + ); + let d = check_text(&text); + assert_eq!( + rules(&d), + ["loader/plugin-secret-written-in-the-file"], + "{key}: {}", + d.render() + ); + } + } + + #[test] + fn a_count_named_after_tokens_is_not_a_secret() { + assert!(!names_a_secret("max-tokens")); + assert!(!names_a_secret("keyboard-layout")); + assert!(names_a_secret("API_KEY")); + assert!(names_a_secret("bearerToken")); + } + + #[test] + fn a_plugin_an_agent_cannot_start_without_is_a_note_naming_it() { + let d = check_text(&WORKSPACE.replace(" if-it-cannot-be-loaded: run-without-it\n", "")); + assert_eq!( + rules(&d), + ["loader/plugin-runs-only-where-loaded"], + "{}", + d.render() + ); + let hit = &d.items()[0]; + assert!( + hit.message.contains("'desk'") && !hit.message.contains("'clerk'"), + "{}", + hit.message + ); + assert!( + !d.has_errors() + && d.items() + .iter() + .all(|x| x.severity == pact_diag::Severity::Note) + ); + } + + #[test] + fn every_agent_means_every_agent_that_runs() { + let text = WORKSPACE + .replace( + " if-it-cannot-be-loaded: run-without-it\n", + " applies-to: every-agent\n", + ) + .replace(" plugins: [web-search]\n", "") + .replace(" clerk:\n", " house:\n base: yes\n clerk:\n"); + let d = check_text(&text); + let hit = &d.items()[0]; + assert!( + hit.message.contains("'desk' and 'clerk'"), + "{}", + hit.message + ); + assert!(!hit.message.contains("'house'"), "{}", hit.message); + } +} diff --git a/crates/pact-loader/src/policy.rs b/crates/pact-loader/src/policy.rs index 8de781b..1632f40 100644 --- a/crates/pact-loader/src/policy.rs +++ b/crates/pact-loader/src/policy.rs @@ -304,6 +304,10 @@ impl Default for Policy { // after the kind, so the folder is the thing and the file // describes it — the shape `agent` and `skill` already have. "program", + // D29. `plugins/web-search/plugin.yaml` is the plugin, as + // `tools/weather/tool.yaml` is the tool: added with the kind + // rather than one round after it. + "plugin", // `redaction` is deliberately NOT here, and the reason is the // `learning.yaml` sentence above rather than an oversight. // `agent.policy` is a NAME (`policy: approvals`), so the cost of diff --git a/crates/pact-loader/src/unnamed.rs b/crates/pact-loader/src/unnamed.rs index b53d5ae..2909ef7 100644 --- a/crates/pact-loader/src/unnamed.rs +++ b/crates/pact-loader/src/unnamed.rs @@ -91,6 +91,13 @@ const KINDS: &[Kind] = &[ // named by nobody, `pact check` printed "OK — loaded cleanly", and `pact // waits` silently lost the whole `needs-permission` entry: a human consent // gate on money, gone, from the list §9.4 G14 obliges a runtime to walk. + // D29. A plugin no agent names and that does not apply to every agent is + // loaded for nobody, and the author who wrote it believes some agent has it. + Kind { + section: "plugins", + what: "plugin", + attach: "`plugins:` with `- {}` under it, on an agent — or `applies-to: every-agent` in the plugin itself", + }, Kind { section: "programs", what: "carried program", diff --git a/docs/01-DECISIONS.md b/docs/01-DECISIONS.md index e351283..a146d25 100644 --- a/docs/01-DECISIONS.md +++ b/docs/01-DECISIONS.md @@ -247,6 +247,73 @@ desirable but is not a failure condition. --- +## D29 — Runtime plugins: named, versioned, governed + +**Date:** 2026-10-26. **Status:** Binding, decided by the project owner. Narrows +R17 (`50-NOT-COPIED.md` §3.3) and D15. + +A workspace may ask whatever runs its agents to load a package for them. +`plugins/.yaml` says which package (`:`), which versions +will do, its own settings, whether an agent may start without it, and which +agents it is for; an agent names the plugins it needs on its `plugins:` line. The +request is part of the document — read, diffed, digested and reviewed like every +other line. The code never is. + +**Consequences:** + +1. `plugins:` is a kind at `S-EXEC` (`workspace.plugins`, `agent.plugins`, the + `plugin` group), so D23 classifies adding or changing one as needing a person. + The learning classifier holds it there with or without + `needs-a-person-to-approve: [plugins]`. +2. Checking never resolves, downloads or opens a package: R5 and D17 hold. + `pact check` holds the request to its registry's grammar — a registry word PACT + knows, an npm or PEP 440 range, no secret written into `config:` — and notes + every agent a `refuse-to-start` plugin keeps to the runtimes that can load it. +3. Runtimes advertise a `plugins` column in the lattice. A runtime that did not + load a `refuse-to-start` plugin stops the agent before its first model call + (`halted: plugins-not-loaded`, with the sentence naming the plugin, its package + and its range); one that did not load a `run-without-it` plugin names it on + `unenforced` on every run. Every transport this repository ships publishes + `plugins: unsupported`, because the reference harness loads nothing; a host + that loads one says what it loaded through `run(loaded_plugins=)`. +4. D15 is narrowed. An agent with a `refuse-to-start` plugin is portable only to + runtimes that load its registry, and `pact discover` publishes each agent's + plugins with that line written out, so a runtime can decline the agent before + it takes it. +5. D27's conformance claim covers agents with no plugins and agents whose plugins + are all `run-without-it`. +6. What a plugin does once loaded is held by the runtime that loaded it. PACT + cannot govern code it never reads, so `allow-egress:`, `redaction.yaml` and + the interceptor chain bound the parts of a run PACT can see and say nothing + about a plugin's own traffic. +7. **The laws of `97-PRIMITIVES.md` §4 hold for a plugin through the interfaces + it implements** (amended 2026-11-11, on master after #4). What a plugin lets an + agent ask is written as an interface the tree declares (`interface: yes`), and + the plugin names it on `implements:`; the plugin supplies the code behind it + and nothing about the contract. + - *Law 4, closed world:* `implements:` names only interfaces + (`loader/plugin-implements-what-is-not-an-interface`). A plugin counts as one + of an interface's implementations, may be a choice of `chosen-by: + run-inputs.`, and an agent using an interface that only plugins it is + not loaded with implement is warned + (`loader/an-interface-only-an-absent-plugin-implements`). A plugin that + implements nothing gives an agent nothing to ask. + - *Law 1, closure:* an agent's computed effect includes the worst action of + every interface its plugins implement, named on `uses:` or not, because the + code is in its runtime either way. Loading a plugin never leaves an agent + reporting a lighter effect. + - *Law 3, exactly once:* an action of an interface a plugin implements that + cannot be undone names its `same-request-key:` + (`loader/a-plugin-action-with-no-claim`). The code that does it is code no + checker sees, so the interface is the one place a runtime retrying the call + can read the claim. + - *Law 8, one checker:* a learning proposal that changes an agent's `plugins:` + is held to `pact check` before it is held for a person, so nobody is asked + to approve a tree that does not load (`checking.check_agent_field`, + `learning.CHECKED_BEFORE_HELD`). + +--- + ## Consolidated impact on the thesis | Thesis element | Change from decisions | @@ -255,7 +322,7 @@ desirable but is not a failure condition. | T2 Evals as oracle | **Strengthened** — D19 adds four no-code on-ramps plus full-fidelity expert upload; D13/D14 make no-code authoring mandatory for *every* feature. | | T3 Two-level lowering | **Confirmed** (D12), with D27 as the pass/fail rule and D26 the cost ceiling. | | T4 Model portability | **Extended** — D11 makes the Resolver a recommender, not only a gate. | -| T5 Expansion Rule | **Promoted to the centre** — D2 makes the tree the native form, so the loader is normative and the Expansion Rule is the execution model's own semantics, not authoring sugar. | +| T5 Expansion Rule | **Promoted to the centre** — D2 makes the tree the native form, so the loader is normative and the Expansion Rule is the execution model's own semantics, not authoring sugar. **Extended** by D29 to `plugins/`: a request for code a runtime loads is one more folder of definitions, and the code itself never enters the tree. | | T6 Learning emits source | **Strengthened** — D22 admits self-authored tools and topology change, so D23's blast-radius classifier and signed provenance become mandatory. | | T7 Structural honesty | Unchanged; D11 adds the recommendation obligation. | | NG1 "not a runtime" | **Revised.** D2 requires the tree to be natively executable and D12 gives PACT loop semantics. PACT is not a *server*, but it does own an execution model that `gaia-ai-runtime` implements. | diff --git a/docs/20-ARCHITECTURE-DRAFT.md b/docs/20-ARCHITECTURE-DRAFT.md index 9d8f48e..d9d51b4 100644 --- a/docs/20-ARCHITECTURE-DRAFT.md +++ b/docs/20-ARCHITECTURE-DRAFT.md @@ -377,8 +377,11 @@ zone computation and blob verification are new work. > it, `open` is gone from `Group` and from `from_doc`, and every field of the shipped > schema carries `surface:` and `tier:` — checked at startup by `governance_is_complete`, > which refuses to validate against a specification that leaves any field without them. -> Point 2 (`spec/extensions/*.yaml`) is not built: there is no extension path at all, and -> `x-` on a field is what an author has. Point 1's sha256 comparison against a compiled-in +> Point 2 (`spec/extensions/*.yaml`) is not built: there is no extension path for the +> schema, and `x-` on a field is what an author has — preserved in the IR and never emitted +> into any substrate. The runtime extension path that does exist is `plugins:` (D29): a +> named, versioned request for code the runtime loads, validated like every other kind and +> never resolved, fetched or run by the checker. Point 1's sha256 comparison against a compiled-in > constant is deliberately NOT built — a constant compiled in beside the file it hashes can > only ever agree with itself, so the digest is COMPUTED and REPORTED instead, and the > property actually verified at startup is the one §8.2 depends on. @@ -8947,6 +8950,7 @@ it would have *"nothing"* in the column headed *"what it decides"*. | `loop:` and `loops/` — `based-on`, `starts-at`, `does`, `says`, `may-use`, `then`, `at-most` | which stage runs at each step, what it is told, what it may reach (G1) | same, via `_loops_of` | | `limits:` — `steps-at-most`, `tool-calls-at-most`, `runs-for-at-most` / `finishes-within`, `cost-per-request-under`, `tokens-at-most`, `when-it-runs-out` | when the run stops, which of the three actions it takes, and the words a person reads (G2) | `test_the_typescript_port_stops_at_the_same_ceiling_and_says_the_same_words`, and `test_both_ports_read_every_way_a_spend_cap_is_written.py` for `cost-per-request-under`. **The second citation was missing for as long as this row existed, and the cell was false without it**: the first test sends the money cap written ` ` and no other way, so four of the six spellings `coerce::money` accepts — `USD 0.05`, `$0.05`, `0.05 usd` and `0.05USD`, all `pact check` rc=0 — produced a different sentence in the two ports with this row reading as held. Nothing structural holds this column: the two tests below check `AGENT_SPEC_FIELDS` and list B, not the Held-by cell | | `answers-with:` | the shape the model is told its answer must have, appended last under one heading (A5) | `test_the_typescript_target_agrees_too` | +| `plugins:` — `if-it-cannot-be-loaded` | whether the turn is refused before a model is called and in what words, and the line each plugin it starts without puts on `unenforced` (D29). Neither port loads a plugin, so the two agree by construction; only the reference port can be handed `loaded_plugins=` by a host that does | `test_a_plugin_nobody_loaded_stops_the_run_before_it_starts`, `test_a_plugin_the_agent_can_do_without_is_named_on_unenforced`, and `test_every_agent_runs_identically_on_the_typescript_target` over `examples/plugin-desk` | Agreement is on the INPUTS as well as the outputs, which is what makes the first four rows evidence rather than assertion: `run-trace.ts` records `told` and `offered` — the system diff --git a/docs/50-NOT-COPIED.md b/docs/50-NOT-COPIED.md index 837129f..d0b95c9 100644 --- a/docs/50-NOT-COPIED.md +++ b/docs/50-NOT-COPIED.md @@ -382,6 +382,19 @@ already covers. To share across workspaces, copy the folder — which is a thing you can read, diff and sign, and which behaves identically wherever it lands. +**Narrowed by D29.** One part of this refusal has been taken back, and it is the +part that was never about definitions: a workspace may now *ask* its runtime for +a package. `plugins/.yaml` names the package with its registry, the +versions that will do, its settings and whether an agent may start without it, +and an agent names the plugins it needs. That is a request written in the folder +— reviewed, diffed and digested like every other line — and not a mount. PACT +never resolves, fetches or runs the package (R5); a runtime that did not load it +refuses to start the agent or names the plugin on every run; and the `plugins` +lattice column says before anything runs which runtimes can load one at all. What +stands is the refusal of packages that bring *definitions* into the tree: a +bundle is still a folder of definitions PACT reads, mounting one is still unbuilt +(§6), and nothing a plugin does at run time is written back into the folder. + ### 3.4 Checks that can only be written as code **What Eve ships.** Evaluations as imperative TypeScript with four judge-based @@ -424,6 +437,7 @@ knows what a Slack block looks like."* | One model summarising another's history | `context-policy.summarised-by:` — which model writes the summary, resolved against the same catalogue | the summarising call itself, through `write_summary()` on the transport. A transport that cannot re-bind to a second model reports `summarised-by` on `unmetered`, so a rung of the author's ladder never silently fails to fire | | Author-set attributes on a trace (`experimental_setAttributes`) | `watch` — which moments to record and what each one carries. A moment is nameable in the format; a span is not, because a span is a shape one tracing vendor chose | the exporter, and the trace itself. PACT names the moment; what a moment becomes on the wire is the host's | | An integration catalogue of servers a machine can reach | `resource.endpoint` — the author names one server, by a name their platform team publishes | the list of what this machine actually has. A registry of it in the format would make the portable artifact depend on one host's inventory (R20) | +| Extensions mounted from packages, with versioned contracts | `plugins:` — the package with its registry, a version range, its settings, `if-it-cannot-be-loaded:` and `applies-to:` | resolving, installing, isolating and loading the package, and the `plugins` column of its lattice that says whether it can (D29) | The line is worth stating once, because it is the same line every time: **PACT holds what a reviewer must be able to read, and what must be identical on two @@ -463,7 +477,7 @@ a restatement, and points at the section that argues it. | R14 | A join rule written as an expression | same as R6; and five named ways to wait cover what teams actually do | `waits-for`, plus a schema edit when a sixth is needed | §2 (G8) | | R15 | A hosted catalogue as the source of model facts | a machine with no network must still be able to validate and resolve, and a fetched catalogue makes that impossible | a local catalogue file; better, say what the model must be able to do and let PACT pick | §3.1 | | R16 | Orchestration code the model writes while it runs | it cannot be reviewed before it runs, diffed, signed, or reproduced — and structural change is exactly what needs a person | `team:` and `teamwork:`; structural change arrives as a diff. Who an agent may bring in while it runs is `teamwork.may-start:`, inside `starts-at-most:` and `nests-at-most:` | §3.2, **narrowed — see its last paragraph** | -| R17 | Extensions as installable packages | naming already solves sharing, and a package layer costs the property that everything the system does is in the folder you were handed | put the shared thing at the workspace root and name it | §3.3 | +| R17 | Extensions as installable packages | naming already solves sharing, and a package layer costs the property that everything the system does is in the folder you were handed | put the shared thing at the workspace root and name it; code the runtime loads is asked for by `plugins:` (D29) | §3.3, **narrowed — see its last paragraph** | | R18 | Checks that can only be written as code | the evaluation suite is what decides whether a port worked, so a suite most authors cannot write makes most agents unportable | example cases, plain rules, or promoted conversations; code stays possible, never necessary | §3.4 | | R24 | An interceptor power a written rule cannot reach — `change-the-request` and `change-the-answer` were choices in `may:` that no sentence in the vocabulary produces, so declaring one and then writing any rule got the rule refused by the next check down | a choice a non-coder can type that nothing can ever use reads as a capability and is worse than an absent one; and offering it made the list of five look like five answers when three of them were the answers | the three that reach something: `hide-values`, `stop-the-run`, `send-elsewhere`. Rewriting a request or an answer stays available to the system running the agent, through the typed escape §5.5 requires — see §6 below | §2 (G5), **amended — see §8.5** | | R25 | A **guessed** number for how much a model can hold, so that `context-policy:` always appears to work | a policy measured against an invented budget tidies at the wrong moment and reports that it tidied, which is the silent degradation T7 exists to name; a spend cap with no price list is already reported rather than guessed and this is the same shape | a **sourced** row in `models/catalog.yaml` — a figure with the place it was read from and the date beside it, or the word `unknown`. A model no row covers, on a transport that will not say, leaves the policy on `unmetered` with `session.limit.failed` naming it | §4 | @@ -700,8 +714,8 @@ section used to count, which was the plan's §1 table: the same inventory *condensed* into eleven areas and 58 bullets, with the other 22 in an exploration output that never reached this repository. -> **101 capabilities are enumerated and 101 are accounted for — 49 expressible, -> 22 declared and delegated, 30 refused or deferred, none outside.** +> **101 capabilities are enumerated and 101 are accounted for — 51 expressible, +> 22 declared and delegated, 28 refused or deferred, none outside.** One row moved this round. **85** — prompt-cache markers — was (a), citing `limits.reuse-context`; that field appeared exactly once in the whole repository, @@ -709,11 +723,16 @@ in the schema itself, so the citation resolved to nothing an author could act on and the field is deleted. Where a cache breakpoint goes is a fact about one provider's wire format, so the row is (c) and its reason is **R46**. +Two rows moved the other way with D29. **84** and **99** — Eve's extensions, +mounted and defined — were (c), against R17 and §6. Naming a package for the +runtime to load is now `workspace.plugins`, so each is (a) for that half, and +loading it is the host's (§8.6). + | | Category | Rows | What holds it | |---|---|---|---| -| **(a)** | An author writes it, in `spec/schema.yaml` | 49 | `authoring_surface.rs` — every field explains itself; `eve_inventory.rs::every_expressible_row_names_a_field_the_schema_actually_has` — the field it points at exists | +| **(a)** | An author writes it, in `spec/schema.yaml` | 51 | `authoring_surface.rs` — every field explains itself; `eve_inventory.rs::every_expressible_row_names_a_field_the_schema_actually_has` — the field it points at exists | | **(b)** | Declared here, executed by the host (§4) | 22 | the schema comment at each point of declaration, and `eve_inventory.rs::every_delegated_row_names_a_row_of_section_four` | -| **(c)** | Refused (§5) or deferred (§6) | 30 | `deliberate_refusals.rs` — the reason is a constraint | +| **(c)** | Refused (§5) or deferred (§6) | 28 | `deliberate_refusals.rs` — the reason is a constraint | | | **Total** | **101** | `crates/pact-cli/tests/eve_inventory.rs` | ### 8.1 Why the count is 101 and not 80 @@ -750,12 +769,12 @@ that fits none of the three letters, and there is not one. | Channels | 9 | 4 | 4 | 1 — one channel writing into another's session (R23) | | Auth | 6 | 1 | 5 | 0 | | Sandbox | 5 | 3 | 2 | 0 | -| Runtime | 11 | 2 | 6 | 3 — five separate kinds of pause (R12); extensions as installable packages (R17); prompt-cache markers placed for the provider that supports them (R46) | +| Runtime | 11 | 3 | 6 | 2 — five separate kinds of pause (R12); prompt-cache markers placed for the provider that supports them (R46) | | Evals | 4 | 2 | 0 | 2 — checks that can only be written as code (R18) | -| Developer experience, catalogue, templates | 11 | 3 | 2 | 6 — the typed client and the framework hooks, and the terminal app (R4); a verb that scaffolds a workspace (§6, deferred); capability sets as installable packages (§6, deferred); the model orchestrating subagents from JavaScript it wrote (R5, R42) | -| **Total** | **101** | **49** | **22** | **30** | +| Developer experience, catalogue, templates | 11 | 4 | 2 | 5 — the typed client and the framework hooks, and the terminal app (R4); a verb that scaffolds a workspace (§6, deferred); the model orchestrating subagents from JavaScript it wrote (R5, R42) | +| **Total** | **101** | **51** | **22** | **28** | -Five of the thirty are **deferred rather than refused** — `toModelOutput`, +Four of the twenty-eight are **deferred rather than refused** — `toModelOutput`, read-before-write, a teammate on another deployment, and a verb that scaffolds a workspace — and sit in §6 with the fixture that lets each back in. The other twenty-four are ledger rows in §5. @@ -781,7 +800,7 @@ refuse. Each of the five had a wrong version that would have passed its own test the sharpest being the last, where the wording was the author's and the contract was correct and the two named different things. -**Four rows are split, and the split is named rather than rounded.** Each is +**Six rows are split, and the split is named rather than rounded.** Each is lettered for the half PACT answers with, and the same cell names the other half: * *markdown-or-module* — the markdown half is (a); a slot written as a module is @@ -794,6 +813,9 @@ lettered for the half PACT answers with, and the same cell names the other half: * *an integration catalogue* — naming one server is (a), `resource.endpoint:`; the list of servers a particular machine has is the host's, §4 and R20. It was lettered (a) against `pact tools list`, a command that does not exist. +* *extensions mounted from packages* and *`defineExtension`* — naming the + package, its range and its settings is (a), `workspace.plugins` and + `plugin.package`; loading it is the host's, §4 (§8.6). ### 8.3 A refusal that was withdrawn @@ -866,6 +888,32 @@ carries the thing it rewrites. A rewriter with nothing to run it leaves the word exactly as they were and says so — half-applying would leave an author believing their program had run. +### 8.6 R17 is narrowed, and this is the row that says so + +**R17 refused extensions as installable packages**, because naming already solves +sharing and a package layer costs the property that everything the system does is +in the folder you were handed. Both halves of that reason are about DEFINITIONS — +the skills, tools and policies a package would carry into a tree — and both still +hold for them: a bundle is a folder PACT reads, and mounting one is unbuilt. + +What D29 admits is the other thing a package can be: code a runtime loads, asked +for by name. Eve's `defineExtension` (row 99) and its mounted extensions (row 84) +are each two powers under one name, and each row is now split the way §8.2's +other four are — lettered for the half PACT answers with, the other half named in +the same cell: + +* *naming the package* — which registry, which versions, its settings and what + an agent does without it — is (a), `workspace.plugins` and `plugin.package`; +* *loading it* is the host's, §4, and the `plugins` lattice column is where a + runtime says whether it can. + +So two rows move from (c) to (a), and R17 is amended rather than deleted for the +reason §8.3 gives about R29: a row that quietly disappears is indistinguishable +from one nobody noticed. Its general refusal stands. Checking a tree is still an +act of reading (R5), an `x-` field is still preserved in the IR and never emitted +into any substrate, and the folder still holds everything a reviewer has to read +— the request for code included, and the code excluded. + ### 8.4 What this section used to say, and why it changed It used to end on an admission: 58 of 58 accounted for, and *"the remaining 22 of diff --git a/docs/90-REVIEW.md b/docs/90-REVIEW.md index ddc1afe..8b479b5 100644 --- a/docs/90-REVIEW.md +++ b/docs/90-REVIEW.md @@ -80,8 +80,8 @@ them and changes nothing. | | | |---|---| | Schema | **43 groups, 262 field entries, 207 distinct field names** (core 123, expert 69, both 15), 40 closed enums, 27 cross-reference constraints, **0 fields missing `tier:` or `surface:`** | -| Rust | 5 crates, **1648 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | -| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3321 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | +| Rust | 5 crates, **1673 tests**, `unsafe_code = forbid`, clippy-deny, edition 2024 | +| Python | **47 modules** (`find adapters/python/src -name "*.py" \| wc -l`), **3340 tests**, **nine transports** — seven model-bound (`anthropic`, `autogen`, `langchain`, `langgraph`, `ollama`, `openai_agents`, `pydantic_ai`), one bound to a remote *agent* (`a2a`), one deterministic mock; `_metering.py`, `_summarise.py` and `_tool_choice.py` are shared code the transports import, which is what the `_` says — offline by default | | TypeScript | 5 modules, 2,305 lines, a second port of **the stepping, the ceilings and the stage path** (`harness.ts:15-21`: *"AC-5.3's bar is the stage path"*) — **not** of the loop entire: no interceptor chain, no durable suspension record, no ledger (`notDoneHere`; a park returns `halted: "suspended"` and writes nothing down). This row is load-bearing for the Phase 3 pricing below (E4) | | CLI | **6 verbs**: `check`, `show`, `waits`, `discover`, `card`, `help` | | Worked example | 42 files, 1128 lines, **132 distinct authored keys**, `pact check` → OK (498 settings) | diff --git a/docs/99-THE-PACT-LANGUAGE.md b/docs/99-THE-PACT-LANGUAGE.md index d6492c3..5d1a4bf 100644 --- a/docs/99-THE-PACT-LANGUAGE.md +++ b/docs/99-THE-PACT-LANGUAGE.md @@ -209,7 +209,7 @@ A line that is none of the three is refused, naming the nearest sentence. header word — `agent`, `tool`, `workflow`, `policy`, `question`, `loop`, `topic`, `event`, `reaction`, `knowledge`, `skill`, `resource`, `program`, `value`, `port`, `shape`, `remembers`, `detector`, `owner`, `context-policy`, -`bundle`, `interceptor`, `watch` — followed by the entry's name: +`bundle`, `interceptor`, `watch`, `plugin` — followed by the entry's name: `tool payments`. An `agent` may add its display name in quotes. The five single documents — `redaction`, `evals`, `learning`, `release`, `models` — are headers with no name. `when ` is a reaction (§4.3). @@ -440,7 +440,7 @@ All seven packages are built (12 November 2026), and an eighth that was not in t author's to move. - **A proposed change is checked against the folder a `.pact` tree stands - for.** `checking.check_instructions` copies the author's tree to try a + for.** `checking.check_agent_field` copies the author's tree to try a learning cycle's proposal on it. A copy written in the language is rewritten as a folder first — `pact write`, then `pact build`, with the files it carries left in place — because that module changes one field of a file and diff --git a/examples/plugin-desk/agents/research-desk/agent.yaml b/examples/plugin-desk/agents/research-desk/agent.yaml new file mode 100644 index 0000000..d20902b --- /dev/null +++ b/examples/plugin-desk/agents/research-desk/agent.yaml @@ -0,0 +1,7 @@ +name: Research desk +description: Answers questions about what is published on the web, and says where it read it. + +# The plugin this agent needs, by its name under `plugins/`. A name that is not +# there is refused here, and the message names the plugins that are. +plugins: + - web-search diff --git a/examples/plugin-desk/agents/research-desk/instructions.md b/examples/plugin-desk/agents/research-desk/instructions.md new file mode 100644 index 0000000..015ff7d --- /dev/null +++ b/examples/plugin-desk/agents/research-desk/instructions.md @@ -0,0 +1,7 @@ +Answer questions about what is published on the web. + +Search before you answer, and say which page each part of the answer came from. +If you have no way to search on this run, say so first, then answer only from +what you are sure of and say that it was not checked against anything. + +Never present something you remember as something you read. diff --git a/examples/plugin-desk/plugins/web-search.yaml b/examples/plugin-desk/plugins/web-search.yaml new file mode 100644 index 0000000..04989ac --- /dev/null +++ b/examples/plugin-desk/plugins/web-search.yaml @@ -0,0 +1,23 @@ +description: Lets the desk search the web and read the pages it finds. + +# Which package, from which registry. The word in front of the colon is the +# portability fact: a runtime that cannot load npm packages cannot honour this +# line, and `pact discover` says so before it takes the agent. +package: npm:@budmh/tool-web + +# Which versions will do, written the way npm writes a range. The release +# candidate the harness ships today and every 0.2 release after it; 0.3 is a +# line somebody edits and a reviewer reads. +version: ^0.2.1-rc.3 + +# The desk can still answer from what it knows, so it may start without the +# plugin — and every run on a runtime that did not load it says the plugin was +# not there, so nobody mistakes an answer from memory for a searched one. Leave +# the line out and the agent refuses to start instead. +if-it-cannot-be-loaded: run-without-it + +# The plugin's own settings, named the way its documentation names them and +# handed over as written. No secret goes here: a key for a search service is +# written `{by-reference: }`, and `pact check` refuses the value itself. +config: + searchMaxResults: 5 diff --git a/examples/plugin-desk/workspace.yaml b/examples/plugin-desk/workspace.yaml new file mode 100644 index 0000000..c030fd6 --- /dev/null +++ b/examples/plugin-desk/workspace.yaml @@ -0,0 +1,23 @@ +# The smallest workspace that asks its runtime to load code (D29). +# +# One agent and one plugin, and nothing else, because the subject is a single +# line: `plugins/web-search.yaml` names a package, a range of versions, the +# plugin's own settings, and what the agent does on a runtime that cannot load +# it. PACT reads those lines and never fetches or runs the package — `pact +# check` holds the request to its registry's grammar, and the runtime says +# whether it loaded it. +name: plugin-desk +description: Answers questions about what is published on the web, with a search plugin its runtime loads. +owner: research-operations + +# NO `allow-egress:` LINE, and the absence is deliberate. That line governs the +# parts of this system PACT can see — models, tools, programs — and a plugin is +# code PACT never reads, so what it reaches once loaded is held by the runtime +# that loaded it, not by a line here. Writing `allow-egress: []` beside a web +# search would tell a reviewer something this file cannot make true. +# +# ONE AGENT, AND NO `team:`, for the reason `examples/mcp-desk/` gives: every +# workspace under `examples/` is compared on seven Python targets and on Node, +# and a run that parks for a teammate suspends in one and errors in the other. +# The plugin is safe in that set: no shipped runtime loads one, so on every +# target the run starts without it and says so, in the same words. diff --git a/research/notes/eve-capabilities.md b/research/notes/eve-capabilities.md index 650890d..a06901d 100644 --- a/research/notes/eve-capabilities.md +++ b/research/notes/eve-capabilities.md @@ -16,7 +16,7 @@ them into eleven areas carrying 58 bullets; the numbered list it was condensed *from* was never in this repository, so `50-NOT-COPIED.md` §8 could account for 58 and had to record 22 as unverifiable. Re-enumerating from source closes that. -**Count:** 101 rows — 49 (a), 22 (b), 30 (c). +**Count:** 101 rows — 51 (a), 22 (b), 28 (c). ## The number is 101, not 80, and that is not padding @@ -69,7 +69,7 @@ constants and test fixtures are not capabilities and are not rows. | **b** | The spec declares it and the host does it | the row of `50-NOT-COPIED.md` §4 that says so | | **c** | PACT will not have it, or has not yet | the `R` ledger row in §5, or §6 for a deferral | -Four rows carry a letter for the half PACT answers with and name the other half +Six rows carry a letter for the half PACT answers with and name the other half in the same cell — that is §8.1's rule, not a new one. **Tooling rows are (a) only when PACT ships that capability as a command that EXISTS**; the cell says which command, and `crates/pact-cli/src/main.rs` is what it is checked against. @@ -219,7 +219,7 @@ were written, which is the argument for them. | 81 | Resetting a session and starting the conversation over | `packages/eve/src/channel/reset-session.ts` : `createResetFn` | b | §4, the store | | 82 | Cron schedules that start a run on their own | `packages/eve/src/public/definitions/schedule.ts` : `defineSchedule` | a | `port.every`, `port.if-still-running` | | 83 | Twenty-eight hook events, all observe-only | `packages/eve/src/public/definitions/hook.ts` : `defineHook` | a | `watch.when`, `watch.writes-to` | -| 84 | Extensions mounted from packages, eleven versioned contracts | `packages/eve/src/compiler/extension-compatibility.ts` : `EXTENSION_CAPABILITY_CONTRACTS` | c | R17 | +| 84 | Extensions mounted from packages, eleven versioned contracts | `packages/eve/src/compiler/extension-compatibility.ts` : `EXTENSION_CAPABILITY_CONTRACTS` | a | `workspace.plugins` and `plugin.package` — the naming half is (a); loading is the host's, §4 | | 85 | Prompt-cache markers placed for the provider that supports them | `packages/eve/src/harness/prompt-cache.ts` : `detectPromptCachePath` | c | R46 | | 86 | An instrumentation slot for OpenTelemetry setup | `packages/eve/src/public/instrumentation/index.ts` : `defineInstrumentation` | b | §4, a stream of run events | @@ -244,7 +244,7 @@ were written, which is the argument for them. | 96 | The CLI editing the author's source file to record a choice | `packages/eve/src/source-change/apply-model-name.ts` : `applyModelNameToSource` | a | `workspace.agents` — the files are the API for every surface (D18) | | 97 | Mount points for Next, Nuxt and SvelteKit | `packages/eve/src/public/next/index.ts` : `EveNextConfig` | b | §4, the connector | | 98 | An integration catalogue — 25 channels, 40 connections, 5 extensions | `packages/eve-catalog/src/index.ts` : `INTEGRATIONS` | a | `resource.endpoint` names the server; which servers a machine has is the host's, §4 | -| 99 | `defineExtension` — tools, connections, skills, instruction fragments and hooks packaged as one installable npm unit | `packages/eve/src/public/definitions/extension.ts` : `defineExtension` | c | §6, deferred — G10 `bundle` is the re-admission fixture; PACT has no distribution unit today | +| 99 | `defineExtension` — tools, connections, skills, instruction fragments and hooks packaged as one installable npm unit | `packages/eve/src/public/definitions/extension.ts` : `defineExtension` | a | `workspace.plugins` and `plugin.package` — the naming half is (a); loading is the host's, §4. What the unit would bring as definitions is still a `bundle`, a folder | | 100 | `experimental_workflow` — the model orchestrates the agent's own subagents from model-authored JavaScript, one durable step, `maxSubagents` 100 | `packages/eve/src/harness/workflow-subagent-limit.ts` : `DEFAULT_WORKFLOW_MAX_SUBAGENTS` | c | R5 and R42 — a spec that names code to run makes `pact check` decide whether that code is safe | | 101 | `experimental_setAttributes` — author-set span attributes on the active trace | `packages/eve/src/public/instrumentation/index.ts` : `experimental_setAttributes` | b | §4, the exporter | diff --git a/site-docs/concepts/what.md b/site-docs/concepts/what.md index d960fd2..d4f602f 100644 --- a/site-docs/concepts/what.md +++ b/site-docs/concepts/what.md @@ -129,7 +129,7 @@ error: 'loop' names 'carefull', and there is no such entry in `loops:`. rule: schema/no-such-name ``` -That is not a special case for loops. **76 fields** declare what they name, and +That is not a special case for loops. **78 fields** declare what they name, and the same machinery produces the same shape of message for all of them. --- diff --git a/site-docs/index.md b/site-docs/index.md index 7b3a051..f1a1192 100644 --- a/site-docs/index.md +++ b/site-docs/index.md @@ -33,7 +33,7 @@ requires it, and deleting it leaves a working agent. - **[What do you write?](concepts/what.md)** Files. One rule — *a directory is a field; a field may be a directory* — - and 68 kinds you never have to memorise. + and 69 kinds you never have to memorise. @@ -47,8 +47,8 @@ so plainly rather than leaving it to be discovered. | | | |---|---| -| Tests | **4,969** — 1648 Rust, 3321 adapter | -| Schema kinds | **68** | +| Tests | **5,013** — 1673 Rust, 3340 adapter | +| Schema kinds | **69** | | Framework targets | **7**, over 2 runtimes | | Eve capabilities accounted for | **101 of 101** | | Worked example | 43 files, 498 settings, 1 payload script | @@ -68,7 +68,7 @@ so plainly rather than leaving it to be discovered. [Gaps and blockers](status/gaps.md), in that order - **Comparing with Vercel Eve?** [Compared to Eve](status/eve.md) — a measured comparison, including where Eve is better -- **Building on it?** [The 68 kinds](reference/kinds.md) and +- **Building on it?** [The 69 kinds](reference/kinds.md) and [Adapters](reference/adapters.md) --- diff --git a/site-docs/reference/adapters.md b/site-docs/reference/adapters.md index 9ab3377..5c9645d 100644 --- a/site-docs/reference/adapters.md +++ b/site-docs/reference/adapters.md @@ -8,15 +8,15 @@ hides a difference is worse than one that admits it, because the difference still exists and now nobody is looking for it. ``` - model_cal | tool_call | text_with | parallel_ | streaming | durable_r | connected -reference native | native | native | native | unsupport | unsupport | unsupport -pydantic-ai native | native | native | native | emulated | unsupport | native -langgraph native | emulated | native | emulated | emulated | native | unsupport -langchain native | native | native | native | emulated | unsupport | unsupport -autogen native | native | emulated | native | emulated | unsupport | unsupport -openai-agents native | native | native | native | emulated | unsupport | unsupport -anthropic native | native | native | native | emulated | unsupport | unsupport -vercel-ai native | native | native | native | emulated | unsupport | unsupport + model_cal | tool_call | text_with | parallel_ | streaming | durable_r | connected | plugins +reference native | native | native | native | unsupport | unsupport | unsupport | unsupport +pydantic-ai native | native | native | native | emulated | unsupport | native | unsupport +langgraph native | emulated | native | emulated | emulated | native | unsupport | unsupport +langchain native | native | native | native | emulated | unsupport | unsupport | unsupport +autogen native | native | emulated | native | emulated | unsupport | unsupport | unsupport +openai-agents native | native | native | native | emulated | unsupport | unsupport | unsupport +anthropic native | native | native | native | emulated | unsupport | unsupport | unsupport +vercel-ai native | native | native | native | emulated | unsupport | unsupport | unsupport ``` Seven targets plus `reference` — the framework-free control arm, which is not a @@ -37,6 +37,12 @@ Three things worth reading: named …` — which is exactly the kind of thing that must be published before a run rather than discovered during one. A row that is `unsupported` here is not a broken adapter; it is an adapter you hand the call back to yourself. +- **`plugins`** says whether the runtime loads the packages a `plugins:` line + names (D29). It is `unsupported` everywhere here, because the harness loads + nothing: a plugin an agent cannot start without stops it before the first + model call (`halted: plugins-not-loaded`), and one it can is named on + `unenforced`. A host that loads packages publishes `native` and hands the + harness what it loaded with `run(loaded_plugins=)`. ## The seams diff --git a/site-docs/reference/kinds.md b/site-docs/reference/kinds.md index e256fd9..06e7280 100644 --- a/site-docs/reference/kinds.md +++ b/site-docs/reference/kinds.md @@ -1,11 +1,11 @@ -# The 68 kinds +# The 69 kinds Generated from `spec/schema.yaml`. Every kind, what it is for, and how many fields it has. You do not need to learn these — `pact check` names the right one when you need it. -**68 kinds.** A kind is a *shape a document can take*, not a file you must +**69 kinds.** A kind is a *shape a document can take*, not a file you must write: most workspaces use a dozen. @@ -14,7 +14,7 @@ write: most workspaces use a dozen. | Kind | Fields | Core fields | |---|---:|---| | `action` | 21 | `description`, `topics`, `takes`, `reads-only`, `needs-a-person` | -| `agent` | 29 | `name`, `description`, `instructions`, `team`, `teamwork` | +| `agent` | 30 | `name`, `description`, `instructions`, `team`, `teamwork` | | `loop` | 4 | `description`, `based-on` | | `knowledge` | 15 | `description`, `topics`, `documents`, `passages-at-most`, `must-cite` | | `skill` | 14 | `name`, `description`, `use-when`, `do-not-use-when`, `if-unsure` | @@ -26,7 +26,7 @@ write: most workspaces use a dozen. | `variant` | 6 | — | | `turn` | 6 | — | | `voice` | 9 | — | -| `workspace` | 39 | `name`, `workspace-id`, `description`, `owner`, `profile` | +| `workspace` | 40 | `name`, `workspace-id`, `description`, `owner`, `profile` | ## Topics, events and reactions @@ -93,6 +93,7 @@ write: most workspaces use a dozen. | Kind | Fields | Core fields | |---|---:|---| | `bundle` | 5 | `description`, `version`, `from`, `brings` | +| `plugin` | 7 | `description`, `if-it-cannot-be-loaded` | | `port` | 22 | `description`, `kind`, `every`, `says`, `if-still-running` | | `resource` | 14 | `resource-kind`, `endpoint`, `auth`, `asks-to-connect`, `description` | | `published` | 5 | — | @@ -126,7 +127,7 @@ write: most workspaces use a dozen. ## How to read a kind -Every kind follows the same shape, so 54 is not 54 things to learn: +Every kind follows the same shape, so 64 is not 64 things to learn: ```yaml description: what this is, in one line # every kind has this diff --git a/site-docs/status/verified.md b/site-docs/status/verified.md index 26741d7..d8734fe 100644 --- a/site-docs/status/verified.md +++ b/site-docs/status/verified.md @@ -12,20 +12,20 @@ from an earlier document. | Claim | Value | How to check | |---|---|---| -| Rust tests | **1648** | `cargo test --workspace` | -| Adapter tests | **3321** | `cd adapters/python && uv run pytest tests/ -q` | -| Total | **4,969** | both of the above | +| Rust tests | **1673** | `cargo test --workspace` | +| Adapter tests | **3340** | `cd adapters/python && uv run pytest tests/ -q` | +| Total | **5,013** | both of the above | | TypeScript type-checks | clean | `cd adapters/typescript && npx tsc --noEmit` | | Lints | clean | `cargo clippy --all-targets` | | Worked example loads | 498 settings | `cargo run -p pact-cli -- check examples/refund-desk` | -| Schema kinds | **68** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | -| Fields that resolve a name | **76** | `grep -cE '^\s*names:' spec/schema.yaml` | +| Schema kinds | **69** | `python3 -c "import yaml;print(len(yaml.safe_load(open('spec/schema.yaml'))['groups']))"` | +| Fields that resolve a name | **78** | `grep -cE '^\s*names:' spec/schema.yaml` | ## What is proven, and what "proven" means ### Framework portability -**58 agents across 27 workspaces**, loaded by the Rust CLI, execute over **seven +**59 agents across 28 workspaces**, loaded by the Rust CLI, execute over **seven targets** across two language runtimes and produce byte-identical traces, tool sequences and model-call counts. The set is generated from `examples/`, so an agent added to the tree joins it by existing rather than by somebody remembering. diff --git a/spec/schema.yaml b/spec/schema.yaml index 2b49167..514036a 100644 --- a/spec/schema.yaml +++ b/spec/schema.yaml @@ -519,6 +519,26 @@ groups: sets of ready-made capabilities this workspace uses. Each says where its folder is and which kinds it may contribute; anything else it contains is refused rather than quietly mounted. + plugins: + # D29, which narrows R17. A bundle above is definitions PACT reads; a + # plugin is CODE, and PACT never reads it, fetches it or runs it (R5). + # What lives here is the request for it — which package, which versions + # will do, its settings, and what an agent does on a runtime that cannot + # load it — so a reviewer reads the request in the folder they were + # handed, and every runtime answers it the same way: it loaded it, it + # refused to start, or the run says the plugin was not there. + # + # S-EXEC because a plugin can do anything its runtime lets it, which is + # what "touching tools" means to D23. A person approves adding one. + type: map of group:plugin + surface: S-EXEC + tier: expert + help: > + code that whatever runs these agents loads for them, one file each + under `plugins/`. Each says which package, which versions will do, its + settings, and whether an agent may start without it. PACT never + fetches or runs one: `pact check` reads the lines, and the runtime says + whether it loaded it. Adding one always needs a person to approve it. interceptors: type: map of group:interceptor surface: S-EXEC @@ -1042,6 +1062,15 @@ groups: worked example the model should copy, including the tool it should call and what came back. These words may carry the same `{{run-inputs.}}` holes as `instructions:`. + plugins: + type: list of text + names: plugins + surface: S-EXEC + tier: expert + help: > + which of the workspace's plugins this agent needs, by name. One whose + `applies-to:` is `every-agent` is loaded for every agent here and need + not be named. variant: describe: something a variant can have @@ -4919,7 +4948,12 @@ groups: `needs-a-person-to-approve:` below is for. needs-a-person-to-approve: type: list of one-of - choices: [tools, permissions, team, limits, evals, policy-clauses] + # `plugins` (D29): a plugin is code its runtime loads, so a change to + # which one an agent needs, or which versions will do, is a change to + # what it can do. The classifier holds it to a person with or without + # this line (`learning.HIGH_RISK_FIELDS`); listing it is how an author + # says so in the file. + choices: [tools, permissions, team, limits, evals, policy-clauses, plugins] surface: S-GOV tier: core help: changes to what it can DO, which always need review @@ -7525,6 +7559,11 @@ groups: - shapes - remembers - owners + # D29. A plugin line is a definition like the rest — a name, a + # package and a range — and a bundle that starts carrying one asks + # every runtime to load code, so it is a kind `brings:` governs + # rather than one it is blind to. + - plugins required: yes surface: S-EXEC tier: core @@ -7544,6 +7583,107 @@ groups: what it defines, read exactly as if you had written it here. You do not type this — it is what is in the bundle's own folder. + # ── PLUGINS (D29) ───────────────────────────────────────────────────────────── + # A request for code the runtime loads, never the code. `pact check` holds each + # line to its own grammar (`crates/pact-loader/src/plugins.rs`) and never + # resolves, downloads or opens the package, so checking a workspace stays an act + # of reading (R5) and stays offline (D17). Whether the package was loaded is the + # runtime's answer, given on every run: the `plugins` lattice column says which + # registries a runtime can load at all, and a plugin it did not load either + # stops the agent before its first step or is named on `unenforced`. + plugin: + describe: something a plugin line can have + fields: + description: + type: text + required: yes + surface: S-GEN + tier: core + help: what it adds, in one line, for whoever has to approve it + package: + # The registry word is the portability fact. A runtime written in + # TypeScript can load an npm package and not a pypi one, so the word in + # front of the name says, before anything runs, which runtimes can + # honour the line at all — and `pact discover` publishes it. + type: text + required: yes + surface: S-EXEC + tier: expert + help: > + the package, written `:` — `npm:@acme/crm-tools` or + `pypi:acme-crm-tools`. The registry word says which runtimes can load + it at all; a runtime with no loader for that registry treats it as a + plugin it cannot load. + version: + # A range and never "latest": an upgrade that changes what an agent can + # do is a line somebody edited and a reviewer read, not something that + # happened on the next install. Held to the registry's own grammar, so + # `^1.4` under pypi or `>=1.4,<2` under npm is refused where it is typed. + type: text + required: yes + surface: S-EXEC + tier: expert + help: > + which versions will do, written the registry's own way — `^1.4.0` for + npm, `>=1.4,<2` for pypi — so an upgrade that changes behaviour is a + line somebody edited rather than something that happened. + if-it-cannot-be-loaded: + # Two answers, because a runtime that cannot load the package has only + # two honest things to do. Refusing is the default: an agent written to + # rely on code it did not get is an agent that is not the one reviewed. + type: one-of + choices: [refuse-to-start, run-without-it] + surface: S-GOV + tier: core + help: > + what happens on a runtime that cannot load it. `refuse-to-start`, which + is what you get if you leave this out, stops the agent before its first + step and says which plugin and why. `run-without-it` starts the agent, + and every run says the plugin was not there. + config: + # Open, like `bundle.contributes`, because the settings are the + # plugin's and not PACT's: a second copy of every plugin's settings + # here would be a schema PACT cannot keep true. What PACT does hold is + # the one rule no plugin may waive — no secret in the file. + type: map of anything + surface: S-EXEC + tier: expert + help: > + the plugin's own settings, handed to it as written. A secret goes in as + `{by-reference: }`, the name your platform team publishes for + where it is kept, never as the secret itself. + applies-to: + # The same polarity `interceptor.applies-to` and `policy.applies-to` + # record, on the same two words. + type: one-of + choices: [the-agents-that-name-it, every-agent] + surface: S-EXEC + tier: expert + help: > + who it is loaded for. `the-agents-that-name-it` is the default and is + for a plugin one job needs. `every-agent` loads it for every agent here, + and an agent cannot opt out by leaving it off its `plugins:` line. + implements: + # Law 4 (closed world) for code PACT never reads. What a plugin lets an + # agent ASK is written here as an interface the tree declares, so the + # contract a policy, an approval and an eval are written against is the + # tree's, and the plugin supplies only the code behind it. The same + # interface then carries law 1 (its actions' effect is the effect of + # every agent the plugin is loaded for) and law 3 (an action it cannot + # undo names its same-request key, because the code that does it is + # code no checker sees). + type: list of text + names: [tools, skills, knowledge] + surface: S-CAP + tier: expert + help: > + the interfaces this plugin's code stands behind, by name — entries in + `tools`, `skills` or `knowledge` that say `interface: yes`. An agent + asks the interface, so what it may do, what needs a person and what + it does to the world are written in this tree, and the plugin is + one way of doing it. Leave it out for a plugin that gives an agent + nothing to ask. + # ── TEAMWORK ──────────────────────────────────────────────────────────────── # How an agent waits for the people it asked for help, and how what it may # spend is shared out between them. diff --git a/tests/golden/schema-fields.txt b/tests/golden/schema-fields.txt index 4549185..ef80b25 100644 --- a/tests/golden/schema-fields.txt +++ b/tests/golden/schema-fields.txt @@ -37,6 +37,7 @@ agent.model list of text | S-CAP | core | optional agent.model-for-checking text | S-CAP | expert | optional agent.name text | S-GEN | core | optional agent.needs group:needs | S-CAP | core | optional +agent.plugins list of text | S-EXEC | expert | optional agent.policy text | S-EXEC | core | optional agent.reactions list of text | S-EXEC | core | optional agent.remembers map of group:state | S-GOV | core | optional @@ -243,6 +244,13 @@ outcome.used-a-tool text | S-CTRL | expert | optional ownership.locked one-of | S-GOV | core | optional ownership.locked-until moment | S-GOV | core | optional ownership.owned-by text | S-GOV | core | optional +plugin.applies-to one-of | S-EXEC | expert | optional +plugin.config map of anything | S-EXEC | expert | optional +plugin.description text | S-GEN | core | required +plugin.if-it-cannot-be-loaded one-of | S-GOV | core | optional +plugin.implements list of text | S-CAP | expert | optional +plugin.package text | S-EXEC | expert | required +plugin.version text | S-EXEC | expert | required policy.applies-to one-of | S-EXEC | core | optional policy.ask-a-person list of group:question-rule | S-EXEC | core | optional port.accepts map of answer-shape | S-CAP | core | optional @@ -547,6 +555,7 @@ workspace.name text | S-GOV | core | required workspace.owner text | S-GOV | core | optional workspace.owners map of group:ownership | S-GOV | core | optional workspace.pact-version text | S-GOV | expert | optional +workspace.plugins map of group:plugin | S-EXEC | expert | optional workspace.policies map of group:policy | S-EXEC | core | optional workspace.ports map of group:port | S-TOPO | core | optional workspace.profile text | S-GOV | core | optional diff --git a/tests/golden/show/plugin-desk.json b/tests/golden/show/plugin-desk.json new file mode 100644 index 0000000..1e7fe09 --- /dev/null +++ b/tests/golden/show/plugin-desk.json @@ -0,0 +1,26 @@ +{ + "name": "plugin-desk", + "description": "Answers questions about what is published on the web, with a search plugin its runtime loads.", + "owner": "research-operations", + "agents": { + "research-desk": { + "name": "Research desk", + "description": "Answers questions about what is published on the web, and says where it read it.", + "plugins": [ + "web-search" + ], + "instructions": "Answer questions about what is published on the web.\n\nSearch before you answer, and say which page each part of the answer came from.\nIf you have no way to search on this run, say so first, then answer only from\nwhat you are sure of and say that it was not checked against anything.\n\nNever present something you remember as something you read." + } + }, + "plugins": { + "web-search": { + "description": "Lets the desk search the web and read the pages it finds.", + "package": "npm:@budmh/tool-web", + "version": "^0.2.1-rc.3", + "if-it-cannot-be-loaded": "run-without-it", + "config": { + "searchMaxResults": 5 + } + } + } +} diff --git a/tests/trees/a-desk-that-needs-a-plugin/README.md b/tests/trees/a-desk-that-needs-a-plugin/README.md new file mode 100644 index 0000000..dd94e22 --- /dev/null +++ b/tests/trees/a-desk-that-needs-a-plugin/README.md @@ -0,0 +1,56 @@ +# a-desk-that-needs-a-plugin + +A desk that cannot do its job without the CRM plugin its runtime loads, and says +so on the plugin's own line by leaving `if-it-cannot-be-loaded:` out. + +``` +workspace.yaml +agents/desk/agent.yaml plugins: [crm-tools] +agents/desk/instructions.md +plugins/crm-tools.yaml package: pypi:acme-crm-tools, version: ">=1.4,<2" +``` + +## What this tree is for + +`refuse-to-start` is the default (D29), and it is the one that changes where an +agent can run at all: on a runtime that cannot load `pypi:` packages this desk +never starts. That is a portability fact, so `pact check` says it before anybody +runs anything — as a note, because it is what the author meant and a shipped +tree that means it must still pass `--deny-warnings`: + +```bash +cargo run -p pact-cli -- check tests/trees/a-desk-that-needs-a-plugin +``` + +``` +note: 'desk' runs only where `pypi:acme-crm-tools` can be loaded: 'crm-tools' says + `refuse-to-start`, so on a runtime that cannot load it the agent stops before + the first step. + rule: loader/plugin-runs-only-where-loaded +OK — a-desk-that-needs-a-plugin loaded cleanly (18 settings). +``` + +`pact discover` publishes the same fact per agent — `plugins: [{name, package, +version, if-it-cannot-be-loaded: refuse-to-start}]` — so a runtime can decline +the agent before it takes it. + +## And the run + +The reference harness loads no plugin, and neither does the second port, so both +end the turn before a model is called: + +``` +halted: plugins-not-loaded +output: This agent cannot start: it needs 'crm-tools' (`pypi:acme-crm-tools`, + versions `>=1.4,<2`), and nothing running it loaded it. fix: run it where a + runtime loads `pypi:` packages, or write `if-it-cannot-be-loaded: + run-without-it` in the plugin's file under `plugins/` if the agent can do + its job without it. +``` + +A host that does load the package says so with `run(loaded_plugins={"crm-tools": +"1.4.2"})`, and the desk runs. Held by +`adapters/python/tests/test_a_plugin_nobody_loaded_stops_the_run_before_it_starts.py`. + +The settings carry the CRM key as `{by-reference: host/crm-key}`. Writing the key +itself there is `loader/plugin-secret-written-in-the-file`. diff --git a/tests/trees/a-desk-that-needs-a-plugin/agents/desk/agent.yaml b/tests/trees/a-desk-that-needs-a-plugin/agents/desk/agent.yaml new file mode 100644 index 0000000..ffaa760 --- /dev/null +++ b/tests/trees/a-desk-that-needs-a-plugin/agents/desk/agent.yaml @@ -0,0 +1,3 @@ +description: Answers questions about a customer's account, out of the CRM. +plugins: + - crm-tools diff --git a/tests/trees/a-desk-that-needs-a-plugin/agents/desk/instructions.md b/tests/trees/a-desk-that-needs-a-plugin/agents/desk/instructions.md new file mode 100644 index 0000000..933c1a5 --- /dev/null +++ b/tests/trees/a-desk-that-needs-a-plugin/agents/desk/instructions.md @@ -0,0 +1,2 @@ +Answer questions about a customer's account. Look the account up in the CRM +before you say anything about it, and never answer from memory. diff --git a/tests/trees/a-desk-that-needs-a-plugin/plugins/crm-tools.yaml b/tests/trees/a-desk-that-needs-a-plugin/plugins/crm-tools.yaml new file mode 100644 index 0000000..9d78a4e --- /dev/null +++ b/tests/trees/a-desk-that-needs-a-plugin/plugins/crm-tools.yaml @@ -0,0 +1,6 @@ +description: Reads customer accounts out of the CRM. +package: pypi:acme-crm-tools +version: ">=1.4,<2" +config: + region: eu + api-key: { by-reference: host/crm-key } diff --git a/tests/trees/a-desk-that-needs-a-plugin/workspace.yaml b/tests/trees/a-desk-that-needs-a-plugin/workspace.yaml new file mode 100644 index 0000000..835a5a5 --- /dev/null +++ b/tests/trees/a-desk-that-needs-a-plugin/workspace.yaml @@ -0,0 +1,2 @@ +name: a-desk-that-needs-a-plugin +description: A desk that cannot do its job without the CRM plugin its runtime loads. diff --git a/tests/trees/a-plugin-with-no-registry/README.md b/tests/trees/a-plugin-with-no-registry/README.md new file mode 100644 index 0000000..441fa18 --- /dev/null +++ b/tests/trees/a-plugin-with-no-registry/README.md @@ -0,0 +1,36 @@ +# a-plugin-with-no-registry + +The CRM desk of `tests/trees/a-desk-that-needs-a-plugin/`, with one line +different: the plugin's `package:` is written without the registry in front. + +``` +plugins/crm-tools.yaml package: acme-crm-tools +``` + +It lives here and not under `examples/` because the refusal is the behaviour +under test: every tree under `examples/` is held to `--deny-warnings`. + +## Why it is refused + +The word in front of the colon — `npm:`, `pypi:` — is what says which runtimes +can load the plugin at all. Without it no runtime can tell whether the line is one +it can honour, so the guess would be made later, by whichever runtime got there +first: + +```bash +cargo run -p pact-cli -- check tests/trees/a-plugin-with-no-registry +``` + +``` +error: 'crm-tools' asks for the package 'acme-crm-tools', which names no registry — + so no runtime can say whether it is one it can load. + --> a-plugin-with-no-registry/plugins/crm-tools.yaml:2:1 + fix: Write one of the registries a runtime loads from — `npm` or `pypi` — in + front of the name: `npm:acme-crm-tools` for an npm package, + `pypi:acme-crm-tools` for a Python one. + rule: loader/plugin-package-has-no-registry +``` + +The version range is not judged until the registry is known, because which +grammar it is read in is the registry's to say. Held by +`crates/pact-cli/tests/a_plugin_is_named_and_never_loaded.rs`. diff --git a/tests/trees/a-plugin-with-no-registry/agents/desk/agent.yaml b/tests/trees/a-plugin-with-no-registry/agents/desk/agent.yaml new file mode 100644 index 0000000..ffaa760 --- /dev/null +++ b/tests/trees/a-plugin-with-no-registry/agents/desk/agent.yaml @@ -0,0 +1,3 @@ +description: Answers questions about a customer's account, out of the CRM. +plugins: + - crm-tools diff --git a/tests/trees/a-plugin-with-no-registry/agents/desk/instructions.md b/tests/trees/a-plugin-with-no-registry/agents/desk/instructions.md new file mode 100644 index 0000000..933c1a5 --- /dev/null +++ b/tests/trees/a-plugin-with-no-registry/agents/desk/instructions.md @@ -0,0 +1,2 @@ +Answer questions about a customer's account. Look the account up in the CRM +before you say anything about it, and never answer from memory. diff --git a/tests/trees/a-plugin-with-no-registry/plugins/crm-tools.yaml b/tests/trees/a-plugin-with-no-registry/plugins/crm-tools.yaml new file mode 100644 index 0000000..0006739 --- /dev/null +++ b/tests/trees/a-plugin-with-no-registry/plugins/crm-tools.yaml @@ -0,0 +1,3 @@ +description: Reads customer accounts out of the CRM. +package: acme-crm-tools +version: ">=1.4,<2" diff --git a/tests/trees/a-plugin-with-no-registry/workspace.yaml b/tests/trees/a-plugin-with-no-registry/workspace.yaml new file mode 100644 index 0000000..9043b59 --- /dev/null +++ b/tests/trees/a-plugin-with-no-registry/workspace.yaml @@ -0,0 +1,2 @@ +name: a-plugin-with-no-registry +description: The CRM desk again, with its plugin's package written without a registry.