Repository navigation
deepeval:g_eval reads evaluation_params and rubric as a suite writes them - #12
Merged
Merged
Conversation
…holder kept across a park A lease (`claim-lasts:`) frees a key a crashed worker held, but cannot stop a call still running when the lease ends. Every grant of a key is now numbered one higher than the key's grant before it (`Taken.fence`); a renewal keeps the number, and expiry and release keep the count, so the system a call writes to can refuse a writer whose number is stale. `SqliteClaims` keeps expired and released rows for their count and gains the column on files written before it. The ledger records the token of each claim it took (`Ledger.fences`). `the-workspace` claims are namespaced by the workspace id a host names (`claims_workspace=`), so two workspaces sharing one store do not refuse each other; without one the key is as before. A parked run comes back as the holder of the claims it took (`Suspension.claim_holder`, carried by `_park_state`, `escalate` and the JSON form): under a new name it could neither renew nor release them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…them G-Eval takes SingleTurnParams members and Rubric objects, which YAML cannot write. A suite that wrote `evaluation_params: [input, actual_output]` was reported absent on every case (DeepEval 4.3.3: "'str' object has no attribute 'value'"), and leaving the field out makes G-Eval grade a trace a case answer does not have, so the one DeepEval score an author defines in their own words could not be written. evaluate_metric now reads the names by DeepEval's spelling or the member's, and each rubric mapping by score-range and expected-outcome; every other score's settings pass through. Found by Bud Meta Harness's DeepEval metric runner, which calls evaluate_metric for every deepeval: score and had to read the names itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the claim's holder AND the reactions; counts recomputed (4822) Both code conflicts were two additions at the same place: Suspension's escalate copy and harness._park_state each gained a line from this branch (claim_holder) and a line from master (reactions). Both are kept. The counts were recomputed by scripts/sync-counts.sh: 1623 Rust + 3199 adapter. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
dittops
marked this pull request as ready for review
November 12, 2026 10:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of UP-7 (eval and selection fixes) from Bud Meta Harness's PACT v97 plan.
What
deepeval:g_evalnow reads the settings a suite writes. G-Eval takesSingleTurnParamsmembers andRubricobjects, and YAML writes neither. Before this change,evaluation_params: [input, actual_output]reached G-Eval as strings, and every case was reported absent ("'str' object has no attribute 'value'", measured on DeepEval 4.3.3). Leaving the field out does not help: G-Eval then grades a trace, and a case answer carries none.providers._as_deepeval_readsreadsevaluation_paramsby DeepEval's spelling (actual_output) or the member's (ACTUAL_OUTPUT), and eachrubric:mapping byscore-range(orscore_range) andexpected-outcome. Every other score's settings pass through untouched, and a name that is no test-case field still gets G-Eval's own refusal.Evidence
tests/test_g_eval_is_measured_as_its_page_writes_it.pyhas 5 tests. Without the change, 4 fail. With it, all pass (scripted offline judge, DeepEval 4.3.3).tests/test_metrics_an_expert_brings.pyandtests/test_the_metric_coverage_is_published.pypass, excepttest_a_deepeval_score_is_never_answered_by_a_hosted_model_the_author_did_not_name. That test fails onmastertoo, with DeepEval 4.3.3 installed, so it is not caused by this change.evaluate_metricfor everydeepeval:score). It reports the reading asdiverges-from-the-referenceuntil this lands. A PAR conformance run on 5 metrics agrees withevaluate_metricwithin 5 points.Draft: opened by the Bud Meta Harness model-finder lane for review.