Say what a variable, a worker, and an export actually covered - #130
Merged
Conversation
Four disclosure gaps, each measured before it was closed. inspect_variable reported statistics over an unstated subset: `sst` returned mean 80.5, and the same field with 145 of its 162 faces masked returned mean 153.0 in the same shape with nothing saying the number describes 17 faces. A fully masked field returned NaN, which is not JSON, after three RuntimeWarnings on stderr the caller never sees. Statistics now come off a boolean index of the finite entries, a partly masked variable also reports n_finite and n_total, and an empty one reports null. mesh_coverage never reached a result computed on HPC, so MESH_NOT_GLOBAL could not fire on a remote reply whatever the mesh. AllCodeStrategies ships one function's code and nothing else, so the measurement is nested inside remote_inspect_mesh and remote_calculate_area, as profile_coverage and source_coverage already are. tests/test_remote_mesh_coverage.py compares the three copies key for key on three grids. Every HPC result failed its own contract. _run_on_hpc stamps tool=func.__name__, nothing declares a contract for remote_calculate_area, so attach_provenance skipped `operation` and validate_response answered malformed_envelope for a reply whose science was fine. The four remote_* functions now alias to the operations they answer. export claimed success without checking anything: no stat, no read-back, no comparison. On a 9-face dataset, CSV dropped all 8 attributes including every unit and wrote its one NaN as an empty field; single-variable NetCDF export dropped the CF crs container while leaving sst pointing at it. Exports now return an export_fidelity block measured on the written file and warn accordingly. The file is still written -- a lossy export is the export that was asked for, and only the silence was wrong. Adds scripts/measure_payload.py, which says where a reply's bytes go. Token counts appear only with tiktoken installed and are never estimated from a bytes-per-token constant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four disclosure gaps, each measured before it was closed. Closes #34.
inspect_variablereported statistics over an unstated subset:sstreturned mean 80.5, and the same field with 145 of its 162 faces masked returned mean 153.0 in the same shape with nothing saying the number describes 17 faces; a fully masked field returned NaN, which is not JSON.mesh_coveragenever reached a result computed on HPC, soMESH_NOT_GLOBALcould not fire on a remote reply whatever the mesh, and every HPC result also failed its own contract because nothing declares one forremote_calculate_area.exportclaimed success without a singlestator read-back: CSV dropped all 8 attributes on a 9-face dataset and wrote its one NaN as an empty field, and single-variable NetCDF export dropped the CFcrscontainer while leavingsstpointing at it.973 passed, 13 skipped; pre-commit 11/11.