Found in epic #324 Phase 15 (#351).
Some sides of these protocols are not seeded or not matched:
benchmarks/reproduce_table1.py: the reference runs in the single-model tier and the multi-model ensemble draw clock-based seeds. On the bundled tier, one reference run of five landed in a lower-likelihood basin, which a rerun may or may not reproduce.
- The ensemble passes
invsigmin 0.0 to the reference (as runamica15.m writes it with %f) and 1e-8 to pamica.
benchmarks/benchmark_dimsweep.py calls AMICA_NumPy without a seed, so its log-likelihood column starts from a different draw than the other backends.
Proposal:
- Seed the reference side (the
seed keyword) and the NumPy side.
- Align
invsigmin.
- Re-run
.context/issue-351/pin_check.py --dump whenever AMICANative's parameter table changes.
- Changing the Table 1 protocol means re-measuring its rows; do it together with the next re-measurement.
Found in epic #324 Phase 15 (#351).
Some sides of these protocols are not seeded or not matched:
benchmarks/reproduce_table1.py: the reference runs in the single-model tier and the multi-model ensemble draw clock-based seeds. On the bundled tier, one reference run of five landed in a lower-likelihood basin, which a rerun may or may not reproduce.invsigmin0.0 to the reference (asrunamica15.mwrites it with%f) and 1e-8 to pamica.benchmarks/benchmark_dimsweep.pycallsAMICA_NumPywithout a seed, so its log-likelihood column starts from a different draw than the other backends.Proposal:
seedkeyword) and the NumPy side.invsigmin..context/issue-351/pin_check.py --dumpwheneverAMICANative's parameter table changes.