Skip to content

Seed every side of the benchmark and Table 1 protocols #360

Description

@neuromechanist

Found in epic #324 Phase 15 (#351).

Some sides of these protocols are not seeded or not matched:

  • benchmarks/reproduce_table1.py: the reference runs in the single-model tier and the multi-model ensemble draw clock-based seeds. On the bundled tier, one reference run of five landed in a lower-likelihood basin, which a rerun may or may not reproduce.
  • The ensemble passes invsigmin 0.0 to the reference (as runamica15.m writes it with %f) and 1e-8 to pamica.
  • benchmarks/benchmark_dimsweep.py calls AMICA_NumPy without a seed, so its log-likelihood column starts from a different draw than the other backends.

Proposal:

  • Seed the reference side (the seed keyword) and the NumPy side.
  • Align invsigmin.
  • Re-run .context/issue-351/pin_check.py --dump whenever AMICANative's parameter table changes.
  • Changing the Table 1 protocol means re-measuring its rows; do it together with the next re-measurement.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions