pathogen-embed and numpy version incompatibility

[joverlee521]

First seen in seasonal-flu build, but also flagged in CI

Traceback (most recent call last):
  File "/usr/local/bin/pathogen-embed", line 8, in <module>
    sys.exit(run_embed())
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pathogen_embed/__main__.py", line 109, in run_embed
    return embed(args)
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pathogen_embed/pathogen_embed.py", line 353, in embed
    from numba.core.errors import NumbaDeprecationWarning
  File "/usr/local/lib/python3.11/site-packages/numba/__init__.py", line 55, in <module>
    _ensure_critical_deps()
  File "/usr/local/lib/python3.11/site-packages/numba/__init__.py", line 42, in _ensure_critical_deps
    raise ImportError("Numba needs NumPy 1.26 or less")
ImportError: Numba needs NumPy 1.26 or less

[joverlee521]

Installing evofr bumped the numpy version

#32 4.360 Installing collected packages: multipledispatch, websockets, uvloop, truststore, soupsieve, python-multipart, python-dotenv, opt_einsum, numpy, itsdangerous, httptools, fastcore, click, watchfiles, uvicorn, starlette, ml_dtypes, httpcore2, beautifulsoup4, jaxlib, httpx2, seaborn, python-fasthtml, jax, numpyro, jaxopt, fastprogress, blackjax, evofr
#32 4.596   Attempting uninstall: numpy
#32 4.624     Found existing installation: numpy 1.26.4
#32 4.660     Uninstalling numpy-1.26.4:
#32 4.832       Successfully uninstalled numpy-1.26.4
#32 9.529 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
#32 9.529 numba 0.58.1 requires numpy<1.27,>=1.22, but you have numpy 2.4.6 which is incompatible.

[victorlin]

Easy workaround is to isolate pathogen-embed in its own venv like fauna. Example: 04d5f6f

[joverlee521]

Ah, ml_dtypes just released v0.6.0, which dropped support for numpy <2.0.
Huh, I would have expected pip to resolve the dependencies by using ml_dtypes v0.5.4

[joverlee521]

Easy workaround is to isolate pathogen-embed in its own venv like fauna. Example: 04d5f6f

Ah I somehow missed your comment @victorlin 🤦‍♀️ That looks cleaner than my workaround in #295.

[victorlin]

The downside to isolation is a larger image, because it'll contain both numpy v1 and numpy v2 (and any other dependencies that could otherwise be shared).

[victorlin]

Another workaround is to install them together pip install pathogen-embed==3.0.0 evofr, forcing proper dependency resolution. It'll combine two smaller layers into one larger one, but the overall image size should stay the same without duplicate dependencies that come with the isolation workaround. Trade offs...

[joverlee521]

Hmm, I like installing them together as the workaround for now. We can revisit isolation if they have conflicts that are impossible to resolve in the future.