Skip to content

COMP: Move ITK Python wrapping to SWIG 4.4.1#6484

Merged
hjmjohnson merged 6 commits into
InsightSoftwareConsortium:mainfrom
hjmjohnson:swig-4.4.1
Jun 23, 2026
Merged

COMP: Move ITK Python wrapping to SWIG 4.4.1#6484
hjmjohnson merged 6 commits into
InsightSoftwareConsortium:mainfrom
hjmjohnson:swig-4.4.1

Conversation

@hjmjohnson

@hjmjohnson hjmjohnson commented Jun 21, 2026

Copy link
Copy Markdown
Member

Move ITK Python wrapping to SWIG 4.4.1. Fixes the import regression in #6479; supersedes the temporary #6480 (4.3.x baseline) and #6481 (WIP CI).

Three focused commits, no WIP — standard GHA + Azure CI run on this PR. The Azure/ARM Python pipelines exercise the vendored 4.4.1 binaries; local pixi run build-python uses conda-forge SWIG/CastXML.

What changed
  • Multi-phase init fix. ITK packs several SWIG modules per shared library and injects C that calls PyInit__<X>Python() and stores the return in sys.modules. SWIG 4.4 (PEP 489 multi-phase init) makes PyInit_* return a raw PyModuleDef, so the proxy failed at import with 'moduledef' object has no attribute 'delete_SwigPyIterator'. The injection now detects a returned def at runtime and instantiates it via PyModule_FromDefAndSpec + PyModule_ExecDef; the single-phase (≤4.3) path is unchanged; all APIs are Limited-API-safe (≥3.5).
  • abi3 floor. Pinned to a single cached source of truth _ITK_MINIMUM_SUPPORTED_LIMITED_API_VERSION (3.11), shared by the Limited-API auto-detect and USE_SABI. 3.11+ build abi3 and load on 3.11; 3.10 keeps the non-Limited-API path.
  • Vendored SWIG 4.4.1 via FetchContent + version assertion. The prebuilt-binary download moves from ExternalProject_Add to FetchContent, so SWIG_EXECUTABLE exists at configure time and itk_assert_swig_version() confirms it reports ≥ swig_version_min (now 4.3.0; too-old system swig is a FATAL_ERROR). Binaries are repackaged from the official, broadly-portable SWIG PyPI wheels and hosted on data.kitware.com (resolved by SHA512). Source-build fallback stays on ExternalProject.
  • Local dev. configure-python / configure-debug-python pass ITK_USE_SYSTEM_SWIG/ITK_USE_SYSTEM_CASTXML; swig (>=4.4.1) + castxml added to the python pixi feature.
How the vendored binaries were produced (reproducible)

The swig<os>-<arch>-4.4.1.zip archives are repackaged from the official swig PyPI package (https://pypi.org/project/swig/, maintained by the SWIG team, built with cibuildwheel). These wheels are the broadest-portability prebuilt SWIG available — Linux manylinux2010 (glibc 2.12) / manylinux2014, macOS 10.9 universal2 (x86_64+arm64 in one file), and Windows — far more portable than building on a dev box. Each wheel contains swig/data/bin/swig[.exe] + swig/data/share/swig/4.4.1/, i.e. ITK's exact layout, so repackaging is just extract-and-rezip (platform-independent — no cross-compiling):

process() {  # $1=pip-platform-tag  $2=itk-archive-name
  wd=$(mktemp -d)
  pip download --no-deps swig==4.4.1 --platform "$1" --only-binary=:all: -d "$wd"
  ( cd "$wd" && unzip -q swig-4.4.1*.whl )
  chmod +x "$wd/swig/data/bin/"*
  ( cd "$wd/swig/data" && zip -r -q "$OLDPWD/$2.zip" bin share )   # bin/ + share/ at archive root
}
process macosx_10_9_universal2   swigmacos-arm64-4.4.1   # universal2 also serves swigmacos-amd64
process manylinux2010_x86_64     swiglinux-amd64-4.4.1
process manylinux2014_aarch64    swiglinux-arm64-4.4.1
process win_amd64                swigwin-amd64-4.4.1

The resulting archives were uploaded to data.kitware.com and are fetched by SHA512:

archive wheel source SHA512
swigmacos-arm64 / -amd64 macosx_10_9_universal2 17c4637d…1ce3cef9
swiglinux-amd64 manylinux2010_x86_64 6447a7b9…d88ca11
swiglinux-arm64 manylinux2014_aarch64 44a6d174…c8902d24
swigwin-amd64 win_amd64 700bd4de…e338f3892

itk_assert_swig_version() runs swig -version on the downloaded binary at configure time and fails if it is not ≥ swig_version_min, so a wrong/corrupt upload is caught immediately. For a future SWIG bump, rerun the recipe with the new version, re-upload, and update ITK_SWIG_VERSION + the per-platform SHA512s.

Validation

Builds and imports on Python 3.11.15 and 3.13.9 (same abi3 build); the std::vector/SwigPyIterator path and 10/10 scoped Python ctests pass. The vendored FetchContent download from data.kitware.com was verified at configure time to fetch 4.4.1, pass the assertion, and build _ITKCommonPython.abi3.so cleanly.

Download mirrors

data.kitware.com (keyed by SHA512) is the primary; dweb.link, ipfs.io, and gateway.pinata.cloud serve the same bytes by content id as IPFS fallbacks, matching ITK's ExternalData gateway set. URL_HASH SHA512 validates whichever mirror responds. The content ids are the reproducible ipfs-car/IPIP-0499 CIDs produced by the ExternalData upload pipeline:

archive CID
swigwin-amd64 bafybeifxjnbydus4eq46bocwb2hcj5ufpbj4etocia5ims426gfiw7krfi
swiglinux-amd64 bafybeiep2wbecrcii7xzfe2cekhfco6mlxqym4jhfdl6xi3p4aieblgz6m
swiglinux-arm64 bafybeifaqaus3yzp5oujxupxk422hfmcg2idf3oyvdtsppzq7ymn64nvze
swigmacos-arm64/amd64 bafybeidllsbllkqwoby6r5w4ueqzky2o7r6qkjbh45q57mczyh2fwfwd3q

The archives are pinned to IPFS and mirrored into ITKTestingData via the ExternalData upload pipeline.

@github-actions github-actions Bot added type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots area:Python wrapping Python bindings for a class labels Jun 21, 2026
@hjmjohnson

Copy link
Copy Markdown
Member Author

Pushed a fix for the vendored-SWIG Python CI failure (ARMBUILD-Python / ITK.*.Python): the FetchContent refactor removed the swig ExternalProject target, but the wrapping build-ordered against it whenever ITK_USE_SYSTEM_SWIG=OFF. Now guarded with if(NOT ITK_USE_SYSTEM_SWIG AND TARGET swig) — prebuilt binaries are fetched at configure time and need no build-order dependency. Verified locally: the vendored path downloads SWIG 4.4.1 from data.kitware.com, passes the configure-time version assertion, and builds _ITKCommonPython.abi3.so cleanly.

@hjmjohnson hjmjohnson force-pushed the swig-4.4.1 branch 4 times, most recently from 78951a1 to b91641b Compare June 21, 2026 12:49
hjmjohnson added a commit to InsightSoftwareConsortium/ITKTestingData that referenced this pull request Jun 21, 2026
Content-addressed (CID) archives repackaged from the official SWIG 4.4.1 PyPI
wheels (macOS 10.9 universal2, manylinux2010/2014, Windows), used as the IPFS/CDN
mirror for ITK's vendored SWIG download. See InsightSoftwareConsortium/ITK#6484.
hjmjohnson added a commit to hjmjohnson/ITKTestingData that referenced this pull request Jun 21, 2026
CID-addressed swig 4.4.1 source used by the ITK_USE_SYSTEM_SWIG=OFF
source-build fallback when no prebuilt binary exists for the host.
See InsightSoftwareConsortium/ITK#6484.
hjmjohnson added a commit to InsightSoftwareConsortium/ITKTestingData that referenced this pull request Jun 21, 2026
CID-addressed swig 4.4.1 source used by the ITK_USE_SYSTEM_SWIG=OFF
source-build fallback when no prebuilt binary exists for the host.
See InsightSoftwareConsortium/ITK#6484.
@hjmjohnson hjmjohnson marked this pull request as ready for review June 21, 2026 16:24
@greptile-apps

This comment was marked as resolved.

Comment thread Wrapping/Generators/Python/CMakeLists.txt
Comment thread Wrapping/Generators/SwigInterface/CMakeLists.txt
Comment thread Wrapping/Generators/SwigInterface/CMakeLists.txt
ITK packs several SWIG modules per shared library and injects C that calls
PyInit__<X>Python() and stores the return in sys.modules. SWIG 4.4 (PEP 489
multi-phase init) makes PyInit_* return a raw PyModuleDef rather than a populated
module, so the proxy fails at import with 'moduledef' object has no attribute
'delete_SwigPyIterator'. Detect a returned def at runtime and instantiate it via
PyModule_FromDefAndSpec + PyModule_ExecDef; the single-phase (<=4.3) path is
unchanged. All APIs used are Limited-API-safe (>=3.5).

Pin the abi3 floor to a single cached source of truth
(_ITK_MINIMUM_SUPPORTED_LIMITED_API_VERSION = 3.11) shared by the Limited-API
auto-detect and the USE_SABI setting, so 3.11+ builds load on Python 3.11; 3.10
keeps the non-Limited-API path.

Assisted-by: Claude Code — root-cause analysis and local 3.11/3.13 validation
Replace the prebuilt-binary ExternalProject_Add download branches with a single
FetchContent_Declare/MakeAvailable that downloads and extracts the binary at
configure time, so SWIG_EXECUTABLE exists immediately and is checked against
swig_version_min by itk_assert_swig_version() (runs 'swig -version'). A wrong or
corrupt upload now fails at configure instead of deep in the build. The
source-build fallback keeps ExternalProject_Add (it compiles SWIG + PCRE2).

Point the download at SWIG 4.4.1 binaries repackaged from the official, broadly
portable SWIG PyPI wheels (Linux manylinux2010/2014, macOS 10.9 universal2,
Windows) and uploaded to data.kitware.com, which resolves each by its SHA512.
Every archive shares one bin/swig[.exe] + share/swig/4.4.1 layout, so the
per-platform swig_dir_subpath collapses to a single value. swig_version_min is
raised 4.2.0 -> 4.3.0 (4.2.x lacks the SWIG_Py_DECREF runtime); too-old system
swig is now a FATAL_ERROR.

Assisted-by: Claude Code — FetchContent refactor and configure-time download+assert verification
Add swig (>=4.4.1) and castxml to the python pixi feature and pass
ITK_USE_SYSTEM_SWIG/ITK_USE_SYSTEM_CASTXML in configure-python and
configure-debug-python, so local 'pixi run build-python' uses the conda-forge
toolchain instead of the internal download/build. CI continues to exercise the
vendored 4.4.1 binaries via the standard Azure/ARM Python pipelines.
swig_archive was only read by the IPFS mirror URLs, which were dropped when the
vendored download moved to data.kitware.com (resolved by SHA512). The macOS
per-architecture branch existed solely to set it (the universal2 archive has one
hash for both), so it collapses too.
swig_cmake_version and ITK_SWIG_VERSION were both 4.4.1 after the consolidation
to PyPI-wheel archives (all platforms share the share/swig/<version> layout). Drop
the duplicate so SWIG_DIR's version segment derives from a single source on both the
prebuilt and source-build paths.
The data.kitware.com + ITKTestingData + IPFS gateway mirror list was duplicated
across the prebuilt FetchContent download and the source-build ExternalProject.
Emit it from a single itk_swig_mirror_urls() helper so the gateway set stays
consistent across both swig acquisition paths.
Comment thread Wrapping/Generators/Python/CMakeLists.txt
@hjmjohnson hjmjohnson requested a review from blowekamp June 22, 2026 00:35
@dzenanz dzenanz requested a review from thewtex June 22, 2026 14:26
@hjmjohnson hjmjohnson merged commit 8e7d35d into InsightSoftwareConsortium:main Jun 23, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Python wrapping Python bindings for a class type:Compiler Compiler support or related warnings type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants