Skip to content

lz4: add build-lz4.yml for riscv64 wheels - #352

Merged
luhenry merged 3 commits into
mainfrom
lz4
Aug 25, 2026
Merged

lz4: add build-lz4.yml for riscv64 wheels#352
luhenry merged 3 commits into
mainfrom
lz4

Conversation

@luhenry

@luhenry luhenry commented Aug 25, 2026

Copy link
Copy Markdown
Member

Adds .github/workflows/build-lz4.yml, porting python-lz4 4.4.5 to riscv64.

Based on upstream's build_dist.yml, narrowed to manylinux_riscv64.

Shape

Build-from-checkout with pypa/cibuildwheel, matrix [cp312, cp313, cp314, cp314t] (per-interpreter — the three extensions link the version-specific ABI; PyPI shows cpXY-cpXY wheels, no abi3 tag).

  • PYLZ4_USE_SYSTEM_LZ4=False (upstream's own CIBW_ENVIRONMENT) builds against the bundled lz4libs/ C sources, so nothing has to provide a system liblz4 in the container.
  • cp314t gets upstream's free-threaded test setup (pytest-run-parallel, --parallel-threads=1 as on their slower arm runner). Upstream builds cp313t, which the riscv64 manylinux image does not ship; cp314t is the free-threaded target here.

Deviations from upstream

Test command. Upstream's is a bare tox -c {project}, which runs envlist = py, flake8, docs against an editable install — so the wheel cibuildwheel just built is never imported, and the wheel test phase also runs flake8 and a sphinx make doctest html. This workflow uses tox -c {package} -e py --installpkg {wheel} instead: same suite, run against the actual wheel.

Vendored-library licensing. patches/lz4/4.4.5/0001-Package-licenses-of-vendored-lz4-and-xxHash-sources.patch carries upstream PR #344 (Upstream-Status: Submitted), refreshed to apply on v4.4.5. The wheels statically link the bundled lz4 1.9.4 and xxHash 0.6.5 sources under lz4libs/, both BSD-2-Clause, whose binary-redistribution clause requires the copyright notice to be reproduced — but only python-lz4's own BSD-3-Clause LICENSE was shipped. The patch adds lz4libs/LICENSE and lz4libs/LICENSE.xxhash and declares all three via license_files, so they land in dist-info/licenses/.

Two notes on the refresh: v4.4.5 predates the license='BSD-3-Clause' line that upstream's setup.py hunk is anchored on, and the xxHash copyright line here follows the vendored lz4libs/xxhash.c header (2012-2016) rather than the wider range used in that PR.

0002-Switch-to-SPDX-license-expression.patch backports #324 (Upstream-Status: Backport), merged just after v4.4.5. Without it the wheel's METADATA carries no License: field at all — only the deprecated classifier — so update_doc.py renders the package's docs entry as license: Unknown.

Applying the patch dirties the checkout, so setuptools_scm derived 4.4.6.dev0 from git describe instead of the tag; the version is now pinned with SETUPTOOLS_SCM_PRETEND_VERSION_FOR_LZ4, and the full-history fetch it needed is gone.

Validation

Ran the full workflow config through cibuildwheel locally on linux/aarch64 (native, Docker) for both cp312-manylinux_aarch64 and cp314t-manylinux_aarch64:

  • version resolved to 4.4.5 from the tag inside the container (setuptools_scm),
  • wheel contains lz4/_version…so, lz4/block/_block…so, lz4/frame/_frame…so and dist-info/licenses/LICENSE,
  • 19804 passed in both cases, coverage reporting 0% on the source tree — i.e. the installed wheel was imported, not the checkout.

For the licensing patch, rebuilt natively (shallow v4.4.5 clone + git apply + the workflow's env) and confirmed the wheel is lz4-4.4.5-… carrying all three extension modules and

lz4-4.4.5.dist-info/licenses/LICENSE
lz4-4.4.5.dist-info/licenses/lz4libs/LICENSE
lz4-4.4.5.dist-info/licenses/lz4libs/LICENSE.xxhash

The riscv64 cp314 artifact from the first patched run confirms the same on the real target: lz4-4.4.5-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl, three extension modules, all three License-File: entries in METADATA, and libc.so.6 as the only linked library — so nothing else needs a licence carried with it.

actionlint clean apart from the usual label "ubuntu-24.04-riscv" is unknown.

Port python-lz4 4.4.5 to riscv64, mirroring upstream's build_dist.yml
cibuildwheel job narrowed to manylinux_riscv64.

The wheel bundles lz4libs (PYLZ4_USE_SYSTEM_LZ4=False), so no system
liblz4 is needed in the container. Tests run upstream's tox suite, but
via `-e py --installpkg {wheel}` so the suite exercises the built wheel
rather than an editable install, and without the flake8/docs envs that
upstream's bare `tox -c {project}` also picks up.
@threexc

threexc commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

I would take the LLM's suggestion and patch the licensing - this is something we need to keep an eye out for since we're distributing the wheels ourselves.

The wheels statically link the bundled lz4 1.9.4 and xxHash 0.6.5 C sources
under lz4libs/, but only ship python-lz4's own BSD-3-Clause LICENSE. Both
vendored libraries are BSD-2-Clause and require their copyright notice to be
reproduced in binary redistributions, so carry upstream PR #344 (refreshed
onto v4.4.5) to add the two notices and declare them via license_files.

Applying the patch dirties the checkout, which makes setuptools_scm derive
4.4.6.dev0 rather than the tag, so pin the version with
SETUPTOOLS_SCM_PRETEND_VERSION_FOR_LZ4 and drop the now-pointless full-history
fetch.
v4.4.5's setup.py declares the licence only through the deprecated
`License :: OSI Approved :: BSD License` classifier, so the wheel's METADATA
carries no `License:`/`License-Expression:` field and update_doc.py renders the
package's docs entry as `license: Unknown`. Backport the upstream commit that
sets `license='BSD-3-Clause'`; the docs entry now names the licence.
@luhenry

luhenry commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

Done — two patches under patches/lz4/4.4.5/, both carrying an upstream link rather than being invented here.

0001-Package-licenses-of-vendored-lz4-and-xxHash-sources.patch (Upstream-Status: Submitted) is upstream PR #344, refreshed onto v4.4.5. It adds lz4libs/LICENSE (lz4 1.9.4) and lz4libs/LICENSE.xxhash (xxHash 0.6.5) and declares all three via license_files, so they land in dist-info/licenses/. Two deliberate differences from that PR: v4.4.5 predates the license='BSD-3-Clause' line its setup.py hunk is anchored on, and the xxHash copyright line here is the one the vendored lz4libs/xxhash.c actually carries (2012-2016) rather than the wider range in the PR — the BSD-2 clause is about reproducing that notice.

0002-Switch-to-SPDX-license-expression.patch (Upstream-Status: Backport) is upstream PR #324, merged just after the tag. Turned out to matter for the same reason you raised: v4.4.5 declares its licence only through the deprecated classifier, so the wheel METADATA has no License: field and update_doc.py was about to write license: Unknown into docs/packages/lz4.yaml.

Verified on the riscv64 artifact, not just locally:

lz4-4.4.5-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl
  lz4-4.4.5.dist-info/licenses/LICENSE
  lz4-4.4.5.dist-info/licenses/lz4libs/LICENSE
  lz4-4.4.5.dist-info/licenses/lz4libs/LICENSE.xxhash

and libc.so.6 is the only library the three extension modules link, so there is nothing else whose licence would need to travel with the wheel.

One side effect worth flagging for the next patched port: git apply leaves the checkout dirty, and setuptools_scm reads a dirty tree at a tag as post-release — the first patched build would have produced lz4-4.4.6.dev0+g59b2d817.d20260825, silently breaking the wheel-name/docs-YAML/patches/<pkg>/<version>/ match. Pinned with SETUPTOOLS_SCM_PRETEND_VERSION_FOR_LZ4, which also made the fetch-depth: 0 redundant. Added to CLAUDE.md as a gotcha.

@luhenry
luhenry merged commit 8e73b0a into main Aug 25, 2026
9 checks passed
@luhenry
luhenry deleted the lz4 branch August 25, 2026 16:03
@luhenry luhenry linked an issue Aug 25, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lz4 riscv64 support

2 participants