DOC/TST: add a how-to page and test packages for dynamic versioning - #845
DOC/TST: add a how-to page and test packages for dynamic versioning#845rgommers wants to merge 3 commits into
Conversation
|
Cirrus says: Cirrus CI shuts down on June 1st 2026. - guess we'll have to deal with that soon. |
setuptools-scm has the advantage of being the only one that is actually e.g. redistributor friendly. So ideally I would like to see information about versioneer be buried forever so that nobody ever uses it again. I don't even want its existence acknowledged. As a downstream builder and repackager of software seeing a project use versioneer is a red flag that I'm in for detrimental experiences and suffering, and that I should think carefully about whether I want to use the project at all, if that project has viable competition. I'd also like to correct you about setuptools-scm's dependencies: Recent work on setuptools-scm has involved splitting out its core logic into "vcs-versioning", which has only one dependency (assuming recent python) and contains the bits you need to parse versions from git without the integration into the setuptools build backend. The setuptools-scm entrypoint and I won't deny that the packaging module isn't strictly necessary for our use cases, but it's still pretty lightweight compared to what you feared (and the packaging module is pretty much ubiquitous as a dependency for any package which is at all pyproject adjacent and doesn't actively have a policy forbidding the use of dependencies).
This is compatible with my long term desires. I want meson to directly support extracting versions from git, and Meson can't have dependencies so this would anyways require building it ourselves. I think that setuptools-scm has lots of cool knobs that... few people really need. :D So reimplementing it won't be a bad thing. Setuptools-scm also has to support a lot of legacy burden in its UX, sooo... Modern setuptools-scm prefers using .git_archival.txt where possible (this required a fix to git itself that allows using git describe in gitattributes export-subst, I'm the one who pushed for that in git upstream; old projects will still have depended on other, legacy workflows in setuptools-scm). My feeling is that meson doesn't need to support anything other than this. So what I'd probably want to see is project(..., version: files('.git_archival.txt'))with special detection of archival information, and then vcs_tag() could "know" when this was used and redetect the version. People could still use custom scripts if they want to do something more complex than this, as they can today. It for many common use cases things can become a lot simpler. |
I agree
I did look at that, and it seems much better in the long term, but as of now it has no docs of its own, and the setuptools-scm docs only say it's for building other packages on top of (https://setuptools-scm.readthedocs.io/en/latest/integrators/) plus they warn explicitly about using setuptools-scm functions like
With that last statement, do you mean some special magic to make that work for direct installs from vcs ( |
|
The file always exists. The way it works is you commit the file with special tokens into git, and then |
|
Thanks, I missed that. The point remains though: |
|
Version already accepts a files() object today -- it is currently expected to have a single line containing the version number. It wouldn't be that hard to detect when a multi-line file has replacement tokens instead of the processed metadata and run I will admit that mostly what exists at the moment is shower thoughts in my head, so don't necessarily treat this as immutable requirements. ;) |
|
Okay, I did a quick check: |
|
To be clear, on a 1.6 gb 20 seconds to write the whole thing to a file is still only 0.189s to do just COPYING. But a single text file wrapped in a tar envelope is trivially small enough to do all operations in a python bytes buffer anyway. I don't like wasted work and I don't like slow-as-heck tools that do tons of work for "convenience". Trust me, this was on my mind. :D |
| for line in f: | ||
| if line.startswith('version ='): | ||
| return line.split('=', 1)[1].strip().strip('\'"') | ||
| raise RuntimeError('version not found in pyproject.toml') |
There was a problem hiding this comment.
Is there any reason to do not use tomllib and fall back to tomli on older Python? tomli is a meson-python dependency that is not going away till we drop Python < 3.11 support, thus it is guaranteed to be present at build time. Because there are good chanced of this being copied in other projects, I think there is value in keeping this as straight forward and as future proof as possible.
There was a problem hiding this comment.
I adapted it from another project that doesn't depend on tomli, and in general this will be rare. So to make this an easy copy-paste for other projects, I avoided it. Can change to tomllib though, because very few projects will still be supporting Python 3.10 at this point - will update.
There was a problem hiding this comment.
What I meant is that tomli is a dependence of meson-python, and this being an example of how to implement something with meson-python, tomli is guaranteed to be there during the build, and this is not supposed to be run at runtime.
There was a problem hiding this comment.
Ah, good point, that's true even for non-isolated builds.
| parser = argparse.ArgumentParser() | ||
| group = parser.add_mutually_exclusive_group(required=True) | ||
| group.add_argument('--print-version', action='store_true') | ||
| group.add_argument('-o', '--outfile') | ||
| args = parser.parse_args() |
There was a problem hiding this comment.
For keeping this as straight forward as possible, why do not simply write to stdout, unless a file path is specified as first argument?
There was a problem hiding this comment.
sure, will update (EDIT: done)
Adapted from how this is done in NumPy
de534d1 to
b8a5375
Compare
b8a5375 to
c7c5f34
Compare
|
All comments addressed now |
|
This has been ready for a while @dnicolodi, shall we get this one in too? |
|
# pyproject.toml
[build-system]
requires = ["meson-python>=0.17", "meson>=1.6", "vcs-versioning>=2.3.2"]
build-backend = "mesonpy"
[project]
dynamic = ["version"]
...# root meson.build
project(
...,
version: run_command(
'python', '-m', 'vcs_versioning',
check: true,
).stdout().strip(),
) |
| project( | ||
| 'version-setuptools-scm', | ||
| version: run_command( | ||
| ['python3', '-m', 'setuptools_scm'], |
There was a problem hiding this comment.
python3 didn't work for me on Windows. I guess a Windows virtual env does not have a python3.exe?
| ['python3', '-m', 'setuptools_scm'], | |
| ['python', '-m', 'setuptools_scm'], |
There was a problem hiding this comment.
Hmm, that is annoying. We have Windows CI and this package passes, so I need to figure out what config is missing. Probably install in venv, then kick off pytest. How did you install Python exactly?
In NumPy et al. we use it via a script rather than invoking python3; Meson guarantees that the shebang of the script is handled on every platform:
https://github.com/numpy/numpy/blob/6e9bc3714bfefe3c3c31e4f70fb010b41b60b9dd/meson.build#L4
A little annoying if that'd be the solution though, because it requires committing a script file. Can't change it to plain python I think, that goes wrong elsewhere - macOS only has python3, and probably some Linux distros as well.
There was a problem hiding this comment.
Here is a reproducer, tiran/pycxxfilt#20 . The test case is using uv pip install:
Run uv pip install --system ".[test]"
Using Python 3.14.7 environment at: C:\hostedtoolcache\windows\Python\3.14.7\x64
Building pycxxfilt @ file:///D:/a/pycxxfilt/pycxxfilt
× Failed to build `pycxxfilt @ file:///D:/a/pycxxfilt/pycxxfilt`
├─▶ The build backend returned an error
╰─▶ Call to `mesonpy.build_wheel` failed (exit code: 1)
[stdout]
+ meson setup D:\a\pycxxfilt\pycxxfilt
D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f
-Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
--native-file=D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f\meson-python-native-file.ini
The Meson build system
Version: 1.12.0
Source dir: D:\a\pycxxfilt\pycxxfilt
Build dir: D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f
Build type: native build
..\meson.build:10:13: ERROR: Command
`C:\hostedtoolcache\windows\Python\3.14.7\x64\python3.exe -m
vcs_versioning` failed with status 1.
A full log can be found at
D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f\meson-logs\meson-log.txt
==== CI platform detected, click here for meson-log.txt
contents. ====
Build started at 2026-09-02T12:10:33.901755
Main binary:
D:\a\_temp\setup-uv-cache\builds-v0\.tmpznY1kJ\Scripts\python.exe
Build Options: -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
'--native-file=D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f\meson-python-native-file.ini'
Python system: Windows
The Meson build system
Version: 1.12.0
Source dir: D:\a\pycxxfilt\pycxxfilt
Build dir: D:\a\pycxxfilt\pycxxfilt\.mesonpy-o8iw1x_f
Build type: native build
Running command: C:\hostedtoolcache\windows\Python\3.14.7\x64\python3.exe
-m vcs_versioning
--- stdout ---
--- stderr ---
C:\hostedtoolcache\windows\Python\3.14.7\x64\python3.exe: No module
named vcs_versioning
..\meson.build:10:13: ERROR: Command
`C:\hostedtoolcache\windows\Python\3.14.7\x64\python3.exe -m
vcs_versioning` failed with status 1.
There was a problem hiding this comment.
cibuildwheel also fails with python3 -m vcs_versioning:
+ python -m build 'C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe' --wheel '--outdir=C:\Users\runneradmin\AppData\Local\Temp\cibw-run-e91uomyi\cp311-win_amd64\built_wheel' -Csetup-args=-Dpython.allow_limited_api=true
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- meson-python>=0.17
- meson>=1.6
- vcs-versioning>=2.3.2
* Getting build dependencies for wheel...
* Building wheel...
+ meson setup C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Dpython.allow_limited_api=true --native-file=C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s\meson-python-native-file.ini
The Meson build system
Version: 1.12.0
Source dir: C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe
Build dir: C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s
Build type: native build
..\meson.build:10:13: ERROR: Command `C:\Users\runneradmin\AppData\Local\Temp\cibw-run-e91uomyi\cp311-win_amd64\build\venv\Scripts\python3.exe -m vcs_versioning` failed with status 1.
A full log can be found at C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s\meson-logs\meson-log.txt
==== CI platform detected, click here for meson-log.txt contents. ====
Build started at 2026-09-02T12:10:45.273167
Main binary: C:\Users\runneradmin\AppData\Local\Temp\build-env-kbz25du8\Scripts\python.exe
Build Options: -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Dpython.allow_limited_api=true '--native-file=C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s\meson-python-native-file.ini'
Python system: Windows
The Meson build system
Version: 1.12.0
Source dir: C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe
Build dir: C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-yw473z_0\pycxxfilt-1.0.1.dev3+g3a52413fe\.mesonpy-_vv00v2s
Build type: native build
Running command: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-e91uomyi\cp311-win_amd64\build\venv\Scripts\python3.exe -m vcs_versioning
--- stdout ---
--- stderr ---
C:\Users\runneradmin\AppData\Local\Temp\cibw-run-e91uomyi\cp311-win_amd64\build\venv\Scripts\python3.exe: No module named vcs_versioning
..\meson.build:10:13: ERROR: Command `C:\Users\runneradmin\AppData\Local\Temp\cibw-run-e91uomyi\cp311-win_amd64\build\venv\Scripts\python3.exe -m vcs_versioning` failed with status 1.
There was a problem hiding this comment.
I had a look at the setuptools-scm docs, and the recommendation there is python -m setuptools_scm. There is no entrypoint for it. Then I had the brighter idea to look for other projects that already use setuptools-scm, and Matplotlib does (and has the solution):
There was a problem hiding this comment.
That works! I have updated my PR to use find_program('python3', 'python', version: '>= 3.11'), and Windows builds are passing.
There was a problem hiding this comment.
Meson guarantees that find_program('python3') will always find a python interpreter because if it cannot find that program name in PATH it will fall back to sys.executable for meson itself (which it knows has to be a 3.x interpreter).
The problems with this are that:
- Meson might not be in the same virtualenv as setuptools-scm / vcs-versioning. :) For Linux distro builds, all build dependencies will be installed to the same environment, of course, and for pip install if Meson isn't installed, it is installed to the isolated environment.
- if python3 doesn't exist in the isolated env, but does exist outside of it / from a different python installation, the version outside always wins and there's no real way to detect the difference.
This is a mess given that different OSes have different ideas about which the "correct" name that must always be present is: python or python3.
To make matters even worse, the BSDs will only have "python3.14", and lack both "python" and "python3".
You basically cannot win.
There was a problem hiding this comment.
I have tried find_program('python3') in tiran/pycxxfilt#21 . It fails on Windows. find_program('python3', 'python', version: '>= 3.11') is passing fine.
There was a problem hiding this comment.
Sure. The error message there is:
Build started at 2026-09-02T13:12:37.204127
Main binary: C:\Users\runneradmin\AppData\Local\Temp\build-env-1s9gao16\Scripts\python.exe
Build Options: -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Dpython.allow_limited_api=true '--native-file=C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-3t1qu6ic\pycxxfilt-1.0.1.dev5+g814e39458\.mesonpy-f6u4fdik\meson-python-native-file.ini'
Python system: Windows
The Meson build system
[...]
Program python3 found: YES (C:\Users\runneradmin\AppData\Local\Temp\cibw-run-0di4fykp\cp311-win_amd64\build\venv\Scripts\python3.exe)
Running command: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-0di4fykp\cp311-win_amd64\build\venv\Scripts\python3.exe -m vcs_versioning
--- stdout ---
--- stderr ---
C:\Users\runneradmin\AppData\Local\Temp\cibw-run-0di4fykp\cp311-win_amd64\build\venv\Scripts\python3.exe: No module named vcs_versioning
From "Main binary" you can see what the fallback python3 is (the python.exe from python -m build is powering Meson and reporting itself as the task manager executable etc.) and also that Meson discovered "Program python3 found: YES" and it is the python3.exe from cibw-run.
That falls under rule 2 above:
- if python3 doesn't exist in the isolated env, but does exist outside of it / from a different python installation, the version outside always wins and there's no real way to detect the difference.
Consequently, the vcs_versioning command succeeded at running python (that is all find_program can really "guarantee") but failed at finding vcs_versioning.
What I said above was maybe a bit confusingly worded. Meson guarantees that find_program('python3') will always find some interpreter, at least -- even if nowhere on PATH is a "python3" command -- because as a last ditch fallback it can use sys.executable even on platforms where "python3" is (alarmingly) never available at all. However it prefers to find one on PATH, if available, because adding a new python install to the front of PATH usually means one wishes to use it. That messes up here because python -m build doesn't create python3, but cibuildwheel does.
|
Great, thanks @tiran! I'm still hoping for |
4 participants
Closes gh-159
Most of our most prominent users (NumPy, pandas, Matplotlib) figured this out, and other packages copy from there if they want dynamic versioning. It's quite a common question (see incoming links to gh-159), so it's past time to document this.
I do feel like the tradeoffs of
versioneer/setuptools-scmare quite poor, and this stuff really isn't so hard that it needs separate dependencies (4 in the case of setuptools-scm?!?!) and thousands of lines of code. So there may be space for a follow-up where we either improve Meson'svcs_tag(or an alternative to it) so that works insideproject(), or we provide something simple to vendor or a CLI script, a la what NumPy does (that just works in O(100) lines of code). But for now, this should help.