Building Windows wheels with libraries fails

[jpakkane]

When building a CapyPDF releases I now get the following error on Windows:

Traceback (most recent call last):
  File "C:\python\lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
    main()
  File "C:\python\lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "C:\python\lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\__init__.py", line 985, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\__init__.py", line 1039, in build_wheel
    return project.wheel(out).name
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\__init__.py", line 890, in wheel
    return builder.build(directory)
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\__init__.py", line 455, in build
    self._install_path(whl, src, dst)
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\__init__.py", line 414, in _install_path
    mesonpy._rpath.fix_rpath(origin, libspath)
  File "C:\Users\Jussi Pakkanen\AppData\Local\Temp\build-env-t_mdgsd0\lib\site-packages\mesonpy\_rpath.py", line 64, in fix_rpath
    raise NotImplementedError(f'Bundling libraries in wheel is not supported on {sys.platform}')
NotImplementedError: Bundling libraries in wheel is not supported on win32

ERROR Backend subprocess exited when trying to invoke build_wheel

Ths is on 3.12.0, but I also tried downgrading to 3.11 and 3.10.11 (which used to work when I made the previous release AFAICR). Buildng the package on macOS 3.10.5 works without problems.

To reproduces:

• Check out CapyPDF
• Edit meson.build to change the default C++ version from c++23 to c++latest (no longer needed once the next Meson release comes out)
• Build either with pip3 install . or python -m build

The error message gives me zero relevant hits on Google.

[dnicolodi]

I'm very surprised that you were able to build the wheel that published on PyPI for CapyPDF 0.5.0. As the error message says, bundling shared libraries into wheels is not supported on Windows, and never was. The reason is that meson-python needs to play tricks with RPATH to have the libraries loaded from the location where they are installed, and AFAIK Windows does not have the equivalent of an RPATH. Without setting the DLL load path, I don't know how capypdf-0.dll in the Windows wheel is able to find the other DLLs it should be linked with. It may be that the ctypes DLL loading mechanism sets the DLL load path. I know very little about dynamic linking on Windows.

The old wheel was build on September first. At that time meson-python 0.13.2 was the latest version. Can check that pinning the version of meson-python in pyproject.toml to 0.13.2 makes the wheel build again?

[jpakkane]

I'll try that soon.

But the way CapyPDF does it is that it does not build a Python extension, but instead a standalone shared library that it then opens with ctypes. The old version had the main capypdf dll as well as its deps as standalone dlls, whereas now we only build a single shared library with the dependencies embedded in the main library so the end result should be fully standalone with no library trickery needed.

[dnicolodi]

The way I would do this is to place the shared library next to the Python module and do not rely on the relocation of the shared library made by meson-python by adjusting the install_dir of the shared library to py.get_install_dir(pure: false).

[jpakkane]

Using 0.13.2 works.

adjusting the install_dir of the shared library to py.get_install_dir(pure: false).

I tried this and it works but it is problematic. The CapyPDF library is meant to be usable also as a standard shared library so it needs to be installed to standard libdir. Needing a toggle switch to choose between the two installation locations is not great. Especially since if you have any dependencies that are also shared libraries must be installed in the same dir as capypdf-0.dll for Windows to find them. If upstream does not provide for this toggle switch (and almost nobody does) then you can't do the install without editing build files, which is bad.

A better approach for this would be for Meson-Python to automatically tell it to install all libraries in the correct location. I tried setting libdir to the expected value but that did not work, I got the same error as above.

[dnicolodi]

Using 0.13.2 works.

This is interesting. We must have tightened the check along the way. Unfortunately, the existing check is correct in the general case, thus relaxing it again is not really an option. For example, the wheel for CapyPDF 0.5.0 includes a few executables that I'm fairly sure fail to load the required DLLs when executed on Windows.

I tried this and it works but it is problematic. The CapyPDF library is meant to be usable also as a standard shared library so it needs to be installed to standard libdir.

Sure. I realize this is problematic, but the intersection of the limitations of the wheel package format and of Windows does not leave space for other solutions. At least I cannot come up with none that works in all use cases. The only thing I can think about is do the DLL relocation also on Windows and use a .pth Python import mechanism hook to execute os.add_dll_directory(). But this does not work for executable installed as part of the wheel.

A better approach for this would be for Meson-Python to automatically tell it to install all libraries in the correct location.

The problem is that there is no "correct location" that works in the general case on Windows. On macOS and on Unix platforms we do just this and we adjust RPATH to make everything work. A possibility would be to add a setting to meson-python that says "I know what I am doing" and enables mapping {libdir} to the same location used on macOS and Unix on Window too. The user would be then responsible to invoke os.add_dll_directory() as required, or use ctypes to load the libraries, or something else that works without meson-python fixing things up.

SciPy uses delvewheel https://pypi.org/project/delvewheel/ to bundle DLLs in wheels on Windows. I haven't investigated how it works.

I tried setting libdir to the expected value but that did not work, I got the same error as above.

This is expected, meson-python needs to lock at the "unexpanded" definition of the install location to work correctly.

[jpakkane]

the wheel for CapyPDF 0.5.0 includes a few executables that I'm fairly sure fail to load the required DLLs when executed on Windows.

Those executables should not be there and are never executed. They just get installed due to --tags=runtime,python-runtime. I have been too lazy to figure out how to get them removed (originally they got there by accident because I did not inspect the installed files by hand, this is tech preview level code after all).

A possibility would be to add a setting to meson-python that says "I know what I am doing" and enables mapping {libdir} to the same location used on macOS and Unix on Window too. The user would be then responsible to invoke os.add_dll_directory() as required, or use ctypes to load the libraries, or something else that works without meson-python fixing things up.

This seems like a reasonable approach and is already how CapyPDF does it (it has heuristics to find out where the shared library is for every setup).

In general having a way to install shared libs on Windows and it "just working" would be nice, but that probably requires assistance from Python upstream.

[rgommers]

I tried this on macOS. With meson-python 0.13.2, the result is an capypdf-0.7.0-py3-none-any.whl and a wheel layout of:

├── .capypdf.mesonpy.libs
│   └── libcapypdf.0.dylib
├── capypdf-0.7.0.dist-info
│   ├── COPYING
│   ├── METADATA
│   ├── RECORD
│   └── WHEEL
└── capypdf.py

Note the name is wrong, since the dylib is specific to macOS arm64 here - this will work for a local from-source install but not for distributing the wheel. It improves when adding find(installation(pure: false) to python/meson.build, but then the tags become more specific than you want (contains cp310 when building on Python 3.10). With latest meson-python (but not 0.13.2), you can add this to pyproject.toml:

[tool.meson-python]
limited-api = true

and that should work but doesn't produce the expected abi3 tag. So we have something to fix for this ctypes + non-py-extension situation.

There's some logic to deal with locating the shared library in capypdf.py:

if sys.platform == 'win32':
    libfile_name = 'capypdf-0.dll'
elif sys.platform == 'darwin':
    libfile_name = 'libcapypdf.0.dylib'
else:
    libfile_name = 'libcapypdf.so'
libfile = None

if 'CAPYPDF_SO_OVERRIDE' in os.environ:
    libfile = ctypes.cdll.LoadLibrary(os.path.join(os.environ['CAPYPDF_SO_OVERRIDE'], libfile_name))

if libfile is None:
    try:
        libfile = ctypes.cdll.LoadLibrary(libfile_name)
    except (FileNotFoundError, OSError):
        pass

if libfile is None:
    # Most likely a wheel installation with embedded libs.
    from glob import glob
    if 'site-packages' in __file__:
        sdir = os.path.split(__file__)[0]
        # Match libcapypdf.so.0.5.0 and similar names too
        globber = os.path.join(sdir, '.*capypdf*', libfile_name + "*")
        matches = glob(globber)
        if len(matches) == 1:
            libfile = ctypes.cdll.LoadLibrary(matches[0])

if libfile is None:
    raise CapyPDFException('Could not locate shared library.')

which looks fragile, and indeed does not work for me without first doing:

export CAPYPDF_SO_OVERRIDE=.capypdf.mesonpy.libs

In general having a way to install shared libs on Windows and it "just working" would be nice, but that probably requires assistance from Python upstream.

Agreed that'd be quite nice. Python (packaging) upstream is unlikely to help us here though, we have to implement support for this ourselves. This issue is pretty much a duplicate of gh-265, where a potential solution direction is discussed.

I've got more experience with that type of solution now that we're distribution OpenBLAS (shared/static libraries + pkg-config files and headers, no Python involved at all) inside a wheel. I think the pre-loading suggestion in gh-265 is doable. It basically matches the "I know what I'm doing" suggested solution direction in the comments above (with a bit more detail already). However, ctypes is a little finicky, one unexpected issue we're having is that in order to make the ctypes preloading robust on all platforms (e.g., Musl-based distros) is that one has to use ctypes.RTLD_GLOBAL - and that then increases the potential of issues due to symbol clashes. Not really an issue for CapyPDF because it has no Python extension module that depends on the shared library, but I think most other packages would have a Python extension module which needs the shared library, so it's something to keep in mind.

In order not to split the discussion, how about we continue the "shared library in wheel" part in gh-265, and close this issue? Plus open a new one for the limited-api thing (assuming you agree it's a bug @dnicolodi)?

[dnicolodi]

I tried this on macOS. With meson-python 0.13.2, the result is an capypdf-0.7.0-py3-none-any.whl and a wheel layout of:

This is a bug in meson-python. meson-python should have detected that the wheel contains platform specific components.

It improves when adding find(installation(pure: false) to python/meson.build, but then the tags become more specific than you want (contains cp310 when building on Python 3.10). With latest meson-python (but not 0.13.2), you can add this to pyproject.toml:

[tool.meson-python]
limited-api = true

and that should work but doesn't produce the expected abi3 tag.

This is correct. The wheel does not contain Python ABi so it should not be marked with the abi3 tag or any other Python ABI tag. I'll have a look at what where things go wrong.

In general having a way to install shared libs on Windows and it "just working" would be nice, but that probably requires assistance from Python upstream.

Agreed that'd be quite nice. Python (packaging) upstream is unlikely to help us here though, we have to implement support for this ourselves. This issue is pretty much a duplicate of gh-265, where a potential solution direction is discussed.

Not really a duplicate. CapyPDF does not distribute a Python extension module, so it does not need an equivalent for the RPATH handling to make the extension module find the shared libraries.

[dnicolodi]

Note the name is wrong, since the dylib is specific to macOS arm64 here

This is an (obvious) oversight in meson-python. Fix in #526

[rgommers]

Not really a duplicate. CapyPDF does not distribute a Python extension module, so it does not need an equivalent for the RPATH handling to make the extension module find the shared libraries.

Fair - partial duplicate at least though, since the place where the shared library ends up in the wheel needs to be exposed, and that's the critical part of the solution to enable the package author to then access that shared library directly (with ctypes, cffi, os.add_dll_directory or some other method - the details there probably don't matter too much).

[dnicolodi]

I meant that I see these as two slightly different use cases: one is implementing a solution akin to the RPATH one on Windows, the other is simply having the shared libraries placed somewhere in the wheel. Having the shared library in a wheel consumed by another wheel maybe can be seen as another variation on the theme.

[jpakkane]

Things get even weirder now. With CapyPDF 0.7.0 and meson-python 0.13.2 on Windows the --skip-subprojects=lcms2 argument seems to be ignored and the executables from that subproject get in the package no matter what I do (on macOS they are correctly not included).

I even tried adding a custom install script that deletes the exes but they still ended up in the final product.

[rgommers]

With CapyPDF 0.7.0 and meson-python 0.13.2 on Windows the --skip-subprojects=lcms2 argument seems to be ignored

This was gh-423, and was fixed in 0.14.0

I even tried adding a custom install script that deletes the exes but they still ended up in the final product.

There is no longer a meson install step, the result of which would be zipped up. Rather, source and built files are included into the wheel directly based on the metadata in intro-install_plan.json.

[jpakkane]

If you grab build targets from the build dir yourself the end result is not guaranteed to work. There may be e.g. remnants of RPATH in the binaries. If you don't do an actual install, the end result can break at any time for whatever reason.

[jpakkane]

Not to mention that projects that need their post-install scripts to be run in order to work obviously won't work.

[dnicolodi]

If you grab build targets from the build dir yourself the end result is not guaranteed to work. There may be e.g. remnants of RPATH in the binaries. If you don't do an actual install, the end result can break at any time for whatever reason.

Meson itself supports running build artifacts from the build directory via the test() facility. Thus we didn't see picking build artifact from the build directory to build the Python wheels as problematic as you describe. meson-python has his own RPATH handling, which was easier to get right starting from binaries that still have the build directory in their RPATH.

[dnicolodi]

Not to mention that projects that need their post-install scripts to be run in order to work obviously won't work.

These projects would indeed not be supported by meson-python, but no one has yet brought up an use case for this.

[jpakkane]

Meson itself supports running build artifacts from the build directory via the test() facility.

That is because we use special platform dependenct magic to make that happen. It is only undone when doing an install. These are the only two supported setups.

Also note that any target that uses install_rpath is probably also broken.

Thus we didn't see picking build artifact from the build directory to build the Python wheels as problematic as you describe.

You can do that if you want, just be aware that if and when it breaks, you get to keep both pieces.

In fact I suspect I have already hit this problem once. When I originally did the macOS extension it did not work due to some weird dylib lookup failure. I fixed it by building deps as static libs so that that only one dylib gets installed.

[dnicolodi]

That is because we use special platform dependenct magic to make that happen. It is only undone when doing an install. These are the only two supported setups.

Yup. meson-python needs to tweak RPATH too so we have very similar code.

Also note that any target that uses install_rpath is probably also broken.

install_rpath does not make much sense for Python wheels. If needed we can add support for it. However, last time I checked, Meson support for install_rpath is broken on macOS, thus I am quite sure it is not widely used.

In fact I suspect I have already hit this problem once. When I originally did the macOS extension it did not work due to some weird dylib lookup failure. I fixed it by building deps as static libs so that that only one dylib gets installed.

I fixed several bugs in this area, thus if you tried that a while ago, I would be curious to know if you can still reproduce these issues. Unfortunately my macOS system is too old to compile CapyPDF.

[dnicolodi]

@jpakkane I would appreciate if you could give #551 a spin. You can do that setting

[build-system]
build-backend = 'mesonpy'
requires = ['meson-python@ git+https://github.com/mesonbuild/meson-python.git@refs/pull/551/head']

[tool.meson-python]
shared-libs-win32 = true

in pyproject.toml. This should enable you to build a win32 wheel with the DLL in the same location as on other platforms.

Suggestions for a better name for the option are welcome.

[jpakkane]

I tried it and it seems to be working. Thanks.

Suggestions for a better name for the option are welcome.

Maybe it should be something along the lines of extension_type = ctypes (as opposed to pymodule) to specify that the project will not build a module and thus is expected to handle its own shared library wrangling?

[dnicolodi]

ctypes is not the only way to get a shared library to be useful on Windows. There is also cffi. Or the DLL could be linked to an extension module but the package may use os.add_dll_directory() or ctypes tricks to load it. Therefore, I would like the name to be generic.