Maturin develop succeeds locally but fails in CI with ModuleNotFoundError

[twitu]

In nautilus trader we're building a python library using pyo3 bindings.

maturin develop successfully builds the library, puts in site packages and the tests pass. But doing the same thing in CI fails.

 ______ ERROR collecting tests/unit_tests/persistence/test_catalog_rust.py ______
tests/unit_tests/persistence/test_catalog_rust.py:24: in <module>
    from nautilus.persistence import ParquetReader
E   ModuleNotFoundError: No module named 'nautilus'

We are using poetry for managing our python environment. This is our pyproject config.

requires = [
    "pip",  # Required for maturin
    "setuptools",
    "poetry-core>=1.4.0",
    "numpy>=1.24.1",
    "Cython==3.0.0a11",
    "maturin>=0.14.10,<0.15",
]

Have you ever encountered a situation where maturin behaves differently in CI?

[messense]

    Compiling nautilus_common v0.2.0 (/home/runner/work/nautilus_trader/nautilus_trader/nautilus_core/common)
   Compiling nautilus_pyo3 v0.2.0 (/home/runner/work/nautilus_trader/nautilus_trader/nautilus_core/pyo3)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 33s

=====================================================================
Nautilus Builder
=====================================================================
System: Linux x86_64
Clang:  Ubuntu clang version 14.0.0-1ubuntu1
Rust:   1.65.0 (8[97](https://github.com/nautechsystems/nautilus_trader/actions/runs/3884087470/jobs/6626186960#step:15:98)e37553 2022-11-02)
Python: 3.9.16
Cython: 3.0.0a11
NumPy:  1.24.1

Starting build...
BUILD_MODE=debug
PROFILE_MODE=False
ANNOTATION_MODE=False
PARALLEL_BUILD=True
COPY_TO_SOURCE=True

Compiling Rust libraries...
(cd nautilus_core && cargo build --all-features)

From this log looks like it's building with cargo build instead of maturin develop to me?

[messense]

the logs indicate the package is installed

Could you paste the relevant logs? I failed to find it.

maturin develop should output something like

$ maturin develop
🔗 Found pyo3 bindings
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
📦 Built wheel to /var/folders/wr/2mh8klw96ylf_s_hkzy79r1c0000gn/T/.tmpnBFMLu/example-0.1.0-py3-none-macosx_11_0_arm64.whl
🛠 Installed example-0.1.0

[cjdsellers]

Apologies, we merged that branch which then auto deleted it. I'll try and push up a new branch soon to show the example.

I did find one of the old action runs though https://github.com/nautechsystems/nautilus_trader/actions/runs/3947970177/jobs/6757416829, some excerpts:

   Compiling nautilus_pyo3 v0.2.0 (/home/runner/work/nautilus_trader/nautilus_trader/nautilus_core/pyo3)
    Finished dev [unoptimized + debuginfo] target(s) in 56.65s
📦 Built wheel for CPython 3.9 to /tmp/.tmp5zpcNy/nautilus_pyo3-0.2.0-cp39-cp39-linux_x86_64.whl
🛠 Installed nautilus_pyo3-0.2.0

 =====================================================================
Nautilus Builder
=====================================================================
System: Linux x86_64
Clang:  14.0.0-1ubuntu1
Rust:   1.66.1 (90743e729 2023-01-10)
Python: 3.9.16
Cython: 3.0.0a11
NumPy:  1.24.1

BUILD_MODE=debug
PROFILE_MODE=False
ANNOTATION_MODE=False
PARALLEL_BUILD=True
COPY_TO_SOURCE=True

Starting build...
Compiling Rust libraries...
(cd nautilus_core && cargo build --all-features)
Building Pyo3 rust module...
(cd nautilus_core/pyo3 && maturin develop --features extension-module)
Creating C extension modules...
define_macros=[('NPY_NO_DEPRECATED_API', 'NPY_1_7_API_VERSION')]
extra_compile_args=['-Wno-parentheses-equality']
build_dir=build/optimized
nthreads=2

and then later

Copied all compiled dynamic library files into source
Build time: 0:07:46.336443
Build completed
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/runner/work/nautilus_trader/nautilus_trader, configfile: pyproject.toml, testpaths: tests
plugins: cov-4.0.0, benchmark-4.0.0, mock-3.10.0, asyncio-0.20.3, xdist-3.1.0
asyncio: mode=strict
collected 3619 items / 2 errors
run-last-failure: 1 known failures not in selected tests

==================================== ERRORS ====================================
______ ERROR collecting tests/unit_tests/persistence/test_catalog_rust.py ______
tests/unit_tests/persistence/test_catalog_rust.py:24: in <module>
    from nautilus.persistence import ParquetReader
E   ModuleNotFoundError: No module named 'nautilus'
______ ERROR collecting tests/unit_tests/persistence/test_catalog_rust.py ______
ImportError while importing test module '/home/runner/work/nautilus_trader/nautilus_trader/tests/unit_tests/persistence/test_catalog_rust.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.9.16/x64/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/unit_tests/persistence/test_catalog_rust.py:24: in <module>
    from nautilus.persistence import ParquetReader
E   ModuleNotFoundError: No module named 'nautilus'
=========================== short test summary info ============================
ERROR tests/unit_tests/persistence/test_catalog_rust.py - ModuleNotFoundError: No module named 'nautilus'
ERROR tests/unit_tests/persistence/test_catalog_rust.py
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 2 errors in 13.71s ==============================

The Cargo.toml:

[package]
name = "nautilus_pyo3"
version.workspace = true
edition.workspace = true
authors.workspace = true
description.workspace = true
documentation.workspace = true

[lib]
name = "nautilus"
crate-type = ["cdylib" ]

[dependencies]
nautilus_persistence = { path = "../persistence" }
pyo3.workspace = true

[features]
extension-module = [
    "pyo3/extension-module",
    "nautilus_persistence/extension-module",
]
default = []

[twitu]

We've gone with the manual builds approach ¹. It allows more flexibility with the module structure and works well with the build script we have in CI.

Footnotes

1. https://pyo3.rs/v0.18.0/building_and_distribution#manual-builds ↩

[messense]

Yep, it's totally fine to build manually, since you only need to support major OSes and CPU architectures, it's manageable.

[cjdsellers]

Thanks so much for your time on this @messense 🙏