NumPy v2 support

[agriyakhetarpal]

Description

Hi all, I'm coming via pyodide/pyodide#4925 where we are trying to update the river package that gets built in-tree for the Pyodide distribution via Emscripten. I noticed that river does not support NumPy 2.0 yet – I'm not aware of the history behind the pin, but I've also seen that Cython is also pinned to <3, which means that building against NumPy v2 as a build-time dependency isn't possible until that pin gets lifted (and this has come up elsewhere as well, such as #1624 (comment) and conda-forge/river-feedstock#28). Considering that a lot of packages in the ecosystem now have wheels that support NumPy v2 and are built against it (please see numpy/numpy#26191), I've been working on a branch locally to test out these changes, and I'm able to build without much trouble on a macOS M-series device with Cython 3.0.11 and NumPy 2.1.3 with most tests passing. Please let me know if it's alright to put together a pull request, and I'll do so! Thanks!

Additionally, it would be great to get a release with NumPy 2.0 compatible wheels for the convenience of downstream packagers like us, so that we can point to the release's source distribution to build from, apply patches, etc.

Additional context for readers

1. The migration guides sourced from the NumPy and Cython documentation websites:
   • NumPy 2.0 migration guide, and
   • Migrating from Cython 0.29 to 3.0.
2. Links to discussions from the PR linked above: Update to NumPy v2 and bump versions for several dependent packages pyodide/pyodide#4925 (comment) and Update to NumPy v2 and bump versions for several dependent packages pyodide/pyodide#4925 (comment)
3. Wheels built against NumPy v2 will remain compatible with both NumPy v1.X and NumPy>=2 versions, so this dependency is required only at build time and not at runtime. The vice versa (the state of wheels built against NumPy v1.X being compatible with NumPy 2.X) is not true.

[MaxHalford]

Hey @gbolmier, could you give a hand with this? I think that we can dump RollingROCAUC in order to support for Cython 3. And then I don't think upgrading to NumPy v2 will be a big deal.

[gbolmier]

Hey @MaxHalford, I'm on a vacation trip until Tuesday... can't guarantee I'll have the time to give a hand on this one

[MaxHalford]

Or maybe @e10e3 if you have some time on your hands?

[e10e3]

I quickly tried upgrading NumPy, I saw no major hurdle, confirming @agriyakhetarpal's experience. Along with NumPy, Scikit-learn needs to be updated (the first compatible version is 1.4.2).
Even building with Cython 3, I saw no problem regarding RollingROCAUC (I compiled on macOS).

The only thing that needs to be adapted are the tests, NumPy's formatting has changed and causes 21 tests to fail (the underlying values are the same).
The failures look like this:

Expected:
    0.994168...
Got:
    np.float64(0.9941683623419704)

However, adapting the tests mean this is not just a build-time change.

[agriyakhetarpal]

If I can receive a green signal, I can open a PR and keep access to my branch enabled for maintainers to push changes to (in the case where I'm unable to finish it) or pick up my changes from. I don't see anything troubling with RollingROCAUC so far that I feel can't be fixed with some revisions to the packaging infrastructure, since it can't find the headers but should be able to compile successfully if it does.

[gbolmier]

I think that would be helpful @agriyakhetarpal, don't hesitate to do so :)

[agriyakhetarpal]

Awesome! Here we go: #1632