Use modern RNG

[flying-sheep]

• Closes leiden alg with igraph flavor causes out of bounds freezing #2969
• Tests included or not required because:

• Release notes not necessary because:

Seems like this code is super performance sensitive: Having a Python implementation of getrandbits in 8572ecb resulted in a slowdown:

Change	Before [0d4554b]	After [1b2d9dd]	Ratio	Benchmark (Parameter)
+	15.2±0.03ms	31.7±0.1ms	2.09	preprocessing.time_highly_variable_genes

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.88%. Comparing base (8d046ff) to head (d9877c9).
Report is 87 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3041      +/-   ##
==========================================
+ Coverage   75.87%   75.88%   +0.01%     
==========================================
  Files         110      110              
  Lines       12536    12542       +6     
==========================================
+ Hits         9512     9518       +6     
  Misses       3024     3024              

Files with missing lines	Coverage Δ
scanpy/_utils/__init__.py	75.41% <100.00%> (+0.31%)	⬆️

[scverse-benchmark]

Benchmark changes

Change	Before [8d046ff]	After [d9877c9]	Ratio	Benchmark (Parameter)
+	15.1±0.05ms	17.2±0.1ms	1.14	tools.time_leiden

Comparison: https://github.com/scverse/scanpy/compare/8d046ff37e024ae88eadfb22ea8fd142a6b95aa1..d9877c996b655a236f14fc242717a637365cd7d8
Last changed: Tue, 4 Jun 2024 12:01:33 +0000

More details: https://github.com/scverse/scanpy/pull/3041/checks?check_run_id=25784166991

[ilan-gold]

@RubenVanEsch could you look at this as well for your issue?

I am not so sure if a 15% slowdown is acceptable given the fact that the answer to solving this could just be "use WSL," which we never got a response on because we go sidetracked. Could @RubenVanEsch or someone else confirm or not that the original problem is reproducible on WSL?

[ilan-gold]

Also @flying-sheep coming back to this - why doesn't this break tests? The underlying number generation mechanism is the same somehow? Or similar enough?

[RubenVanEsch]

@ilan-gold turns out i cant install WSL on my laptop after all, so unfortunately i cant check this.

[flying-sheep]

The other way round: If using WSL would be a viable workaround, we don’t need this.

So if you can’t use WSL, it’s even more important that you help us by checking if this PR fixes things for you.

[patrick-nicodemus]

Hi, I cloned this repo, switched to modern-rng, and installed it with pip. I was able to reproduce the same error.

Exception ignored in: <class 'ValueError'>
Traceback (most recent call last):
    File "numpy\random\_generator.pyx", line 622, in numpy.random._generator.Generator.integers
    File "numpy\random\_bounded_integers.pyx", line 2881, in numpy.random._bounded_integers._rand_int32"
ValueError: high is out of bounds for int32

I am using numpy 1.26, which is the numpy version required by this branch.

[patrick-nicodemus]

The exception is raised from the C module in the igraph library which actually computes the clustering algorithm, GraphBase.community_leiden. So it may be a bug in igraph, or the incorrect arguments are being passed to igraph.

Here is some sample code to reproduce.

import numpy as np
import anndata as ad
import scanpy as sc

rng = np.random.default_rng()
counts = rng.integers(low=-1000,high=100,size=(100,1000))
counts = np.maximum(counts , 0)
adata = ad.AnnData(counts)
sc.tl.pca(adata)
sc.pp.neighbors(adata)
sc.tl.leiden(adata,flavor='igraph',n_iterations=2)

[flying-sheep]

Hi Patrick, we’re quite busy with the scverse conference right now, so don’t be distraught if we can’t investigate this more right away.

I’ll link to your reproducers here from the issue, as a closed PR is a place few people ever look, and we’ll come back to this in a week or so.