-
-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make benchmarking 25% faster by parallelizing benchmarking venv setup #375
Conversation
This is cool. Thanks for taking this on. Linking to a related bug. There's a few things I'm puzzled by. On our benchmark machines and locally for me, creating all of the virtual environments takes ~3min not 30, which is I why I decided to work on benchmark running time first. I wonder what the difference is -- I don't think you're machine would be that much worse. I seem to be getting pip 24.2 -- maybe yours is different? If we can get to the bottom of that, maybe we can get the 10x improvement without introducing race conditions (see below). My timings are with a hot package cache so it's not doing a lot of downloading, but if downloading were the bottleneck, I wouldn't expect this patch to make things faster (which it does for both of us, though by a smaller margin for me). Are we sure that |
I don't think it's installing into the same venv though. I was under the assumption that it's installing into separate venvs.
Oh no. Can you please give the version of Python you're using and your OS? For context, a lot of things don't fit into the common venv for me. Edit: I'm also using pip 22.3 |
Ah found it. It's the pip version! 22 is significantly slower than >= 23. New pip brings down benchmark setup to just 1min30s on my computer. Thanks new pip and pypa team :)! |
For each benchmark, it tries to install in the "common" venv, and if that fails (usually due to a version conflict) it creates a new benchmark-specific venv. When I have it build for all of pyperformance + python_macrobenchmarks, I get the following under the
How many do you get?
I'm using Python 3.11 as the "driver" of pyperformance, but benchmarking with CPython main. I see the same timings across all of the OS's -- Windows about 6 minutes rather than 3, but that's not surprising on something I/O bound. These are the OS details:
|
Oh, that's cool. Glad that helps. I think it's safest to just stick with that for now. Your fix here may be useful in the future if we need it and/or pip guarantees thread-safety. uv apparently already does. |
Ok. Closing this and will re-open if we switch to uv. But uv is probably fast enough anyways that we wouldn't need this. |
Fixes #374.
On my computer it takes roughly 90 minutes for a single pyperformance run.
Setting up (compiling and venv installation) takes 45 minutes. We can cut that down.
Before (no compilation, just installation):
After:
Roughly 5x faster setup (without compilation). Saving roughly 25 minutes, and thus 25% (roughly) of benchmarking time.
Edit: this only applies to older pip (22 and below)