-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support uv compiled requirements files #10040
base: main
Are you sure you want to change the base?
Conversation
python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/file_updater/pip_compile_file_updater.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/file_updater/pip_compile_file_updater.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb
Outdated
Show resolved
Hide resolved
we at @Shiphero are looking forward to using this functionality as soon as it becomes available. Currently we use dependabot only as an alert to recompile manually as we have to manage everything via uv (and it generates slightly different results than pip-compile). |
python/spec/dependabot/python/file_updater/pip_compile_file_updater_spec.rb
Outdated
Show resolved
Hide resolved
python/spec/dependabot/python/file_updater/pip_compile_file_updater_spec.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/file_updater/pip_compile_file_updater.rb
Outdated
Show resolved
Hide resolved
python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb
Outdated
Show resolved
Hide resolved
915a9ec
to
58f992e
Compare
@edgarrmondragon what is missing to merge this? it is a very valuable feature. |
python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb
Outdated
Show resolved
Hide resolved
First time I contribute here so it is very likely we are missing something. Edit as you see fit, I just thought this was a good way to get the ball rolling. |
options.join(" ") | ||
if (requirements_file = compiled_file_for_filename(filename)) | ||
if requirements_file.content.include?("uv pip compile") | ||
options += uv_pip_compile_options_from_compiled_file(requirements_file) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @avilaton , looks like you added 2 new functions uv_pip_compile_options_from_compiled_file
, pip_compile_options_from_compiled_file
in pip_compile_version_resolver.rb
, i think they are being referenced from pip_compile_file_updater.rb
can you please fix this (IDE is showing it as no function exists in Dependabot::Python::UpdateChecker::PipCompileVersionResolver
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be neat to have a way not to repeat this code, it seems to me as if it would be more resilient to have them centralized. What do you suggest we should do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@avilaton , if you think we might benefit from moving them to new location/class and see a additional functionality that could be used across. please do so.
options += uv_pip_compile_options_from_compiled_file(requirements_file) | ||
command = "pyenv exec uv pip compile" | ||
else | ||
options += pip_compile_options_from_compiled_file(requirements_file) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@avilaton , same as above
Hey! I am waiting for a review on the documentation pr I made, and I believe @avilaton has some changes pending on this pull request (as requested by a dev here). I could take a look at what is needed here and propose a patch. Will have a look later today! |
I know there are some internal gates on the docs side, but I can help us work through those. Want to make sure that the code changes are moving forward though :) |
I don't want to delay this a lot, I meant it to be a surgical change on the command and it snowballed into supporting other options which I don't fully understand. I need help here. The current pip-compile way of doing this has options parsed in both the |
@avilaton hi! I think the tests are failing due to your recent changes (I believe they were already failing before tho). Im sorry to hear that this got a bit out for control for you. Ruby is still a bit new to me, but will try to spend the day today figuring out how to run and test dependabot locally and what changes still need to be done. Again, thanks for getting this rolling! |
def uv_pip_compile_options_from_compiled_file(requirements_file) | ||
options = ["--output-file=#{requirements_file.name}"] | ||
|
||
options << "--no-emit-index-url" unless requirements_file.content.include?("index-url http") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pip-tools defaults to --emit-index-url
, but will skip the default index.
uv defaults to --no-emit-index-url
so I think this should be
options << "--no-emit-index-url" unless requirements_file.content.include?("index-url http") | |
options << "--emit-index-url" if requirements_file.content.include?("index-url http") |
if (resolver = RESOLVER_REGEX.match(requirements_file.content)) | ||
options << "--resolver=#{resolver}" | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think there's any reason to pass this to uv, since it's just there for better UX
@avilaton, @beagold, @jonjanego, @sachin-sandhu, @RazerM -- to help out the team here, i applied the comments/fixed bugs in PR #10688. Hope that helps land this faster! Happy weekend :) |
OK, quick update- this isn't going to work as is. There are other bugs in it also, and now taking a deeper look at the way pdm/uv/pip-compile/etc are supported (or not) here, that's showing up a bit more to do. will come back to it shortly. @jonjanego - can you reach out to me please? thanks! |
Hi all; any further progress on this PR? |
It snowballed into something I cannot really keep up with. No further
progress from my end. All I wanted was to swap pip install by uv pip
install. Turned out to be pretty hard in the end apparently, not sure why
we found so much overhead.
UV support should be doable in full support by just copying what you have
for poetry as they are the same from where dependabot can see them IMHO.
El mar, 15 de oct de 2024, 11:45 a. m., Jon Janego ***@***.***>
escribió:
… Hi all; any further progress on this PR?
—
Reply to this email directly, view it on GitHub
<#10040 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQ3J7XVEFQVTXL3CHWR7LLZ3UTBJAVCNFSM6AAAAABJR3YZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUGE2DANJZGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I would like to take over this PR, @avilaton can you tell me which bugs are still in here? |
Hey, wondering if there could be a way to tell dependabot which command to run, maybe in a .dependabot file (or similar) in place of the normal An alternative would be to implement semantics on the uv side, meaning we'd tell More generally, at the dependabot level, it'd be great to allow people to alter the behavior/command(s) execute by dependabot to support their workflow, whether it's passing specific arguments or running multiple commands. |
Probably not the place to ask for this, but dependabot will use the same flags that were used then the file was initially generated, so it has implicit support for this |
I agree but now that we're derailed I'll keep pushing :) So dependabot will read the comment at the top of I remember digging in the docs to try and figure out how to pass specific arguments to pip-compile or specify multiple files and figured it wasn't possible. Good to know, not sure where it might be or should be documented as I think most software engineers wouldn't intuitively think that the bot is parsing the comments in a file to come up with a plan of what commands to execute. Side note: it explains some of the magic behavior I've seen in some repos and found myself wondering what incantations were running behind the scene. |
fwiw, we get a backwards compatible output using --no-annotate --no-header:
|
What are you trying to accomplish?
Trying to draft support for using https://github.com/astral-sh/uv as a replacement for pip-tools in dependabot. The reason for this is that uv is much faster and many projects have already started switching to it. UV is a pip-tools compatible replacement written in rust.
This is a proposal for:
Anything you want to highlight for special attention from reviewers?
Even if
uv
isn't adopted right now, it might be the long term solutions for pip-compile slowness. We use it for generating requirements.txt and each time dependabot does it withpip-compile
we get a slightly different output which we later correct manually.How will you know you've accomplished your goal?
The change I introduced is fairly simple. First I look at the requirements file header to identify if uv was used to generate it. If it was, I change the command used from
pyenv exec pip-compile
topyenv exec uv pip compile
.Checklist