Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Windows test execution time #13710

Closed
wants to merge 54 commits into from
Closed

CI: Windows test execution time #13710

wants to merge 54 commits into from

Conversation

comphead
Copy link
Contributor

@comphead comphead commented Dec 10, 2024

Which issue does this PR close?

Related to #13726
Closes #.

Rationale for this change

Test Windows compilation time improvements

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the development-process Related to development process of DataFusion label Dec 10, 2024
@comphead
Copy link
Contributor Author

Executed "map.slt". Took 460.5636141s
Executed "window.slt". Took 217.240136s
Executed "struct.slt". Took 1316.049712s
Executed "array.slt". Took 2407.7054095s

Thats really weird, a simple array SLT takes 40 mins

@comphead comphead changed the title CI: Windows compilation time CI: Windows test execution time Dec 10, 2024
@korowa
Copy link
Contributor

korowa commented Dec 15, 2024

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

@comphead
Copy link
Contributor Author

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

Thanks @korowa for the hint I was playing with lots of combination including different Windows images but nothing seems to work.

@comphead
Copy link
Contributor Author

Some projects give up rust testing on Windows like DataDog/orchestrion#415

@alamb wondering should we pause the Windows flow temporarily? The build time is incredibly slow and we holding GH resources for way longer time

@alamb
Copy link
Contributor

alamb commented Dec 15, 2024

Another thing that would benefit the project as a whole is to reduce build time.

One thing I noticed a while ago is that each binary built (e.g. every example) was around 100MB

I was thinking that we could reduce / conslidate the number of examples / binaries built which would speed up all runners (not just Windows)

For example consolidating https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/regexp.rs and other examples of using functions into dataframe_api.rs might be a good win

@comphead
Copy link
Contributor Author

Closing it as there is no any progress after all the experiments

@comphead comphead closed this Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development-process Related to development process of DataFusion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants