Improve CI run time #8348
Improve CI run time #8348
Conversation
johnny-m-young
force-pushed
the
improve-ci-run-time
branch
from
619d6d6 to
6e25cef
Compare
|
I've got some food for thought on this: With the current implementation, MiniTest tests are manually assigned to nodes. This potentially puts us at risk of not running the full suite: if we've missed some tests, or if future tests get added to locations which lie outside of the assignment. This could be solved by randomly assigning tests across nodes, potentially resulting in a small increase in run time. The random assignment of tests would make it easier to scale up - as a node can be added without worrying about re-allocating tests across nodes - but increases complexity. Here's a draft PR to show what random test assignments would look like with 4 nodes. |
johnny-m-young
force-pushed
the
improve-ci-run-time
branch
from
13311dc to
6e25cef
Compare
johnny-m-young
force-pushed
the
improve-ci-run-time
branch
from
6e25cef to
56eaf94
Compare
|
this is great 🚀 One thought. If we add a new pattern etc. which uses a new directory. We would have to remember to add it to a node or the tests wouldn't run on CI right? Edit: sorry i missed this #8348 (comment) |
johnny-m-young
force-pushed
the
improve-ci-run-time
branch
4 times, most recently
from
891156f to
b9f7494
Compare
This separates out the two runs done by integration tests: one with preview_design_system permissions and one without. Each is run in a separate workflow. I opted to make separate worksflows to make it easy to delete the preview_design_system workflow once fully ported over to the design system.
This configures the MiniTest workflow to run using a matrix strategy. It run separate parts of the test suite on different nodes and hence speeds up total run time of the CI/CD pipeline.
johnny-m-young
force-pushed
the
improve-ci-run-time
branch
from
b9f7494 to
07cd144
Compare
|
Seems like a good approach to me. Makes me feel a lot more comfortable about us not missing tests 👍 |
What
Trello ticket link
This PR improves the time it takes our CI tests to run by:
cucumber:oktask and one process runs thecucumber:preview_design_systemtaskBenchmark: Approximately 17m39s (average of the 10 most recent merge to main CI runs)
With the changes proposed, this PR reduces CI run time to approximately 7 minutes.
Why
This is to help reduce developer toil and time to deploy changes. The current run time is excessive and slows down development.
Additional information
Order of implementation
To avoid having a period where there are no Minitest or integration checks in our CI/CD pipeline, these changes must be implemented in the following order: