Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): Replace busybox with ubuntu to avoid "device or resource busy" failures #7686

Merged
merged 1 commit into from
Oct 6, 2023

Conversation

teor2345
Copy link
Contributor

@teor2345 teor2345 commented Oct 5, 2023

Motivation

Some CI jobs are regularly failing with "device or resource busy" errors.

Close #7659

Solution

The busybox image is really old, so I replaced it with a newer ubuntu image.

This could have been a kernel/OS incompatibility. Or this PR might fix the issue because it slows down automatic checks on the disks, allowing the mount command to win the race.

Review

This is urgent so I'm going to admin-merge it if it works. It needs to admin-merge so the checkpoint rebuild job can run.

We need to run it at least 3 times to test that it works.

Reviewer Checklist

  • Are the PR labels correct?
  • Does the code do what the ticket and PR says?
    • Does it change concurrent code, unsafe code, or consensus rules?
  • How do you know it works? Does it have tests?

Follow Up Work

Work out if we can use ubuntu-slim instead to save image download & launch time.

@github-actions github-actions bot added the C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG label Oct 5, 2023
@teor2345 teor2345 changed the title Try ubuntu GCP image fix(ci): Replace busybox with ubuntu to avoid "device or resource busy" failures Oct 6, 2023
@teor2345 teor2345 self-assigned this Oct 6, 2023
@teor2345 teor2345 added C-bug Category: This is a bug A-devops Area: Pipelines, CI/CD and Dockerfiles P-Critical 🚑 I-integration-fail Continuous integration fails, including build and test failures labels Oct 6, 2023
@teor2345
Copy link
Contributor Author

teor2345 commented Oct 6, 2023

These runs were successful, except for the known issue with the checkpoint rebuild:

This seems to be better than the current main branch, so maybe it's worth merging this change.

@teor2345 teor2345 marked this pull request as ready for review October 6, 2023 05:09
@teor2345 teor2345 requested a review from a team as a code owner October 6, 2023 05:09
@teor2345 teor2345 requested review from gustavovalverde and removed request for a team October 6, 2023 05:09
@teor2345 teor2345 merged commit f3238fc into main Oct 6, 2023
107 of 113 checks passed
@teor2345 teor2345 deleted the gcp-ubu branch October 6, 2023 05:11
@upbqdn upbqdn mentioned this pull request Oct 13, 2023
38 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-devops Area: Pipelines, CI/CD and Dockerfiles C-bug Category: This is a bug C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG I-integration-fail Continuous integration fails, including build and test failures
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI failing with mounting volume error
1 participant