CI failing with mounting volume error

[oxarbitrage]

https://github.com/ZcashFoundation/zebra/actions/runs/6382508836/job/17321768540?pr=7653#step:13:188

docker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/fully-synced-rpc-85b855b/_data': failed to mount local volume: mount /dev/sdb:/var/lib/docker/volumes/fully-synced-rpc-85b855b/_data: device or resource busy.
Error: Process completed with exit code 125.

This seems to be happening for every PR we have opened recently. We can post more links to failures and more information in this ticket.

[oxarbitrage]

It is not the first time in the CI history we see this error. One option is to find what we did at that time and try to repeat it or get information from it.

[oxarbitrage]

In a closer inspection, I've observed that the error is not present in every open pull request, but specifically in the one linked here.

The error is visible in the Zebra Tip JSON-RPC job, while none of the other open pull requests encounter this issue in the same job or with the specific error message.

All other pull requests pass this job and start failing in the lightwalletd tip update job with a 'The operation was canceled.' error.

So, we should pay attention only to what is happening in this specific pull request in this ticket:

• It is using this image: zebrad-cache-7633-merge-7b222f7-v25-mainnet-tip-u-174603. You can find this information in the Find fully-synced-rpc cached state disk job.
• The job is failing in Zebra Tip JSON-RPC, but it failed only once in this particular pull request and nowhere else.

I suggest deleting the possibly corrupted image from gcloud and restarting the CI for this pull request. Another image will be selected, if this was a one time issue, then the CI should pass the Zebra Tip JSON-RPC job.

This will help us determine if the problem was related to the image, and it may also lower the priority of the ticket.

@gustavovalverde , please let me know your thoughts on this. Any additional input is welcome.

[teor2345]

Have we tried restarting CI without deleting any images?
It might have been a temporary issue on that Google Cloud machine, and we'll get a new machine when we restart.

[teor2345]

(It is unlikely that a "device or resource busy" error would be caused by a specific image, because they are usually about open files or devices.)

[oxarbitrage]

Makes sense, restarting all jobs at https://github.com/ZcashFoundation/zebra/actions/runs/6382508836?pr=7653 for the PR with the issue.

[teor2345]

a closer inspection, I've observed that the error is not present in every open pull request, but specifically in the one linked here.

The error is visible in the Zebra Tip JSON-RPC job, while none of the other open pull requests encounter this issue in the same job or with the specific error message.

Sorry about that, I thought I had checked multiple PRs, but I might have accidentally checked the same PR multiple times.

[teor2345]

I'm also seeing this bug in PRs:

• ref(docker): combine test and prod entrypoints into one #7660: https://github.com/ZcashFoundation/zebra/actions/runs/6398108030/job/17367882179?pr=7660
• fix(note-commitment-trees): Populate subtree fields in the NoteCommitmentTrees structs #7636: https://github.com/ZcashFoundation/zebra/actions/runs/6397871155/job/17367204613?pr=7636
• build(deps): bump webpki from 0.22.1 to 0.22.2 #7658: https://github.com/ZcashFoundation/zebra/actions/runs/6385810453/job/17331628795?pr=7658

So maybe it is multiple PRs, but only after a recent change?

[gustavovalverde]

Yes...restarting works.

Failed here:

• https://github.com/ZcashFoundation/zebra/actions/runs/6339916337/job/17294920418#step:13:194
• https://github.com/ZcashFoundation/zebra/actions/runs/6398108030/job/17367882179?pr=7660#step:13:189

Worked here:

• https://github.com/ZcashFoundation/zebra/actions/runs/6339916337/job/17330803774#step:13:187
• https://github.com/ZcashFoundation/zebra/actions/runs/6398108030/job/17368726883#step:13:189

[gustavovalverde]

This PR did not work:

• fix(ci): wait for disk to be mounted in VM #7662

[teor2345]

We could re-run the entire docker command a limited number of times until it succeeds?

[teor2345]

This wasn't completely fixed by PR #7686, but it's a lot better now.

Maybe we can drop it down from critical to high priority?

[gustavovalverde]

I’m waiting for the latest commit to run, but I was able to found the issue while deploying the instances manually, as running dmesg was outputting the following message when I tried to mount /dev/sbd in Docker:

/dev/sdb: Can't open blockdev

And this only happened after creating the Docker volume.

I added a new commit (which I’ve tested at least 3 times), and it’s not failing to mount: 398c2f1

But I’ll keep testing to confirm

[teor2345]

This is still failing to mount in:
https://github.com/ZcashFoundation/zebra/actions/runs/6435775989/job/17478042314?pr=7636#step:13:194

[teor2345]

Failed here as well:
https://github.com/ZcashFoundation/zebra/actions/runs/6450609361/job/17510268834?pr=7633#step:12:193

[teor2345]

What if using block storage is part of our issue?
It's only recommended for experts in the docker docs:
https://docs.docker.com/storage/volumes/#block-storage-devices

Is there a way to let docker handle the devices automatically, without us having to initialise them?