devops: CI failure in tee when Rust tests pass

[oxarbitrage]

Describe the issue or request

It seems this is a new issue in the CI. I am not sure how often it is happening, i only saw it once by now but it might worth to have a tciket in case it gets repeated.

A pull request failed with

tee: 'standard output': Broken pipe
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 44 filtered out; finished in 2.11s
sudo docker exit status: 101
Error: Process completed with exit code 101.

at https://github.com/ZcashFoundation/zebra/actions/runs/6165092427/job/16733752210?pr=7515

Apparently, "The Rust test passes, but Docker returns an error status. This is probably an issue with the self-hosted runner changes." (that was mentioned by @teor2345 in slack).

Expected Behavior

No failure or more details on what is the failure inside docker.

Current Behavior

Failure with sudo docker exit status: 101

Possible Solution

We need to figure out if this was a temporal issue, we might have to do nothing to resolve the ticket other than just a short comment of what went wrong, provide what to do if it happens again.

Additional Information/Context

It looks like the failure is unrelated to what the pull request was doing.

Is this happening on PRs?

yes

Is this happening on the main branch?

not seen yet

[mpguerra]

@gustavovalverde any ideas?

[teor2345]

This is happening multiple times a week.

[gustavovalverde]

I'll have a look at this today.

[gustavovalverde]

A bit of context on this error:

The error tee: 'standard output': Broken pipe typically occurs when one process in a pipeline terminates before the other process has finished reading its input. In this case, the error is likely caused by the grep command finding a match and exiting (due to --max-count=1), while tee is still trying to write to the pipe.

As re-running the same workflow just works as expected, this might be the reason.

[teor2345]

It's possible that this error happened in some PRs because we were running multiple test commands, but that should be fixed now.

Does CI always fail when there is a tee error, or can that error happen without CI failing?
If CI always fails, removing the set -e might help here.

[gustavovalverde]

that error happen without CI failing?

Yes, it happens if CI is not failing.

[teor2345]

This failed in PR #7663 even though it included the fix in PR #7580 https://github.com/ZcashFoundation/zebra/actions/runs/6405648679/job/17392070905#step:9:5457

I suggest we do what we did previously to fix this, and remove the pipefail.

[teor2345]

@mpguerra this bug might need to be re-scheduled, because it's causing occasional CI failures in unrelated PRs.

[mpguerra]

@mpguerra this bug might need to be re-scheduled, because it's causing occasional CI failures in unrelated PRs.

I've scheduled this for next sprint for now since it's not happening all the time

[teor2345]

This is still happening, but I've only seen it in the CD tests, see the tagged PRs for details.

[gustavovalverde]

I'm almost certain this was improved (or maybe fixed) by fbd6b0f#diff-4f5cabe26761257a4d685a6edc7a43e0fe0f78762f50eeb48530f2bd3b3ee7caR186

Which might be also why this week the full sync did not crash wit this error

[teor2345]

I haven't seen this recently, are you happy to close it?