Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): handle tee trying to write to a closed pipe #7580

Merged
merged 1 commit into from
Sep 20, 2023

Conversation

gustavovalverde
Copy link
Member

Motivation

This issue happens even if there's not an actual failure in integration tests.

tee: 'standard output': Broken pipe
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 44 filtered out; finished in 2.11s
sudo docker exit status: 101
Error: Process completed with exit code 101.

Fixes #7564

Complex Code or Requirements

trap '' PIPE is a not a very intuitive command.

Solution

Use trap '' PIPE as a command in the pipeline might exit early and we want the script to continue executing, as using grep --max-count=1 might exit after finding the first match, causing a broken pipe error with tee. This is being combined with set -o pipefail to have strict error handling, as we want the script to fail if any command in the pipeline fails.

Review

We should confirm this part of CI is being tested and passes as expected.

Reviewer Checklist

  • Will the PR name make sense to users?
    • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
  • Are the PR labels correct?
  • Does the code do what the ticket and PR says?
    • Does it change concurrent code, unsafe code, or consensus rules?
  • How do you know it works? Does it have tests?

Follow Up Work

N/A

Use `trap '' PIPE` as a command in the pipeline might exit early and we want the script to continue executing, as using `grep --max-count=1` might exit after finding the first match, causing a broken pipe error with `tee`.

This is being combined with  `set -o pipefail` to have strict error handling and we want the script to fail if any command in the pipeline fails.
@gustavovalverde gustavovalverde added C-bug Category: This is a bug A-devops Area: Pipelines, CI/CD and Dockerfiles P-High 🔥 I-integration-fail Continuous integration fails, including build and test failures labels Sep 18, 2023
@gustavovalverde gustavovalverde self-assigned this Sep 18, 2023
@gustavovalverde gustavovalverde requested a review from a team as a code owner September 18, 2023 21:37
@gustavovalverde gustavovalverde requested review from teor2345 and removed request for a team September 18, 2023 21:37
@github-actions github-actions bot added the C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG label Sep 18, 2023
@teor2345
Copy link
Contributor

This is being combined with set -o pipefail to have strict error handling, as we want the script to fail if any command in the pipeline fails.

This is just for interest, I want to avoid any other changes in a bug fix PR:
http://redsymbol.net/articles/unofficial-bash-strict-mode/

Other possible settings are:

  • set -u: fail if variables are unset
  • set -x: show all the code that is executed (GitHub already does this, but only for top-level commands), this will also interfere with the exit status check

@teor2345 teor2345 removed their request for review September 19, 2023 02:11
@teor2345
Copy link
Contributor

We just agreed that I wouldn't do DevOps reviews any more 😅

@mpguerra mpguerra requested a review from arya2 September 20, 2023 08:20
Copy link
Contributor

@arya2 arya2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

mergify bot added a commit that referenced this pull request Sep 20, 2023
@mergify mergify bot merged commit 2b0b2a4 into main Sep 20, 2023
114 checks passed
@mergify mergify bot deleted the fix-broken-pipeline branch September 20, 2023 18:58
@upbqdn upbqdn mentioned this pull request Sep 22, 2023
38 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-devops Area: Pipelines, CI/CD and Dockerfiles C-bug Category: This is a bug C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG I-integration-fail Continuous integration fails, including build and test failures
Projects
None yet
Development

Successfully merging this pull request may close these issues.

devops: CI failure in tee when Rust tests pass
3 participants