Retry each crane command on 429 after a sleep #10
Conversation
| @@ -89,7 +76,25 @@ node('multiarch-' + env.BASHBREW_ARCH) { ansiColor('xterm') { | |||
| sh """#!/usr/bin/env bash | |||
| set -Eeuo pipefail | |||
|
|
|||
| ${ shell } | |||
| commands=( ${ shell } ) | |||
There was a problem hiding this comment.
Won't this split the expanded Groovy variable shell on whitespace?
There was a problem hiding this comment.
Yup. That's why it does crane_deploy_commands | @sh to create it. The function will return a stream of crane commands and then each one will be shell quoted to be a single shell string.
crane_deploy_commands
crane copy ...
crane copy ...
-> @sh
'crane copy ...'
'crane copy ...'
There was a problem hiding this comment.
And to be clear, this won't be shell doing expansion/splitting, but rather the Groovy will embed the generated and appropriately quoted shell code directly in the string it passes to the shell for invocation, so the end result of what the shell actually executes will be literally something like:
commands=( 'crane copy ...'
'crane copy ...' )
yosifkit
force-pushed
the
deploy-retries
branch
from
fc5d239 to
715f2ed
Compare
yosifkit
force-pushed
the
deploy-retries
branch
from
715f2ed to
d53f6f2
Compare
| # input: list of build objects i.e., builds.json | ||
| # output: stream of crane copy command strings | ||
| def crane_deploy_commands: | ||
| reduce (.[] | select(.build.resolved and .build.arch == env.BASHBREW_ARCH)) as $i ({}; |
There was a problem hiding this comment.
In the future, it might be worth splitting this select out to be the responsibility of the caller, but I don't think we should do that here 👀
(Especially since the use cases I have in mind aren't well formed yet, so it's not worth doing anything concrete yet)
| tries=3 | ||
| while ! output="$( { $c; } |& tee /dev/stderr )"; then | ||
| # check to see if we hit the docker hub rate limit | ||
| if grep -q 'TOOMANYREQUESTS' <<<"$output"; then |
There was a problem hiding this comment.
Is TOOMANYREQUESTS something crane outputs, or something that just happens to be in the error text from Docker Hub right now (and thus might change in the future)?
I don't think this is worth doing much about (ie, this is fine as-is because the failure mode would be that we just ... don't retry and thus the problem becomes obvious), but something to consider in the future.
move the crane commands generator to a jq file instead of embedded in the Jenkinsfile for easier debugging
yosifkit
force-pushed
the
deploy-retries
branch
from
d53f6f2 to
8a4c026
Compare
|
Fire in the hole! |
|
We're seeing some weirdness in how this actually runs, and it's probably thanks to https://www.docker.com/increase-rate-limits/ (these jobs are actually hitting the "normal" 429 that only resets every 6 hours and for the user on that host only gives 200 / 6 hours and our other usage on that system has been mitigating that in other ways until now 🙃). Gonna look at patching our |
|
With a bit more hacking, this is now doing much better (the bandaid was extremely successful). 👍 |
Also, move the crane commands generator to a
.jqfile instead of embedded in the Jenkinsfile for easier debugging