Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

401 error while pushing since 1.2.11 #214

Open
nudgegoonies opened this issue Jan 9, 2020 · 7 comments
Open

401 error while pushing since 1.2.11 #214

nudgegoonies opened this issue Jan 9, 2020 · 7 comments
Assignees
Milestone

Comments

@nudgegoonies
Copy link

Since our update to lstags 1.2.11 we get 401 errors on pushing when images have changed.

I turned on tracing and verbosity with 1.2.10 and 1.2.11 and copied the relevant sections for the image/tag that fails with 1.2.11.
This is the working 1.2.10:

b6874ea|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
b6874ea|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:08 GMT]
b6874ea|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
b6874ea|@HEADER: X-Content-Type-Options                   = [nosniff]
b6874ea|@HEADER: Content-Length                           = [2278]
b6874ea|@HEADER: Connection                               = [keep-alive]
b6874ea|@HEADER: Content-Type                             = [application/json; charset=utf-8]
b6874ea|--- BODY BEGIN ---
b6874ea|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
b6874ea|
b6874ea|--- BODY END ---
[...]
12c6fdb|@URL: https://$ARTIFACTORY/v2/kibana/kibana/tags/list
12c6fdb|@HEADER: Connection                               = [keep-alive]
12c6fdb|@HEADER: Server                                   = [Artifactory/6.16.0]
12c6fdb|@HEADER: X-Artifactory-Id                         = [$ID]
12c6fdb|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
12c6fdb|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:15 GMT]
12c6fdb|@HEADER: Content-Type                             = [application/json]

This is the failing 1.2.11:

d55957a|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
d55957a|@HEADER: Content-Type                             = [application/json; charset=utf-8]
d55957a|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:03:50 GMT]
d55957a|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
d55957a|@HEADER: X-Content-Type-Options                   = [nosniff]
d55957a|@HEADER: Content-Length                           = [2278]
d55957a|@HEADER: Connection                               = [keep-alive]
d55957a|--- BODY BEGIN ---
d55957a|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
d55957a|
d55957a|--- BODY END ---

Bad response status: 401 Unauthorized >> https://$ARTIFACTORY/v2/kibana/kibana/tags/list

Because this error comes from authentication there is no useful log from Artifactory. The nginx log shows these:

[09/Jan/2020:08:03:15 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 200 119 "-" "Go-http-client/1.1"
[09/Jan/2020:08:09:56 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 401 101 "-" "Go-http-client/1.1"

I suspect this has to do with the new token handling that was merged into 1.2.11.

@ivanilves
Copy link
Owner

Thanks for this info! I demoted 1.2.11 to "pre-release", so latest would be 1.2.10 again.

Hope to have some time in nearest future to investigate this.

@nudgegoonies
Copy link
Author

Thank you. We reverted to 1.2.10.

@ivanilves
Copy link
Owner

Please forgive me this long delay. I had a lot of involvement with sick kids and so on. 😓

I've released https://github.com/ivanilves/lstags/releases/tag/v1.2.14

So you can force BASIC auth by adding a parameter:

-B external.docker.mamdev.server.lan username:password

Still I fail to understand what was really broken since 1.2.11.

But here we just try to see if an issue could at least worked around 🤷‍♂️

@nudgegoonies
Copy link
Author

No problem. I was ill too. Since yesterday i started homeoffice. Yes, with the parameter it works.

@nudgegoonies
Copy link
Author

My team-lead and i discussed today about the authentication problem since 1.2.11.

It seems that lstags constructs a jwt token altough Artifactory has responded with a plain token.

Please have a look at my team-leads comments in this containerd issue:
containerd/containerd#3556 (comment)
containerd/containerd#3556 (comment)

See also these docker links:
https://docs.docker.com/registry/spec/auth/token/

jwt is an optional token method for docker registries:
https://docs.docker.com/registry/spec/auth/jwt/

@ivanilves
Copy link
Owner

Thank you very much for this input! lstags has always respected the remote registry authentication preferences and the only thing I could potentially blame here is my connection cache addition somehow messed up its decision making. Need to play more with Artifactory on my local PC to understand it... 😄

@ivanilves ivanilves self-assigned this Mar 22, 2020
@ivanilves ivanilves added this to the v1.4 milestone Mar 22, 2020
@cablespaghetti
Copy link
Collaborator

This also seems to be an issue on ECR. Downgrading to 1.2.10 fixes it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants