Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeconfig: Append /clusterURL to the AuthInfo added by crc #4020

Merged
merged 1 commit into from
Feb 14, 2024

Conversation

anjannath
Copy link
Member

@anjannath anjannath commented Feb 6, 2024

this appends /api.crc.testing:6443 i.e the API server URL to the name key of the AuthInfos list

this prevents the following error from oc when kubeconfig has two entries with same name in the AuthInfos list:

error: error loading config file ".kube/config": error converting *[]NamedAuthInfo into *map[string]*api.AuthInfo: duplicate name "user" in list:

Fixes #3940

Copy link

openshift-ci bot commented Feb 6, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from anjannath. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gbraad gbraad changed the title kubeconfig: Append /cluster to the AuthInfo added by crc kubeconfig: Append /clusterURL to the AuthInfo added by crc Feb 6, 2024
@praveenkumar
Copy link
Member

@anjannath #3940 (comment) check the commit from this comment because we don't want to modify the complete kubeconfig file instead just want to make sure our entry is according to oc login.

@anjannath
Copy link
Member Author

anjannath commented Feb 6, 2024

@anjannath #3940 (comment) check the commit from this comment because we don't want to modify the complete kubeconfig file instead just want to make sure our entry is according to oc login.

checked the commit, and i think the changes here are similar to what you posted there, since we have two different functions for updating the kubeconfig one for microshift (mergeKubeconfig) and one for ocp (writeKubeconfig) the changes you posted only handles the openshift preset

the other additional function/changes in this PR to the mergeKubeConfig handles the microshift preset

i did test with an existing kubeconfig file with entries from minikube and it was only touching the entries added by crc

Copy link
Contributor

@gbraad gbraad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplication prevents easy maintenance

@praveenkumar
Copy link
Member

@anjannath #3940 (comment) check the commit from this comment because we don't want to modify the complete kubeconfig file instead just want to make sure our entry is according to oc login.

checked the commit, and i think the changes here are similar to what you posted there, since we have two different functions for updating the kubeconfig one for microshift (mergeKubeconfig) and one for ocp (writeKubeconfig) the changes you posted only handles the openshift preset

On microshift side we don't create kubeconfig (as part of crc codebase), we just merge what is created by microshift so should we fix it because we are merging or microshift should fix that kubeconfig?

the other additional function/changes in this PR to the mergeKubeConfig handles the microshift preset

i did test with an existing kubeconfig file with entries from minikube and it was only touching the entries added by crc

@cfergeau
Copy link
Contributor

cfergeau commented Feb 6, 2024

On microshift side we don't create kubeconfig (as part of crc codebase), we just merge what is created by microshift so should we fix it because we are merging or microshift should fix that kubeconfig?

Since microshift document using these files outside of the machine running the cluster ( https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.14/html/configuring/microshift-kubeconfig ), imo it would be nice if the users in there were suffixed, especially as they create multiple kubeconfig depending on the hostname the cluster is going to be accessed from.

@gbraad
Copy link
Contributor

gbraad commented Feb 6, 2024

it would be nice if the users in there were suffixed

does this need an issue?

According to the document they generate these files in:

/var/lib/microshift/resources/kubeadmin/

I think this strategy makes sense too, as it prevents any conflict. the file is specific for the microshift usecase. why not write ours in: ~/.crc/kubeconfig ?

@gbraad
Copy link
Contributor

gbraad commented Feb 7, 2024

should be <username>/api-crc-testing:6443 instead <username>/api.crc.testing:6443

Is it necessary to have the port :6443 included in the suffix?

@anjannath
Copy link
Member Author

On microshift side we don't create kubeconfig (as part of crc codebase), we just merge what is created by microshift so should we fix it because we are merging or microshift should fix that kubeconfig?

Since microshift document using these files outside of the machine running the cluster ( https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.14/html/configuring/microshift-kubeconfig ), imo it would be nice if the users in there were suffixed, especially as they create multiple kubeconfig depending on the hostname the cluster is going to be accessed from.

in the case of standalone kubeconfig file that is only containing one pair of cluster and user, there's no provision for having conflicts, the conflict may occur when we merge microshift generated kubeconfig to the user's kubeconfig, so i think its more important that we namespace it (add the cluster suffix) when the merging happens.

and we can also open an issue in microshift repo to know if they think this is something they would consider doing.

@anjannath
Copy link
Member Author

I think this strategy makes sense too, as it prevents any conflict. the file is specific for the microshift usecase. why not write ours in: ~/.crc/kubeconfig ?

we need to merge the kubeconfig onto the default kubeconfig location, i.e ~/.kube/config otherwise oc, kubectl and also PD extension won't see the contexts for microshift, ocp from crc

@gbraad
Copy link
Contributor

gbraad commented Feb 7, 2024

otherwise oc, kubectl and also PD extension won't see the contexts for microshift, ocp from crc

The kubeconfig can be set by environment variable... something I also expect: oc-env to do, right?

I would also expect PD to pick this up. Something the extension should do?

Perhaps even they should allow additional files to be loaded in PD. Especially as the microshift documentation specifies they will use /var/lib/microshift/resources/kubeadmin/ for this.

@anjannath
Copy link
Member Author

otherwise oc, kubectl and also PD extension won't see the contexts for microshift, ocp from crc

The kubeconfig can be set by environment variable... something I also expect: oc-env to do, right?

currently crc oc-env only updates PATH to include oc binaries that we ship, its not setting KUBECONFIG

I would also expect PD to pick this up, and perhaps even allow additional files to be loaded. Especially as the microshift documentation specifies they will use /var/lib/microshift/resources/kubeadmin/ for this.

this only says that when the microshift service is started it generates the files in this location, they don't mention how to use it with oc or kubectl the user is free to either set KUBECONFIG or use the --kubconfig flag or merge with the existing ~/.kube/config

@gbraad
Copy link
Contributor

gbraad commented Feb 7, 2024

they don't mention how to use it

regardless, seems to be 'convention' as it is documented. Although, something for PD+extension to handle.

its not setting KUBECONFIG

Shouldn't we do this? /cc: @praveenkumar @cfergeau and therefore not 'merge' our config in by default (or as default now, but allow this to be configured).

@cfergeau
Copy link
Contributor

cfergeau commented Feb 7, 2024

its not setting KUBECONFIG
Shouldn't we do this? /cc: @praveenkumar @cfergeau and therefore not 'merge' our config in by default (or as default now, but allow this to be configured).

If kubectl/oc know how to merge multiple files in ~/.kube, we definitely should make use of this instead of modifying the main file ourselves.
Not sure about forcing KUBECONFIG if this means cutting easy access to other clusters which are already configured in ~/.kube

@gbraad
Copy link
Contributor

gbraad commented Feb 7, 2024

If kubectl/oc know how to merge multiple files in ~/.kube,

is this possible? if not, file an issue for something like a kubeconfig.d approach.

cutting easy access to other clusters

I believe this is what happens... but in that case suggesting merging content to kubeconfig makes sense. Guess this constitutes a separate discussion.

However, I still did not get an answer to: #4020 (comment) about the port in the suffix. Is this necessary?

@cfergeau
Copy link
Contributor

cfergeau commented Feb 7, 2024

However, I still did not get an answer to: #4020 (comment) about the port in the suffix. Is this necessary?

Nothing is necessary, even appending a random string would work instead of the cluster name. Having the port number makes it consistent with the example in #3940.

@anjannath
Copy link
Member Author

anjannath commented Feb 7, 2024

KUBECONFIG env variable can contain list of config files, so we could set KUBECONFIG=$HOME/.kube/config:~/.crc/kubeconfig and oc kubectl will merge the config files by itself, and will be able to access users, clusters defined in either of the files..

but i think it'd still be good to append /clustername to the users/AuthInfos as according to kubeconfig merge rules:

The first file to set a particular value or map key wins.

https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#merging-kubeconfig-files

@gbraad
Copy link
Contributor

gbraad commented Feb 7, 2024

Nothing is necessary,

Don't want to lock the user in so he can only use port 6443, so it wouldn't work anymore if remapped to 8443, etc. Although I understand the matching happens on the actual key/auth.

but i think it'd still be good to append /clustername to the users/AuthInfos

the identifier is more of a 'nice to have' for the user, but 'needs to be unique' due to how they store it in code when read/parsed. it does not add anything else than user-facing info.

@praveenkumar
Copy link
Member

Nothing is necessary,

Don't want to lock the user in so he can only use port 6443, so it wouldn't work anymore if remapped to 8443, etc. Although I understand the matching happens on the actual key/auth.

We are not hardcoding 6443 but just using the api server url and if url is having different port then we use that even that is just the placeholder.

but i think it'd still be good to append /clustername to the users/AuthInfos

the identifier is more of a 'nice to have' for the user, but 'needs to be unique' due to how they store it in code when read/parsed. it does not add anything else than user-facing info.

Right, Also since we are merging microshift kubeconfig file to global one it also remove the override in case same user exist for different cluster.

Shouldn't we do this? /cc: @praveenkumar @cfergeau and therefore not 'merge' our config in by default (or as default now, but allow this to be configured).

most of the tools do it (like kind/minikube ..etc.) but they don't merge but add the context like we are doing for OCP by generating the required field. Technically on microshift side we are getting the required info from the generated kubeconfig by parsing it and then appending it to global kubeconfig (something same might oc/kubectl can do behind the scene when KUBECONFIG env variable set for multiple kubeconfigs) . With our current approach (merging/appending) user can use oc command on OCP/microshift cluster without everytime setting the KUBECONFIG env which is big plus since most of the time user don't even use oc-env since they already have cli tool installed just want to explore the cluster.

@gbraad
Copy link
Contributor

gbraad commented Feb 8, 2024

and if url is having different port then we use that even that is just the placeholder.

I can read the code, but I am questioning if this is necessary. It might be better to just say:

<username>/crc-api-testing

and therefore substring on the port/semi-colon.

@gbraad
Copy link
Contributor

gbraad commented Feb 8, 2024

Also since we are merging microshift kubeconfig

So adding the preset as part of the identifier makes more sense?

<username>/crc-api-testing-ocp or <username>/crc-api-testing-microshift, because OKD would have the same ports and cause issues?

@anjannath
Copy link
Member Author

Also since we are merging microshift kubeconfig

So adding the preset as part of the identifier makes more sense?

<username>/crc-api-testing-ocp or <username>/crc-api-testing-microshift, because OKD would have the same ports and cause issues?

ahh, yes it'd cause issues since it'd be the same kubeadmin/api-crc-testing:6443 for both okd and ocp

but we allow only one preset to run at a time, so kubeconfig will not have entries from both, as to move to okd preset from ocp we'd have to run crc delete which cleans the kubeconfig of entries that were added by crc

@praveenkumar
Copy link
Member

Also since we are merging microshift kubeconfig

So adding the preset as part of the identifier makes more sense?
<username>/crc-api-testing-ocp or <username>/crc-api-testing-microshift, because OKD would have the same ports and cause issues?

ahh, yes it'd cause issues since it'd be the same kubeadmin/api-crc-testing:6443 for both okd and ocp

but we allow only one preset to run at a time, so kubeconfig will not have entries from both, as to move to okd preset from ocp we'd have to run crc delete which cleans the kubeconfig of entries that were added by crc

there is no kubeadmin user for microshift it should be user because this is what system:admin role associated to so it shouldn't conflict also the whole point to have port number as part of <username> to consist what other entry in kubeconfig.

@gbraad
Copy link
Contributor

gbraad commented Feb 9, 2024

to consist what other entry in kubeconfig.

Do not understand this part. To be consistent with the other entries in kubeconfig ?

this adds /api.crc.testing:6443 i.e the API server URL to the
name key of the AuthInfos list

this prevents the following error from `oc` when kubeconfig has
two entries with same name in the AuthInfos list:

```
error: error loading config file ".kube/config": error converting *[]NamedAuthInfo into *map[string]*api.AuthInfo: duplicate name "user" in list:
```
Copy link

openshift-ci bot commented Feb 12, 2024

@anjannath: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-crc a711b26 link true /test e2e-crc
ci/prow/e2e-microshift-crc a711b26 link true /test e2e-microshift-crc
ci/prow/integration-crc a711b26 link true /test integration-crc

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@praveenkumar
Copy link
Member

to consist what other entry in kubeconfig.

Do not understand this part. To be consistent with the other entries in kubeconfig ?

so whenever we do oc login <any_cluster then it add entry to kubeconfig with <username>/api_url_along_with_port so this is what I mean around consistency.

@praveenkumar praveenkumar merged commit 39a033d into crc-org:main Feb 14, 2024
17 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

kubeconfig: Add /cluster suffix to the user AuthInfos entries
4 participants