Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not Able To Create More Than 2 Clusters #674

Open
infinitydon opened this issue Aug 12, 2024 · 4 comments
Open

Not Able To Create More Than 2 Clusters #674

infinitydon opened this issue Aug 12, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@infinitydon
Copy link

Hi,

I am trying to create 3 clusters using LB but so far only 2 of the clusters are able to come up properly, I have checked the logs, could not find a logic as to why, sample cluster template:

apiVersion: k0smotron.io/v1beta1
kind: Cluster
metadata: 
  name: core-k8s-ctrl
  namespace: k0smotron
spec:
  version: v1.30.3-k0s.0
  persistence:
    type: emptyDir
  service:
    type: LoadBalancer
---
apiVersion: k0smotron.io/v1beta1
kind: Cluster
metadata:
  name: edge-k8s-ctrl
  namespace: k0smotron
spec:
  version: v1.30.3-k0s.0
  persistence:
    type: emptyDir
  service:
    type: LoadBalancer
---
apiVersion: k0smotron.io/v1beta1
kind: Cluster
metadata: 
  name: regional-k8s-ctrl
  namespace: k0smotron
spec:
  version: v1.30.3-k0s.0
  persistence:
    type: emptyDir
  service:
    type: LoadBalancer

 kubectl -n k0smotron get po
NAME                                            READY   STATUS    RESTARTS      AGE
k0smotron-controller-manager-6686bcc6f9-gcdcb   2/2     Running   0             73m
kmc-core-k8s-ctrl-0                             1/1     Running   0             6m24s
kmc-core-k8s-ctrl-etcd-0                        1/1     Running   0             6m24s
kmc-edge-k8s-ctrl-0                             1/1     Running   0             6m27s
kmc-edge-k8s-ctrl-etcd-0                        1/1     Running   0             6m27s
kmc-regional-k8s-ctrl-0                         0/1     Running   5 (90s ago)   6m22s
kmc-regional-k8s-ctrl-etcd-0                    1/1     Running   0             6m22s


kubectl -n k0smotron get svc
NAME                                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                           AGE
k0smotron-controller-manager-metrics-service   ClusterIP      10.100.90.16     <none>        8443/TCP                          38h
kmc-core-k8s-ctrl-etcd                         ClusterIP      None             <none>        2379/TCP,2380/TCP                 6m38s
kmc-core-k8s-ctrl-lb                           LoadBalancer   10.111.252.252   10.0.1.240    30443:31152/TCP,30132:31385/TCP   6m49s
kmc-edge-k8s-ctrl-etcd                         ClusterIP      None             <none>        2379/TCP,2380/TCP                 6m41s
kmc-edge-k8s-ctrl-lb                           LoadBalancer   10.106.31.26     10.0.1.241    30443:30104/TCP,30132:31115/TCP   6m49s
kmc-regional-k8s-ctrl-etcd                     ClusterIP      None             <none>        2379/TCP,2380/TCP                 6m36s
kmc-regional-k8s-ctrl-lb                       LoadBalancer   10.97.86.228     10.0.1.242    30443:31749/TCP,30132:31040/TCP   6m45s

kubectl -n k0smotron describe cluster
Name:         core-k8s-ctrl
Namespace:    k0smotron
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  internal.kpt.dev/upstream-identifier: k0smotron.io|Cluster|k0smotron|CLUSTER_NAME
API Version:  k0smotron.io/v1beta1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-08-12T15:49:22Z
  Generation:          2
  Resource Version:    470989
  UID:                 f6e9fad6-4953-4e90-adf2-880b8b89227c
Spec:
  Etcd:
    Image:  quay.io/k0sproject/etcd:v3.5.13
    Persistence:
      Size:           1Gi
      Storage Class:
  External Address:   10.0.1.240
  Image:              k0sproject/k0s
  Monitoring:
    Enabled:           false
    Prometheus Image:
    Proxy Image:
  Persistence:
    Type:    emptyDir
  Replicas:  1
  Resources:
  Service:
    API Port:           30443
    Konnectivity Port:  30132
    Type:               LoadBalancer
  Version:              v1.30.3-k0s.0
Status:
  Ready:                  true
  Reconciliation Status:  Reconciliation successful
Events:                   <none>
----
Name:         edge-k8s-ctrl
Namespace:    k0smotron
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  internal.kpt.dev/upstream-identifier: k0smotron.io|Cluster|k0smotron|CLUSTER_NAME
API Version:  k0smotron.io/v1beta1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-08-12T15:49:22Z
  Generation:          2
  Resource Version:    470986
  UID:                 8312c806-84f7-47be-bdb3-1c23e23c78d3
Spec:
  Etcd:
    Image:  quay.io/k0sproject/etcd:v3.5.13
    Persistence:
      Size:           1Gi
      Storage Class:
  External Address:   10.0.1.241
  Image:              k0sproject/k0s
  Monitoring:
    Enabled:           false
    Prometheus Image:
    Proxy Image:
  Persistence:
    Type:    emptyDir
  Replicas:  1
  Resources:
  Service:
    API Port:           30443
    Konnectivity Port:  30132
    Type:               LoadBalancer
  Version:              v1.30.3-k0s.0
Status:
  Ready:                  true
  Reconciliation Status:  Reconciliation successful
Events:                   <none>
----
Name:         regional-k8s-ctrl
Namespace:    k0smotron
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  internal.kpt.dev/upstream-identifier: k0smotron.io|Cluster|k0smotron|CLUSTER_NAME
API Version:  k0smotron.io/v1beta1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-08-12T15:49:22Z
  Generation:          2
  Resource Version:    471060
  UID:                 c87f3ddd-3915-4d48-b9d3-044801753a92
Spec:
  Etcd:
    Image:  quay.io/k0sproject/etcd:v3.5.13
    Persistence:
      Size:           1Gi
      Storage Class:
  External Address:   10.0.1.242
  Image:              k0sproject/k0s
  Monitoring:
    Enabled:           false
    Prometheus Image:
    Proxy Image:
  Persistence:
    Type:    emptyDir
  Replicas:  1
  Resources:
  Service:
    API Port:           30443
    Konnectivity Port:  30132
    Type:               LoadBalancer
  Version:              v1.30.3-k0s.0
Status:
  Ready:                  true
  Reconciliation Status:  Reconciliation successful
Events:                   <none>

Logs of the failing POD kmc-regional-k8s-ctrl-0 is given below

k0smotron-regional-cluster-logs.txt

I used kubectl apply --server-side=true -f https://docs.k0smotron.io/v1.0.3/install.yaml for the k0smotron installation
@infinitydon
Copy link
Author

Separating the Clusters into different namespaces seems to stabilize the PODs..

kubectl get cluster -A
NAMESPACE   NAME                AGE
core        core-k8s-ctrl       57m
edge        edge-k8s-ctrl       57m
regional    regional-k8s-ctrl   57m


kubectl get po -Ao wide | egrep "core|edge|regional"
core             kmc-core-k8s-ctrl-0                             1/1     Running   0             58m     10.244.0.137   nephio-mgmt-k8s   <none>           <none>
core             kmc-core-k8s-ctrl-etcd-0                        1/1     Running   0             58m     10.244.0.143   nephio-mgmt-k8s   <none>           <none>
edge             kmc-edge-k8s-ctrl-0                             1/1     Running   0             58m     10.244.0.139   nephio-mgmt-k8s   <none>           <none>
edge             kmc-edge-k8s-ctrl-etcd-0                        1/1     Running   0             58m     10.244.0.144   nephio-mgmt-k8s   <none>           <none>
regional         kmc-regional-k8s-ctrl-0                         1/1     Running   0             58m     10.244.0.141   nephio-mgmt-k8s   <none>           <none>
regional         kmc-regional-k8s-ctrl-etcd-0                    1/1     Running   0             58m     10.244.0.145   nephio-mgmt-k8s   <none>           <none>

Still not sure about the logic behind this

@makhov makhov added the bug Something isn't working label Aug 13, 2024
@jnummelin
Copy link
Member

hmm, that is weird indeed. I don't immediately get either why separating clusters into other namespaces would make a difference.

Did you see anything weird on k0smotron controller logs?

Based on the logs you shared, it seems the CP pod is actually killed. So I wonder if there's some failing health check or something going on.

@jnummelin
Copy link
Member

The other thing I'm thingking of is that maybe we have some naming mismatch, i.e. k0smotron would use same name for some related object (configmaps,secrets,...) for two clusters. Now that you've got the clusters separated into namespaces, could you check ALL the objects in each namespace to see if there's some naming mismatches? You can dump ALL the objects with something like:

kubectl api-resources --verbs=list --namespaced -o name \    
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

@infinitydon
Copy link
Author

I did not see anything weird in the controller-logs, it too was complaining of not able to connect to the LB address of the 3rd cluster endpoint.

I did the api-respources dump, did not see a conflict there but you can also have a look
regional-ns-objects.txt
core-ns-objects.txt
edge-ns-objects.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@makhov @jnummelin @infinitydon