Gateway "Waiting for controller"

[tim5901]

I have been following the How-To guide on Routing Traffic to Your Application. I created the gateway and the httproute by copying the code on the page. When I do a describe on the gateway the Message element under Status -> Conditions says "Waiting for controller". I am very new to all of this and have no idea what to do next.

Name:         cafe
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2024-06-11T20:32:22Z
  Generation:          1
  Resource Version:    18201
  UID:                 b3c13dc3-9911-4a03-900f-c3adb3f7e42c
Spec:
  Gateway Class Name:  nginx
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Conditions:
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Accepted
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Programmed
Events:                    <none>

Name:         coffee
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2024-06-11T20:32:25Z
  Generation:          1
  Resource Version:    18204
  UID:                 5ab0833b-54b5-4545-9b99-0dc38a3c9ded
Spec:
  Hostnames:
    cafe.example.com
  Parent Refs:
    Group:  gateway.networking.k8s.io
    Kind:   Gateway
    Name:   cafe
  Rules:
    Backend Refs:
      Group:   
      Kind:    Service
      Name:    coffee
      Port:    80
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
Events:         <none>

NAME                                   READY   STATUS    RESTARTS       AGE
coffee-6b8b6d6486-2qtgk                1/1     Running   0              36m
coffee-6b8b6d6486-l67mn                1/1     Running   0              36m
kubernetes-bootcamp-f95c5b745-9zszt    1/1     Running   0              58m
kubernetes-bootcamp-f95c5b745-s4m4c    1/1     Running   0              61m
nginx-app-5777b5f95-skfrh              1/1     Running   3 (135m ago)   13d
notes-app-deployment-d4fcc5ccd-x5hvj   1/1     Running   0              84m
notes-app-deployment-d4fcc5ccd-xnslj   1/1     Running   0              84m

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
coffee                 ClusterIP      10.101.64.38    <none>        80/TCP           36m
kubernetes             ClusterIP      10.96.0.1       <none>        443/TCP          13d
kubernetes-bootcamp    NodePort       10.111.76.4     <none>        8080:30481/TCP   60m
nginx-app              NodePort       10.108.249.3    <none>        80:31365/TCP     13d
notes-app-deployment   LoadBalancer   10.104.172.90   <pending>     80:30471/TCP     85m

[pleshakov]

Hi @tim5901

Have you deployed NGINX Gateway Fabric (NGF) in the cluster? https://docs.nginx.com/nginx-gateway-fabric/installation/installing-ngf/

Could you describe the pod of NGF? kubectl -n nginx-gateway describe pod <pod_name>

Could you also share logs from nginx-gateway container https://docs.nginx.com/nginx-gateway-fabric/how-to/monitoring/troubleshooting/#logs ?

[tim5901]

Yes, I did install ngf using the Helm install.

Name:             ngf-nginx-gateway-fabric-65c7cf876c-gqkxk
Namespace:        nginx-gateway
Priority:         0
Service Account:  ngf-nginx-gateway-fabric
Node:             k8sworker1/192.168.100.101
Start Time:       Tue, 11 Jun 2024 14:48:14 -0400
Labels:           app.kubernetes.io/instance=ngf
                  app.kubernetes.io/name=nginx-gateway-fabric
                  pod-template-hash=65c7cf876c
Annotations:      cni.projectcalico.org/containerID: 16e55294db67acbdfffd64092984d5d6b02020342dddeaad851a8bfcfdfd85e3
                  cni.projectcalico.org/podIP: 172.16.230.197/32
                  cni.projectcalico.org/podIPs: 172.16.230.197/32
                  prometheus.io/port: 9113
                  prometheus.io/scrape: true
Status:           Running
IP:               172.16.230.197
IPs:
  IP:           172.16.230.197
Controlled By:  ReplicaSet/ngf-nginx-gateway-fabric-65c7cf876c
Containers:
  nginx-gateway:
    Container ID:  containerd://fb4a96cb71142326a56b87140d896eb98c51d470d678ed26773406ce70e27e46
    Image:         ghcr.io/nginxinc/nginx-gateway-fabric:1.3.0
    Image ID:      ghcr.io/nginxinc/nginx-gateway-fabric@sha256:d3702a72b110338d986b87c7d7b6a3ea7e0e8de48cd717f00fa7e070eccd4436
    Ports:         9113/TCP, 8081/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      static-mode
      --gateway-ctlr-name=gateway.nginx.org/nginx-gateway-controller
      --gatewayclass=nginx
      --config=ngf-config
      --service=ngf-nginx-gateway-fabric
      --metrics-port=9113
      --health-port=8081
      --leader-election-lock-name=ngf-nginx-gateway-fabric-leader-election
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 12 Jun 2024 08:11:23 -0400
      Finished:     Wed, 12 Jun 2024 08:11:23 -0400
    Ready:          False
    Restart Count:  209
    Readiness:      http-get http://:health/readyz delay=3s timeout=1s period=1s #success=1 #failure=3
    Environment:
      POD_IP:          (v1:status.podIP)
      POD_NAMESPACE:  nginx-gateway (v1:metadata.namespace)
      POD_NAME:       ngf-nginx-gateway-fabric-65c7cf876c-gqkxk (v1:metadata.name)
    Mounts:
      /etc/nginx/conf.d from nginx-conf (rw)
      /etc/nginx/includes from nginx-includes (rw)
      /etc/nginx/module-includes from module-includes (rw)
      /etc/nginx/secrets from nginx-secrets (rw)
      /var/run/nginx from nginx-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jnqw (ro)
  nginx:
    Container ID:   containerd://387f59b8a935e2fefd232bbae8e946ba8612f6483e5347a040b9b4294dbb5a42
    Image:          ghcr.io/nginxinc/nginx-gateway-fabric/nginx:1.3.0
    Image ID:       ghcr.io/nginxinc/nginx-gateway-fabric/nginx@sha256:23339146dbdf1f5e40457dc3b83022b8bf89ea77a805b0b4cf3a2dbf4e24733d
    Ports:          80/TCP, 443/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 11 Jun 2024 14:48:25 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/nginx/conf.d from nginx-conf (rw)
      /etc/nginx/includes from nginx-includes (rw)
      /etc/nginx/module-includes from module-includes (rw)
      /etc/nginx/secrets from nginx-secrets (rw)
      /var/cache/nginx from nginx-cache (rw)
      /var/lib/nginx from nginx-lib (rw)
      /var/run/nginx from nginx-run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6jnqw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  module-includes:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-secrets:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-run:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-cache:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-lib:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  nginx-includes:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-6jnqw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  2m15s (x5009 over 17h)  kubelet  Back-off restarting failed container nginx-gateway in pod ngf-nginx-gateway-fabric-65c7cf876c-gqkxk_nginx-gateway(3f6e1702-677b-41d3-87d6-da6938899be1)

ngf-gateway-container.log

[pleshakov]

failed to start control loop: failed to prepare the first batch: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v1"

based on this error, you might be running version of Gateway API < 1.1.0.

To know for sure, you can look at the gateway.networking.k8s.io/bundle-version annotation of Gateway API CRDs. For example:

kubectl describe crd gatewayclasses.gateway.networking.k8s.io | grep bundle-version
              gateway.networking.k8s.io/bundle-version: v1.1.0

Based on the logs, you running 1.3.0 version of NGF:

  1 {"level":"info","ts":"2024-06-12T12:16:24Z","msg":"Starting NGINX Gateway Fabric in static mode","version":"1.3.0","commit":"3a372747333fb1db372af7cf0b18ed7eef7c91f7","date":"2024-06-11T16:58:02Z"}

Latest release of NGF (1.3.0) requires Gateway API 1.1.0. Specifically, it requires GRPCRoute gateway.networking.k8s.io/v1, which is not present in Gateway 1.0.0 standard channel CRDs.

[pleshakov]

Running
kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v1.3.0" | kubectl apply -f -
will install Gateway API 1.1.0

[tim5901]

Awesome!!! That got the getway in to stable state.

I am still confused about the GW_IP and GW_PORT. Where do I get those?

[tim5901]

I am sorry, again. I am confused. Is this correct?

NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ngf-nginx-gateway-fabric   LoadBalancer   10.104.190.173   <pending>     80:30518/TCP,443:30457/TCP   23h

Should the EXTERNAL_IP status be ?
Or does that get fixed when I run a port-forward to associate it with my external IP address?
kubectl port-forward svc/frontend 8080:80

[jasonwilliams14]

@tim5901 I may have missed it, but where are you deploying your clusters?
If you are using a cloud provider (Azure, AWS, GKE), when you deploy NGF and the type.loadbalancer service, the provider will automatically provision a simple L4 cloud loadbalancer with an external IP attached it. Having an external IP automatically created for you (cloud providers make it easy) you would then create your DNS records to resolve to the cloud loadbalancer, which would then simply forward the traffic to NGF.

now if you are doing some think local, kind, minikube, k3d for example, they all behave a bit different in how you would assign an external IP to the service.

If you are using say a local onprem type clusters, you will need to use something like metalLB to take care of capabilities of assigning the IP to the loadbalancer service.

kubectl port-forward is a quick and easy way to test access to applications within your cluster, so you can verify the application is working correctly for example.

Does that help?

[tim5901]

So there is another piece I need. I installed Kubenetes on VirtualBox VMs using kubeadm. I have a Master and two Worker VMs. Is there a way to load balance with that configuration?

[jasonwilliams14]

@tim5901 you will need to setup a static IP on your type.loadbalancer that would match the network your VMs are running on. You can set the IP for the service that would match the IP of the node in your cluster (eth0 for example).
Another way to do this would be using metalLB (does everything for you).

Either way, you will need to get that static IP set so outside traffic can traverse into your k8 cluster, through NGF

[tim5901]

I setup metallb and this worked. Thanks for all the help. I really appreciate it.