diff --git a/README.md b/README.md index c149a1413..eaa7f874b 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,13 @@ curl -v -p --proxy-key certs/frontend/private/proxy-client.key --proxy-cert cert ### Running on kubernetes See following [README.md](examples/kubernetes/README.md) +### Running on a local kubernetes cluster with `kind` +See this [README.md](examples/kind/README.md) for an example that creates a local kubernetes cluster using` kind` and +deploys the proxy agent on a worker node and the proxy server on a control plane node. + +See this [README.md](examples/kind-multinode/README.md) for a similar example that creates a `kind` cluster with a +user-configurable number of control plane and worker nodes and optionally sideloads custom proxy agent and server images. + ### Clients `apiserver-network-proxy` components are intended to run as standalone binaries and should not be imported as a library. Clients communicating with the network proxy can import the `konnectivity-client` module. diff --git a/examples/kind-multinode/README.md b/examples/kind-multinode/README.md new file mode 100644 index 000000000..77aa06321 --- /dev/null +++ b/examples/kind-multinode/README.md @@ -0,0 +1,82 @@ +# Set up KIND cluster with multiple KCP and worker nodes running konnectivity + +Change to the `examples/kind-multinode-kcp` folder and run `./quickstart-kind`. This script +performs the following operations: + +1. Render config templates in `templates/` using provided values. +2. Create a new `kind` cluster with the desired number of KCP and worker nodes. +3. Changes `kubectl` context to point to the new `kind` cluster. +4. Deploys `konnectivity` proxy servers and agents to the KCP and worker nodes. + + +## `./quickstart-kind.sh` command-line flags +- `--cluster-name `: Name of the `kind` cluster to be created Default: `knp-test-cluster` +- `--overwrite-cluster`: Overwrite existing `kind` cluster if necessary. Default: do not overwrite. +- `--server-image [:]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-server:master` +- `--agent-image [:]`: Proxy server image to deploy. Default: `gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master` +- `--num-kcp-nodes `: Number of control plane nodes to spin up. Default: 2. +- `--num-worker-nodes `: Number of worker nodes to spin up. Default: 1. +- `--server-count-override `: If this flag is >= 0, override the `--serverCount` flag in the proxy server's configuration to the provided number. Default: set `--serverCount` to equal the number of KCP nodes. +- `--sideload-images`: Use `kind load ...` to sideload custom proxy server and agent images with the names set by `--server-image` and `--agent-image` into the kind cluster. Default: do not sideload. + - Use this if you don't want to publish your custom KNP images to a public registry. + - NOTE: You MUST specify an image tag (i.e. `my-image-name:my-image-tag` and not just `my-image-name`) and the image tag MUST NOT be `:latest` for this to work! See [`kind` docs](https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster) for why this is necessary. + +## Example usage to deploy custom local KNP images +In the repo root, build KNP and its docker images with the following: +```shell +make clean +make certs +make gen +make build +make docker-build +``` + +Verify that the new images are available in the local docker registry with `docker images`. Then, bring up the cluster: + +```shell +cd examples/kind-multinode + +# These are the default values of the registry, image name, and tag used by the Makefile. +# Edit them if necessary. +REGISTRY=gcr.io/$(gcloud config get-value project) +TAG=$(git rev-parse HEAD) +TARGET_ARCH="amd64" +SERVER_IMAGE="$REGISTRY/proxy-server-$TARGET_ARCH:$TAG" +AGENT_IMAGE="$REGISTRY/proxy-agent-$TARGET_ARCH:$TAG" + +# Bring up the cluster! +./quickstart-kind.sh --cluster-name custom-knp-test --server-image "$SERVER_IMAGE" --agent-image "$AGENT_IMAGE" \ + --num-kcp-nodes 3 --num-worker-nodes 2 --sideload-images +``` + +## Making sure the script worked + +Check that the `konnectivity` pods are up and running: +```shell +kubectl --namespace kube-system get pods | grep konnectivity +# Output: +# konnectivity-agent-4db5j 1/1 Running 0 34m +# konnectivity-agent-c7gj5 1/1 Running 0 34m +# konnectivity-agent-h86l9 1/1 Running 0 34m +# konnectivity-server-9bl45 1/1 Running 0 34m +# konnectivity-server-dcfz8 1/1 Running 0 34m +# konnectivity-server-klww5 1/1 Running 0 34m +# konnectivity-server-nrfz8 1/1 Running 0 34m +``` + +Then create a test pod on a worker node and verify you can get logs from it: +```shell +kubectl run test --image httpd:2 +# Output: +# pod/test created +kubectl get pods +# Output: +# NAME READY STATUS RESTARTS AGE +# test 1/1 Running 0 34s +kubectl logs test +# Output: +# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message +# AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.5.3. Set the 'ServerName' directive globally to suppress this message +# [Wed Jun 12 20:42:06.471169 2024] [mpm_event:notice] [pid 1:tid 139903660291968] AH00489: Apache/2.4.59 (Unix) configured -- resuming normal operations +# [Wed Jun 12 20:42:06.471651 2024] [core:notice] [pid 1:tid 139903660291968] AH00094: Command line: 'httpd -D FOREGROUND' +``` diff --git a/examples/kind-multinode/egress_selector_configuration.yaml b/examples/kind-multinode/egress_selector_configuration.yaml new file mode 100644 index 000000000..eda4be511 --- /dev/null +++ b/examples/kind-multinode/egress_selector_configuration.yaml @@ -0,0 +1,15 @@ +apiVersion: apiserver.k8s.io/v1beta1 +kind: EgressSelectorConfiguration +egressSelections: +- name: cluster + connection: + proxyProtocol: GRPC + transport: + uds: + udsName: /etc/kubernetes/konnectivity-server/konnectivity-server.socket +- name: controlplane + connection: + proxyProtocol: Direct +- name: etcd + connection: + proxyProtocol: Direct diff --git a/examples/kind-multinode/quickstart-kind.sh b/examples/kind-multinode/quickstart-kind.sh new file mode 100755 index 000000000..fb464b5e8 --- /dev/null +++ b/examples/kind-multinode/quickstart-kind.sh @@ -0,0 +1,121 @@ +#!/bin/bash + +set -e + +# DEFAULT ARGS +CLUSTER_NAME="knp-test-cluster" +AGENT_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master" +SERVER_IMAGE="gcr.io/k8s-staging-kas-network-proxy/proxy-server:master" +NUM_WORKER_NODES=1 +NUM_KCP_NODES=2 +OVERWRITE_CLUSTER=false +SIDELOAD_IMAGES=false +SERVER_COUNT_OVERRIDE=-1 + +# Provide usage info +usage() { + printf "USAGE:\n./quickstart-kind.sh\n\t[--cluster-name ]\n\t[--server-image [:]]\n\t[--agent-image [:]]\n\t[--num-worker-nodes ]\n\t[--num-kcp-nodes ]\n\t[--overwrite-cluster]\n" +} + +# ARG PARSING +VALID_ARGS=$(getopt --options "h" --longoptions "sideload-images,cluster-name:,agent-image:,server-image:,num-worker-nodes:,num-kcp-nodes:,help,overwrite-cluster,server-count-override:" --name "$0" -- "$@") || exit 2 + +eval set -- "$VALID_ARGS" +while true; do + case "$1" in + --cluster-name) + CLUSTER_NAME=$2 + shift 2 + ;; + --agent-image) + AGENT_IMAGE=$2 + shift 2 + ;; + --server-image) + SERVER_IMAGE=$2 + shift 2 + ;; + --num-worker-nodes) + NUM_WORKER_NODES=$2 + shift 2 + ;; + --num-kcp-nodes) + NUM_KCP_NODES=$2 + shift 2 + ;; + --overwrite-cluster) + OVERWRITE_CLUSTER=true + shift 1 + ;; + --sideload-images) + SIDELOAD_IMAGES=true + shift 1 + ;; + --server-count-override) + SERVER_COUNT_OVERRIDE=$2 + shift 2 + ;; + --) + shift + break + ;; + *|-h|--help) + usage + exit + ;; + esac +done + +# RENDER CONFIG TEMPLATES +echo "Rendering config templates..." +if [ ! -d rendered ]; then + echo "Creating ./rendered" + mkdir rendered +fi +echo "Adding $NUM_KCP_NODES control plane nodes and $NUM_WORKER_NODES worker nodes to kind.config..." +cp templates/kind/kind.config rendered/kind.config +for i in $(seq 1 "$NUM_KCP_NODES") +do + cat templates/kind/control-plane.config >> rendered/kind.config +done +for i in $(seq 1 "$NUM_WORKER_NODES") +do + cat templates/kind/worker.config >> rendered/kind.config +done + +SERVER_COUNT=$NUM_KCP_NODES +if [ "$SERVER_COUNT_OVERRIDE" -ge 0 ]; then + echo "Overriding default server count from $NUM_KCP_NODES to $SERVER_COUNT_OVERRIDE" + SERVER_COUNT=$SERVER_COUNT_OVERRIDE +fi + +echo "Setting server image to $SERVER_IMAGE, agent image to $AGENT_IMAGE, and --server-count flag on server to $SERVER_COUNT" +sed -e "s|image: .*|image: $AGENT_IMAGE|" rendered/konnectivity-agent-ds.yaml +sed -e "s|image: .*|image: $SERVER_IMAGE|" -e "s/--server-count=[0-9]\+/--server-count=$SERVER_COUNT/" rendered/konnectivity-server.yaml + + + +# CLUSTER CREATION +if [ $OVERWRITE_CLUSTER = true ] && kind get clusters | grep -q "$CLUSTER_NAME"; then + echo "Deleting old cluster $CLUSTER_NAME..." + kind delete clusters "$CLUSTER_NAME" +fi + +echo "Creating cluster $CLUSTER_NAME..." +kind create cluster --config rendered/kind.config --name $CLUSTER_NAME + +echo "Successfully created cluster. Switching kubectl context to kind-$CLUSTER_NAME" +kubectl cluster-info --context kind-$CLUSTER_NAME + +# SIDELOAD IMAGES IF REQUESTED +if [ $SIDELOAD_IMAGES = true ]; then + echo "Sideloading images into the kind cluster..." + kind --name "$CLUSTER_NAME" load docker-image "$SERVER_IMAGE" + kind --name "$CLUSTER_NAME" load docker-image "$AGENT_IMAGE" +fi + +# DEPLOY KONNECTIVITY +echo "Requesting creation of konnectivity proxy servers on cluster $CLUSTER_NAME..." +kubectl apply -f rendered/konnectivity-server.yaml +echo "Requesting creation of konnectivity proxy agents on cluster $CLUSTER_NAME..." +kubectl apply -f rendered/konnectivity-agent-ds.yaml diff --git a/examples/kind-multinode/templates/k8s/konnectivity-agent-ds.yaml b/examples/kind-multinode/templates/k8s/konnectivity-agent-ds.yaml new file mode 100644 index 000000000..779fdeb85 --- /dev/null +++ b/examples/kind-multinode/templates/k8s/konnectivity-agent-ds.yaml @@ -0,0 +1,95 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: konnectivity-agent + namespace: kube-system + labels: + kubernetes.io/cluster-service: "true" +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + k8s-app: konnectivity-agent + namespace: kube-system + name: konnectivity-agent +spec: + selector: + matchLabels: + k8s-app: konnectivity-agent + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + k8s-app: konnectivity-agent + spec: + priorityClassName: system-cluster-critical + tolerations: + - key: "CriticalAddonsOnly" + operator: "Exists" + - operator: "Exists" + effect: "NoExecute" + nodeSelector: + kubernetes.io/os: linux + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: konnectivity-agent-container + image: gcr.io/k8s-staging-kas-network-proxy/proxy-agent:master + resources: + requests: + cpu: 50m + limits: + memory: 30Mi + command: [ "/proxy-agent"] + args: [ + "--logtostderr=true", + "--ca-cert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt", + "--proxy-server-host=konnectivity-server.kube-system.svc.cluster.local", + "--proxy-server-port=8091", + "--sync-interval=5s", + "--sync-interval-cap=30s", + "--sync-forever", + "--probe-interval=5s", + "--service-account-token-path=/var/run/secrets/tokens/konnectivity-agent-token", + "--agent-identifiers=ipv4=${HOST_IP}" + ] + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: HOST_IP + valueFrom: + fieldRef: + fieldPath: status.hostIP + livenessProbe: + httpGet: + scheme: HTTP + port: 8093 + path: /healthz + initialDelaySeconds: 15 + timeoutSeconds: 15 + readinessProbe: + httpGet: + scheme: HTTP + port: 8093 + path: /readyz + initialDelaySeconds: 15 + timeoutSeconds: 15 + volumeMounts: + - mountPath: /var/run/secrets/tokens + name: konnectivity-agent-token + serviceAccountName: konnectivity-agent + volumes: + - name: konnectivity-agent-token + projected: + sources: + - serviceAccountToken: + path: konnectivity-agent-token + audience: system:konnectivity-server diff --git a/examples/kind-multinode/templates/k8s/konnectivity-server.yaml b/examples/kind-multinode/templates/k8s/konnectivity-server.yaml new file mode 100644 index 000000000..46f938e9a --- /dev/null +++ b/examples/kind-multinode/templates/k8s/konnectivity-server.yaml @@ -0,0 +1,127 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: system:konnectivity-server + labels: + kubernetes.io/cluster-service: "true" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:auth-delegator +subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: system:konnectivity-server +--- +apiVersion: v1 +kind: Service +metadata: + name: konnectivity-server + namespace: kube-system +spec: + selector: + k8s-app: konnectivity-server + clusterIP: None + ports: + - protocol: TCP + port: 8091 + targetPort: 8091 +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + k8s-app: konnectivity-server + namespace: kube-system + name: konnectivity-server +spec: + selector: + matchLabels: + k8s-app: konnectivity-server + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + k8s-app: konnectivity-server + spec: + priorityClassName: system-cluster-critical + tolerations: + - key: "CriticalAddonsOnly" + operator: "Exists" + - operator: "Exists" + nodeSelector: + node-role.kubernetes.io/control-plane: "" + hostNetwork: true + containers: + - name: konnectivity-server-container + image: gcr.io/k8s-staging-kas-network-proxy/proxy-server:master + resources: + requests: + cpu: 1m + securityContext: + allowPrivilegeEscalation: false + runAsUser: 0 + command: [ "/proxy-server"] + args: [ + "--log-file=/var/log/konnectivity-server.log", + "--logtostderr=true", + "--log-file-max-size=0", + "--uds-name=/etc/kubernetes/konnectivity-server/konnectivity-server.socket", + "--delete-existing-uds-file", + "--cluster-cert=/etc/kubernetes/pki/apiserver.crt", + "--cluster-key=/etc/kubernetes/pki/apiserver.key", + "--server-port=0", + "--agent-port=8091", + "--health-port=8092", + "--admin-port=8093", + "--keepalive-time=1h", + "--mode=grpc", + "--agent-namespace=kube-system", + "--agent-service-account=konnectivity-agent", + "--kubeconfig=/etc/kubernetes/admin.conf", + "--authentication-audience=system:konnectivity-server", + "--server-count=2", + ] + livenessProbe: + httpGet: + scheme: HTTP + host: 127.0.0.1 + port: 8092 + path: /healthz + initialDelaySeconds: 10 + timeoutSeconds: 60 + ports: + - name: serverport + containerPort: 8090 + hostPort: 8090 + - name: agentport + containerPort: 8091 + hostPort: 8091 + - name: healthport + containerPort: 8092 + hostPort: 8092 + - name: adminport + containerPort: 8093 + hostPort: 8093 + volumeMounts: + - name: varlogkonnectivityserver + mountPath: /var/log/konnectivity-server.log + readOnly: false + - name: kubernetes + mountPath: /etc/kubernetes + readOnly: true + - name: konnectivity-home + mountPath: /etc/kubernetes/konnectivity-server + volumes: + - name: varlogkonnectivityserver + hostPath: + path: /var/log/konnectivity-server.log + type: FileOrCreate + - name: kubernetes + hostPath: + path: /etc/kubernetes + - name: konnectivity-home + hostPath: + path: /etc/kubernetes/konnectivity-server + type: DirectoryOrCreate diff --git a/examples/kind-multinode/templates/kind/control-plane.config b/examples/kind-multinode/templates/kind/control-plane.config new file mode 100644 index 000000000..8aee7c193 --- /dev/null +++ b/examples/kind-multinode/templates/kind/control-plane.config @@ -0,0 +1,25 @@ +- role: control-plane + kubeadmConfigPatchesJSON6902: + - kind: ClusterConfiguration + patch: | + - op: add + path: /apiServer/certSANs/- + value: konnectivity-server.kube-system.svc.cluster.local + kubeadmConfigPatches: + - | + kind: ClusterConfiguration + apiServer: + extraArgs: + "egress-selector-config-file": "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml" + extraVolumes: + - name: egress-selector-config-file + hostPath: "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml" + mountPath: "/etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml" + readOnly: true + - name: konnectivity-server + hostPath: "/etc/kubernetes/konnectivity-server" + mountPath: "/etc/kubernetes/konnectivity-server" + readOnly: true + extraMounts: + - hostPath: ./egress_selector_configuration.yaml + containerPath: /etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml diff --git a/examples/kind-multinode/templates/kind/kind.config b/examples/kind-multinode/templates/kind/kind.config new file mode 100644 index 000000000..aaee43767 --- /dev/null +++ b/examples/kind-multinode/templates/kind/kind.config @@ -0,0 +1,5 @@ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +networking: + ipFamily: ipv4 +nodes: diff --git a/examples/kind-multinode/templates/kind/worker.config b/examples/kind-multinode/templates/kind/worker.config new file mode 100644 index 000000000..3bbf8a75f --- /dev/null +++ b/examples/kind-multinode/templates/kind/worker.config @@ -0,0 +1 @@ +- role: worker