Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix github action e2e test. #555

Merged
merged 1 commit into from
Jan 26, 2024
Merged

Fix github action e2e test. #555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 88 additions & 77 deletions .github/workflows/e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ jobs:
# kubectl
curl -L https://dl.k8s.io/${{ env.K8S_VERSION }}/bin/linux/amd64/kubectl -o ${TMP_DIR}/kubectl
# kind
curl -Lo ${TMP_DIR}/kind https://kind.sigs.k8s.io/dl/v0.17.0/kind-linux-amd64
curl -Lo ${TMP_DIR}/kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
liangyuanpeng marked this conversation as resolved.
Show resolved Hide resolved
# Install
sudo cp ${TMP_DIR}/ginkgo /usr/local/bin/ginkgo
sudo cp ${TMP_DIR}/e2e.test /usr/local/bin/e2e.test
Expand All @@ -103,6 +103,12 @@ jobs:
ipFamily: ${IP_FAMILY}
nodes:
- role: control-plane
liangyuanpeng marked this conversation as resolved.
Show resolved Hide resolved
kubeadmConfigPatchesJSON6902:
- kind: ClusterConfiguration
patch: |
- op: add
path: /apiServer/certSANs/-
value: konnectivity-server.kube-system.svc.cluster.local
kubeadmConfigPatches:
liangyuanpeng marked this conversation as resolved.
Show resolved Hide resolved
- |
kind: ClusterConfiguration
Expand All @@ -121,7 +127,10 @@ jobs:
extraMounts:
- hostPath: ./examples/kind/egress_selector_configuration.yaml
containerPath: /etc/kubernetes/konnectivity-server-config/egress_selector_configuration.yaml
- role: worker
- role: worker
EOF

# dump the kubeconfig for later
/usr/local/bin/kind get kubeconfig --name ${{ env.KIND_CLUSTER_NAME}} > _artifacts/kubeconfig.conf

Expand All @@ -143,79 +152,81 @@ jobs:
kubectl apply -f examples/kind/konnectivity-server.yaml
kubectl apply -f examples/kind/konnectivity-agent-ds.yaml

# - name: Get Cluster status
# run: |
# # wait network is ready
# sleep 5
# /usr/local/bin/kubectl get nodes -o wide
# /usr/local/bin/kubectl get pods -A
# /usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods --namespace=kube-system -l k8s-app=konnectivity-agent
# /usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods --namespace=kube-system -l k8s-app=konnectivity-server
# # smoke test
# /usr/local/bin/kubectl run test --image httpd:2
# /usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods test
# /usr/local/bin/kubectl logs test

# - name: Workaround CoreDNS for IPv6 airgapped
# if: ${{ matrix.ipFamily == 'ipv6' }}
# run: |
# # Patch CoreDNS to work in Github CI
# # 1. Github CI doesn´t offer IPv6 connectivity, so CoreDNS should be configured
# # to work in an offline environment:
# # https://github.com/coredns/coredns/issues/2494#issuecomment-457215452
# # 2. Github CI adds following domains to resolv.conf search field:
# # .net.
# # CoreDNS should handle those domains and answer with NXDOMAIN instead of SERVFAIL
# # otherwise pods stops trying to resolve the domain.
# # Get the current config
# original_coredns=$(/usr/local/bin/kubectl get -oyaml -n=kube-system configmap/coredns)
# echo "Original CoreDNS config:"
# echo "${original_coredns}"
# # Patch it
# fixed_coredns=$(
# printf '%s' "${original_coredns}" | sed \
# -e 's/^.*kubernetes cluster\.local/& net/' \
# -e '/^.*upstream$/d' \
# -e '/^.*fallthrough.*$/d' \
# -e '/^.*forward . \/etc\/resolv.conf$/d' \
# -e '/^.*loop$/d' \
# )
# echo "Patched CoreDNS config:"
# echo "${fixed_coredns}"
# printf '%s' "${fixed_coredns}" | /usr/local/bin/kubectl apply -f -

# - name: Run tests
# run: |
# export KUBERNETES_CONFORMANCE_TEST='y'
# export E2E_REPORT_DIR=${PWD}/_artifacts

# # Run tests
# /usr/local/bin/ginkgo --nodes=25 \
# --focus="\[Conformance\]" \
# --skip="Feature|Federation|machinery|PerformanceDNS|DualStack|Disruptive|Serial|Slow|KubeProxy|LoadBalancer|GCE|Netpol|NetworkPolicy|NodeConformance" \
# /usr/local/bin/e2e.test \
# -- \
# --kubeconfig=${PWD}/_artifacts/kubeconfig.conf \
# --provider=local \
# --dump-logs-on-failure=false \
# --report-dir=${E2E_REPORT_DIR} \
# --disable-log-dump=true

# - name: Upload Junit Reports
# if: always()
# uses: actions/upload-artifact@v2
# with:
# name: kind-junit-${{ env.JOB_NAME }}-${{ github.run_id }}
# path: './_artifacts/*.xml'

# - name: Export logs
# if: always()
# run: |
# /usr/local/bin/kind export logs --name ${KIND_CLUSTER_NAME} --loglevel=debug ./_artifacts/logs

# - name: Upload logs
# if: always()
# uses: actions/upload-artifact@v2
# with:
# name: kind-logs-${{ env.JOB_NAME }}-${{ github.run_id }}
# path: ./_artifacts/logs
- name: Get Cluster status
run: |
# wait network is ready
sleep 5
/usr/local/bin/kubectl get nodes -o wide
/usr/local/bin/kubectl get pods -A
/usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods --namespace=kube-system -l k8s-app=kube-dns
# smoke test
/usr/local/bin/kubectl run test --image httpd:2
/usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods test
/usr/local/bin/kubectl get pods -A -owide
/usr/local/bin/kubectl wait --timeout=1m --for=condition=ready pods --namespace=kube-system -l k8s-app=konnectivity-agent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth also waiting for -l k8s-app=konnectivity-server to be ready.

(In this 1 control-plane node cluster is is nearly the same, but in a multi control-plane cluster it would be needed.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming you're talking about waiting the pod of ANP server with
readiness probe.(now, the pod of ANP server have not readiness probe)

konnectivity-server'readiness indicates that at least one Konnectivity Agent is connected.

konnectivity-agent'readiness indicates that the client is connected to at least one proxy server.

Therefore, if readiness is added at the same time, both will enter an infinite loop.

The reason is ANP agent is using kubernetes svc to connect ANP server and ANP server is running with daemonset.

Remind me if i missed something,Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming you're talking about waiting the pod of ANP server with readiness probe.(now, the pod of ANP server have not readiness probe)

konnectivity-server'readiness indicates that at least one Konnectivity Agent is connected.

konnectivity-agent'readiness indicates that the client is connected to at least one proxy server.

Yes to above.

Therefore, if readiness is added at the same time, both will enter an infinite loop.

I don't expect this, because waiting on agent (or any) readiness does not depend on control plane egress.

The reason is ANP agent is using kubernetes svc to connect ANP server and ANP server is running with daemonset.

Remind me if i missed something,Thanks.

What really matters (for a given proxy request ,like kubectl logs) is whether the apiserver that handles the request has a konnectivity-server with at least one useful agent. Since the agent readiness only covers "at least 1", it is insufficient in this case. (There was discussion on that feature to add agent readiness mode "connected to all servers" but it is not implemented.) This would be the gap, fixed by waiting for all konnectivity-server to be ready.

/usr/local/bin/kubectl logs test


- name: Workaround CoreDNS for IPv6 airgapped
if: ${{ matrix.ipFamily == 'ipv6' }}
run: |
# Patch CoreDNS to work in Github CI
# 1. Github CI doesn´t offer IPv6 connectivity, so CoreDNS should be configured
# to work in an offline environment:
# https://github.com/coredns/coredns/issues/2494#issuecomment-457215452
# 2. Github CI adds following domains to resolv.conf search field:
# .net.
# CoreDNS should handle those domains and answer with NXDOMAIN instead of SERVFAIL
# otherwise pods stops trying to resolve the domain.
# Get the current config
original_coredns=$(/usr/local/bin/kubectl get -oyaml -n=kube-system configmap/coredns)
echo "Original CoreDNS config:"
echo "${original_coredns}"
# Patch it
fixed_coredns=$(
printf '%s' "${original_coredns}" | sed \
-e 's/^.*kubernetes cluster\.local/& net/' \
-e '/^.*upstream$/d' \
-e '/^.*fallthrough.*$/d' \
-e '/^.*forward . \/etc\/resolv.conf$/d' \
-e '/^.*loop$/d' \
)
echo "Patched CoreDNS config:"
echo "${fixed_coredns}"
printf '%s' "${fixed_coredns}" | /usr/local/bin/kubectl apply -f -

- name: Run tests
run: |
export KUBERNETES_CONFORMANCE_TEST='y'
export E2E_REPORT_DIR=${PWD}/_artifacts

# Run tests
/usr/local/bin/ginkgo --nodes=25 \
--focus="\[Conformance\]" \
--skip="Feature|Federation|machinery|PerformanceDNS|DualStack|Disruptive|Serial|Slow|KubeProxy|LoadBalancer|GCE|Netpol|NetworkPolicy|NodeConformance" \
/usr/local/bin/e2e.test \
-- \
--kubeconfig=${PWD}/_artifacts/kubeconfig.conf \
--provider=local \
--dump-logs-on-failure=false \
--report-dir=${E2E_REPORT_DIR} \
--disable-log-dump=true

- name: Upload Junit Reports
if: always()
uses: actions/upload-artifact@v2
with:
name: kind-junit-${{ env.JOB_NAME }}-${{ github.run_id }}
path: './_artifacts/*.xml'

- name: Export logs
if: always()
run: |
/usr/local/bin/kind export logs --name ${KIND_CLUSTER_NAME} --loglevel=debug ./_artifacts/logs

- name: Upload logs
if: always()
uses: actions/upload-artifact@v2
with:
name: kind-logs-${{ env.JOB_NAME }}-${{ github.run_id }}
path: ./_artifacts/logs
7 changes: 7 additions & 0 deletions examples/kind/konnectivity-agent-ds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,13 @@ spec:
path: /healthz
initialDelaySeconds: 15
timeoutSeconds: 15
readinessProbe:
httpGet:
scheme: HTTP
port: 8093
path: /readyz
initialDelaySeconds: 15
timeoutSeconds: 15
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: konnectivity-agent-token
Expand Down
Loading