[BUG] CNF Test Suite cannot run today vs Clusters where Pod Security Standard restricted is enforced

[collivier]

Describe the bug

CNF Test Suite cannot run today vs Clusters where Pod Security Standard restricted is enforced.
To work vs hardened clusters in production, Functest Kubernetes has to patch all 3 namespaces before running CNF Test Suite.
https://github.com/opnfv/functest-kubernetes/blob/master/functest_kubernetes/cnf_conformance/conformance.py#L67-L86

To Reproduce

Enforce Pod Security Standard restricted ad follows

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "restricted"
      enforce-version: "latest"
      audit: "restricted"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces:
        - kube-system
        - local-path-storage

Here is one simple issue:
v129-control-plane/pods/kube-system_kube-controller-manager-v129-control-plane_3ec392f33c3dbdc70f705c7b917bf2c0/kube-controller-manager/0.log:2024-02-13T14:12:55.808318165Z stderr F I0213 14:12:55.808202 1 event.go:376] "Event occurred" object="cnf-testsuite/cluster-tools" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"cluster-tools-d9rw8\" is forbidden: violates PodSecurity \"restricted:latest\": host namespaces (hostNetwork=true, hostPID=true), privileged (container \"cluster-tools\" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container \"cluster-tools\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"cluster-tools\" must set securityContext.capabilities.drop=[\"ALL\"]), restricted volume types (volumes \"containerd-volume\", \"proc\", \"dockerd-volume\", \"systemd\" use restricted volume type \"hostPath\"), runAsNonRoot != true (pod or container \"cluster-tools\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"cluster-tools\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"

Expected behavior

cnf-testsuite cert in success when Kubernetes Cluster is hardened

How will this be tested? aka Acceptance Criteria (optional)

cnf-testsuite cert in success + see #1887

[collivier]

A simple way to setup an hardened cluster as described in https://kubernetes.io/docs/tutorials/security/cluster-level-pss/

$ cat kind.v1.29.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.29.0
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          admission-control-config-file: /etc/config/cluster-level-pss.yaml
        extraVolumes:
          - name: accf
            hostPath: /etc/config
            mountPath: /etc/config
            readOnly: false
            pathType: "DirectoryOrCreate"
  extraMounts:
  - hostPath: /home/opnfv/K8S
    containerPath: /etc/config
    # optional: if set, the mount is read-only.
    # default false
    readOnly: false
    # optional: if set, the mount needs SELinux relabeling.
    # default false
    selinuxRelabel: false
    # optional: set propagation mode (None, HostToContainer or Bidirectional)
    # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
    # default None
    propagation: None
- role: worker
  image: kindest/node:v1.29.0
- role: worker
  image: kindest/node:v1.29.0

$ cat /home/opnfv/K8S/cluster-level-pss.yaml 
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "restricted"
      enforce-version: "latest"
      audit: "restricted"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces:
        - kube-system
        - local-path-storage