Issues with RabbitMQ Cluster Operator and Secrets Store CSI Driver Integration

[dannotes]

I am facing challenges integrating RabbitMQ with Azure Key Vault using the Secrets Store CSI Driver in my Kubernetes environment.

Environment

• Kubernetes Version: 1.30.5
• RabbitMQ Version: 3.12
• Secrets Store CSI Driver Version: v2.11.1

Reproduction steps

1. Created a SecretProviderClass to fetch TLS certificates and credentials from Azure Key Vault.
2. Attempted to deploy RabbitMQ Cluster Operator.

SecretServiceProvider.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: rabbitmq-keyvault-secrets
  namespace: rabbitmq-system
spec:
  provider: azure
  secretObjects:
  - data:
    - key: tls.crt
      objectName: rabbitmq-server-cert
    - key: tls.key
      objectName: rabbitmq-server-key
    secretName: rabbitmq-server-certs
    type: kubernetes.io/tls
  - data:
    - key: admin-password
      objectName: rabbitmq-admin-password
    - key: erlang-cookie
      objectName: rabbitmq-erlang-cookie
    secretName: rabbitmq-credentials
    type: Opaque
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "f2120383-a285-489e-aa69-45a1283b02c9"
    keyvaultName: "aks-vault-jkadbhabadjad"
    cloudName: AzurePublicCloud
    objects: |
      array:
        - |
          objectName: rabbitmq-server-cert
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-server-key
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-admin-password
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-erlang-cookie
          objectType: secret
          objectVersion: ""
    tenantId: "f19a22fe-f19f-4a89-8879-face6c476625"

rabbitmq-cluster-definitions.yaml

# rabbitmq-definitions.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-definitions
  namespace: rabbitmq-system
data:
  definitions.json: |-
    {
      "users": [
        {
          "name": "monitoring",
          "password_hash": "pssEXpJo+paqFO4HMFdSRy0gNGe9vzJZB+e/n7OyF6G34ntN",
          "tags": "monitoring"
        }
      ],
      "vhosts": [
        {
          "name": "/"
        }
      ],
      "permissions": [
        {
          "user": "monitoring",
          "vhost": "/",
          "configure": "^$",
          "write": "^$",
          "read": ".*"
        }
      ],
      "policies": [
        {
          "name": "ha-policy",
          "pattern": ".*",
          "vhost": "/",
          "priority": 0,
          "definition": {
            "ha-mode": "all",
            "ha-sync-mode": "automatic",
            "queue-master-locator": "min-masters"
          }
        }
      ]
    }

rabbitmq-cluster.yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: production-rabbitmq
  namespace: rabbitmq-system
  labels:
    app.kubernetes.io/component: rabbitmq
spec:
  replicas: 3
  image: rabbitmq:3.12-management
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"
  persistence:
    storageClassName: rabbitmq-premium-ssd
    storage: "50Gi"
  tls:
    secretName: rabbitmq-server-certs  # TLS secret with cert and key
    caSecretName: rabbitmq-ca-cert-secret  # Separate secret for CA certificate
  rabbitmq:
    additionalPlugins:
      - rabbitmq_prometheus
      - rabbitmq_shovel
      - rabbitmq_shovel_management
      - rabbitmq_federation
      - rabbitmq_federation_management
    additionalConfig: |
      load_definitions = /etc/rabbitmq/custom/definitions.json
      cluster_partition_handling = autoheal
      vm_memory_high_watermark.relative = 0.8
      disk_free_limit.absolute = 5GB
      queue_master_locator = min-masters
      prometheus.tcp.port = 15692
      collect_statistics_interval = 10000
      ssl_options.verify = verify_peer
      ssl_options.fail_if_no_peer_cert = true
      listeners.ssl.default = 5671
      ssl_options.cacertfile = /etc/rabbitmq-certs/ca.crt  # Path to CA cert
      ssl_options.certfile = /etc/rabbitmq-certs/tls.crt    # Path to server cert
      ssl_options.keyfile = /etc/rabbitmq-certs/tls.key      # Path to server key
  override:
    statefulSet:
      spec:
        template:
          spec:
            volumes:
              - name: definitions
                configMap:
                  name: rabbitmq-definitions
              - name: secrets-store-inline
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: rabbitmq-keyvault-secrets
            containers:
              - name: rabbitmq
                env:
                  - name: RABBITMQ_ERLANG_COOKIE
                    valueFrom:
                      secretKeyRef:
                        name: rabbitmq-credentials  # Kubernetes secret created by SecretProviderClass
                        key: erlang-cookie
                  - name: RABBITMQ_DEFAULT_PASS
                    valueFrom:
                      secretKeyRef:
                        name: rabbitmq-credentials  # Kubernetes secret created by SecretProviderClass
                        key: admin-password
                  - name: RABBITMQ_DEFAULT_USER
                    value: "admin"
                volumeMounts:
                  - name: definitions
                    mountPath: /etc/rabbitmq/custom/
                  - name: secrets-store-inline
                    mountPath: "/etc/rabbitmq-certs"  # Mount path for certificates
                    readOnly: true
  service:
    type: ClusterIP
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "15692"

Expected behavior

The RabbitMQ cluster should start successfully with secrets populated from Azure Key Vault.

Additional context

The RabbitMQ deployment fails due to missing secrets, even though they are configured in Azure Key Vault and referenced in the SecretProviderClass.

Additional Information

• I have ensured that the managed identity has access to read secrets from Azure Key Vault by creating a test pod.
• I manually created the secret with the annotation secrets-store.csi.k8s.io/used: "true", but the secrets remain empty in Kubernetes even after deploying using cluster operator that references the SecretProviderClass.

Please advise on how to resolve this integration issue or if there are any known limitations with this setup.

[michaelklishin]

@dannotes RabbitMQ 3.12.x is out of community support.

RabbitMQ is entirely unaware of Azure or any specific Kubernetes flavor. We won't help you troubleshoot Azure Kubernetes-specific behavior even if it was with 4.0.x, and you haven't shared any logs for us to work with anyway.

[dannotes]

@dannotes RabbitMQ 3.12.x is out of community support.

RabbitMQ is entirely unaware of Azure or any specific Kubernetes flavor. We won't help you troubleshoot Azure Kubernetes-specific behavior even if it was with 4.0.x, and you haven't shared any logs for us to work with anyway.

Thank you for your guidance. I understand that RabbitMQ operates independently of specific cloud providers, including Azure Kubernetes Service, and I apologize for not providing the necessary logs earlier.

I was able to resolve the issue by creating a SecretProviderClass to fetch TLS certificates and configuring Kubernetes secrets with respect to the SecretProviderClass, followed by deploying the RabbitMQ Cluster Operator. This approach successfully integrated RabbitMQ with Azure Key Vault using the Secrets Store CSI Driver in my Kubernetes environment. Note that the Kubernetes secrets will not contain data themselves; however, when used with the pods and RabbitMQ operations, they will automatically fetch data from the vault. All that's required is to mount the secrets. I hope this solution is helpful to others facing similar challenges.

Adding the corrected YAML files:

# Creating dummy secrets since RabbitMQ cannot create based on the `SecretProviderClass`
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-server-certs
  namespace: rabbitmq-system
type: kubernetes.io/tls
data:
  tls.crt: ""  # Replace with the actual base64 encoded value of rabbitmq-server-cert
  tls.key: ""  # Replace with the actual base64 encoded value of rabbitmq-server-key
---
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-credentials
  namespace: rabbitmq-system
type: Opaque
data:
  admin-password: ""  # Replace with the actual base64 encoded value of rabbitmq-admin-password
---
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-ca-cert
  namespace: rabbitmq-system
type: Opaque
data:
  ca.crt: ""  # Replace with the actual base64 encoded value of rabbitmq-ca-cert
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: rabbitmq-keyvault-secrets
  namespace: rabbitmq-system
spec:
  provider: azure
  secretObjects:
  - data:
    - key: tls.crt
      objectName: rabbitmq-server-cert
    - key: tls.key
      objectName: rabbitmq-server-key
    secretName: rabbitmq-server-certs
    type: kubernetes.io/tls
  - data:
    - key: ca.crt
      objectName: rabbitmq-ca-cert  # Add CA cert object
    secretName: rabbitmq-ca-cert    # New secret for CA cert
    type: Opaque
  - data:
    - key: admin-password
      objectName: rabbitmq-admin-password
    - key: erlang-cookie
      objectName: rabbitmq-erlang-cookie
    secretName: rabbitmq-credentials
    type: Opaque
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxxxxx-a285-489e-aa69-45a1283b02c9"
    keyvaultName: "aks-vault-JKADJBDA"
    cloudName: AzurePublicCloud
    objects: |
      array:
        - |
          objectName: rabbitmq-server-cert
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-server-key
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-ca-cert    # Add CA cert to objects
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-admin-password
          objectType: secret
          objectVersion: ""
        - |
          objectName: rabbitmq-erlang-cookie
          objectType: secret
          objectVersion: ""
    tenantId: "f19a22fe-f19f-4a89-8879-face6c476625"