Merge pull request #38 from tum-gis/feature/clamav

Enable clamav for scanning files

charts/sddi-ckan/Chart.yaml

@@ -48,6 +48,10 @@ dependencies:
     condition: cert-manager.enabled
     version: "^1"
     repository: https://charts.jetstack.io
+  - name: clamav
+    condition: clamav.enabled
+    version: "~2.8.0"
+    repository: https://wiremind.github.io/wiremind-helm-charts
   - name: common
     repository: oci://registry-1.docker.io/bitnamicharts
     tags:

charts/sddi-ckan/README.md

@@ -30,8 +30,9 @@ Kubernetes: `>= 1.23.0-0`
 |  | postgis | * |
 |  | redis | * |
 |  | solr | * |
-| https://charts.jetstack.io | cert-manager(cert-manager) | ^1 |
-| https://kubernetes.github.io/ingress-nginx | ingress-nginx(ingress-nginx) | ^4 |
+| https://charts.jetstack.io | cert-manager(cert-manager) | ~1.11.0 |
+| https://kubernetes.github.io/ingress-nginx | ingress-nginx(ingress-nginx) | ~4.4.0 |
+| https://wiremind.github.io/wiremind-helm-charts | clamav | ~2.8.0 |
 | oci://registry-1.docker.io/bitnamicharts | common | 2.x.x |
 
 ## Values
@@ -40,6 +41,14 @@ Kubernetes: `>= 1.23.0-0`
 |-----|------|---------|-------------|
 | cert-manager.enabled | bool | `false` | Enable/disable cert-manager. |
 | certIssuer.enabled | bool | `true` | Enable/disable namespace Issuers for cert-manager. |
+| clamav.clamdConfig | string | `"###############\n# General\n###############\n\nDatabaseDirectory /data\nTemporaryDirectory /tmp\nLogTime yes\n# CUSTOM: Use pid file in tmp\nPidFile /tmp/clamd.pid\nLocalSocket /tmp/clamd.sock\n# CUSTOM: Set local socket group to defined group id\nLocalSocketGroup 2000\nTCPSocket 3310\nForeground yes\nStreamMaxLength 4000M\nLogVerbose yes\nBytecodeTimeout 1000\n\n###############\n# Results\n###############\n\nDetectPUA yes\nExcludePUA NetTool\nExcludePUA PWTool\nHeuristicAlerts yes\nBytecode yes\n\n###############\n# Scan\n###############\n\nScanPE yes\nDisableCertCheck yes\nScanELF yes\nAlertBrokenExecutables yes\nScanOLE2 yes\nScanPDF yes\nScanSWF yes\nScanMail yes\nPhishingSignatures yes\nPhishingScanURLs yes\nScanHTML yes\nScanArchive yes\n\n###############\n# Scan\n###############\n\nMaxScanSize 150M\nMaxFileSize 30M\nMaxRecursion 10\nMaxFiles 15000\nMaxEmbeddedPE 10M\nMaxHTMLNormalize 10M\nMaxHTMLNoTags 2M\nMaxScriptNormalize 5M\nMaxZipTypeRcg 1M\nMaxPartitions 128\nMaxIconsPE 200\nPCREMatchLimit 10000\nPCRERecMatchLimit 10000\n"` |  |
+| clamav.enabled | bool | `true` | Enable/disable [ClamAV](https://www.clamav.net/) virus scanning of uploaded files. |
+| clamav.freshclamConfig | string | `"###############\n# General\n###############\n\nDatabaseDirectory /data\nUpdateLogFile /dev/stdout\nLogTime yes\n# CUSTOM: Use pid file in tmp\nPidFile /tmp/freshclam.pid\n# CUSTOM: Set defined user\nDatabaseOwner 2000\n\n###############\n# Updates\n###############\n\nDatabaseMirror database.clamav.net\nScriptedUpdates yes\nNotifyClamd /etc/clamav/clamd.conf\nBytecode yes\n"` |  |
+| clamav.fullnameOverride | string | `"clamav"` |  |
+| clamav.resources.limits.cpu | string | `"4000m"` | [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| clamav.resources.limits.memory | string | `"8Gi"` | [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| clamav.resources.requests.cpu | string | `"1500m"` | [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
+| clamav.resources.requests.memory | string | `"2Gi"` | [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
 | datapusher.enabled | bool | `true` | Enable/disable Datapusher |
 | fullnameOverride | string | `""` | Override fullname |
 | global.datapusher.db.auth.password | string | `"changeMe"` | Jobs database password. If set, this values will overwrite the value in the Datapusher chart. |

charts/sddi-ckan/charts/ckan/README.md

@@ -49,6 +49,11 @@ A Helm chart for SDDI enabled CKAN.
 | autoscaling.targetCPUUtilizationPercentage | string | `nil` | [HorizontalPodAutoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). |
 | autoscaling.targetMemoryUtilizationPercentage | string | `nil` | [HorizontalPodAutoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). |
 | backgroundImage | string | `"../base/images/hero.jpg"` | Set URL or path to [CKAN SDDI background image](https://github.com/tum-gis/ckanext-grouphierarchy-sddi#personalisation). |
+| clamav.enabled | bool | `true` |  |
+| clamav.host | string | `"clamav"` |  |
+| clamav.port | int | `3310` |  |
+| clamav.timeout | int | `360` |  |
+| clamav.uploadUnscanned | string | `"False"` |  |
 | component | string | `"ckan"` | Role of CKAN in this chart |
 | datapusher.apiToken | string | `nil` | Datapusher API token, see [CKAN Datapusher settings](https://docs.ckan.org/en/latest/maintaining/configuration.html#datapusher-settings) |
 | datapusher.callback_url_base | string | `"http://ckan:5000/"` | This should be set to cluster internal ckan service domain. [CKAN DataPusher settings](https://docs.ckan.org/en/latest/maintaining/configuration.html#ckan-datapusher-callback-url-base) |
@@ -90,7 +95,7 @@ A Helm chart for SDDI enabled CKAN.
 | ingress.cors.enabled | bool | `true` | Enable/disable [CORS](https://de.wikipedia.org/wiki/Cross-Origin_Resource_Sharing). See [ingress-nginx cors settings](https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#enable-cors) for details on CORS configuration and default settings. Use `ingress.annotations` to overwrite the default configuration annotations. |
 | ingress.domains | list | `[]` | List of [FQDNs](https://de.wikipedia.org/wiki/Fully-Qualified_Host_Name) for this Ingress. Note: All FQDNs will be used for Ingress hosts and TLS certificate. Note: Set `siteUrl` accordingly! |
 | ingress.enabled | bool | `true` | Enable/disable Ingress. |
-| ingress.stickySessions.enabled | bool | `true` | Enable/disable sticks sessions, see [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/). |
+| ingress.stickySessions.enabled | bool | `false` | Enable/disable sticks sessions, see [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/). |
 | ingress.stickySessions.sessionCookie.affinityMode | string | `"balanced"` | [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/) |
 | ingress.stickySessions.sessionCookie.changeOnFailure | string | `"true"` | [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/) |
 | ingress.stickySessions.sessionCookie.maxAge | string | `"172800"` | [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/) |

charts/sddi-ckan/charts/ckan/templates/ckan-configMap-env.yml

@@ -75,6 +75,14 @@ data:
   {{- if .Values.webassets.path }}
   CKAN__WEBASSETS__PATH: {{ .Values.webassets.path | quote }}
   {{- end }}
+  {{- if .Values.clamav.enabled }}
+  # CKANEXT__CLAMAV__SOCKET_PATH: /your/path/to/socket.file
+  CKANEXT__CLAMAV__UPLOAD_UNSCANNED: {{ .Values.clamav.uploadUnscanned | quote }}
+  CKANEXT__CLAMAV__SOCKET_TYPE: tcp
+  CKANEXT__CLAMAV__TCP__HOST: {{ .Values.clamav.host | quote }}
+  CKANEXT__CLAMAV__TCP__PORT: {{ .Values.clamav.port | quote }}
+  CKANEXT__CLAMAV__TIMEOUT: {{ .Values.clamav.timeout | quote }}
+  {{- end }}
   # Additional env vars from values.yaml
   {{- with .Values.extraEnv }}
     {{- toYaml . | nindent 2 }}

charts/sddi-ckan/charts/ckan/templates/ckan-ingress.yml

@@ -30,6 +30,12 @@ metadata:
     {{- end }}
     nginx.ingress.kubernetes.io/proxy-body-size: "{{ .Values.maxUploadSizeMB }}m"
     nginx.org/client-max-body-size: "{{ .Values.maxUploadSizeMB }}m"
+    nginx.ingress.kubernetes.io/proxy-connect-timeout: "360"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "360"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "360"
+    nginx.org/proxy-connect-timeout: "360"
+    nginx.org/proxy-read-timeout: "360"
+    nginx.org/proxy-send-timeout: "360"
     nginx.ingress.kubernetes.io/configuration-snippet: {{- .Values.ingress.configurationSnippet | toYaml | indent 4 }}
     {{- if .Values.ingress.stickySessions.enabled }}
     # https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

charts/sddi-ckan/charts/ckan/values.yaml

@@ -134,7 +134,7 @@ ingress:
   stickySessions:
     # -- Enable/disable sticks sessions, see
     # [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/).
-    enabled: true
+    enabled: false
     sessionCookie:
       # -- [Nginx Ingress Controller Sticky sessions](https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/)
       affinityMode: "balanced"
@@ -498,6 +498,25 @@ activityStreams:
   # -- [CKAN config activity stream](https://docs.ckan.org/en/latest/maintaining/configuration.html#activity-streams-settings)
   emailNotifications: True
 
+clamav:
+  enabled: true
+  uploadUnscanned: "False"
+  host: clamav
+  port: 3310
+  timeout: 360
+
+resources:
+  limits:
+    # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+    cpu: 500m
+    # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+    memory: 1Gi
+  requests:
+    # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+    cpu: 250m
+    # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+    memory: 256Mi
+
 # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
 resources: {}
 # -- [k8s: Assign pods to nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

charts/sddi-ckan/values.yaml

@@ -144,3 +144,109 @@ cert-manager:
 certIssuer:
   # -- Enable/disable namespace Issuers for cert-manager.
   enabled: true
+
+clamav:
+  # -- Enable/disable [ClamAV](https://www.clamav.net/) virus scanning of uploaded files.
+  enabled: true
+
+  fullnameOverride: clamav
+
+  resources:
+    limits:
+      # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+      cpu: 4000m
+      # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+      memory: 8Gi
+    requests:
+      # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+      cpu: 1500m
+      # -- [k8s: Resource management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+      memory: 2Gi
+
+  clamdConfig: |
+    ###############
+    # General
+    ###############
+
+    DatabaseDirectory /data
+    TemporaryDirectory /tmp
+    LogTime yes
+    # CUSTOM: Use pid file in tmp
+    PidFile /tmp/clamd.pid
+    LocalSocket /tmp/clamd.sock
+    # CUSTOM: Set local socket group to defined group id
+    LocalSocketGroup 2000
+    TCPSocket 3310
+    Foreground yes
+    StreamMaxLength 4000M
+    LogVerbose yes
+    BytecodeTimeout 1000
+
+    ###############
+    # Results
+    ###############
+
+    DetectPUA yes
+    ExcludePUA NetTool
+    ExcludePUA PWTool
+    HeuristicAlerts yes
+    Bytecode yes
+
+    ###############
+    # Scan
+    ###############
+
+    ScanPE yes
+    DisableCertCheck yes
+    ScanELF yes
+    AlertBrokenExecutables yes
+    ScanOLE2 yes
+    ScanPDF yes
+    ScanSWF yes
+    ScanMail yes
+    PhishingSignatures yes
+    PhishingScanURLs yes
+    ScanHTML yes
+    ScanArchive yes
+
+    ###############
+    # Scan
+    ###############
+
+    MaxScanSize 150M
+    MaxFileSize 30M
+    MaxRecursion 10
+    MaxFiles 15000
+    MaxEmbeddedPE 10M
+    MaxHTMLNormalize 10M
+    MaxHTMLNoTags 2M
+    MaxScriptNormalize 5M
+    MaxZipTypeRcg 1M
+    MaxPartitions 128
+    MaxIconsPE 200
+    PCREMatchLimit 10000
+    PCRERecMatchLimit 10000
+
+  ## Ref: https://linux.die.net/man/5/freshclam.conf
+  ## Note: will completely override default clamd.conf file at https://github.com/Mailu/Mailu/tree/master/optional/clamav/conf
+  freshclamConfig: |
+    ###############
+    # General
+    ###############
+
+    DatabaseDirectory /data
+    UpdateLogFile /dev/stdout
+    LogTime yes
+    # CUSTOM: Use pid file in tmp
+    PidFile /tmp/freshclam.pid
+    # CUSTOM: Set defined user
+    DatabaseOwner 2000
+
+    ###############
+    # Updates
+    ###############
+
+    DatabaseMirror database.clamav.net
+    ScriptedUpdates yes
+    NotifyClamd /etc/clamav/clamd.conf
+    Bytecode yes