Helm: Flux doesn't change drifted resources

[Andrea-Gallicchio]

Describe the bug

After having released a Helm Release that installs for example a deployment, If I manually edit the deployment itself, Flux doesn't change the resource to the state of the Git repository, even if I run flux reconcile hr .....

This is really weird because I'm sure it worked fine in previous versions of Flux v2.

Steps to reproduce

1. Install Flux in a Kubernetes cluster
2. Deploy a Helm Release through Flux; i.e I've used https://github.com/deliveryhero/helm-charts/blob/master/stable/kube-downscaler/values.yaml
3. Manually change something in the deployment; i.e change the value of an ENV variable
4. Wait to see if Flux overwrites the manual changes or also try to reconcile the Helm Release

Expected behavior

Flux changes the edited resource back to the state shown in the Git repos.

Screenshots and recordings

No response

OS / Distro

Linux

Flux version

v0.37.0

Flux check

► checking prerequisites
✗ flux 0.37.0 <0.38.2 (new version is available, please upgrade)
✔ Kubernetes 1.24.7-eks-fb459a0 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.27.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.27.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.23.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.31.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.29.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.32.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

GitHub

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[kingdonb]

Duplicate of the pinned issue on Helm Controller:

• Cluster-state manifest drift correction helm-controller#186

This is on the roadmap for Flux 2.1 to solve. https://fluxcd.io/roadmap/#flux-helm-ga-q2-2023

(Its roadmap item is the one called "Atomic reconciliation of HelmReleases"

[kingdonb]

Not sure if I should "close this as completed" (nor as "not planned") – would edit the title to say "Helm Controller doesn't correct drift" if we are keeping it open here, (just to avoid accumulating more new reports of more duplicates from others who didn't think to check Helm Controller for an open issue.)

[benjimin]

@Andrea-Gallicchio If your git repo contains a HelmRelease, it is the responsibility of flux to ensure that the HelmRelease in the cluster matches the HelmRelease in the repo.

If the corresponding Helm chart specifies for a deployment (or other kubernetes resources) to be created, it should be the responsibility of the helm controller (not fluxcd itself) to keep the direct child resources in sync with the HelmRelease.

This is analogous to how it is the responsibility of a ReplicaSet controller (not flux) to keep pods in sync with the ReplicaSet spec (and similarly for the Deployment controller). Unfortunately the helm controller does not currently perform this function (presumably owing to Helm having its own model for change management, different than both kubernetes and flux/gitops), but that's a separate issue (which has already been linked in the comment above).

[stefanprodan]

You need to enable Helm drift detection, see https://fluxcd.io/flux/cheatsheets/bootstrap/#enable-helm-drift-detection