Document best practices for editing dashboards in grafrana while the operator is active

[gberche-orange]

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

When the operator is reconciling (version 5.8.1, packaged by bitnami helm chart 4.1.0), it is overriding changes made to dashboards using grafana ui.

(If applicable)If your feature request solves a bug please provide a link to the community issue

Describe the solution you'd like

A documented workflow on how to author dashboards using the UI while working with grafana operator

Here is the workflow what I'm currently using, but it is tedious and error prone

Initial version of a dashboard:

• author a new dashboard in grafana UI (ignored by operator as it is not declared)
• export the modified dashboard as json (button share, and export tag and Export for sharing externally)
• Annotate dashboard to map variables (e.g. datasource variable to local datasource). See https://grafana.github.io/grafana-operator/docs/examples/datasource_mapping/readme/

  datasources:
    - inputName: datasource
      datasourceName: DS_PROMETHEUS

• update the expected state by operator (say configmap pulled from git repo)

2nd version of the dashboard edited through grafana (typically adding new panels or applying non trivial changes):

• temporary disable the grafana operator kubectl scale --replicas=0 deployment grafana-operator
• optionally make a copy of the dashboard to modify it (named dashboard-copy)
• edit the dashboard-copy
• export the dashboard-copy as json
• edit the Json to:
  • restore original name
  • restore original dashboard gui
• re-enable the grafana operator kubectl scale --replicas=1 deployment grafana-operator
• update the expected state by operator (say configmap pulled from git repo)

2nd version of the dashboard with multiple predictable changes to promql expressions (i.e. no need for webui ):

• edit the dashboard json in the gitops repo
• commit & push
• wait for grafana operator to apply changes
• manually check in grafana the expected result

Describe alternatives you've considered

Checking dashboard CRD api for existing undocumented support for pausing the reconciliation during authoring of new dashboards (similar to flux suspend/resume)

The operator could also detects drifts (i.e. newer versions) and only issue a warning without overriding the dashboard. Potentially related to #1106

Additional context
Add any other context or screenshots about the feature request here.

Existing solutions
If applicable please provide a link to an existing solution from a different project

[Baarsgaard]

I know this is not a feature request, but this is a workflow I would like to see:
My usecase: Central Grafana instance where dashboards are developed for multiple other instances.

Not a GitOps approach, but the ability to define a Grafana CR as a source, similar to how you can add grafanaCom.id as a dashboard source.
Essentially:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: central-development
  namespace: monitoring-system
spec:
  external:
    apiKey:
      key: apikey
      name: central-grafana-key
    url: https://grafana.example.com
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: node-exporter
  namespace: shared
spec:
  instanceSelector:
    matchLabels:
      node-exporter: "true"
  # Overwrite the folder of the source dashboard
  folder: Provisioned Dashboards
  resyncPeriod: "1m"
  grafanaRef:
    name: central-development
    dashboardUID: "bdawaz2a8d"
    viewOnly: true
    # Allow generating UIDs similar to how folders are created and referenced on alert rules.
    # maskUID: true
    # To source multiple dashboards, match their labels
    # dashboardSelectorLabels:
    #   - node-exporter
    # additionalLabels:
    #   - provisioned

The above could allow a workflow where you edit a dashboard in the referenced Grafana instance with a specific DashboardUID or label.
Save it, and wait x minutes for it to reconcile.
Then, if your central Grafana instance has the node-exporter: "true" label, it will get a folder Provisioned Dashboards with the dashbords matching the dashboardUID/dashboardSelectorLabels

maskUID and folder are useful here as they ensure dashboards do not conflict.

[theSuess]

@Baarsgaard this sounds very interesting! Can you submit a proposal for this one, as it's not trivial and there are some concerns around how the final spec should look like?

[Baarsgaard]

Sure, it will take me a bit as I need to think it through over a couple of days to fully understand what it would solve/enable vs what I want from it.