[BUG][opensearch] cant connect to metrics endpoint

[gabriel-suela]

Describe the bug
Im trying to see the metrics for opensearch but it seems that it isn't working.

To Reproduce

serviceMonitor:
  enabled: true

Expected behavior
See all metrics

Chart Name
Specify the Chart which is affected?

Screenshots

Host/Environment (please complete the following information):

• Helm Version: 3.15.4
• Kubernetes Version: 1.29.6

Additional context
Also, prometheus is alerting that the metrics are unreachable

[BernhardGruen]

Same problem here. Thank you for your report.

[eyenx]

TBH this looks more like a configuration issue. I have the same problem. Looking into it, probably opensearch is either missing a configuration option or even the prometheus-exporter-plugin

[eyenx]

Looking at what port is by default configured (9600) I'd think the idea is to use the performance analyzer which is already included and installed as plugin on the opensearch container, but not enabled for what I can tell
https://opensearch.org/docs/latest/monitoring-your-cluster/pa/index/

[eyenx]

Okay, when trying to start it inside the container I just see that the address is already bound but not reachable.

I'm gonna continue troubleshoot this.

sh-5.2$ OPENSEARCH_HOME=/usr/share/opensearch/ OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk/ OPENSEARCH_PATH_CONF=/usr/share/opensearch/config/ bin/opensearch-performance-analyzer/performance-analyzer-agent-cli &
[1] 1181
sh-5.2$ 09:05:50.833 [main] ERROR org.opensearch.performanceanalyzer.PerformanceAnalyzerWebServer - Could not create HttpServer on port 9600
java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.bind(Net.java:565) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:344) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:301) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocke
        ```

[eyenx]

According to the initial PR, the SM is meant to be used in conjunction with https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch

I'll try to integrate it and make sure it's installed by default when enabling serviceMonitor

Reference: #537 (comment)

[gabriel-suela]

According to the initial PR, the SM is meant to be used in conjunction with https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch

I'll try to integrate it and make sure it's installed by default when enabling serviceMonitor

Reference: #537 (comment)

with the plugin i managed to access the metrics on port 9200, but the ServiceMonitor manifest still have a port pointing to metrics port name:

 spec:                                                                                                                                                                                                                                                                    │
│   endpoints:                                                                                                                                                                                                                                                             │
│   - interval: 10s                                                                                                                                                                                                                                                        │
│     path: /_prometheus/metrics                                                                                                                                                                                                                                           │
│     port: metrics    

i believe that is why prometheus keeps warning me that the opensearch metrics are unreachable even so i can access it through port forward.

[eyenx]

You can configure the port with metricsPort

[KristijanL]

i have try to make servicemonitor work for a while now, but i think the implementation is broken. we have
service.metricsPortName and metricsPort

but setting either of those, breaks deployment, example:

service:
  metricsPortName: "http"
metricsPort: 9200

* Service "opensearch-cluster-master-headless" is invalid: spec.ports[2].name: Duplicate value: "http"
* Service "opensearch-cluster-master" is invalid: spec.ports[2].name: Duplicate value: "http"

* Service "opensearch-cluster-master-headless" is invalid: spec.ports[2]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9200, TargetPort: intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}                                                                                                                                                       
* Service "opensearch-cluster-master" is invalid: spec.ports[2]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9200, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}

the best way for this to work, would be to leave the users an option to edit the servicemonitor but add defaults to values
example:

  endpoints:
  - targetPort: {{ .Values.serviceMonitor.targetPort }}
    path: {{ .Values.serviceMonitor.path }}
    interval: {{ .Values.serviceMonitor.interval }}
    scrapeTimeout: {{ .Values.serviceMonitor.scrapeTimeout }}
    honorLabels: {{ .Values.serviceMonitor.honorLabels }}
    {{- with .Values.serviceMonitor.endpointAdditionalProperties }}
    {{- toYaml . | nindent 4 }}
    {{- end }}

another issue that i notice, i get duplicate endpoints, beacuse the selector selects both headless and non headless service, so the selector needs to be updated to only one service.

[gabriel-suela]

@KristijanL i found out that serviceMonitor needs to use basic auth to properly connect to prometheus endpoint:

  - apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    labels:
      app.kubernetes.io/component: opensearch-cluster-master
      app.kubernetes.io/instance: opensearch
    metadata:
      namespace: opensearch
      name: opensearch-cluster-master-service-monitor
    spec:
      endpoints:
        - basicAuth:
            password:
              key: password
              name: '{{ .Release.Name }}-initial-admin-password'
            username:
              key: username
              name: '{{ .Release.Name }}-initial-admin-password'
          interval: 30s
          path: /_prometheus/metrics
          port: http
      selector:
        matchLabels:
          app.kubernetes.io/instance: opensearch
          app.kubernetes.io/name: opensearch
          name: opensearch-headless

you can disable the chart default serviceMonitor and put a custom one in the extraObjects.

[KristijanL]

That is certainly an option. However, at present, the charts ServiceMonitor appears to be non-functional and effectively redundant. If it is intended for use, it will need modifications to make it functional, or detailed documentation should be provided on how to use it effectively, as its current purpose is unclear.

[gabriel-suela]

totally agree

[eyenx]

Related #593 and #594

[muhammeddanish]

Hi Team,

Is it required to install the plugin "prometheus-exporter-plugin-for-opensearch" for the opensearch container to expose the metrics?

May be adding below content to values.yaml, will fix the issue? How we can make this plugin to expose metrics on 9600?

plugins:
  enabled: true
  installList: []
  - prometheus-exporter-plugin-for-opensearch

Thanks,
Danish

[eyenx]

@muhammeddanish yes see #593 We got prom metrics running with that plugin.