Allow exporting JetStream metadata as prometheus labels

[jkraml-staffbase]

What motivated this proposal?

We would like to route NATS-related alerts to responsible teams, but would not want teams to have to configure alert routing (complex and error prone). Instead, we'd like to enable them to add suitable metadata to JetStream streams/consumers (i.e. "team", "domain", "product", …) and have generic routing rules based on those.

What is the proposed change?

Provide configuration options to allow exporting select metadata values as Prometheus labels.

Who benefits from this change?

Anyone who'd like to group Prometheus metrics by some metadata like product/domain/etc. This is common based on e.g. k8s labels.

What alternatives have you evaluated?

• Tedious manual config of alert routing
• Building an additional tool visualize JetStream resource ownership (does not actually help with alert routing)

[jkraml-staffbase]

I took the liberty of providing an example implementation.

[jkraml-staffbase]

One might discuss whether stream and consumer labels should coalesce or not.

That is, if both stream and consumer have a metadata value for "product", should the resulting Prometheus metrics have two labels for this (e.g. stream_meta_product and consumer_meta_product) or should they be combined into one (e.g. meta_product, assuming the consumer label "wins")

[jkraml-staffbase]

Another discussion point:

The exported values should be explicitly defined so as to avoid exporting all metadata as labels and accidentally creating high metric cardinality.

[ripienaar]

Agree with either specifically defined or prefix based metadata selection,

We're adding a bunch of metadata and I have a monitoring solution - that I am considering adding to the exporters - that does health checks on a per resource based based on metadata and it would become a lot

    "metadata": {
      "_nats.server.api_level": "1",
      "_nats.server.version": "2.11.0-dev",
      "io.nats.monitor.enabled": "1",
      "io.nats.monitor.lag-critical": "100",
      "io.nats.monitor.msgs-critical": "500",
      "io.nats.monitor.msgs-warn": "999",
      "io.nats.monitor.peer-expect": "3",
      "io.nats.monitor.seen-critical": "5m"
    },

[jkraml-staffbase]

The linked PR currently allows defining stream and consumer metadata fields which are separately exported as metrics.
e.g. running with --jsz_consumer_meta_keys=foo --jsz_stream_meta_keys=bar will get you metrics with labels stream_meta_bar="…" consumer_meta_foo="…"