kprom: Support for multiple clients

[lukasrynt]

Hi,

I'm trying to use the kprom plugin with two producers running in the same program to expose separate metrics for each producer. I expected the WithClientLabel configuration option to allow this, but when I register both producers via the hooks option to the same metric, only the most recent producer gets registered. Here's an example:

metrics := kprom.NewMetrics("test", kprom.WithClientLabel())

// Prepare producers
primary, err := kgo.NewClient(
		kgo.SeedBrokers(brokers...),
		kgo.ClientID("primary"),
		kgo.WithHooks(metrics),
	)
...
secondary, err := kgo.NewClient(
		kgo.SeedBrokers(brokers...),
		kgo.ClientID("secondary"),
		kgo.WithHooks(metrics),
	)
...
mux := nethttp.NewServeMux()
mux.Handle("/metrics", metrics.Handler())

// Run the actual HTTP server with metrics
...
// Only metrics about secondary client are seen (although I suspect it's sum of both)

However, I believe that supporting separate metrics for multiple consumers/producers within a single program is a valid use case for this plugin, especially since kgo allows multiple client instances in the same application. I would love to see this use case supported, I can even create a pull request for it if you'll find this issue valid.

[twmb]

I'm open to this, what's the proposed API?

[lukasrynt]

Thanks for the swift response. I created a PR for this. I would retain the API, feel free to correct me if you don't like it.

[mehranmeidani]

@lukasrynt Why do you want to pass the same kprom.NewMetrics instance to both client? Why not create a new one and register both to a single prometheus registry and serve metrics from there? We are doing this in our application with multiple clients and it's working very well.

[lukasrynt]

@mehranmeidani Thanks for the suggestion. I already tried passing the metrics to one common Prometheus registry, however the solution felt a bit hacky. Since kprom already creates its own registry, one would have to create an extra structure that implements the Collector interface, that another prometheus.Registry can register.

I might be missing something though. Would you be so kind as to provide some example code with usage of two kprom metrics instances? Thanks a lot!

[mehranmeidani]

This is how we do it:

func newClient(ctx context.Context, opt ...kgo.Opt) (*kgo.Client, error) {
	metrics := kprom.NewMetrics("kafka_stats",
		kprom.Registerer(prometheus.DefaultRegisterer),
		kprom.WithClientLabel(),
	)
        KgoClient, err := kgo.NewClient(opt...)
        ...
}

We create multiple clients by calling the newClient function, and then, in the main function:

r := http.NewServeMux()
r.Handle("/metrics", promhttp.Handler())

[lukasrynt]

@mehranmeidani Thanks for sharing your example; the Registerer option is quite handy and I missed that. Building on your approach, though, I'd set up two clients with different client IDs, that both write to same topic. This causes the program to panic due to duplicate metrics collector registration attempted.

Clients definition:

primaryCl := newClient(kgo.SeedBrokers(brokers...), kgo.ClientID("primary"))
secondaryCl := newClient(kgo.SeedBrokers(brokers...), kgo.ClientID("secondary"))
...

func newClient(opts ...kgo.Opt) *kgo.Client {
	metrics := kprom.NewMetrics("kafka_stats",
		kprom.Registerer(prometheus.DefaultRegisterer),
		kprom.WithClientLabel(),
	)
	opts = append(opts, kgo.WithHooks(metrics))
	cl, err := kgo.NewClient(opts...)
	if err != nil {
		panic(err)
	}
	return cl
}

This actually makes sense, as there's no way to distinguish between these metrics in this setup. I think that your solution is good as long as each client writes to different topic. In my case, though, both clients are writing to the same topic, but use different brokers, which isn't enough to differentiate the metrics.

This is the source of my problems and the need to create this issue. However I agree, that the related PR could be improved with the usage of Registerer option in mind.