[metrics-generator] Add flexibility to exclude externalLabels

[yehaotian]

Is your feature request related to a problem? Please describe.
Current metrics generator will add __metrics_gen_instance external label with hostname, which will significantly introduce additional cardinality.

	hostname, _ := os.Hostname()
	externalLabels["__metrics_gen_instance"] = hostname

Describe the solution you'd like
Have a config to exclude such external labels to just include needed metrics labels/dimensions.

Describe alternatives you've considered
N/A

Additional context
N/A

[javiermolinar]

Hi, the reason for adding __metrics_gen_instance is to ensure that the series generated by each metrics-generator are unique between them. I'm not sure if it makes sense to exclude it.

[yehaotian]

Hi @javiermolinar, it would be better to let the user decide if this level of uniqueness is needed. For example when we have millions of time series and hundreds of metrics generator, as the storage/infra owner, I would like to exclude all unnecessary labels to reduce overall cardinality and improve query efficiency. At this moment I dont see any customer using __metrics_gen_instance label.

You can always have these external labels included as default, and provide a config which would take a list of strings that could be excluded from the externalLabels,
does this make sense?

[javiermolinar]

Fair enough! We're always open to contributions. If you'd like, feel free to open a pull request so we can explore this feature further

[yehaotian]

We will use streaming aggregation to resolve this for now, let me know if anyone works on this