Allow others OTEL deployment patterns

[jvanz]

@kubewarden/kubewarden-developers , during the tests I've figured out that our current approach to deploy OTEL collector using sidecar, prevents us from using the Kubernetes Attributes processor. Which pretty important to fetch metadata from the cluster and allow the association of the metrics/traces with the resources running in the cluster:

The Kubernetes Attributes Processor is one of the most important components for a collector running in Kubernetes. Any collector receiving application data should use it. Because it adds Kubernetes context to your telemetry, the Kubernetes Attributes Processor lets you correlate your application’s traces, metrics, and logs signals with your Kubernetes telemetry, such as pod metrics and traces.

This is configuration required for Stackstate as show in the documentation example. Therefore, I think we need a second iteration on this to improve our OTEL deployment allowing us to enable this processor available only in DaemonSet and Deployment deployments mode. This may require changes in policy server and Kubewarden controller because these components always consider the collector accessible in the same pod.

Originally posted by @jvanz in #581 (comment)

---

During a call the team decided to change the controller and policy server to allow the users send data to Otel collectors running somewhere else. Not only for the sidecar in the pod. Therefore, Kubewarden stack can take advantage of the Otel stack already running in the user environment.

Acceptance criteria

• Update the controller code to allow it to send data to OTEL collector running somewhere else. Adds a configuration to allow defining the OTEL collector listening endpoint. Allowing the removal of the sidecar when necessary
• Update the policy server code to allow it to send data to OTEL collector running somewhere else. Adds a configuration to allow defining the OTEL collector listening endpoint. Allowing the removal of the sidecar when necessary

[flavio]

Do we need to implement that for 1.19?

[jvanz]

@flavio, I don't think this is a blocker for the 1.19 because we can send the current Kubewarden metrics for the Stackstate collector. But it misses some k8s metadata that can be useful to filtering and association with the resources. Therefore, this should be addressed to for a better Stackstate collaboration in the near future.

[jvanz]

Just sharing an ideia on how to address this issue, we could have a Otel collector deployment in the Kubewarden namespace to receive the data from the sidecar collectors. And this deployment otel collector send the data to jaeger, prometheus and stackstate. This approach will not require changes in the policy-server and Kubewarden controller