Relevant data collection after load/perf test

[trepel]

Summary

Document what data to collect after load test and how to do it. Also how to produce a report so that the two load test runs are easy to compare.

For this we might use reporting feature of the tooling that will be used for load/perf testing so it is partially dependent on #506 but on top of that we also need to collect other data, most likely from Prometheus instance(s) where various interesting metrics are scraped by default. The outcome of this ticket might be a set of promQL queries that collects required data.

Istio standard metrics:
https://istio.io/latest/docs/reference/config/metrics/

k8s controller runtime metrics:
https://book.kubebuilder.io/reference/metrics-reference

What to collect?

• requests per second, throughput, latency
• vCPU/memory consumption of pods of services identified as relevant
• it would also be good to know what alerts were pending/firing during the load/pert test

Related Epic

#132