Understand histogram percentile data generated with TFB

[kusumachalasani]

Benchmark : TFB

How the benchmark runs? :
Benchmark is run for 3 iterations. Each iteration is run for 6 minutes in total in which first 3 mins is considered as warmup and the next 3 mins for measurement. Before the start of each iteration, the pod is deployed again.
TFB benchmark is run with 1 instance of application server (Quarkus) in openshift and standalone postgres DB on RHEL.

Queries used:
Below queries are used to collect the percentile data of application server are below. Because it is a single instance, I collected the data using histogram_quantile and the basic query to understand how the data looks like.

query1=http_server_requests_seconds{status="200",uri="/db",quantile="1.0",}
query2=histogram_quantile(1.0, rate(http_server_requests_seconds_bucket{status="200",uri="/db"}[3m]))'
query3=http_server_requests_seconds_max{status="200",uri="/db"}

percentile data:
Below is the percentile data collected using query1 and query2 and the max server_request time using query3 for 3 iterations.
All the data mentioned below is in seconds.
I picked the max_value during the run for the max_server_request ( query3 ) as this value resets for every 2 mins.

query1 percentile data				query2 histogram percentile data				query3 max_server_request
95p	99.999p	100p		95p	99.999p	100p
0.002	0.080	0.088		0.002	0.089	30.000		0.192
0.003	0.088	0.096		0.003	0.089	12.885		0.095
0.003	0.084	0.096		0.003	0.089	30.000		0.174

From the above data, 95p and 99.999p from query1 and query2 are almost at the same level. But 100th percentile in histogram gives a very high data point.
From the multiple runs I did, atleast one iteration shows 30 secs as 100th percentile with query2.
The max server_request time is < 1 sec in all these cases.

How can histogram 100th percentile data (query2) gives 30sec if the max_server_request time is < 1 sec ?
Any thoughts of what would have caused this ?

@dinogun @johnaohara @ebullient

[johnaohara]

@kusumachalasani do you have a link to the '/db' endpoint src?

[kusumachalasani]

@kusumachalasani do you have a link to the '/db' endpoint src?

https://github.com/kusumachalasani/FrameworkBenchmarks/blob/tfb_mf/frameworks/Java/quarkus/resteasy-hibernate/src/main/java/io/quarkus/benchmark/resource/DbResource.java