High latency due to dns refresh on https every x minutes

[waszak]

Elastic.Clients.Elasticsearch version: 8.15.8

Elasticsearch version: 8.14.0

.NET runtime version: 8.x

Operating system version: Linux

Description of the problem including expected versus actual behavior:
Depending on parameter DnsRefreshTimeout we see longer requests every X minutes.
I attached example with settings 10 minutes and 5 minutes so there is strong correlation

The issue doesn't occure on HTTP connection its only on HTTPS.
Also this is new index/cluster so there is not many request to it yet.

I would provide more diagnostic details if you tell me what to check. All I know its caused by Https and DNS refresh
but i just can't find any proof its due to client or some network configuration issue. ever
We had no issue like that with older elastic versions like 6

Steps to reproduce:
0. Its hard to reproduce

1. There should be enabled https
2. DnsRefreshTimeout set to some value like 5 minutes
3. .Run the same request in loop for idk 1h and you will see huge spikes every 5 minutes

Expected behavior
No timeouts/long requests during queries to elatic

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

[flobernd]

Hi @waszak, I'll transfer this issue over to the https://github.com/elastic/elastic-transport-net repository.

[flobernd]

When the DNS refresh happens, the underlaying HttpMessageHandler is recreated (to make sure any subsequent request is correctly routed to the new server after a potential DNS update). This involves establishing a new connection and a higher latency is definitely expected.

HTTPS introduces massive overhead over plain HTTP as it has to perform a TLS handshake. This again involves cryptographic operations which are not cheap.

Another factor could be a proxy that intercepts SSL traffic for deep packet inspection or any firewall in general.

Is this latency spike problematic for your concrete usecase? If I read your (second) graph correctly, the duration for a complete request is still below 350ms.

[waszak]

@flobernd the 2 seconds tiemout are from Kubernetes Cluster k8s and this cluster in theory should be faster and is faster than AKS.
With exception of issue of https with elastic. On k8s which is close to us we have issue with this high latenecy.
So i think its something with k8s but we are still testing this.

From Azure Kubernetes we got the better times 350ms.