ReadException "too many concurrent streams"

[visox]

Hi i am running a service which makes lots of calls to some other service, lets say 10 calls per second.

The service runs properly but eventually there are periods when it starts to crash with

sttp.client3.SttpClientException$ReadException: Exception when sending request: POST https://xxxxx
	at sttp.client3.SttpClientExceptionExtensions.defaultExceptionToSttpClientException(SttpClientExceptionExtensions.scala:25)
	at sttp.client3.SttpClientExceptionExtensions.defaultExceptionToSttpClientException$(SttpClientExceptionExtensions.scala:9)
	at sttp.client3.SttpClientException$.defaultExceptionToSttpClientException(SttpClientException.scala:24)
	at sttp.client3.HttpClientAsyncBackend.adjustExceptions$$anonfun$1(HttpClientAsyncBackend.scala:147)
	at sttp.client3.SttpClientException$$anon$1.applyOrElse(SttpClientException.scala:35)
	at sttp.client3.SttpClientException$$anon$1.applyOrElse(SttpClientException.scala:34)
	at zio.ZIO.tryRescue$1$$anonfun$1(ZIO.scala:359)
	at scala.util.Either.fold(Either.scala:190)
	at zio.ZIO.tryRescue$1(ZIO.scala:359)
	at zio.ZIO.catchSome$$anonfun$1(ZIO.scala:361)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1121)
	at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
	at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
	at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
	at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:476)
Caused by: java.io.IOException: too many concurrent streams
	at java.net.http/jdk.internal.net.http.Http2Connection.reserveStream(Http2Connection.java:484)
	at java.net.http/jdk.internal.net.http.Http2ClientImpl.getConnectionFor(Http2ClientImpl.java:106)
	at java.net.http/jdk.internal.net.http.ExchangeImpl.get(ExchangeImpl.java:94)
	at java.net.http/jdk.internal.net.http.Exchange.establishExchange(Exchange.java:369)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:553)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:406)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsync(Exchange.java:398)
	at java.net.http/jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:409)
	at java.net.http/jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:342)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)

  lazy val sttpV = "3.8.+"

        "com.softwaremill.sttp.client3" %% "core" % sttpV,
      "com.softwaremill.sttp.client3" %% "zio" % sttpV,

Also its a POST request with some timeout a json body and some auth headers

in sttp code i do see

case e: java.io.IOException                   => Some(new ReadException(request, e))

and also from the stack trace one can see its an IO exception with too many concurrent streams

Any obvious solution ?

[kciesielski]

@visox I'd like to look into this, if it's still relevant. Does the problem still occur?

[hochgi]

@kciesielski I stumbled on this as well.
Also with zio backend. (sttp client3 version: 3.9.1)

Caused by: sttp.client3.SttpClientException$ReadException: Exception when sending request: GET https://<REDACTED>
	at sttp.client3.SttpClientExceptionExtensions.defaultExceptionToSttpClientException(SttpClientExceptionExtensions.scala:25)
	at sttp.client3.SttpClientExceptionExtensions.defaultExceptionToSttpClientException$(SttpClientExceptionExtensions.scala:11)
	at sttp.client3.SttpClientException$.defaultExceptionToSttpClientException(SttpClientException.scala:24)
	at sttp.client3.HttpClientAsyncBackend.$anonfun$adjustExceptions$1(HttpClientAsyncBackend.scala:147)
	at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:35)
	at sttp.client3.SttpClientException$$anonfun$adjustExceptions$1.applyOrElse(SttpClientException.scala:34)
	at zio.ZIO.$anonfun$catchSome$1(ZIO.scala:359)
	at scala.util.Either.fold(Either.scala:190)
	at zio.ZIO.tryRescue$1(ZIO.scala:359)
	at zio.ZIO.$anonfun$catchSome$4(ZIO.scala:361)
	... 6 more
Caused by: java.io.IOException: too many concurrent streams
	at java.net.http/jdk.internal.net.http.Http2Connection.reserveStream(Http2Connection.java:449)
	at java.net.http/jdk.internal.net.http.Http2ClientImpl.getConnectionFor(Http2ClientImpl.java:104)
	at java.net.http/jdk.internal.net.http.ExchangeImpl.get(ExchangeImpl.java:93)
	at java.net.http/jdk.internal.net.http.Exchange.establishExchange(Exchange.java:343)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:475)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:380)
	at java.net.http/jdk.internal.net.http.Exchange.responseAsync(Exchange.java:372)
	at java.net.http/jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:408)
	at java.net.http/jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:341)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
	... 1 more

[hochgi]

@kciesielski & @visox a workaround for me was just to retry:

sttpClient.send(httpRequest).retry(Schedule.fibonacci(1.second).whileOutput(60.seconds.>))

I'm guessing in a burst of too many requests this may happen, so eventually it'll succeed if we retry enough.