Intermittent "response truncated" error on SSL connection

[pitfield]

On an SSL connection, operations against a PostgreSQL server may intermittently generate the following error:

serverError(description: "no data available from server")

and the following log record:

[2019-07-20T21:22:02.275Z Connection-1 warning] Response truncated; no data available

• This issue only occurs on SSL connections.

• This issue only occurs on macOS/iOS clients (not Linux).

• This issue occurs more frequently when processing SELECT results containing large rows (for example, a text or bytea column with a very long value).

• This issue occurs less frequently when the PostgreSQL server is on the same local network as the client, and more frequently when the PostgreSQL server is distant (e.g. on a Linode or EC2).

[pitfield]

The issue appears to be caused by a bug in the socket library used by PostgresClientKit.

I have reported this in Socket.read(...) unexpectedly returns 0 in the IBM-Swift/BlueSSLService repository.

I have a workaround in mind, but will wait a couple of days for the BlueSSLService community to respond to this issue.

[pitfield]

Fixed with a workaround for the underlying BlueSSLService issue. If Socket.read(...) returns 0, then wait 10 ms and try again, timing out after 30 seconds.

[ebsanford]

Any additional info on this issue?

My API is getting "no data available from server" messages under heavy load (requests less than 1 second apart) with the following code snippet. If calls are at a more leisurely pace, the error doesn't occur. So I put the 'if' statement in the 'catch' block to deal with it. It's a hack, but otherwise works. My API is connecting to an AWS hosted database using an SSL connection. I've tried enclosing this block in a serial dispatch queue, but it didn't help. I'm also making use of the connection pool if that's an issue.

do {
   let stmt = try hndl.prepareStatement(text: command)
   defer
      { stmt.close() }

    let curs = try stmt.execute(parameterValues: using)
    curs.close()

    self.Message_ = ""
    return
    }

catch let error as PostgresError {
   // Not expecting data anyway...
   if (error.Message != "no data available from server")
      { self.Message_ = error.Message } else
      { self.Message_ = ""            }

   return
   }

[pitfield]

I haven't heard any updates on the underlying BlueSSLService bug. But the workaround in PostgresClientKit should still work. I am not seeing this issue in my environment, but, as noted above, the original problem reproduced more consistently against a "distant" PostgreSQL server.

Could you supply more info about your environment?

• What release of PostgresClientKit are you using?
• What is the client OS (iOS, macOS, Linux...) and version?
• By "AWS hosted database", do you mean AWS Aurora, RDS, or something else?

What is the value of socketTimeout in ConnectionConfiguration?

With the default socketTimeout of 0, the workaround will retry the socket read for 30 seconds before timing out and raising the "no data available from server" error. (See refillReadBuffer in Connection.swift.). Are you seeing this 30 second pause prior to the error?

You could also try making the BlueSSLService fix that I proposed in that bug. It's a one-line change.

[ebsanford]

Hi David, The quick answers are: - I’m using the “master” branch, so I assume this is the latest. I did an “Update To Latest Package Versions” two days ago. - The client boxes are 2018 mac Minis with 6-core i7 and 32GB memory running Venture 13.2. - I’m going through an AWS RDS merchant, ElephantSQL (elephantsql.com <http://elephantsql.com/>), with the database being hosted at "amazon-web-services::us-east-1" The mac Minis are hosted in a data center in Atlanta, and connection times to the database is typically 500ms or less. I just looked and my code and I’ve have: conn.socketTimeout = 10 conn.ssl = false And I’m seeing the socket error after approximately 10 seconds. My experience has been that it doesn’t matter if ’ssl’ is set or not. After lunch I’ll try changing it to 0 and see what happens. —Barry

…

On Feb 15, 2023, at 11:10 AM, David Pitfield [dbp] ***@***.***> wrote: I haven't heard any updates on the underlying BlueSSLService bug <Kitura/BlueSSLService#79>. But the workaround in PostgresClientKit should still work. I am not seeing this issue in my environment, but, as noted above, the original problem reproduced more consistently against a "distant" PostgreSQL server. Could you supply more info about your environment? What release of PostgresClientKit are you using? What is the client OS (iOS, macOS, Linux...) and version? By "AWS hosted database", do you mean AWS Aurora, RDS, or something else? What is the value of socketTimeout in ConnectionConfiguration? With the default socketTimeout of 0, the workaround will retry the socket read for 30 seconds before timing out and raising the "no data available from server" error. (See refillReadBuffer in Connection.swift.). Are you seeing this 30 second pause prior to the error? You could also try making the BlueSSLService fix that I proposed in that bug <Kitura/BlueSSLService#79>. It's a one-line change. — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTRL5EMJ2AC6FBDLA5ENOLWXUEWZANCNFSM4IFPZ3GA>. You are receiving this because you commented.

[pitfield]

From #41:

We're almost 3 hours into the change of timeout value back to 0, and the behavior has be much, much better. I’ll know more after 24 hours or so. But for now, it looks like the correct change.

Glad to hear it's working better. Since you were encountering this error for both SSL and non-SSL connections, it's probably unrelated to the BlueSSLService bug.

So the question is: why does changing the socketTimeout seem to help? I suppose there could be some other BlueSocket issue that's masked by the PostgresClientKit workaround tied to socketTimeout.

But, at this point, it seems more likely that a 10 second socket timeout is simply too short for your environment. Any SQL statement that takes more than 10 seconds to execute would time out, producing the "no data available from server" error. Whereas a socketTimeout of 0 means "wait forever".

Since what you are seeing isn't related to the SSL issue originally described by this bug, I'm going to re-close it. Feel free to create a new issue if you are still getting the "no data available from server" error.