Enhance RPC Health Status

[sreuland]

What problem does your feature solve?

• RPC's getHealth response doesn't represent the actual run time status levels that can be present on RPC.
• RPC's getHealth requires POST and json-rpc request payload, and parsing of the json by client afterwards to determine status, which puts more lifting on the operator side to support doing the health checks, in general terms, health endpoints are usually exposed as HTTP GET endpoints to simplify what the operator side needs to perform to determine the status from response, i.e. just check the HTTP response code.

What would you like to see?

note - these requirements were elided from design discussion on https://github.com/stellar/kube/pull/2098#pullrequestreview-2005913742

• RPC provides a /health/{qos_level} endpoint that is retrievable via HTTP GET.
• RPC provides the http service that publishes this endpoint immediately after the rpc process is started, no delayed start of the http service.
• RPC provides a 503 or 200 HTTP response code in response to a HTTP GET /health/{qos_level}, to indicate the level is active or not.

The new endpoint supports a notion of QoS levels for representing the different potential run time states that RPC can be in:

level 1 - service is completely unhealthy, the process is running but ingestion isn't stable yet to network, unable to process requests.
level 2 - service is running and forward ingestion with network is happening, data retention window is not fully caught up yet, but can process some json-rpc request endpoints.
level 3 - service is running, forward ingestion with network is happening, data retention window is full, all rpc request endpoints are up.

What alternatives are there?

use the current json-rpc getHealth

[overcat]

I strongly support this feature. When I was configuring failover for sorobanrpc.com, I had to write an additional simple API service to proxy the getHealth interface, and then have the health checker access this API service. If soroban-rpc supported direct GET access to the getHealth interface, I wouldn't need to an extra API service.

(I'm unsure how many health checkers support posting JSON body during their health checks.)

[mollykarcher]

After @overcat's comment, I'm realizing that the link referenced in the issue description is to an internal repository, which defines the k8s manifests for SDF's service deployments. So it's not broadly public what we do, nor do we public any recommendations about this anywhere in our docs (which we should fix also!). We currently deploy it via k8s, and define the readinessProbe as follows:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - |
        curl -s --location --request POST 'http://127.0.0.1:8000/' \
          --header 'Content-Type: application/json' \
          --data-raw '{
            "jsonrpc": "2.0",
            "id": 10235,
            "method": "getHealth"
          }' | jq -es 'if (. | length) == 0 then null else .[0] end | .result | .status == "healthy" and (.latestLedger - .oldestLedger >= (.ledgerRetentionWindow - 10))' > /dev/null;
  failureThreshold: 1
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 2   

So we are effectively parsing out the ledger range in order to determine health of the instance.

[sreuland]

additional scope to consider for QoS readiness levels are to capture 'live' simulation and 'proof' simulation capability.