Merge pull request #563 from FgForrest/dev

Readiness & liveness probes
FgForrest · May 11, 2024 · f00fa03 · f00fa03
2 parents 5ac0611 + 34874aa
commit f00fa03
Show file tree

Hide file tree

Showing 228 changed files with 4,855 additions and 2,124 deletions.
diff --git a/.idea/inspectionProfiles/Evita_DB_Defaults.xml b/.idea/inspectionProfiles/Evita_DB_Defaults.xml
diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -6,12 +6,16 @@ ARG VERSION=not_set
 ARG RELEASE_DATE=not_set
 ENV EVITA_HOME="/evita"
 ENV EVITA_JAR_NAME="$EVITA_JAR_NAME"
+ENV SYSTEM_API_PORT="5557"
 
 # Labels with dynamic information based on environment variables and current date
 LABEL vendor="FG Forrest, a.s." \
       io.evitadb.version="${VERSION}" \
       io.evitadb.release-date="${RELEASE_DATE}"
 
+HEALTHCHECK --start-period=5m --interval=10s --timeout=2s --retries=3 \
+    CMD curl -f http://localhost:$SYSTEM_API_PORT/system/liveness || exit 1
+
 USER root
 
 # Install bash

diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -27,6 +27,7 @@ set -e
 if [ "$1" = "" ]; then
     set -x
     exec java \
+        -javaagent:${EVITA_HOME}/bin/${EVITA_JAR_NAME} \
         $EVITA_JAVA_OPTS \
         -jar "${EVITA_HOME}/bin/${EVITA_JAR_NAME}" \
         "-DconfigFile=$EVITA_CONFIG_FILE" \

diff --git a/documentation/user/en/operate/monitor.md b/documentation/user/en/operate/monitor.md
@@ -98,6 +98,106 @@ The layout is the `io.evitadb.server.log.AppLogJsonLayout` layout to log app log
 </configuration>
 ```
 
+## Readiness and liveness probes
+
+The evitaDB server provides endpoints for Kubernetes [readiness and liveness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). The liveness probe is also c
+onfigured as [healthcheck](https://docs.docker.com/reference/dockerfile/#healthcheck) by default in our Docker image. By default the health check waits `30s` before it
+starts checking the server health, for larger databases you may need to increase this value using environment variable 
+`HEALTHCHECK_START_DELAY` so that they have enough time to be loaded into memory.
+
+<Note type="warning">
+
+<NoteTitle toggles="false">
+
+##### When you change system API port don't forget to set `SYSTEM_API_PORT` environment variable
+</NoteTitle>
+
+The healthcheck in the Docker image is configured to use the default system API port, which is `5557`. If you change 
+the port, the health check will immediately report an unhealthy container because it won't be able to reach the probe 
+endpoint. You need to specify the new port using the `SYSTEM_API_PORT` environment variable of the Docker container.
+
+</Note>
+
+Both probes are available in the `system` API and are accessible at the following endpoints:
+
+### Readiness probe
+
+```shell
+curl -k "http://localhost:5557/system/readiness" \
+     -H 'Content-Type: application/json'
+```
+
+The probe will return `200 OK` if the server is ready to accept traffic, otherwise it will return `503 Service Unavailable`.
+Probe internally calls all enabled APIs via HTTP call on the server side to check if they are ready to serve traffic. 
+Example response:
+
+```json
+{
+  "status": "READY",
+  "apis": {
+	"rest": "ready",
+	"system": "ready",
+	"graphQL": "ready",
+	"lab": "ready",
+	"observability": "ready",
+	"gRPC": "ready"
+  }
+}
+```
+
+The overall status may be one of the following constants:
+
+<dl>
+    <dt>STARTING</dt>
+    <dd>At least one API is not yet ready.</dd>
+    <dt>READY</dt>
+    <dd>The server is ready to serve traffic.</dd>
+    <dt>STALLING</dt>
+    <dd>At least one API that was ready is not ready anymore.</dd>
+    <dt>SHUTDOWN</dt>
+    <dd>Server is shutting down. None of the APIs are ready.</dd>
+</dl>
+
+Each of the enabled APIs has its own status so that you can see which particular API is not ready in case of `STARTING` 
+or `STALLING` status.
+
+### Liveness probe
+
+```shell
+curl -k "http://localhost:5557/system/liveness" \
+     -H 'Content-Type: application/json'
+```
+
+If the server is healthy, the probe will return `200 OK`. Otherwise, it will return `503 Service Unavailable`.
+Example response:
+
+```json
+{
+  "status": "healthy",
+  "problems": []
+}
+```
+
+If the server is unhealthy, the response will list the problems.
+
+<dl>
+    <dt>MEMORY_SHORTAGE</dt>
+    <dd>Signalized when the consumed memory never goes below 85% of the maximum heap size and the GC tries to free 
+    the old generation at least once. This leads to repeated attempts of expensive old generation GC and pressure on 
+    host CPUs.</dd>
+    <dt>INPUT_QUEUES_OVERLOADED</dt>
+    <dd>Signalized when the input queues are full and the server is not able to process incoming requests. The problem
+	is reported when there is ration of rejected tasks to accepted tasks >= 2. This flag is cleared when the rejection
+	ratio decreases below the specified threshold, which signalizes that server is able to process incoming requests 
+	again.</dd>
+    <dt>JAVA_INTERNAL_ERRORS</dt>
+    <dd>Signaled when there are occurrences of Java internal errors. These errors are usually caused by the server
+    itself and are not related to the client's requests. Java errors signal fatal problems inside the JVM.</dd>
+    <dt>EXTERNAL_API_UNAVAILABLE</dt>
+    <dd>Signalized when the readiness probe signals that at least one external API, that is configured to be enabled
+	doesn't respond to internal HTTP check call.</dd>
+</dl>
+
 ## Client and request identification
 
 In order to monitor which requests each client executes against evitaDB, each client and each request can be identified by

diff --git a/documentation/user/en/operate/run.md b/documentation/user/en/operate/run.md
@@ -324,7 +324,11 @@ you will also see the ports configuration here.
 
 ### Check statuses of the APIs
 
-You can check statuses of the GraphQL and the REST API by using the `curl` command.
+To check the status of all enabled APIs, use the `curl` command and our [readiness probe](./monitor.md#readiness-probe).
+This probe verifies that APIs are ready to serve requests via internal HTTP calls and returns their status in a single 
+response.
+
+You can also check statuses of the GraphQL and the REST API manually by using the `curl` command.
 
 #### GraphQL
 
@@ -377,19 +381,15 @@ You need to replace `__path_to_log_file__` with the path to your logback configu
 The running (and named) container can be stopped and restarted using the following commands:
 
 ```shell
-# shut the container down
-docker stop evitadb
-# bring the container up in frontend mode
-docker start evitadb
-# bring the container up in daemon mode
-docker start evitadb -d
+# restart the running container
+docker restart evitadb
 ```
 
 Alternatively, you can use `docker ps` to get the ID of a running container and restart it using the
 [UUID short identifier](https://docs.docker.com/engine/reference/run/#name---name):
 
 ```shell
-docker start b0c7b140c6a7
+docker restart b0c7b140c6a7
 ```
 
 ### Docker Compose
@@ -418,4 +418,4 @@ All previously documented options for using Docker apply to Docker Compose:
 
 - use [environment variables](#configure-the-evitadb-in-the-container) to configure evitaDB
 - use [volumes](#configure-database-persistent-storage) to set the data folder
-- use [ports](#open--remap-ports) for mapping ports in the docker composition
+- use [ports](#open--remap-ports) for mapping ports in the docker composition
diff --git a/...er/en/query/requirements/examples/facet/facet-groups-negation.evitaql.string.md b/...er/en/query/requirements/examples/facet/facet-groups-negation.evitaql.string.md
@@ -1,15 +1,15 @@
 ```md
 Facet summary:
 	parameterValues: 'ram-memory' [525]:
-		[ ] 'ram-memory-24' (4214) 0
-		[ ] 'ram-memory-64' (4118) 0
-		[ ] 'ram-memory-4' (3820) 0
-		[ ] 'ram-memory-48' (4223) 0
+		[ ] 'ram-memory-24' (4214) -11
+		[ ] 'ram-memory-64' (4118) -107
+		[ ] 'ram-memory-4' (3820) -405
+		[ ] 'ram-memory-48' (4223) -2
 	parameterValues: 'rom-memory' [1527]:
-		[ ] 'rom-memory-64' (267) 0
-		[ ] 'rom-memory-4' (83) 0
-		[ ] 'rom-memory-0-064' (6) 0
-		[ ] '1024' (912) 0
-		[ ] '4094' (20) 0
-		[ ] '2048' (365) 0
+		[ ] 'rom-memory-64' (267) -3958
+		[ ] 'rom-memory-4' (83) -4142
+		[ ] 'rom-memory-0-064' (6) -4219
+		[ ] '1024' (912) -3313
+		[ ] '4094' (20) -4205
+		[ ] '2048' (365) -3860
 ```
diff --git a/...user/en/query/requirements/examples/facet/facet-groups-negation.graphql.json.md b/...user/en/query/requirements/examples/facet/facet-groups-negation.graphql.json.md
@@ -14,8 +14,8 @@
           "requested": false,
           "count": 4214,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -11,
+            "matchCount": 4214,
             "hasSense": true
           },
           "facetEntity": {
@@ -29,8 +29,8 @@
           "requested": false,
           "count": 4118,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -107,
+            "matchCount": 4118,
             "hasSense": true
           },
           "facetEntity": {
@@ -44,8 +44,8 @@
           "requested": false,
           "count": 3820,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -405,
+            "matchCount": 3820,
             "hasSense": true
           },
           "facetEntity": {
@@ -59,8 +59,8 @@
           "requested": false,
           "count": 4223,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -2,
+            "matchCount": 4223,
             "hasSense": true
           },
           "facetEntity": {
@@ -85,8 +85,8 @@
           "requested": false,
           "count": 267,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -3958,
+            "matchCount": 267,
             "hasSense": true
           },
           "facetEntity": {
@@ -100,8 +100,8 @@
           "requested": false,
           "count": 83,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -4142,
+            "matchCount": 83,
             "hasSense": true
           },
           "facetEntity": {
@@ -115,8 +115,8 @@
           "requested": false,
           "count": 6,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -4219,
+            "matchCount": 6,
             "hasSense": true
           },
           "facetEntity": {
@@ -130,8 +130,8 @@
           "requested": false,
           "count": 912,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -3313,
+            "matchCount": 912,
             "hasSense": true
           },
           "facetEntity": {
@@ -145,8 +145,8 @@
           "requested": false,
           "count": 20,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -4205,
+            "matchCount": 20,
             "hasSense": true
           },
           "facetEntity": {
@@ -160,8 +160,8 @@
           "requested": false,
           "count": 365,
           "impact": {
-            "difference": 0,
-            "matchCount": 4225,
+            "difference": -3860,
+            "matchCount": 365,
             "hasSense": true
           },
           "facetEntity": {