Prometheus Setup

Prometheus Monitoring Setup Guide

Introduction

Quarkchain now has built-in monitoring tools implemented through Prometheus.

Currently you can use it to monitor the total balance of a specific token as well as the balance of each shard.

You can run the prometheus client with an existing cluster with JSONRPC port enabled, or start a cluster with prometheus enabled to monitor itself.

We recommend following the instruction wiki to start clusters using docker.

Run Prometheus Client to Monitor Existing Cluster

Cluster Env Setup

• Update the cluster with latest pyquarkchain version
• Setup port forwarding for UDP/TCP 38291 and UDP/TCP 8000. Port 8000 is the default prometheus client exposing port, your can change to other available port for your need but remember to change all relative port settings in following steps.

Script Env Setup

To run a prometheus client, you should have the python package prometheus_client installed, we recommend the latest version.

pip install prometheus_client

Then you can run the script pyquarkchain/quarkchain/cluster/prom.py with following options:

• --enable_count_balance: Enable the prometheus client to monitor total balance.
• --tokens: Specified the tokens to be monitored, separated by comma.
• --interval: Time gap between two consecutive queries for monitoring, lower interval may cause performance degradation, default is 30s.
• --host: Socket of the cluster, as well as the JSONRPC port.
• --port: Port where prometheus expose the data to localhost, default is 8000.

The following command will run the prometheus client to monitor the socket "cluster.address.com:39291" for token "QKC" and "BTC", the interval will be 60s and the port is 8000 by default.

python prom.py --host "cluster.address.com:38291" --enable_count_balance --tokens "QKC,BTC" --interval 60

Then you can open the address "localhost:8000" (localhost should be changed to the actual address where the script is running, and the port should change to the port you actually use) to see the metrics exposed by the prometheus.

Start a Cluster with Prometheus client

If you wanna start a cluster with a prometheus client, your should do some extra steps before you run the run_cluster.sh file in the instruction wiki

1. Edit cluster configuration file

Open mainnet/singularity/cluster_config_template.json and add the prometheus configuration, your can change to your own parameter value, remember to add comma after previous configuration:

  ...
  },
  "PROMETHEUS": {
    "INTERVAL": 30,
    "TOKENS": "QKC,BTC",
    "PORT": 8000
  }
}

2. Edit the shell script

Open the run_cluster.sh script and add the prometheus parameters:

#!/bin/bash

${PYTHON:=python3} quarkchain/cluster/cluster.py --enable_prometheus --enable_count_balance --cluster_config $(realpath mainnet/singularity/cluster_config_template${QKC_CONFIG_EXT:=}.json) "$@"

Now the cluster has started with prometheus monitoring itself for the balance counting. If you wanna run a prometheus client for other cluster in the same time, remember using different port number to avoid confiliction.

Prometheus Server and Grafana Server

As the metrics had been exposed to the port you specified, you can now run a prometheus server to fetch the data following the official instructions, and setup grafana to visualize your result as well.