The nodegraph-provider works with the Grafana datasource "Node Graph API Datasource Plugin", https://github.com/exaco/nodegraph-api-plugin.
The nodegraph-provider use a RedisGraph, https://github.com/RedisGraph/RedisGraph, to keep a graph model of what to expose to the Node Graph API datasource.
To create a model the nodegraph-provider expose api endpoints to create and update nodes and edges. The attributes that can be used is limited to the attributes of the Grafana Node Graph panel plugin, https://grafana.com/docs/grafana/latest/visualizations/node-graph/.
For available configuration check out the config.yaml. All configuration can be overridden by environment variables
except the graph_schemas.
Prefix all configuration varaibles with NODEGRAPH_PROVIDER_ and replace . with _, e.g. set the redis server:
export NODEGRAPH_PROVIDER_REDIS_HOST=my_redis_host
The graph model must be compatible to the data model for nodes and edges in the Node Graph panel. The attributes allowed is specified at https://grafana.com/docs/grafana/latest/visualizations/node-graph/#data-api.
In the nodegraph-provider each instance schema of a model is described in the configuration file, e.g.
graph_schemas:
micro:
node_fields:
- field_name: "id"
type: "string"
- field_name: "title"
type: "string"
- field_name: "subTitle"
type: "string"
- field_name: "mainStat"
type: "string"
displayName: "CPU"
- field_name: "secondaryStat"
type: "number"
- field_name: "arc__failed"
type: "number"
color: "red"
- field_name: "arc__passed"
type: "number"
color: "green"
- field_name: "detail__role"
type: "string"
displayName: "Role"
edge_fields:
- field_name: "id"
type: "string"
- field_name: "source"
type: "string"
- field_name: "target"
type: "string"
- field_name: "mainStat"
type: "number"
- field_name: "secondaryStat"
type: "number"
- field_name: "detail__traffic"
type: "string"The graph_schemas can have multiple named schema definitions. Above is a schema called micro.
The name is used as a key in all API accesses to support multiple instances. E.g. when setting up the URL in the
Node Graph API Datasource Plugin we define it as part of the path, http://localhost:9393/micro.
This enables nodegraph-provider to support multiple data sources in a single instance.
The schema defines the different fields and the type, string or number, that are allowed for the specific schema.
All endpoint expects the header "Content-Type" set to "application/json"
GET /{graph_schema}/api/graph/data?query=<WHERE clause>
The above query parameter is optional, If empty or not set all nodes and edges are returned. If set, it must be a valid Cypher statement, https://neo4j.com/docs/cypher-manual/current/clauses/where/, but with some limitation documented in https://neo4j.com/docs/cypher-manual/current/clauses/where/. For example the Redis Cypher do not support regular expression.
The
WHEREshould not be part of the query string, and the string must be urlencoded.
The WHERE clause can only operate where source nodes, edges and target nodes are returned, e.g.
the statement edge.mainStat > 0 will be in encoded url:
query=edge.mainStat%20%3E%200
The source nodes are referred to as source, target nodes are referred to as target and edges are
referred to as edge
GET /{graph_schema}/api/graph/fields
GET /{graph_schema}/api/health
POST /api/nodes/{graph_schema}
GET /api/nodes/{graph_schema}/{id}
PUT /api/nodes/{graph_schema}/{id}
DELETE /api/nodes/{graph_schema}/{id}
POST /api/edges/{graph_schema}
GET /api/edges/{graph_schema}/{source_id}/{target_id}
PUT /api/edges/{graph_schema}/{source_id}/{target_id}
DELETE /api/edges/{graph_schema}/{source_id}/{target_id}
POST operations expect a json body. The content should only include the field names in the graph schema.
Node example:
{
"id": "lb-01",
"title": "lb01",
"subTitle": "instance:#01",
"arc__failed": 0.0,
"arc__passed": 0.0,
"detail__role": "load",
"mainStat": 0.0,
"secondaryStat": 0.0
}PUT takes query parameters with the fields to be updated and id as a path parameter.
DELETE and GET do not have any query parameters.
curl -s -i -H "Content-Type: application/json" -X PUT "localhost:9393/api/nodes/micro/lb-01?arc__failed=0.1?arc__passed=0.9"The api endpoints will operate on a complete graph. The POST will first delete before create. For a client that have the full "picture" of the graph model, this is the most effective endpoint to use.
POST /api/graphs/{graph_schema}
DELETE /api/graphs/{graph_schema}
The graph endpoints in singulars are deprecated:
POST /api/graphs/{graph_schema}
DELETE /api/graphs/{graph_schema}
The POST endpoint require a body of a list of nodes and edges, e.g.
{
"nodes": [
{
"id": "lb-1",
"title": "lb",
"subTitle": "instance:#01",
"detail__role": "load",
"arc__failed": 0,
"arc__passed": 1,
"mainStat": 0,
"secondaryStat": 0
},
....
],
"edges": [
{
"source": "lb-1",
"target": "cust-svc-1",
"mainStat": 0,
"secondaryStat": 0
},
....
]
}Please see the examples/graph.json and examples/setup_graph.sh for a
complete example.
Internal metrics are exposed on endpoint /metrics. Go metrics and nodegraph-provider specific metrics are exposed
on the endpoint in Prometheus format.
The following api are deprecated:
POST /api/controller/{graph_schema}/delete-all
- 200
- Successful - PUT, DELETE, GET
- 201
- Create successful - POST
- 400
- Not a valid json - POST
- Not a valid field name - POST, PUT
- 404
- Object do not exist - PUT, DELETE, GET
- 409
- Object already exists - POST
- 415
- Invalid value for header "Content-Type"
Install Grafana and datasource https://github.com/exaco/nodegraph-api-plugin. Start grafana to allow unsigned plugins
export GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=hamedkarbasi93-nodegraphapi-datasource ; ./bin/grafana-serve
Start redis with module RedisGraph. Simple way just use docker.
docker run -p 6379:6379 redislabs/redismod
Start nodegraph-provider, configuration file is default config.yml
go build -o build/nodegraph-provider *.go
./build/nodegraph-provider
To see all options
./build/nodegraph-provider -h
Create a data source in Grafana with the nodegraph-api-plugin and set url to http://localhost:9393/micro.
Name it to Micro.
Create a dashboard and select the "Node Graph" plugin. Select the data source Micro.
Load the simple graph model by create nodes and edges:
./examples/setup_test.sh
Or run the example to create a complete graph:
./examples/setup_graph.sh
In Grafana you should now see this graph.
Add a new node with id cust-svc-2
curl -s -i -H "Content-Type: application/json" -X POST localhost:9393/api/nodes/micro -d @examples/node_create.json
Create an edge between lb-1 and cust-svc-3
curl -s -i -H "Content-Type: application/json" -X POST localhost:9393/api/edges/micro -d @examples/edge_create.json
Update metrics on lb-1
curl -s -i -H "Content-Type: application/json" -X PUT "localhost:9393/api/nodes/micro/book-svc-1?mainStat=$RANDOM&secondaryStat=$RANDOM&arc__failed=0.1&arc__passed=0.9"
Update metrics on edge between lb-1 to `cust-svc-1
curl -s -i -H "Content-Type: application/json" -X PUT "localhost:9393/api/edges/micro/lb-1/cust-svc-1?mainStat=$RANDOM&secondaryStat=$RANDOM"
You should now see something like this.
Delete the whole graph
curl -s -i -H "Content-Type: application/json" -X DELETE "localhost:9393/api/graph/micro"
Use the Dockerfile in the root directory of the project
docker build --tag nodegraph-provider .
To run the image a config file must be mounted
docker run -p 9393:9393 -v $(pwd)/config_tempo.yaml:/app/config.yaml nodegraph-provider
Environment variables can be set to override defaults and config file
docker run -p 9393:9393 -v $(pwd)/config_tempo.yaml:/app/config.yaml -e NODEGRAPH_PROVIDER_PORT=9393 -e NODEGRAPH_PROVIDER_REDIS_HOST=localhost -e NODEGRAPH_PROVIDER_REDIS_PORT=6379 nodegraph-provider
Checkout the tempo_trace_aggregation project where nodegraph-provider is used to create a "dynamic service map" based on aggregated traces stored in Grafana Tempo.