Errors with hub/spoke Docker deployment of InfluxDB and Scrutiny

[nosoop4u]

Hoping this is the right place for this; apologies if not...

I am attempting to deploy using the hub/spoke method described here, given that I have multiple servers I want to monitor, so I want to deploy multiple collectors.

I deployed the influxdb container as follows (password and token replaced):

docker run -d -p 8086:8086 \
  -v `pwd`/influxdb2:/var/lib/influxdb2 \
  -e DOCKER_INFLUXDB_INIT_MODE=setup \
  -e DOCKER_INFLUXDB_INIT_USERNAME=admin \
  -e DOCKER_INFLUXDB_INIT_PASSWORD=####### \
  -e DOCKER_INFLUXDB_INIT_ORG=homelab \
  -e DOCKER_INFLUXDB_INIT_BUCKET=scrutiny \
  -e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=######## \
  --name scrutiny-influxdb \
  --restart=always \
  influxdb:2.2

This looks good... I can log into the Influx UI by pointing a browser to port 8086 on my docker host, using the credentials I specified above...

I populated scrutiny/scrutiny.yaml as follows based on the example here

version: 1
web:
  listen:
    host: localhost
    port: 8080
    basepath: ''
  database:
    location: /opt/scrutiny/config/scrutiny.db
  src:
    frontend:
      path: /opt/scrutiny/web
  influxdb:
    scheme: 'http'
    host: docker1.<my.domain>
    port: 8086
    token: '####'
    org: 'homelab'
    bucket: 'scrutiny'
    retention_policy: true
    insecure_skip_verify: true

Note I have the actual docker host hostname as the "host" for influxdb, but "localhost" as the "host" for scrutiny-web... This got me the furthest, as things just didn't work otherwise (see additional details below).

The next step, starting the Scrutiny web, is where I'm hung up. I run:

docker run -d -p 8080:8080 \
   -v $(pwd)/scrutiny:/opt/scrutiny/config \
   --name scrutiny-web \
   --restart=always \
   ghcr.io/analogj/scrutiny:master-web

I can see that the listener starts up on port 8080...

root@docker1:~# lsof -i
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
...
docker-pr 5284 root    4u  IPv4 1601824      0t0  TCP *:http-alt (LISTEN)
docker-pr 5290 root    4u  IPv6 1601827      0t0  TCP *:http-alt (LISTEN)

At this point, I would expect to be able to connect to the scrutiny web UI (I know it would be empty since no data was collected), but I can't seem to talk to it, from a web browser on my network, or even locally:

root@docker1:~# telnet localhost 8080
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

If I try to run the container in the foreground (no "-d") option, I get the following output (timestamps removed for clarity):

Loading configuration file: /opt/scrutiny/config/scrutiny.yaml
...
`github.com/AnalogJ/scrutiny                             dev-0.7.1
...
Start the scrutiny server
level=info msg="Trying to connect to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n" type=web
level=info msg="Successfully connected to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n" type=web
level=info msg="InfluxDB certificate verification: true\n" type=web
level=info msg="Database migration starting. Please wait, this process may take a long time...." type=web
level=info msg="Database migration completed successfully" type=web
level=info msg="SQLite global configuration migrations starting. Please wait...." type=web
level=info msg="SQLite global configuration migrations completed successfully" type=web

Am I doing something wrong? Can anyone help me troub;eshoot?

---

What happens when I specify different "host" values in the YAML...

If I specify the actual hostname as my scrutiny-web host, I get the following during container startup...

ERROR: listen tcp 192.168.1.245:8080: bind: cannot assign requested address

If I specify localhost as the influx hostname, I get this error:

panic: failed to check influxdb setup status - Get "http://localhost:8086/api/v2/setup": dial tcp 127.0.0.1:8086: connect: connection refused

---

Environment is Docker running in a Debian LXC container on Proxmox 8.0.3:

# uname -a
Linux docker1 6.2.16-5-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-6 (2023-07-25T15:33Z) x86_64 GNU/Linux

# docker -v
Docker version 24.0.6, build ed223bc

# dpkg -l | grep -i docker
ii  docker-buildx-plugin          0.11.2-1~debian.12~bookworm    amd64        Docker Buildx cli plugin.
ii  docker-ce                     5:24.0.6-1~debian.12~bookworm  amd64        Docker: the open-source application container engine
ii  docker-ce-cli                 5:24.0.6-1~debian.12~bookworm  amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras     5:24.0.6-1~debian.12~bookworm  amd64        Rootless support for Docker.
ii  docker-compose-plugin         2.21.0-1~debian.12~bookworm    amd64        Docker Compose (V2) plugin for the Docker CLI.

[AnalogJ]

you might want to run with DEBUG mode enabled to get extra logs and see whats going on. It looks like Scrutiny was able to successfully connect to your influxdb database.

[nosoop4u]

Set up debug mode:

root@docker1:~# grep -A4 log scrutiny/scrutiny.yaml
log:
  #file: 'web.log' #absolute or relative paths allowed, eg. web.log
  file: '/opt/scrutiny/config/scrutiny-web.log'
  level: DEBUG

Restarted the scrutiny-web container.
Output from the log (below) is basically the same as what I listed above in the original ticker. I obfuscated the domain name but otherwise it is as-is.

root@docker1:~# cat scrutiny/scrutiny-web.log 
time="2023-09-30T05:53:12Z" level=debug msg="{\"log\":{\"file\":\"/opt/scrutiny/config/scrutiny-web.log\",\"level\":\"DEBUG\"},\"notify\":{\"urls\":[]},\"version\":1,\"web\":{\"database\":{\"location\":\"/opt/scrutiny/config/scrutiny.db\"},\"influxdb\":{\"bucket\":\"scrutiny\",\"host\":\"docker1.<domain>\",\"init_password\":\"password12345\",\"init_username\":\"admin\",\"insecure_skip_verify\":true,\"org\":\"homelab\",\"port\":8086,\"retention_policy\":true,\"scheme\":\"http\",\"tls\":{\"insecure_skip_verify\":false},\"token\":\"69amazingmets\"},\"listen\":{\"basepath\":\"\",\"host\":\"localhost\",\"port\":8080},\"src\":{\"frontend\":{\"path\":\"/opt/scrutiny/web\"}}}}<nil>" type=web
time="2023-09-30T05:53:12Z" level=info msg="Trying to connect to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n" type=web
time="2023-09-30T05:53:12Z" level=info msg="Successfully connected to scrutiny sqlite db: /opt/scrutiny/config/scrutiny.db\n" type=web
time="2023-09-30T05:53:12Z" level=debug msg="InfluxDB url: http://docker1.<domain>:8086" type=web
time="2023-09-30T05:53:12Z" level=info msg="InfluxDB certificate verification: true\n" type=web
time="2023-09-30T05:53:12Z" level=debug msg="Determine Influxdb setup status..." type=web
time="2023-09-30T05:53:12Z" level=info msg="Database migration starting. Please wait, this process may take a long time...." type=web
time="2023-09-30T05:53:12Z" level=info msg="Database migration completed successfully" type=web
time="2023-09-30T05:53:12Z" level=info msg="SQLite global configuration migrations starting. Please wait...." type=web
time="2023-09-30T05:53:12Z" level=info msg="SQLite global configuration migrations completed successfully" type=web
time="2023-09-30T05:53:12Z" level=debug msg="basepath: " type=web

[nosoop4u]

Figured out the issue. I somehow was using an old example.scrutiny.yaml file. Had to specify 0.0.0.0 as the web/listen/host value instead of localhost. That made the service bind correctly so I could access it via http://docker1:8080/.