20,000 interfaces, possible with snmp_exporter ?

[jseifeddine]

Host operating system: output of uname -a

docker

snmp_exporter version: output of snmp_exporter -version

ts=2023-09-04T09:42:07.920Z caller=main.go:196 level=info msg="Starting snmp_exporter" version="(version=0.24.0, branch=HEAD, revision=c9fd025309572b81eaca65100de4ca38ea406da9)" concurrency=1

What device/snmpwalk OID are you using?

  if_mib:
    max_repetitions: 2000  # How many objects to request with GET/GETBULK, defaults to 25.
    retries: 3   # How many times to retry a failed request, defaults to 3.
    timeout: 10m # Timeout for each walk, defaults to 10s.
    walk:
    - 1.3.6.1.2.1.2
    - 1.3.6.1.2.1.31.1.1
    get:
    - 1.3.6.1.2.1.1.3.0
    - 1.3.6.1.2.1.1.5.0

What did you do that produced an error?

start snmp_exporter

What did you expect to see?

Gather metrics for prometheus

What did you see instead?

Timeout

Wondering what is the limitations of snmp_exporter, I have a Juniper JNP10003 [MX10003] Edge Router with roughly 20,000 interfaces (including sub interfaces)

No matter what I've tried, it times out.... I understand that it's quite a lot of data for a single HTTP request so i'm just wondering how to tackle this ?

Thank you in advance

[SuperQ]

Some Juniper devices are very slow to respond to SNMP.

• What is the latency between the exporter and the device?
• Have you tried with curl (which doesn't have a timeout by default)?
• Have you tried with --log.level=debug?

I've made a JunOS-specific if_mib that splits up the walk into smaller pieces. This tends to work better.

https://github.com/SuperQ/tools/blob/master/snmp_exporter/junos/generator.yml

[jseifeddine]

No good,

Your if_mib_junos module works with other devices with less interfaces, but on this particular device it just times out

curl 'http://localhost:9116/snmp?target=X.X.X.X&auth=XXX&module=if_mib_junos'

Can you think of a way to achieve this ?
I guess it's just a case of too much data to process ?

Thanks in advance

[SuperQ]

If you change the command in compose, you need to include the config path change.

  snmp-exporter:
    image: prom/snmp-exporter:latest
    pull_policy: always
    restart: unless-stopped
    volumes:
      - './snmp_exporter/snmp.yml:/etc/snmp_exporter/snmp.yml'
    ports:
      - '9116:9116'
    command:
      - '--log.level=debug'
      - '--config.file=/etc/snmp_exporter/snmp.yml'

How long does this take?

time snmpbulkwalk -v2c -c 'XXX' X.X.X.X -m +JUNIPER-MIB 1.3.6.1.2.1.2 -Cr5000 > walk

When you curl, use time curl so you can report when it times out.

It would also help to see the snmp_... metrics scraped from :9116/metrics.

Also make sure you're running the latest version (v0.24.1) since it includes several bug fixes for metrics.

You have configured a very high max_repetitions value. Juniper recommends this be 10, which is the default for snmpbulkwalk. The snmp_exporter defaults this to 25. Maybe try the recommended Juniper value.

[jseifeddine]

It's all good, just takes 11 minutes to scrape, which is not great but I guess that a limitation of the Juniper not snmp_exporter...

Thank you

[jseifeddine]

For context,

this is the working Prometheus global config:

global:
  scrape_interval: 15m
  evaluation_interval: 15m
  scrape_timeout: 15m

and snmp.yml

modules:
  if_mib_junos:
    max_repetitions: 1000  # How many objects to request with GET/GETBULK, defaults to 25.
    retries: 3   # How many times to retry a failed request, defaults to 3.
    timeout: 15m # Timeout for each walk, defaults to 10s.
    ...

Thanks so much for your fast response! Great project !

[SuperQ]

You might consider something like https://github.com/czerwonk/junos_exporter, rather than SNMP.