why does 1 scrape provide 20 samples?

[vcarceler]

We have several switches with scrapes that last several minutes. So I have defined a job in prometheus like this:

  - job_name: snmp-slow
    scrape_interval: 1800s
    scrape_timeout: 900s
    static_configs:
      - targets:
        - 192.168.0.50    # R1-P1 Servidores (stack 3 unidades) (12m)
        - 192.168.0.58    # R3 (stack 2 unidades) (4m) 
        - 192.168.240.50  # R3-Wifi-POE DGS-1100-26MP 
        - 192.168.240.51  # Biblioteca-POE DGS-1100-26MP SG0V1GB000158
        - 192.168.240.54  # R2-PoE R02-0.18-POE-WIF DGS-1100-24PV2 Gigabit Ethernet Switch QSA510B000474
    metrics_path: /snmp
    params:
      auth: [public_v2]
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.0.129:9116  # The SNMP exporter's real hostname:port.

I expect a sample every 30 minutes, but every 30 minutes I find a series of 20 samples separated by 15 seconds.

Is this the expected behavior? What sense does it have?

[SuperQ]

That is expected behavior.

It is related to the way Prometheus lookback/staleness handling works. Prometheus timestamps are accurate to the millisecond. So it looks back in time from the point in the graph, up to 5 minutes, in order to present the data at time X,Y,Z.

Because your scrape interval is much slower, you're always going to run into this staleness issue.

Grafana has a default "step", configured in the datasource. This is the "Step" in the per-query Options in your dashboard. If you set the "Min Step" to 30m, it will better align your data points.

Here's a good video on Prometheus data handling.

[vcarceler]

Thanks a lot! :-)