Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block submission metrics #625

Merged
merged 4 commits into from
Jun 4, 2024
Merged

Block submission metrics #625

merged 4 commits into from
Jun 4, 2024

Conversation

avalonche
Copy link
Collaborator

@avalonche avalonche commented May 22, 2024

📝 Summary

Migrating the metrics to OpenTelemetry and set up more detailed profiling to measure the request latencies of block submission and proposer api.

⛱ Motivation and Context

Improve observability and latency issues in boost relay

📚 References


✅ I have run these commands

  • make lint
  • make test-race
  • go mod tidy
  • I have seen and agree to CONTRIBUTING.md

@avalonche avalonche requested a review from 0x416e746f6e May 23, 2024 20:21
services/api/service.go Fixed Show fixed Hide fixed
@avalonche avalonche force-pushed the block-submission-metrics branch from 9781986 to 4ab0fa0 Compare May 23, 2024 20:32
metrics/metrics.go Outdated Show resolved Hide resolved
Comment on lines 1198 to 1201
otelapi.WithAttributes(
attribute.Int64("slot", s),
attribute.Int64("floorSecIntoSlot", msIntoSlot/1000),
),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remark:

not so relevant if we just drop this counter, but generally this approach is not desirable as each new combination of labels (slot and floorSecIntoSlot in this case) will cause prometheus to start a fresh counter (which will eventually explode as the size of the scrape payload will keep growing with time).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we label metrics not using attributes then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there will be labels added automatically by the scraper (things like instance, instance_name, and so on).

important is to keep all possible label permutations limited.

e.g. in the current case floorSecIntoSlot would be still fine (and maybe interesting to capture, to be able to see the the breakdown of requests per second => and there would only be 12 possible labels for that.

but the slot would change every 12s => therefore infinite count of labels. unless we somehow make the scraper "forget" old labels, so that scrape request size would be limited in size.

services/api/service.go Outdated Show resolved Hide resolved
services/api/service.go Outdated Show resolved Hide resolved
@avalonche avalonche force-pushed the block-submission-metrics branch from 9e59e24 to e95e2fe Compare May 24, 2024 16:36
Copy link
Contributor

@0x416e746f6e 0x416e746f6e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's goooo

@avalonche avalonche merged commit 7dc9c0d into main Jun 4, 2024
4 checks passed
@avalonche avalonche deleted the block-submission-metrics branch June 4, 2024 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants