GRPC Ingest #183
Draft
GRPC Ingest #183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kespinola
force-pushed
the
grpc-ingest
branch
3 times, most recently
from
b5d93b9 to
a31344f
Compare
kespinola
commented
Apr 15, 2024
|
|
||
| # Cargo.lock | ||
| - name: Check lock file | ||
| run: | | ||
| cargo tree | ||
| git checkout Cargo.lock | ||
| cargo tree --frozen |
There was a problem hiding this comment.
yellowstone crates are referenced using git so require network fetch to retrieve which violates frozen flag.
There was a problem hiding this comment.
https://lib.rs/crates/yellowstone-grpc-client
NVM crate is published
|
Went over it quickly and it looks good. One question though, how would this work if we want to backfill some accounts or run account flushes to reindex some accounts? |
* feat: allows complete reindex * chore: allow deprecated borsh package --------- Co-authored-by: Kyle Espinola <[email protected]>
* feat: mock snapshots code * refactor: remove dead code * feat: getting snapshot stream in the works * feat: adds snapshot stream * feat: adds config-ingester settings * Remove dead config * Add snapshot config in ingester --------- Co-authored-by: Kevin Rodriguez <[email protected]>
* Include prometheus metric explorer * Only xadd flush on a timeer * Follow prom metric naming conventions * Logging changes for ping-pong grpc
* feat: Add NFT single backfill command (ops) to fetch mint,tokenAcc,metadata for a nft and backfill it to the db * chore: cleanup * chore: optimize
* feat: move backfill to bubblegum crate and verfiy to it * feat: grpc ingest monitor
…le each stream (#168) * feat : separate global stream to multiple streams depending on programs * WIP : add metrics for each program subscription * refactor: add structs with builder for SubscriptionTask and add better shutdown handling * refactor : separate subscription builder to work on each subscription * refactor: change pipeline to handle single stream instead of multiple streams * Update: readme for grpc-ingest, index-readme, ops (#172) * Update: readme for grpc-ingest, index-readme, ops * Update : README * fix: prometheus ports * chore: resolve comments * chore: add comments to config files in grpc-ingest (#173) * change tag name of subscriptions * Remove topograph from grpc2redis --------- Co-authored-by: Kyle Espinola <[email protected]>
* time the program transformer tasks * add consumer labels to metrics
Co-authored-by: Ahzam Akhtar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goal
Reduce the operational cost and complexity of running DAS by switching to Dragonmouth grpc stream for account and transaction updates.
Upgrade the ingestion engine by refactoring configuration, workers, and metrics.
Approach
Create a new crate named das-grpc-ingest which has 3 primary subcommands.
stream - connects to Dragonmouth and pushes relevant events into a redis queue
ingest - subscribes to redis streams, spawns workers, and processes events using program_transformers crate.
download - fetches metadata json for asset_data replacement to bgtasks
Based on #144
closes #83
Next Steps
Diagram