Add local ingestion client to indexer-alt #19924
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
| @@ -57,9 +62,18 @@ impl IngestionService { | |||
| metrics: Arc<IndexerMetrics>, | |||
| cancel: CancellationToken, | |||
| ) -> Result<Self> { | |||
| // TODO: Potentially support a hybrid mode where we can fetch from both local and remote. | |||
There was a problem hiding this comment.
Curious to hear more about this might work (what scenario you're thinking of).
There was a problem hiding this comment.
Local file sources are not as reliable as remote sources, so if we use local file source, it is always better to have a fallback remote source. For instance, if the FN just started up and is still warming up, or if the FN gets stuck or even crashes for whatever reason, we would want to keep the ingestion alive.
There was a problem hiding this comment.
Is this assuming the co-located inotify based approach? I had thought that was somewhat off the table for the near term.
Onboard for the idea of having a fallback though -- this is similar to what I had in mind for the remote ingestion set-up as well -- where the indexer would periodicially check with a fullnode whether it should switch to the fullnode's REST API for ingestion (if the fullnode is responsive and leading the indexer by a short amount), and would otherwise fallback to the ingestion service.
|
All feedback addressed. |
lxfind
force-pushed
the
indexer-alt-add-local-client
branch
from
ec6b428 to
a2d7663
Compare
Description
This PR adds an IngestionClientTrait that abstracts over checkpoint fetching.
The existing one is moved into RemoteIngestionClient.
Added a new one LocalIngestionClient to read from files.
The retry logic is kept in a top-level struct, which then calls into the traits. Each trait decides the type of error for various cases.
Test plan
Added a test.
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.