Rework of storage [mtg-868] #310

StanChe · 2024-11-12T08:48:03Z

No description provided.

* feat: add more metrics to the synchronizer * feat: more metrics

* feat: move fung dump to different thread * feat: add more metrics

* feat: save fungible updates in regular sync * feat: change migration file and add indexes drop/create during dump load * fix: fungible tokens * chore: code style

Now all the assets-related data is persisted in a single column

…ment over the reference, or over 50% over the initial merge

Current benchmarking results for 1M assets: Dumping Group/2k batch size time: [4.8857 s 4.9288 s 4.9785 s] Dumping Group/get_assets time: [5.0407 ms 5.0694 ms 5.0968 ms] Dumping Group/get_assets_individually (taking 1000 assets in a loop one after another) time: [31.137 ms 33.730 ms 38.050 ms]

kstepanovdev · 2024-11-25T16:37:47Z

rocks-db/src/asset.rs

+    let mut static_details = existing_val.and_then(|a| a.static_details());
+    // creating a copy of every single field of the rest of the asset fields including the pubkeys to properly select the latest ones and reconstruct the asset
+    let mut dynamic_details_pubkey = existing_val
+        .and_then(|a| a.dynamic_details())


I'm not quite sure I understand what a and d mean here. Asset and Details?

kstepanovdev · 2024-11-25T16:40:31Z

rocks-db/src/asset.rs

+                        if (existing_static_details.specification_asset_class()
+                            == fb::SpecificationAssetClass::FungibleToken
+                            || existing_static_details.specification_asset_class()
+                                == fb::SpecificationAssetClass::FungibleAsset)
+                            && (new_static_details.specification_asset_class()
+                                == fb::SpecificationAssetClass::Nft
+                                || new_static_details.specification_asset_class()
+                                    == fb::SpecificationAssetClass::ProgrammableNft)


maybe some sort of array's any()? would be more convenient to read?

kstepanovdev · 2024-11-25T16:45:55Z

rocks-db/src/asset.rs

+            // Merge authority
+            if let Some(new_authority) = new_val.authority() {
+                if authority.map_or(true, |current_authority| {
+                    new_authority.compare(&current_authority) == Ordering::Greater


fb::AssetAuthority::compare() looks like a manual implementation of Ord

kstepanovdev · 2024-11-25T16:48:44Z

rocks-db/src/asset.rs

+
+fn merge_field<'a, T>(existing_field: &mut Option<T>, new_field: Option<T>)
+where
+    T: PartialOrd + 'a,


I'm not sure that lifetime annotation is needed.

kstepanovdev · 2024-11-25T16:53:35Z

rocks-db/src/asset.rs

+        .cloned()
+        .rev() // Reverse the iterator for max_by to get the first-most element for the case of multiple equal values
+        .max_by(|a, b| {
+            if let (Some(a_write_version), Some(b_write_version)) = unsafe {


at least a couple of comments regarding unsafe (also, below) would be useful

kstepanovdev · 2024-11-25T16:54:38Z

rocks-db/src/asset.rs

+    builder: &mut FlatBufferBuilder<'a>,
+    iter: Vec<fb::AssetOwner<'a>>,
+) -> Option<WIPOffset<fb::AssetOwner<'a>>> {
+    let pk = iter


iter.iter looks weird, Vec isn't Iter

kstepanovdev · 2024-11-25T16:56:38Z

rocks-db/src/asset.rs

+        .filter_map(|owner| owner.pubkey())
+        .next()
+        .map(|k| builder.create_vector(k.bytes()));
+    pk?;


We need to ensure that pk is Some then pass it later as Option. Why?

…-support [MTG-868] support for IPFS files

…reship-fix [MTG-964] Ownership model fix using a history of owners

&& called cleaning up from the ingester instead of the synchronizer

* fix: a few issues with core assets * fix: add missed None

* fix: a few issues with core assets * fix: add missed None * feat: add integration e to e tests * fix: add cfg and link to original tests in readme * Update integration_tests/README.md Co-authored-by: Stanislav Cherviakov <[email protected]> * chore: change log to tracing loging crate --------- Co-authored-by: Stanislav Cherviakov <[email protected]>

…-storage # Conflicts: # nft_ingester/Cargo.toml # nft_ingester/src/api/backfilling_state_consistency.rs # nft_ingester/src/bin/ingester/main.rs # nft_ingester/src/index_syncronizer.rs # nft_ingester/tests/api_tests.rs # postgre-client/src/asset_index_client.rs # rocks-db/src/asset_client.rs # rocks-db/src/batch_client.rs # rocks-db/src/dump_client.rs # rocks-db/src/lib.rs # rocks-db/src/storage_traits.rs

…MTG-868-slots-storage # Conflicts: # nft_ingester/src/bin/ingester/main.rs # nft_ingester/tests/api_tests.rs # rocks-db/src/asset_client.rs

* feat: fix a few issues with cNFT assets * feat: comments fixes * feat: add new 2 test cases for cNFT (#328)

RequescoS · 2024-12-09T08:41:04Z

Makefile

@@ -3,11 +3,14 @@
 SHELL := /bin/bash

 build:
-	@docker compose -f docker-compose.yaml build ingester raw-backfiller das-api synchronizer core-indexing
+	@docker compose -f docker-compose.yaml build ingester raw-backfiller das-api synchronizer core-indexing slot-persister


Do we still need core-indexing as a separate bin?

RequescoS · 2024-12-09T09:09:25Z

nft_ingester/src/api/dapi/rpc_asset_convertors.rs

-            print_max_supply: e.max_supply,
-            edition_number: e.edition_number,
-        }),
+        Interface::V1NFT => {


Is V1NFT the only Interface that can have editions?

RequescoS · 2024-12-09T09:13:23Z

nft_ingester/src/backfiller.rs

@@ -952,7 +617,6 @@ where
        slot: u64,
        block: solana_transaction_status::UiConfirmedBlock,
    ) -> Result<(), BlockConsumeError> {
-        self.rocks_client.consume_block(slot, block.clone()).await?;


Don't we need to consume the block in slot storage?

…uring mint account update (#332)

n00m4d · 2024-12-09T21:51:16Z

nft_ingester/src/config.rs

@@ -280,6 +270,10 @@ pub struct SynchronizerConfig {
    pub heap_path: String,
 }

+fn default_native_mint() -> String {


NIT: maybe it's better to import it from some Solana crate

StanChe and others added 30 commits October 10, 2024 13:32

drop the mutex as soon as needed

e79ccd4

spawn_blocking instead of the regular spawn for blocking code

9e63ab2

removed parallel processing from synchronizer reading

91bfdee

More metrics for synchronizer (#281)

b629102

* feat: add more metrics to the synchronizer * feat: more metrics

MTG-742 Move fungible assets dump to different thread (#282)

52e511a

* feat: move fung dump to different thread * feat: add more metrics

MTG-747 Save updates for fungible assets during regular sync (#283)

f61d79d

* feat: save fungible updates in regular sync * feat: change migration file and add indexes drop/create during dump load * fix: fungible tokens * chore: code style

feat: pass optional hashMap to get_asset_indexes

06d85a8

fix: optimize memory usage

fbc242d

a complete rework of the rocks storage

78b291f

Now all the assets-related data is persisted in a single column

renamed the merge function

d1963f0

a small rocks-db explorer for debugging purposes

61affec

fixed readme formatting

db55652

made clippy happy

1770d1d

using axum

e2d72c6

deps for axum

b643d8b

removed the unnecessary primary db opening

10952fa

flatbuffer types + merged types + optimized dump client = 30% improve…

824fc1f

…ment over the reference, or over 50% over the initial merge

batch client using flatbuffer

a05855d

insert_gaped_data updated in batch client

d7d9b6b

minor refactor

31aca0f

more benchmarks

edcff2f

get_asset_indexes optimized

936ebda

full refactor to use flatbuffer

ff7e94e

code cleanup

89a6571

crazy merge function

73531ff

moved the code around

4221d94

merge function tests

522cb1a

more randomized assets generation in tests

885fbba

another attempt to the merge function

3ea1df2

kstepanovdev reviewed Nov 25, 2024

View reviewed changes

StanChe and others added 14 commits November 25, 2024 17:34

Merge pull request #323 from metaplex-foundation/feature/MTG-961-IPFS…

7919627

…-support [MTG-868] support for IPFS files

Merge pull request #322 from metaplex-foundation/feature/MTG-964-owne…

578a167

…reship-fix [MTG-964] Ownership model fix using a history of owners

Implemented cleaning up asset slot update idx && added tests

2f8e7ee

Simplify cleaning up asset update index

49f0096

simplified interfaces and their number for assets idx cleaning up

3e3f044

&& called cleaning up from the ingester instead of the synchronizer

reorganize cleaners into a module

eadeb0d

Fix receiving shutdown signal in ingester and cleaning up service

089528a

MTG-959 fix a few issues with core assets (#324)

f7ba181

* fix: a few issues with core assets * fix: add missed None

Merge commit 'afeb76ee4de287ca3707d4db419a196cfcbe374b' into feature/…

68bf6a0

…MTG-868-slots-storage # Conflicts: # nft_ingester/src/bin/ingester/main.rs # nft_ingester/tests/api_tests.rs # rocks-db/src/asset_client.rs

revert the formating of migration that broke prod

f8ab715

MTG-980 Fix a few issues with cNFT assets (#327)

c334114

* feat: fix a few issues with cNFT assets * feat: comments fixes * feat: add new 2 test cases for cNFT (#328)

chore: drop unused data struct (#330)

3fe1e04

StanChe marked this pull request as ready for review December 6, 2024 14:35

This was referenced Dec 6, 2024

MTG-886 Change config for TCP and Redis messengers #313

Open

Add env feature for clap #314

Open

RequescoS reviewed Dec 9, 2024

View reviewed changes

n00m4d added 2 commits December 9, 2024 12:21

feat: fix one integration test and drop hardcoding mutability field d…

a971484

…uring mint account update (#332)

feat: add new debug tool (#335)

e3974da

n00m4d reviewed Dec 9, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework of storage [mtg-868] #310

Rework of storage [mtg-868] #310

StanChe commented Nov 12, 2024

kstepanovdev Nov 25, 2024 •

edited

Loading

kstepanovdev Nov 25, 2024

kstepanovdev Nov 25, 2024

kstepanovdev Nov 25, 2024

kstepanovdev Nov 25, 2024

kstepanovdev Nov 25, 2024

kstepanovdev Nov 25, 2024

RequescoS Dec 9, 2024 •

edited

Loading

RequescoS Dec 9, 2024

RequescoS Dec 9, 2024

n00m4d Dec 9, 2024

Rework of storage [mtg-868] #310

Are you sure you want to change the base?

Rework of storage [mtg-868] #310

Conversation

StanChe commented Nov 12, 2024

kstepanovdev Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RequescoS Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kstepanovdev Nov 25, 2024 •

edited

Loading

RequescoS Dec 9, 2024 •

edited

Loading