Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework of storage [mtg-868] #310

Open
wants to merge 145 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
e79ccd4
drop the mutex as soon as needed
StanChe Oct 10, 2024
9e63ab2
spawn_blocking instead of the regular spawn for blocking code
StanChe Oct 10, 2024
91bfdee
removed parallel processing from synchronizer reading
StanChe Oct 10, 2024
b629102
More metrics for synchronizer (#281)
n00m4d Oct 10, 2024
52e511a
MTG-742 Move fungible assets dump to different thread (#282)
n00m4d Oct 11, 2024
f61d79d
MTG-747 Save updates for fungible assets during regular sync (#283)
n00m4d Oct 14, 2024
06d85a8
feat: pass optional hashMap to get_asset_indexes
n00m4d Oct 14, 2024
fbc242d
fix: optimize memory usage
n00m4d Oct 15, 2024
78b291f
a complete rework of the rocks storage
StanChe Oct 17, 2024
d1963f0
renamed the merge function
StanChe Oct 22, 2024
61affec
a small rocks-db explorer for debugging purposes
StanChe Oct 22, 2024
db55652
fixed readme formatting
StanChe Oct 22, 2024
1770d1d
made clippy happy
StanChe Oct 22, 2024
e2d72c6
using axum
StanChe Oct 22, 2024
b643d8b
deps for axum
StanChe Oct 22, 2024
10952fa
removed the unnecessary primary db opening
StanChe Oct 23, 2024
824fc1f
flatbuffer types + merged types + optimized dump client = 30% improve…
StanChe Oct 24, 2024
d978784
dump client optimized further, asset client refactored
StanChe Oct 28, 2024
a05855d
batch client using flatbuffer
StanChe Oct 28, 2024
d7d9b6b
insert_gaped_data updated in batch client
StanChe Oct 28, 2024
31aca0f
minor refactor
StanChe Oct 28, 2024
edcff2f
more benchmarks
StanChe Oct 28, 2024
936ebda
get_asset_indexes optimized
StanChe Oct 28, 2024
ff7e94e
full refactor to use flatbuffer
StanChe Oct 28, 2024
89a6571
code cleanup
StanChe Oct 28, 2024
73531ff
crazy merge function
StanChe Oct 29, 2024
4221d94
moved the code around
StanChe Oct 29, 2024
522cb1a
merge function tests
StanChe Oct 29, 2024
885fbba
more randomized assets generation in tests
StanChe Oct 30, 2024
3ea1df2
another attempt to the merge function
StanChe Oct 30, 2024
de2e42f
using the recent iteration of merge function
StanChe Oct 30, 2024
98eb391
Merge commit '88e5d12b86483126a5aa571072dca93b5d4df52f' into feature/…
StanChe Oct 30, 2024
44df86e
updated streaming client to use flatbuffers
StanChe Oct 31, 2024
53c4a68
fixed tests
StanChe Oct 31, 2024
ce73e95
fixed a test and dropped a debug line from another test
StanChe Oct 31, 2024
1e25381
cleanup benches
StanChe Oct 31, 2024
5247299
fixed the imports
StanChe Oct 31, 2024
08062da
fixed migrator, some tests
StanChe Nov 1, 2024
ab77ec4
fixed an issue with mpl core collections getting into response with m…
StanChe Nov 4, 2024
7342535
dump client fixed for fungibles
StanChe Nov 4, 2024
4123d05
fix for tests setup accounting for authority being optional in some t…
StanChe Nov 4, 2024
fd0ca32
Merge commit '962eaf9560de83ae1f917ec168b461666c3d7000' into feature/…
StanChe Nov 4, 2024
cae65de
making fmt and clippy happier
StanChe Nov 4, 2024
620d842
less randomness in tests
StanChe Nov 4, 2024
3474b40
dropped some commented out code
StanChe Nov 4, 2024
d8855c4
Merge commit 'b410996aaf3584ed572b2dea699d2a66f94359b3' into feature/…
StanChe Nov 4, 2024
d4baf7d
Merge commit 'd2f3e0af6b2cb438a77f45e640e2e5731c1f0977' into feature/…
StanChe Nov 4, 2024
6459da1
a small backfiller cli tool
StanChe Nov 4, 2024
8848d08
reverted the raw slots removal
StanChe Nov 4, 2024
d62ee3a
reworked to a concurrent version
StanChe Nov 5, 2024
f711dfd
fixed the message
StanChe Nov 5, 2024
1fc1555
graceful stop, async worker model reworked
StanChe Nov 5, 2024
849428a
fmt+clippy
StanChe Nov 5, 2024
17411f7
raw slot persister
StanChe Nov 7, 2024
bee1001
added existing column families
StanChe Nov 7, 2024
545d3ba
more logging
StanChe Nov 7, 2024
1502677
a slot checker cli tool
StanChe Nov 8, 2024
3b4ea49
faster iteration in persister
StanChe Nov 8, 2024
f14115b
Make big table optional (#312)
n00m4d Nov 13, 2024
68b05dc
ingester reworked to use a separate slots storage, column remover upd…
StanChe Nov 9, 2024
1286ecb
check whether slots are present in the db
StanChe Nov 9, 2024
17e79f0
persist the specified slots
StanChe Nov 9, 2024
3811815
backfill particular slots
StanChe Nov 11, 2024
d2a7701
dropped raw slots from columns used in the main db
StanChe Nov 11, 2024
20e96cc
offsetting the slot persister for 2 minutes from the last finalized s…
StanChe Nov 11, 2024
ffa289e
tmp: commented out the migrator for now
StanChe Nov 11, 2024
9a54f6b
more metrics
StanChe Nov 11, 2024
514fbd8
fucking fungibles
StanChe Nov 12, 2024
6e27910
fixing other then fungible assets request times
StanChe Nov 12, 2024
a04792d
removed synchronizer from ingester and no starting of tcp receivers w…
StanChe Nov 12, 2024
c02b9aa
refactored ingester code
StanChe Nov 12, 2024
41efee0
minor refactor
StanChe Nov 12, 2024
8d58106
fixed messages on metrics server and fixed synchronizer
StanChe Nov 13, 2024
c1e45b6
add the args to ingester
StanChe Nov 13, 2024
aa29911
Merge branch 'main' into feature/MTG-868-slots-storage
RequescoS Nov 13, 2024
4bf3999
compose update to include slots paths + removed the grpc server from …
StanChe Nov 13, 2024
6efcec4
fix: rm unwraps in dump_csv()
RequescoS Nov 13, 2024
d205eef
Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' i…
RequescoS Nov 13, 2024
9276076
fix: if asset.static_details().is_none() continue
RequescoS Nov 13, 2024
21daeeb
fix: iter.next()
RequescoS Nov 13, 2024
111ed85
fix: static_details().is_none() continue
RequescoS Nov 13, 2024
3dae65a
Merge branch 'feature/rocks-explorer' into feature/MTG-868-slots-storage
StanChe Nov 13, 2024
f1f2a8d
a more flexible db object creation on the columns that exist in rocks
StanChe Nov 13, 2024
7cbec47
merge functions in explorer
StanChe Nov 14, 2024
c44c711
using unique short flag
StanChe Nov 14, 2024
fdbf020
actually using cf options
StanChe Nov 14, 2024
1009d71
fix for synchronizer putting fungibles into the main index
StanChe Nov 14, 2024
6aa03d2
fmt
StanChe Nov 14, 2024
061f88f
filter out empty urls from getting into pg index
StanChe Nov 14, 2024
c7d1514
optimized the dump synchronizer further by introducing the mpl core a…
StanChe Nov 14, 2024
2b9cb82
Merge pull request #315 from metaplex-foundation/feature/mpl-core-col…
StanChe Nov 14, 2024
e089de9
token22 fix
StanChe Nov 16, 2024
5f73cc8
fix: display options default values
RequescoS Nov 16, 2024
daf7fd0
Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' i…
RequescoS Nov 16, 2024
8fa17ae
fix: display options default values
RequescoS Nov 16, 2024
0624262
fixes for build
StanChe Nov 16, 2024
c9a9a19
fix: grand_total_query
RequescoS Nov 16, 2024
dba1a7a
Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' i…
RequescoS Nov 16, 2024
898c13a
fix: grand_total_query
RequescoS Nov 16, 2024
f6efe2e
fix: native mint to config
RequescoS Nov 16, 2024
a0cd305
fix: collect::<Vec<_>>()
RequescoS Nov 16, 2024
99e5678
fix: token_price_fetcher
RequescoS Nov 16, 2024
e4e8235
extended explorer with an additional method
StanChe Nov 18, 2024
edf3357
yet another debug tool
StanChe Nov 18, 2024
8217fab
fix for transactions not being closed
StanChe Nov 18, 2024
0d3d71e
an additional debug tool allowing to lookup asset by an owner using a…
StanChe Nov 19, 2024
34a5b49
additional mode for synchronizer utils
StanChe Nov 19, 2024
eac6267
MTG-949 additional debug tool
StanChe Nov 19, 2024
3b54d19
MTG-949 fixed the downloader considering empty metadata valid
StanChe Nov 19, 2024
fa12a8d
MTG-950 json downloader less strict content type filters
StanChe Nov 19, 2024
fdc18ee
a proper usage of tokentype filter
StanChe Nov 20, 2024
a21d651
using separated in sqlx syntax
StanChe Nov 20, 2024
bfd6b4d
merge function updating asset class if a new one was provided for a r…
StanChe Nov 20, 2024
a229539
Merge pull request #318 from metaplex-foundation/feature/MTG-947-toke…
StanChe Nov 20, 2024
053528d
mapping the asset class on indexing given it's unrelyable in rocks
StanChe Nov 20, 2024
1d05e1f
mapping the asset class before returning it from rocks
StanChe Nov 20, 2024
ef7c6d6
rearranged the sql code into proper places
StanChe Nov 20, 2024
7333979
encode fungibles properly on dump
StanChe Nov 20, 2024
94a6f80
super hacky way to ingest images when those are provided instead of m…
StanChe Nov 21, 2024
24b21f9
Merge pull request #320 from metaplex-foundation/feature/MTG-947-asse…
StanChe Nov 22, 2024
2e7935d
Merge pull request #319 from metaplex-foundation/feature/MTG-947-merg…
StanChe Nov 22, 2024
27ef2ed
fix: write version merge
RequescoS Nov 22, 2024
f668af7
change if condition
RequescoS Nov 25, 2024
2c74caa
adding a history of owners for an asset, tracking any ownership chang…
StanChe Nov 25, 2024
15fcec8
support for IPFS files
StanChe Nov 25, 2024
99960af
storing the prebuilt metadata for metadata urls that are actually images
StanChe Nov 25, 2024
f00d1b4
Merge pull request #321 from metaplex-foundation/fix/merge-write-version
StanChe Nov 25, 2024
bd0c704
nit: new line in fbs
StanChe Nov 25, 2024
872cb65
removed old todo
StanChe Nov 25, 2024
7919627
Merge pull request #323 from metaplex-foundation/feature/MTG-961-IPFS…
StanChe Nov 25, 2024
578a167
Merge pull request #322 from metaplex-foundation/feature/MTG-964-owne…
StanChe Nov 25, 2024
2f8e7ee
Implemented cleaning up asset slot update idx && added tests
kstepanovdev Nov 26, 2024
49f0096
Simplify cleaning up asset update index
kstepanovdev Nov 26, 2024
3e3f044
simplified interfaces and their number for assets idx cleaning up
kstepanovdev Nov 26, 2024
eadeb0d
reorganize cleaners into a module
kstepanovdev Nov 26, 2024
089528a
Fix receiving shutdown signal in ingester and cleaning up service
kstepanovdev Nov 27, 2024
f7ba181
MTG-959 fix a few issues with core assets (#324)
n00m4d Nov 27, 2024
810c162
MTG-941 add integration e to e tests (#326)
n00m4d Nov 27, 2024
ca12aae
Merge branch 'clean-asset-slot-update-idx' into feature/MTG-868-slots…
StanChe Nov 27, 2024
68bf6a0
Merge commit 'afeb76ee4de287ca3707d4db419a196cfcbe374b' into feature/…
StanChe Nov 27, 2024
f8ab715
revert the formating of migration that broke prod
StanChe Nov 28, 2024
c334114
MTG-980 Fix a few issues with cNFT assets (#327)
n00m4d Nov 29, 2024
3fe1e04
chore: drop unused data struct (#330)
n00m4d Dec 3, 2024
a971484
feat: fix one integration test and drop hardcoding mutability field d…
n00m4d Dec 9, 2024
e3974da
feat: add new debug tool (#335)
n00m4d Dec 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,9 @@ INGESTER_RPC_HOST='https://rpc:port'
INGESTER_BACKFILLER_SOURCE_MODE=RPC #RPC or Bigtable
INGESTER_BIG_TABLE_CONFIG='{creds="/usr/src/app/creds.json", timeout=1000}'

INGESTER_DISABLE_SYNCHRONIZER=true
INGESTER_RUN_SEQUENCE_CONSISTENT_CHECKER=true
# Optional, required only if it needs to run fork cleaner, default is false. Unstable as it removes forked items, but also removes some valid leafs. Recommended to use only! for testing purposes.
INGESTER_RUN_FORK_CLEANER=false
INGESTER_RUN_DUMP_SYNCHRONIZE_ON_START=false
INGESTER_RUN_BUBBLEGUM_BACKFILLER=true

INGESTER_BACKFILLER_MODE=PersistAndIngest
Expand Down
167 changes: 156 additions & 11 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading