Database: Add `--txIndex` to kaspad #2103

D-Stacks · 2022-06-21T21:28:07Z

closes #2092

This allows kaspads to be run with an optional --txindex flag. This stores TXIDs with the acceptingBlocHash. this allows for exaustive querying of tx associated data, which is useful to fledge out rpc commands, increase information available to the wallet, and to ease confirmation calculations, or even work them directly into kaspad rpc. the ability to query tx associated blockdata, is a much requested functionality from external developers, the latter of which specifically from exchanges.

it follows the example of the utxoindex.

Currently untested, and unintegrated, into kaspad's actual functioning through the rpc manager.

To Do

turn pruningpoint change into consensus event
--> needed to prune txindex
integrate into the rpc manager
--> bare minimum to update and reset the index accordingly
minimal rpc
--> for minimal usecase and testing
implement proper removal logic into the txindex database.
tests
- test for accurate dataretrival
- test for txindex pruning
- test logic (reorg & double spend cases)

…o add-tx-index

codecov · 2022-06-22T22:51:06Z

Codecov Report

Base: 58.95% // Head: 56.86% // Decreases project coverage by -2.08% ⚠️

Coverage data is based on head (e9f7169) compared to base (26c7db2).
Patch coverage: 6.16% of modified lines in pull request are covered.

❗ Current head e9f7169 differs from pull request most recent head a2c851f. Consider uploading reports for the commit a2c851f to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #2103      +/-   ##
==========================================
- Coverage   58.95%   56.86%   -2.09%     
==========================================
  Files         679      702      +23     
  Lines       33060    34376    +1316     
==========================================
+ Hits        19490    19548      +58     
- Misses      10736    11982    +1246     
- Partials     2834     2846      +12

Impacted Files	Coverage Δ
app/appmessage/message.go	`71.42% <ø> (ø)`
...appmessage/rpc_get_accepting_blockhashes_of_txs.go	`0.00% <0.00%> (ø)`
app/appmessage/rpc_get_txs.go	`0.00% <0.00%> (ø)`
app/appmessage/rpc_get_txs_confirmations.go	`0.00% <0.00%> (ø)`
...p/appmessage/rpc_modify_notifying_addresses_txs.go	`0.00% <0.00%> (ø)`
...e/rpc_modify_notifying_txs_confirmation_changed.go	`0.00% <0.00%> (ø)`
app/appmessage/rpc_notify_addresses_txs.go	`0.00% <0.00%> (ø)`
.../appmessage/rpc_notify_txs_confirmation_changed.go	`0.00% <0.00%> (ø)`
app/appmessage/rpc_submit_transaction.go	`100.00% <ø> (ø)`
app/rpc/rpc.go	`63.88% <ø> (ø)`
... and 37 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

D-Stacks · 2022-06-29T22:41:56Z

Converting back to draft. I will apply a notification stream and implement all testing which does not require dag params to be overwritten.

…o add-tx-index

someone235 · 2022-10-15T12:09:09Z

app/appmessage/rpc_get_including_block_of_tx.go

@@ -0,0 +1,43 @@
+package appmessage


I wouldn't expose the including block to the user if it's different between kaspads

someone235 · 2022-10-15T12:10:37Z

app/appmessage/rpc_get_including_blockhash_of_tx.go

@@ -0,0 +1,41 @@
+package appmessage


I wouldn't expose the including block to the user if it's different between kaspads.

someone235 · 2022-10-15T12:12:00Z

app/appmessage/rpc_get_including_blockhashes_of_txs.go

@@ -0,0 +1,41 @@
+package appmessage


I wouldn't expose the including block to the user if it's different between kaspads

someone235 · 2022-10-15T12:12:10Z

app/appmessage/rpc_get_including_blocks_of_txs.go

@@ -0,0 +1,43 @@
+package appmessage


I wouldn't expose the including block to the user if it's different between kaspads

someone235 · 2022-10-15T12:15:09Z

app/appmessage/rpc_get_tx_confirmations.go

@@ -0,0 +1,41 @@
+package appmessage


It seems redundant to have duplicate commands for multiple/single transactions (relevant for other commands as well)

someone235 · 2022-10-16T07:08:00Z

domain/txindex/txindex.go

+}
+
+// TXIncludingBlockHashes returns the including block hashes for for the given txI
+func (ti *TXIndex) TXIncludingBlockHashes(txIDs []*externalapi.DomainTransactionID) (


Redundant function

someone235 · 2022-10-16T07:08:08Z

domain/txindex/txindex.go

+}
+
+// TXIncludingBlocks returns the including block hashes for for the given txIDs
+func (ti *TXIndex) TXIncludingBlocks(txIDs []*externalapi.DomainTransactionID) (


Redundant function

someone235 · 2022-10-16T07:08:48Z

go.mod


 go 1.18

+//replace github.com/kaspanet/kaspad/domain/txindex => /home/ds/Coding/kaspad/kaspad/domain/txindex


Please unchange this file

someone235 · 2022-10-16T07:10:30Z

infrastructure/network/netadapter/server/grpcserver/protowire/messages.proto

@@ -139,6 +139,49 @@ message KaspadMessage {
    GetMempoolEntriesByAddressesResponseMessage getMempoolEntriesByAddressesResponse = 1085;
    GetCoinSupplyRequestMessage getCoinSupplyRequest = 1086;
    GetCoinSupplyResponseMessage getCoinSupplyResponse= 1087;
+


Remember to remove from here as well all of the requested commands

someone235 · 2022-10-16T07:15:05Z

infrastructure/network/netadapter/server/grpcserver/protowire/rpc.proto

@@ -666,6 +666,18 @@ message GetInfoResponseMessage{
  RPCError error = 1000;
 }

+
+//Kaspad most be started with the `--txindex` flag for this Request to work.


most -> must. And you should always add a space after //

D-Stacks and others added 9 commits June 20, 2022 16:32

starting checkpoint

3a33c0f

preliminary draft

345eaac

mostly remove old code from first round.

c339398

remove leftover from copy pasting and bug splattering

540d77f

remove smaller bugs and unused code.

d9b5f0e

Merge branch 'dev' into add-tx-index

da7a9dd

proper naming and fmt / formatting

02326dc

Merge branch 'add-tx-index' of https://github.com/D-Stacks/kaspad int…

66809e9

…o add-tx-index

turn pruning point changes into a consensus event

346bd26

D-Stacks added 7 commits June 24, 2022 10:52

add minimal rpc for usecase and testing.

463db63

include removal logic in txindex

e059e88

fix cases of nil pointers...

b4754d4

linting and deal with pruning point commitment

560a1a9

fix not found sending err, add to kaspactl

09b2c53

fix pruning notification

600f39f

logger to tracef

8bacd2e

D-Stacks marked this pull request as ready for review June 25, 2022 08:40

Merge branch 'dev' into add-tx-index

edb6962

D-Stacks marked this pull request as draft June 29, 2022 22:43

D-Stacks added 2 commits August 29, 2022 15:51

update with dev

576a24a

fic virtual change nil

ca01c1f

D-Stacks marked this pull request as ready for review September 2, 2022 08:26

D-Stacks and others added 6 commits September 2, 2022 10:27

Merge branch 'dev' into add-tx-index

de5f551

Merge branch 'dev' into add-tx-index

87c5ad0

save changes

f7fcffe

pass tests

11f5a49

generate the protos

9b6c2e3

Merge branch 'dev' into add-tx-index

3b317fc

D-Stacks and others added 17 commits October 10, 2022 10:01

pass tests - linting & fmt

06c6706

set up appmessage converters

38e9ab8

checkpoint: started work on handlers

e8e57fc

finish handlers

4c9aa95

set-up client commnds

8f1348b

fix fmt and linting

3c94c86

include commands in kaspactl

1b5a236

some fixes, not all commands working

1aa93d2

fix batched calls

325d2fd

fix fmt

7336b2d

store accepting + including block + index

d401cac

expose including & accepting blocks

6f1456c

Merge branch 'dev' into add-tx-index

4ac225c

implement protobuf txindex notifications

32205a2

Merge branch 'add-tx-index' of https://github.com/D-Stacks/kaspad int…

a905e34

…o add-tx-index

update listener protos

22aeb04

add listener converters.

1f8fe53

someone235 requested changes Oct 16, 2022

View reviewed changes

D-Stacks added 12 commits October 16, 2022 11:11

downsize up rpc commands

8875d66

remove excess functions from txindex

231633f

remove some more

f820acd

add to verbose data and getInfo

735f9c2

handle error properly

d6f0a16

set-up app messages for notifications

f6d1493

fix converters

e9f7169

fix fmt and linting

ee09eb5

start work on listening logic

cdcdfc4

add addrsindex to txindex.

fa5423e

add access to addr index

cc05cfa

finish accesss function

a2c851f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Database: Add `--txIndex` to kaspad #2103

Database: Add `--txIndex` to kaspad #2103

D-Stacks commented Jun 21, 2022 •

edited

Loading

codecov bot commented Jun 22, 2022 •

edited

Loading

D-Stacks commented Jun 29, 2022

someone235 Oct 15, 2022

someone235 Oct 15, 2022

someone235 Oct 15, 2022

someone235 Oct 15, 2022

someone235 Oct 15, 2022

someone235 Oct 16, 2022

someone235 Oct 16, 2022

someone235 Oct 16, 2022

someone235 Oct 16, 2022

someone235 Oct 16, 2022


		go 1.18

		//replace github.com/kaspanet/kaspad/domain/txindex => /home/ds/Coding/kaspad/kaspad/domain/txindex

Database: Add --txIndex to kaspad #2103

Are you sure you want to change the base?

Database: Add --txIndex to kaspad #2103

Conversation

D-Stacks commented Jun 21, 2022 • edited Loading

codecov bot commented Jun 22, 2022 • edited Loading

Codecov Report

D-Stacks commented Jun 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Database: Add `--txIndex` to kaspad #2103

Database: Add `--txIndex` to kaspad #2103

D-Stacks commented Jun 21, 2022 •

edited

Loading

codecov bot commented Jun 22, 2022 •

edited

Loading