Issue transactions are not (or slowly) accepted in the blockchain

[cebe]

I am generating a lot of issue transactions using issuemorefrom API call.

This was working okay when the chain was small, but now I see that nearly all of them are getting rejected with debug.log showing:

2018-02-06 12:30:19 ERROR: AcceptToMemoryPool : inputs already spent
2018-02-06 12:30:19 Tx e8bfb5b419be6a318e3ffd87e6ba3882cc1176e3726d165bfb786eadb07745a1 was not accepted to mempool, setting INVALID flag

An example API call:

{
    "id":"bo5a79a12363437",
    "method":"issuemorefrom",
    "params": [
        "1EaV1JMtbW1AwtcPE8cqGHgiPx1nUiHctU4wVe",
        "1P1jZrdugtWdqwHHCKyZ6ACgeCrSBArDSEhkve",
        "alpcoin",
        6.48332685,
        0,
        {"description":"exchange from Alpcoin Classic","exchange_rate":"5"}
    ]
}

As far as I understand a coin issue has no inputs to spend so it should not fail. I would consider this a bug in the implementation, or is there something I do not understand correctly?

[gidgreen]

First every transaction has at least one input which spends a previous output, otherwise there's no way of knowing who signed it.

Second it looks like you may be generating simultaneous issuemorefrom transactions from multiple nodes sharing the private key for the sending address. If that's the case, I'm afraid you just can't do it. If it's not the case please let us know.

[cebe]

If that's the case, I'm afraid you just can't do it.

This is the case, could you please explain why this is a problem? What is the reason these transactions are rejected?

[gidgreen]

They are double spends against each other, i.e. two transactions trying to spend the same UTXO. See the 'Approaches for writing' section of this page for possible workarounds: https://www.multichain.com/developers/clustering-high-availability/

[cebe]

Thanks for the fast reply!

[cebe]

I understand the impact of using UTXO of the transactions better now. When sending from a single node, all transactions use the correct UTXO of the previous transaction, so they will eventually get accepted on the blockchain. When sending a lot of issue transactions (e.g. 720 per hour, or one transaction every 5 seconds) I see that it takes a very long time for these to eventually get accepted in a block, in some cases the node that generates them even becomes unresponsive to API requests after a certain amount of time.

A graph of the size property in getmempoolinfo RPC call looks like the following:

I am sending the issue transactions via node 9201 in this case. Transactions where created from 12:00 to approx. 15:00.

I am not familiar with how multichain implements the mining process but I have the following thoughts:

• I would expect multichain to order the transactions by dependency before putting them into a block.
• it looks to me as if there is a problem when transactions depend on each other in this way, i.e. they all spend each others output, so they have to be added in a strict order.
• does multichain look on the dependencies between transactions when creating blocks or are transactions added randomly as they come in?

[cebe]

sending transactions at a rate of 60 per 15 minutes (sending 1 transaction every second for the first minute and then wait 14 minutes) seems to work quite smoothly. If I do this every 10 minutes instead of 15 minutes, I get the above behavior and the system gets stuck eventually.

[cebe]

The ideal solution for this issue would be to make multichain work as expected, however if there are limitations that can not be changed, I am thinking about how I can distribute assets in a better way.
The current transactions look like this, every transaction depends on the previous tx output:

I do want to have a single transaction per output address, so I want to avoid a single issue transaction that directly creates an output for the target address. I want to add metadata to every transaction to make the issue more transparent. I would create the following schema, but that is currently not possible with the multichain API:

That would issue all outputs for the admin transaction that performs the issue, then I would generate multiple send transactions that distribute the assets to the target addresses. The multichain API currently only allows me to generate a transaction which issues multiple outputs to different addresses, not to the same address. Is this even possible in general (i.e. if I would create the raw transaction myself)? Or would I hit other problems with this model?

[gidgreen]

What you describe is not the expected behavior. Which version of MultiChain are you using, and can you please post the output of getblockchainparams.

[cebe]

getblockchainparams:

{
    "chain-protocol" : "multichain",
    "chain-description" : "AlpCoin",
    "root-stream-name" : "root",
    "root-stream-open" : true,
    "chain-is-testnet" : false,
    "target-block-time" : 30,
    "maximum-block-size" : 1048576,
    "default-network-port" : 9200,
    "default-rpc-port" : 9300,
    "anyone-can-connect" : true,
    "anyone-can-send" : true,
    "anyone-can-receive" : true,
    "anyone-can-receive-empty" : true,
    "anyone-can-create" : false,
    "anyone-can-issue" : false,
    "anyone-can-mine" : false,
    "anyone-can-activate" : false,
    "anyone-can-admin" : false,
    "support-miner-precheck" : true,
    "allow-arbitrary-outputs" : false,
    "allow-p2sh-outputs" : true,
    "allow-multisig-outputs" : true,
    "setup-first-blocks" : 60,
    "mining-diversity" : 0.60000000,
    "admin-consensus-upgrade" : 0.51000000,
    "admin-consensus-admin" : 0.51000000,
    "admin-consensus-activate" : 0.51000000,
    "admin-consensus-mine" : 0.51000000,
    "admin-consensus-create" : 0.51000000,
    "admin-consensus-issue" : 0.51000000,
    "lock-admin-mine-rounds" : 10,
    "mining-requires-peers" : true,
    "mine-empty-rounds" : 2.00000000,
    "mining-turnover" : 0.50000000,
    "first-block-reward" : -1,
    "initial-block-reward" : 0,
    "reward-halving-interval" : 52560000,
    "reward-spendable-delay" : 1,
    "minimum-per-output" : 0,
    "maximum-per-output" : 100000000000000,
    "minimum-relay-fee" : 0,
    "native-currency-multiple" : 100000000,
    "skip-pow-check" : false,
    "pow-minimum-bits" : 8,
    "target-adjust-freq" : -1,
    "allow-min-difficulty-blocks" : false,
    "only-accept-std-txs" : true,
    "max-std-tx-size" : 524288,
    "max-std-op-returns-count" : 10,
    "max-std-op-return-size" : 262144,
    "max-std-op-drops-count" : 5,
    "max-std-element-size" : 8192,
    "chain-name" : "alpcoin",
    "protocol-version" : 10009,
    "network-message-start" : "faccfdff",
    "address-pubkeyhash-version" : "00dee13f",
    "address-scripthash-version" : "05936ca5",
    "private-key-version" : "809c9ccc",
    "address-checksum-value" : "49f6ff49",
    "genesis-pubkey" : "03d169eaeba95c8ed00ede3c445a969922d86c8b6c7bdddccd4825cd67a1a043a7",
    "genesis-version" : 1,
    "genesis-timestamp" : 1510317720,
    "genesis-nbits" : 536936447,
    "genesis-nonce" : 341,
    "genesis-pubkey-hash" : "647515d4b3b315fbc2deb369a0b5c6148232efaa",
    "genesis-hash" : "00a58489a3a51feb0592e53ee13c6f16ab3fcf21e127d4ee3a60ac16fafaa528",
    "chain-params-hash" : "f4bddba0928e1dfe093945cd44c1fd64d745089d5a9ab85e606daf5ec46853eb"
}

Multichain Version:

MultiChain 1.0.2.1 Daemon (protocol 10004-10009)

[gidgreen]

Thanks – nothing unusual I can see in the blockchain parameters. Just to confirm, you are sending all the issue transactions from one node, and no other node is generating transactions at the same time? I ask because you should be able to send 100s of issue transactions per second, sustained, under normal circumstances.

[cebe]

Just to confirm, you are sending all the issue transactions from one node, and no other node is generating transactions at the same time?

that is correct, only one node is creating the issue transactions. There might be a few non-issue transactions coming from other nodes using send between addresses but this is completely unrelated to the issue tx and should not affect the process as far as I see. Also by "a few" I mean at max 2-3 per hour.

The issuemore rpc call usually takes a few millisecond to finish but after sending a certain amount of transactions it gets slower and slower sometimes taking > 5 minutes to respond.

I ask because you should be able to send 100s of issue transactions per second, sustained, under normal circumstances.

How long do you expect the time for these transactions to show up in a block under normal circumstances? We have "target-block-time" : 30 but in these cases it takes up to 30 minutes for transactions to become part of a block. Even though blocks are created, inbetween most of them have only the single miner signature transaction in them.

[gidgreen]

OK, thanks. One more thing – please show the output of getwalletinfo.

[cebe]

{
    "walletversion" : 10500,
    "balance" : 0.00000000,
    "walletdbversion" : 2,
    "txcount" : 145566,
    "utxocount" : 2,
    "keypoololdest" : 1518651012,
    "keypoolsize" : 2
}

[cebe]

btw, in the stable situation with 60 tx every 15minutes, this is how blocks are created typically (the generation of transactions started at 7:45, running for ~1 minute):

height	miner	time	#tx
49781	18zkQcvRrDU7krVuw4CNXFfard2A5QA1W565wV	2018-02-21 07:59:43	1
49780	17cbcXN3nPLWkk3uteAvb31huefckHNr7Yf7DQ	2018-02-21 07:59:22	1
49779	1EaV1JMtbW1AwtcPE8cqGHgiPx1nUiHctU4wVe	2018-02-21 07:58:33	1
49778	1GyeeVEHheDkc9cK8ahb69gEASQ9XAhcYpcF4G	2018-02-21 07:58:10	1
49777	1Xk7mjXxAUQQCtuRggdDZNAmVKkxyzSrYfj4Hw	2018-02-21 07:58:07	1
49776	18zkQcvRrDU7krVuw4CNXFfard2A5QA1W565wV	2018-02-21 07:57:44	2
49775	1D28CrXPQLjSSHyFTh6XQEgH9rFs2G3adhjXxa	2018-02-21 07:57:23	1
49774	1EaV1JMtbW1AwtcPE8cqGHgiPx1nUiHctU4wVe	2018-02-21 07:56:45	1
49773	1MBT3L7rSPi2iAdfxdRxxKM1geqYHQtq7sTGua	2018-02-21 07:56:20	1
49772	1Xk7mjXxAUQQCtuRggdDZNAmVKkxyzSrYfj4Hw	2018-02-21 07:55:48	1
49771	18zkQcvRrDU7krVuw4CNXFfard2A5QA1W565wV	2018-02-21 07:55:34	1
49770	1GyeeVEHheDkc9cK8ahb69gEASQ9XAhcYpcF4G	2018-02-21 07:51:41	33
49769	1EaV1JMtbW1AwtcPE8cqGHgiPx1nUiHctU4wVe	2018-02-21 07:50:11	1
49768	1MBT3L7rSPi2iAdfxdRxxKM1geqYHQtq7sTGua	2018-02-21 07:50:01	1
49767	1Xk7mjXxAUQQCtuRggdDZNAmVKkxyzSrYfj4Hw	2018-02-21 07:49:46	1
49766	18zkQcvRrDU7krVuw4CNXFfard2A5QA1W565wV	2018-02-21 07:49:34	1
49765	1GyeeVEHheDkc9cK8ahb69gEASQ9XAhcYpcF4G	2018-02-21 07:49:30	1
49764	1DHGpZaLAEJMtd1sTBWWmnGynrGd9fLdiCGSJQ	2018-02-21 07:49:18	1
49763	1MBT3L7rSPi2iAdfxdRxxKM1geqYHQtq7sTGua	2018-02-21 07:48:00	5
49762	1Xk7mjXxAUQQCtuRggdDZNAmVKkxyzSrYfj4Hw	2018-02-21 07:47:13	3
49761	1EaV1JMtbW1AwtcPE8cqGHgiPx1nUiHctU4wVe	2018-02-21 07:45:35	22

it seems to work fine at the beginning, but then 33 tx need 5 minutes to get accepted in a block while inbetween empty blocks are created.

[gidgreen]

OK, your utxocount looks fine too. When you are "issuing more" for the asset, how many previous issuances have already been made for that asset? If there are multiple assets, please consider the one for which this answer will be the highest.

[cebe]

there is one asset and the number of issuances is roughly equal to the txcount in the wallet, currently 145556. The problems started way earylier than this number. A month ago we had about 10000 issuances and the problem already existed back then.

[gidgreen]

OK, well I suspect that's the problem. We didn't design the "follow-on issuance" feature to be used at these sorts of volumes. I'll confirm back with the dev team that this is the issue and if there's something we can do about it. In the short term, your best workaround is to adjust your workflow somewhat. Instead of issuing new units every time, issuing them every once in a while to the same address that is used for the issuances, and distribute them from there in send transactions. The difference is purely cosmetic because in either case it's the same address that has the ability to create new units, or distribute ones it previously created.

[gidgreen]

So... I've confirmed that this is the underlying cause of your performance problems. Please see the suggestion in the previous post.