fix(mirror): fix issues with limits on transaction speed (#11916)

There has been an issue where the rate of transactions sent can only go
so high, and when we try to raise it with the "tx_batch_interval"
option, it hits a bottleneck and fails to send transactions fast enough.
What is happening is that the `main_loop()` function handles many other
things than sending transactions, and shares a thread with the indexer
for the target chain. The biggest bottleneck is the fact that building
new sets of transactions to send in `queue_txs()` after each batch of
transactions we send actually can take a long time, so that the early
break in that function when we're 20 milliseconds away from needing to
send more transactions is not enough. Then also sometimes we are held up
by the indexer sending a new block and us processing it.

So to fix this, quite a lot needs to be refactored so that we can put
these components in different threads. We pass around a
`Mutex<TxTracker>`, and modify `on_target_block()` so that it's not
async, because otherwise we wouldn't be able to lock the mutex and call
that function in a new thread, since the resulting future would not be
`Send`. The main thread handles queueing new blocks worth of
transactions, and calling all the async `TxTracker` functions, and we
start a new thread to run the loop that sends transactions, and another
to run the indexer and call `TxTracker::on_target_block()`

pytest/tests/mocknet/helpers/neard_runner.py

@@ -911,7 +911,7 @@ def start_neard(self, batch_interval_millis=None):
     # Configure the logs config file to control the level of rust and opentelemetry logs.
     # Default config sets level to DEBUG for "client" and "chain" logs, WARN for tokio+actix, and INFO for everything else.
     def configure_log_config(self):
-        default_log_filter = 'client=debug,chain=debug,actix_web=warn,mio=warn,tokio_util=warn,actix_server=warn,actix_http=warn,info'
+        default_log_filter = 'client=debug,chain=debug,mirror=debug,actix_web=warn,mio=warn,tokio_util=warn,actix_server=warn,actix_http=warn,info'
         log_config_path = self.target_near_home_path('log_config.json')
 
         logging.info("Creating log_config.json with default log filter.")

tools/mirror/Cargo.toml

@@ -13,6 +13,7 @@ workspace = true
 
 [dependencies]
 actix.workspace = true
+actix-rt.workspace = true
 anyhow.workspace = true
 async-trait.workspace = true
 borsh.workspace = true