Setting up t8n compatible Nethermind EVM inside Retesteth's docker container

[debjit-bw]

Building the Dockerfile in this PR fails due to several package conflicts (version, et.al). An alternate way is to download the old and built retesteth docker container they provide (without Nethermind), and then building Nethermind's EVM inside it.

1. Install Docker if you don't have it

sudo apt install -y wget docker docker.io 

2. Get and load the retesteth Docker image as a tar file

wget http://retesteth.ethdevops.io/dretesteth.tar
sudo docker image load --input dretesteth.tar

3. Download the retesteth script

wget https://raw.githubusercontent.com/ethereum/retesteth/master/dretesteth.sh
chmod +x dretesteth.sh

4. Get the container running indefinately

docker run --platform linux/amd64 -e BESU_PATH=/besu -e ETHEREUMJS_PATH=/ethereumjs -e PYSPECS_PATH=/execution-spec-tests -e PYT8N_PATH=/pyt8n -v ~/Drawer/gnosis/retest/tests:/tests --entrypoint sh -d retesteth -c "sleep infinity"

5. Get into the container and set it up

Attach a terminal to the container any way you please (either using docker exec -it or using VS Code's dev containers). Once inside, set up dotnet and Nethermind's env.

NETHERMIND_SRC="https://github.com/NethermindEth/nethermind.git"
NETHERMIND="feature/t8n"
git clone $NETHERMIND_SRC /nethermind
cd /nethermind && git fetch && git checkout $NETHERMIND
git submodule init
git submodule update

6. Build the Evm

dotnet build ./src/Nethermind/Nethermind.sln
dotnet build ./src/Nethermind/EthereumTests.sln
dotnet build ./tools/Evm/Evm.sln

---

You can run Nethermind's t8n tool with ./tools/Evm/bin/Debug/net8.0/Evm t8n.

[debjit-bw]

Adding a new config for tests

Following the docs on retesteth, copy over the t8ntool inside tests/config/t8ntool to tests/config/t8ntool2. Inside tests/config/t8ntool2/start.sh, replace evm with the path to Nethermind's evm (usually /nethermind/tools/evm/bin/Debug/net8.0/Evm).

Nethermind's evm also uses chainId instead of chainid, so add the following line inside the for index in ... block:

        # Replace --state.chainid with --state.chainId
        if [ $index = "--state.chainid" ]; then
            index="--state.chainId"
        fi

Facing problems

Using retesteth

The following command fails, even though this succeeds with Geth's evm.

retesteth -t GeneralStateTests/stChainId  -- --singletest chainId  --testpath /tests --datadir /tests/config --filltests --clients t8ntool2 --chainid 100

The error message is:

Tool Error:

makeRPCError The command '/tests/config/t8ntool2/start.sh --state.fork Cancun --state.reward 0 --state.chainid 100 --input.alloc /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/alloc.json --input.txs /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/txs.rlp --input.env /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/env.json --output.basedir /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2 --output.result out.json --output.alloc outAlloc.json --output.errorlog /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/error.json' exited with 512 code.

Error: The command '/tests/config/t8ntool2/start.sh --state.fork Cancun --state.reward 0 --state.chainid 100 --input.alloc /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/alloc.json --input.txs /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/txs.rlp --input.env /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/env.json --output.basedir /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2 --output.result out.json --output.alloc outAlloc.json --output.errorlog /dev/shm/a11d82b7-6bcd-45fb-b16c-f6c46cde9aa2/error.json' exited with 512 code. (GeneralStateTests/stExample/add11_yml, fork: Cancun, TrInfo: d: 0, g: 0, v: 0)

--------
TestOutputHelper detected 1 errors during test execution!
/retesteth/retesteth/helpers/TestOutputHelper.cpp(240): error: in "GeneralStateTests/stExample": 

*** 1 failure is detected in the test module "EthereumTests"
*** Total Tests Run: 0

Using t8n tool directly

Next I copied over the alloc, env and txs JSON files (which were printed to the console with verbose output), and directly executed the t8n tool with them.

./tools/evm/bin/Debug/net8.0/Evm t8n --input.alloc /example/alloc.json --input.env /example/env.json --input.txs /example/txs.json --state.chainId 100 --output.basedir /example/output

It says Expected genesis spec to be Frontier for blockchain tests, which doesn't go away even when I pass in --state.fork Frontier. (related?)

[debjit-bw]

Test filling works, chain filling doesn't

retesteth -t GeneralStateTests/stChainId  -- --singletest chainId  --filltests --testpath /tests --datadir /tests/config --clients t8ntool2 --chainid 100

The above command now WORKS for generating the Filled state test (ref: 2 tests should be generated in total, but a different error occurs when generating the Blockchain state test with the following command:

retesteth -t GeneralStateTests/stChainId  -- --singletest chainId --fillchain  --testpath /tests --datadir /tests/config --clients t8ntool2 --chainid 100

Error logs:

Finishing retesteth run
*** Total Tests Run: 0

Error: The command '/tests/config/t8ntool2/start.sh --state.fork Cancun --state.reward -1 --state.chainid 100 --input.alloc /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/alloc.json --input.txs /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/txs.rlp --input.env /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/env.json --output.basedir /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a --output.result out.json --output.alloc outAlloc.json --output.errorlog /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/error.json' exited with 768 code. (GeneralStateTests/stChainId/chainId, step: test_setChainParams: Cancun)
Error: ERROR OCCURED FILLING TESTS: The command '/tests/config/t8ntool2/start.sh --state.fork Cancun --state.reward -1 --state.chainid 100 --input.alloc /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/alloc.json --input.txs /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/txs.rlp --input.env /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/env.json --output.basedir /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a --output.result out.json --output.alloc outAlloc.json --output.errorlog /dev/shm/e9d9b0ae-9015-4298-9b20-574270a51f5a/error.json' exited with 768 code. (GeneralStateTests/stChainId/chainId, step: test_setChainParams: Cancun)

One thing to note, if the command is executed with --verbosity 10, we can see the pre and post allocs. While the beacon roots account is added (compared to --filltests), it doesn't have the rewards implementation contract etc (which are additional calls made in Gnosis).