Crash on rpc error(s)

[Shungy]

Should probably gracefully handle this:

thread 'tokio-runtime-worker' panicked at /home/runner/work/helios/helios/execution/src/evm.rs:76:47:
called `Result::unwrap()` on an `Err` value: rpc error on method: get_proof, message: Deserialization Error: invalid type: string "too many requests, we recommend you to use free api key", expected struct JsonRpcError at line 1 column 67. Response: {"error": "too many requests, we recommend you to use free api key","id":39, "project": "11307768e6d77951256a33cc9abd40851f6f5ce8"}

Location:
    execution/src/rpc/http_rpc.rs:54:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

[ncitron]

Ah yeah that is a good idea. We do handle retries right now via ethers but it seems depending on what type of RPC provider you are using it may fail. We should add a fallback retry in case ethers retries blows up.

[varun-doshi]

Is this still open?

[amiller]

adding that im currently observing an intermittent panic, presumably caused by alchemy on base throwing get_proof errors. Here's what it looks like:

2024-12-08T15:42:50.345212Z  INFO helios_opstack::consensus: unsafe head updated: block=23441611 age=1s
2024-12-08T15:42:52.477125Z  INFO helios_opstack::consensus: unsafe head updated: block=23441612 age=1s
thread 'tokio-runtime-worker' panicked at /home/runner/work/helios/helios/core/src/execution/evm.rs:69:47:
called `Result::unwrap()` on an `Err` value: rpc error on method: get_proof, message: server returned an error response: error code -32602: distance to target block exceeds maximum proof window

Location:
    /home/runner/work/helios/helios/core/src/execution/rpc/http_rpc.rs:58:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)