Update about current situation on the incident of the zkEVM Mainnet Beta

Announcements
1

Update about current situation on the incident of the zkEVM Mainnet Beta

  • [2024/03/22 - 19:39:35 CET] A reorg affecting a GlobalLobalExitRoot update tx happened in L1, but the trusted sequencer synchronizer was not able to handle it correctly.

  • This caused the timestamp and GER used for the block production to be incorrect.

  • The network continued to process transactions with an incorrect virtual state in terms of timestamping

  • [2024/03/22 - 20:21:00 CET] The aggregator reached a state in which it could not prove batches, as it could not generate a correct L1InfoTree.

  • [2024/03/23 - 10:02:00 CET] We did a resync on a permissionless detecting a different GER (L1InfoTree) being the correct one so that we decided to do a resync of the network from L1 sequenced data

  • [2024/03/22 - 11:59:00 CET] The result of the resync was having some of the L2 batches being invalid because they did not have a correct timestamp.

  • [2024/03/22 - 12:18:00 CET] A bug with the execution of some of these batches was found, not correctly processing some invalid transactions.

We’ve identified the bugs and we’re currently working on having a new version of the executor in the following hours to fix this situation and stabilize the network as soon as possible.We’ll keep you updated, sorry for all the inconvenience

2

[IMP] zkEVM mainnet beta incident

We have successfully restored the RPC, sequencer, and sequences in L1. However, during our inspection, we have found an issue in the proving system. This requires us to develop a new version of the zkEVM ROM and upgrade the verifier system. As a result, the bridge service is not currently operational, and withdrawals cannot be made until the upgrade is complete. Please note that zkProofs are also currently not operational. Also as a consequence of this, there may be transactions that have been confirmed and are no longer there, or that have not been processed at all since we have had a reorg in the network. We apologize for any inconvenience this may cause and appreciate your patience during the upgrade process.

In order to fix the permissionless nodes, you must follow the steps below:

  1. Stop the RPC, Synchronizer and Executor components (keep the state DB running)
  2. Update node (RPC and Synchronizer) to v0.6.3 (keep RPC and Synchronizer stopped)
  3. Update Executor to v5.0.9 (keep Executor stopped)
  4. Connect to the state DB and run the following queries:
    delete from state.batch where batch_num >= 2001300;
    delete from state.block where block_num >= 19490457;
  5. Start the components in this order:
    5.1 Executor
    5.2 Synchronizer
    5.3 RPC

we will try to fix this situation as soon as possible, we are very sorry for all the inconveniences caused

1 Like