Recovery of Heimdall after Aalborg Hardfork

sandeep · September 9, 2023, 3:23pm

Hello All,

If you have failed to upgrade Mumbai Testnet Heimdall to v1.0.0-beta before the Aalborg HF (block no 18035772), you might be encountering AppHash errors and seeing these messages Wrong Block.Header.AppHash in the logs.

Follow the steps to recover your Heimdall node:

Ugrade you Heimdall using the steps mentioned in the previous forum post:
Bor v1.0.0-beta and Heimdall v1.0.0-beta Mumbai Release
Stop your Heimdall
sudo service heimdalld stop
Check your Heimdall version
/usr/bin/heimdalld version
Make sure this is at v1.0.0-beta
Run the following rollback command to take heimdall back by 1 block:
sudo /usr/bin/heimdalld rollback --home=/var/lib/heimdall/ --chain-id=heimdall-80001
Note: If you have configured a different heimdall-home dir, use that path in --home above.
Start heimdall service:
sudo service heimdalld start

Update: There have been multiple reports of this method not working. We have been able to reproduce the issue in one of the nodes as well and are still investigating the issue with the rollback command. If the above steps do not work, the alternative approach (which is more time consuming) is to restore the snapshot of heimdall data after the Aalborg fork (as mentioned in Heimdall & Bor Snapshots | Polygon Wiki).

Thanks,
Polygon Labs Team

ian · September 9, 2023, 6:28pm

This did not work for me unfortunately. Attempting to roll back repeatedly didn’t make a difference

ian · September 9, 2023, 6:28pm

that link to the notion page also does not work for me

pgo · September 9, 2023, 10:51pm

There appears to be a permissions problem with the Notion page. Please try this link: Bor_v1.0.0-beta_and_Heimdall_v1.0.0-beta_mumbai_release.pdf - Google Drive

sandeep · September 10, 2023, 4:49am

Apologies for posting the notion link. I have updated the link to the forum post which is accessible to all.

@ian , did you upgrade heimdall before running rollback? Can you post the output after you ran rollback?

nick · September 10, 2023, 6:33am

can confirm this doesnt work are there any other means of recovering?

sandeep · September 10, 2023, 10:05am

An alternative approach would be to restore heimdall from the latest snapshot (which was taken after the Aalborg hardfork) by following the instructions here: Heimdall and Bor Snapshots | Polygon Wiki

cshintov · September 10, 2023, 3:40pm

I did upgrade. First bor then heimdall.

bash-5.1# heimdalld version
1.0.0-beta
bash-5.1# /usr/bin/heimdalld rollback --home=/data/heimdall --chain-id=heimdall-80001
Rolled back state to height 18035776 and hash 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52AINFO [2023-09-10|14:07:36.388] Heimdall services stopped                    module=cmd/heimdalld

Same error!


ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52A, got E308BB741EA1F6CDD2FBE15B72E4ED931E13077A95B6DD278DD1CE393686D1F9

ian · September 10, 2023, 4:52pm

I did upgrade, and this is the same error I got. If you try to repeat the command it rolls back to the same state but no further. I was able to use a snapshot to get it back up though

cshintov · September 10, 2023, 7:14pm

Ya, for fullnodes ,yes. What to do for archives?!!

sandeep · September 11, 2023, 4:58am

There is no concept of archive node in heimdall. So you can directly use heimdall-mumbai-snapshot in https://snapshots.polygon.technology/

MrFreezeDZ · September 12, 2023, 12:58pm

Is there a hidden flag or something to pass the block number to which heimdall should rollback? Because it only rolls back to block number 18035776 but here in the first post block number 18035772 is mentioned.

MrFreezeDZ · September 12, 2023, 1:41pm

On another node the rollback ended for me at 18035775

cshintov · September 13, 2023, 8:04am

Same for me.

And no, there’s no flag. Check here.

github.com

maticnetwork/heimdall/blob/01b33bb559803db646914de29ca440547d92d650/cmd/heimdalld/service/rollback.go#L40


      
          		Args: cobra.NoArgs,
          		RunE: func(_ *cobra.Command, _ []string) error {
          			config := ctx.Config
          			config.SetRoot(viper.GetString(cli.HomeFlag))
          
          			db, err := sdk.NewLevelDB("application", config.DBDir())
          			if err != nil {
          				return err
          			}
          
          			height, hash, err := commands.RollbackState(config)
          
          			if err != nil {
          				return fmt.Errorf("failed to rollback tendermint state: %w", err)
          			}
          			// rollback the multistore
          			cms := rootmulti.NewStore(db)
          			cms.RollbackToVersion(height)
          			fmt.Printf("Rolled back state to height %d and hash %X", height, hash)
          			return nil
          		},

It’s calling Tendermint’s rollback function.

I think it’s automatically detecting the last healthy block to revert back to.

andrewwill · October 15, 2023, 4:22pm

Your reply is comprehensive and informative, and it includes all of the necessary steps to recover a Heimdall node after failing to upgrade to v1.0.0-beta before the Aalborg HF. You have also included a note about the rollback command not working in some cases, and provided an alternative approach. This is very helpful for me.