Recovery of Heimdall after Aalborg Hardfork

Hello All,

If you have failed to upgrade Mumbai Testnet Heimdall to v1.0.0-beta before the Aalborg HF (block no 18035772), you might be encountering AppHash errors and seeing these messages Wrong Block.Header.AppHash in the logs.

Follow the steps to recover your Heimdall node:

  1. Ugrade you Heimdall using the steps mentioned in the previous forum post:
    Bor v1.0.0-beta and Heimdall v1.0.0-beta Mumbai Release
  2. Stop your Heimdall
    sudo service heimdalld stop
  3. Check your Heimdall version
    /usr/bin/heimdalld version
    Make sure this is at v1.0.0-beta
  4. Run the following rollback command to take heimdall back by 1 block:
    sudo /usr/bin/heimdalld rollback --home=/var/lib/heimdall/ --chain-id=heimdall-80001
    Note: If you have configured a different heimdall-home dir, use that path in --home above.
  5. Start heimdall service:
    sudo service heimdalld start

Update: There have been multiple reports of this method not working. We have been able to reproduce the issue in one of the nodes as well and are still investigating the issue with the rollback command. If the above steps do not work, the alternative approach (which is more time consuming) is to restore the snapshot of heimdall data after the Aalborg fork (as mentioned in Heimdall & Bor Snapshots | Polygon Wiki).

Polygon Labs Team


This did not work for me unfortunately. Attempting to roll back repeatedly didn’t make a difference

that link to the notion page also does not work for me

There appears to be a permissions problem with the Notion page. Please try this link: Bor_v1.0.0-beta_and_Heimdall_v1.0.0-beta_mumbai_release.pdf - Google Drive

Apologies for posting the notion link. I have updated the link to the forum post which is accessible to all.

@ian , did you upgrade heimdall before running rollback? Can you post the output after you ran rollback?

can confirm this doesnt work are there any other means of recovering?

An alternative approach would be to restore heimdall from the latest snapshot (which was taken after the Aalborg hardfork) by following the instructions here: Heimdall and Bor Snapshots | Polygon Wiki

I did upgrade. First bor then heimdall.

bash-5.1# heimdalld version
bash-5.1# /usr/bin/heimdalld rollback --home=/data/heimdall --chain-id=heimdall-80001
Rolled back state to height 18035776 and hash 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52AINFO [2023-09-10|14:07:36.388] Heimdall services stopped                    module=cmd/heimdalld

Same error!

ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52A, got E308BB741EA1F6CDD2FBE15B72E4ED931E13077A95B6DD278DD1CE393686D1F9

I did upgrade, and this is the same error I got. If you try to repeat the command it rolls back to the same state but no further. I was able to use a snapshot to get it back up though

Ya, for fullnodes ,yes. What to do for archives?!!

There is no concept of archive node in heimdall. So you can directly use heimdall-mumbai-snapshot in

1 Like

Is there a hidden flag or something to pass the block number to which heimdall should rollback? Because it only rolls back to block number 18035776 but here in the first post block number 18035772 is mentioned.

On another node the rollback ended for me at 18035775

Same for me.

And no, there’s no flag. Check here.

It’s calling Tendermint’s rollback function.

I think it’s automatically detecting the last healthy block to revert back to.

1 Like

Your reply is comprehensive and informative, and it includes all of the necessary steps to recover a Heimdall node after failing to upgrade to v1.0.0-beta before the Aalborg HF. You have also included a note about the rollback command not working in some cases, and provided an alternative approach. This is very helpful for me.

1 Like