If you have failed to upgrade Mumbai Testnet Heimdall to v1.0.0-beta before the Aalborg HF (block no 18035772), you might be encountering AppHash errors and seeing these messages Wrong Block.Header.AppHash in the logs.
Check your Heimdall version /usr/bin/heimdalld version
Make sure this is at v1.0.0-beta
Run the following rollback command to take heimdall back by 1 block: sudo /usr/bin/heimdalld rollback --home=/var/lib/heimdall/ --chain-id=heimdall-80001 Note: If you have configured a different heimdall-home dir, use that path in --home above.
Start heimdall service: sudo service heimdalld start
Update: There have been multiple reports of this method not working. We have been able to reproduce the issue in one of the nodes as well and are still investigating the issue with the rollback command. If the above steps do not work, the alternative approach (which is more time consuming) is to restore the snapshot of heimdall data after the Aalborg fork (as mentioned in Heimdall and Bor Snapshots | Polygon Wiki).
bash-5.1# heimdalld version
bash-5.1# /usr/bin/heimdalld rollback --home=/data/heimdall --chain-id=heimdall-80001
Rolled back state to height 18035776 and hash 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52AINFO [2023-09-10|14:07:36.388] Heimdall services stopped module=cmd/heimdalld
ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected 0250F27568A6F527904BA9BCCEABD0DF1910EDFE83FFC594E74EA5DD8B8AA52A, got E308BB741EA1F6CDD2FBE15B72E4ED931E13077A95B6DD278DD1CE393686D1F9
Is there a hidden flag or something to pass the block number to which heimdall should rollback? Because it only rolls back to block number 18035776 but here in the first post block number 18035772 is mentioned.