Hi All,
As you are aware, Heimdall went through a co-ordinated upgrade during the maintenance window between 2PM and 5PM UTC on 10th Jul. Although many validators upgraded in time, there were a few that took longer but around an hour after the maintenance window, Heimdall-v2 chain was up and running.
Due to the staggered upgrade of heimdall, Bor nodes on the network were seeing different versions of state-sync records. The difference was between those whose heimdall was already upgraded vs the ones whose heimdall was still down. This caused many Bor nodes to encounter bad block errors.
The plan is to roll out a new version of Bor that assumes no state sync txs from the time heimdall v1 reached halt height (around 2PM UTC) till around 2AM UTC 11th Jul, so that all Bor nodes agree upon the state sync records and hence get on to the canonical chain. At the start of the first sprint after this point, all pending state sync txs will be included in the first block of the sprint. We have also identified an issue in querying state syncs from heimdall taking too long and have a fix for Heimdall also available.
All node operators would have to upgrade Bor/Erigon first and then upgrade Heimdall to these new versions before 1AM UTC (Bor Block #73826700) to ensure the network is stable and all functionalities including state syncs, milestones, checkpoints are completely restored.
The version of Bor is v2.2.8, for Erigon it is v3.0.14. Once Bor/Erigon is upgraded, Heimdall should be upgraded to v0.2.8 in the same order.
Note 1: Please do not upgrade Heimdall before Bor.
2 : If you still see bad block errors on Bor, restart bor service. This is expected to happen until majority of the nodes in the network have completed these upgrades.
Bor
This release fixes a bug in committing state sync into bor. We now explicitly do not allow any state sync events into bor (Mainnet) for a given block range from 73812433
to 73826700
. This is to allow syncing bor from a block which was prior to the heimdall (v1 to v2) migration.
Without this, bor cannot sync as it will reject blocks with bad block error.
Steps for upgrading Bor node
-
Stop bor service
sudo service bor stop
-
Install Bor with a version tag, network name (amoy, mainnet), and node type (sentry, validator, or archive).
# Replace the node type curl -L https://raw.githubusercontent.com/maticnetwork/install/main/bor.sh | bash -s -- v2.2.8 <network> <node_type>
-
Check bor version
/usr/bin/bor version # It should print # v2.2.8
-
Restart bor service
sudo service bor start
Heimdall
This version of heimdall-v2 for mainnet must be installed only after Bor is upgraded to v2.2.8 and Erigon is upgraded to v3.0.14
Once Bor/Erigon is upgraded, Heimdall-v2 should be upgraded to v0.2.8 in the same order.
This version contains an optimization that makes state sync queries much faster in Heimdall.
Steps for Upgrading Heimdall
Before proceeding, please create a backup of your heimdall
config file whose default location is: /var/lib/heimdall/config/config.toml
. This might differ if you have setup heimdall
at a different location.
-
Stop
heimdalld
servicesudo service heimdalld stop
-
Install
heimdall
with a version tag, network name (mainnet), and node type (sentry or validator).# Replace the network and node type curl -L https://raw.githubusercontent.com/maticnetwork/install/heimdall-v2/heimdall-v2.sh | bash -s -- v0.2.8 mainnet <node_type>
-
Check
heimdall
version/usr/bin/heimdalld version # It should print # 0.2.8
-
Restart
heimdall
servicesudo service heimdalld start
-
Restart the telemetry services
sudo service telemetry restart
What’s Changed - Bor
- consensus/bor/heimdall: increased the internalFetchWithTimeout to 30 seconds by @pratikspatil024.
- consensus/bor: skipping override state sync if value is 0 by @pratikspatil024.
- mainnet: added blocks in the OverrideStateSyncRecordsInRange for heimdall (v1 → v2) migration by @pratikspatil024.
- mainnet: updated the OverrideStateSyncRecordsInRange for mainnet by @pratikspatil024.
Full Changelog: v2.2.5…v2.2.8
What’s Changed - Heimdall
- Fix migration script and shasum paths in doc by @kaustubhkapatral in #341
- update docs with final v2 version by @marcello33 in #343
- update docs with telemetry upgrade info by @marcello33 in #345
- update order of instructions for docker by @marcello33 in #346
- update instructions with ports protocols by @marcello33 in #347
- update instructions with open ports for sentries by @marcello33 in #348
- update instructions for docker vals by @marcello33 in #349
- update send-ack instruction by @marcello33 in #350
- update send-ack instruction - fix by @marcello33 in #351
- update pex info by @marcello33 in #352
- Updating sha512 and migrate.sh by @djpolygon in #353
- x/clerk: fix TestGetRecordListWithTime_Pagination by @Raneet10 in #355
- Optimize state sync query by @avalkov in #354
- Dummy commit by @avalkov in #356
New Contributors
- @kaustubhkapatral made their first contribution in #341
Full Changelog: v0.2.7…v0.2.8
Docker Images
You can find the latest docker images here:
Bor: https://hub.docker.com/r/0xpolygon/bor/tags
Heimdall:https://hub.docker.com/r/0xpolygon/heimdall-v2/tags
Thanks,
Polygon Team