Aalborg Hardfork - Introducing Faster Deterministic Finality with Milestones [Action Required]

Polygon Mainnet (PoS)
1

Hello All,

After a successful rollout of Milestones feature with Aalborg Hardfork on Mumbai Testnet, new versions of Bor v1.0.4 and Heimdall v1.0.1 have been released for Mainnet.

This includes Aalborg hardfork on Mainnet Heimdall which is scheduled for Heimdall block number 15,950,759. Since the block time in Heimdall depends on various factors, based on the current average block time, the hardfork might occur around October 11th, 2023. However, it is recommended to have all nodes upgraded before October 9th, 2023. With this hardfork, the “Milestones” feature will be activated which offers faster deterministic finality.

For more information please refer PIP-11: Deterministic finality via Milestones

Updates as on 9th Oct 2023:

  1. Based on the current Heimdall block height and average block time, the hardfork block (#15,950,759) might be mined on October 13th: One-stop validator portal — Validator.Info
  2. New versions of Bor, Heimdall and Erigon (containing bug fixes) have been released. Please refer the following forum posts:

Instructions to Upgrade

Note 1: Instructions to upgrade Bor and Heimdall are given below. If you are using Erigon instead of Bor client, Milestones feature is available after v2.50.0 (Release v2.50.0 Milestones · ledgerwatch/erigon · GitHub). There are few known issues in Erigon which are being fixed currently and you can expect more patch versions in the coming days.

Note 2: Please upgrade Bor to v1.0.4 before upgrading Heimdall to v1.0.1 on all nodes (validator, sentry/full nodes and archive nodes).

Steps for Upgrading Bor

  1. Stop bor service

    sudo service bor stop

  2. Install Bor with a version tag, network name (mainnet), and node type (sentry, validator, or archive).

    # Replace the node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/bor.sh | bash -s -- v1.0.4 mainnet <node_type>

  3. Check bor version

    /usr/bin/bor version

# It should print 
# 1.0.4

  4. Restart bor service

    sudo service bor start

Additional Notes:

Once you upgrade your nodes, expect to see a few error/warning logs until the hard fork kicks in on heimdall network (at block #15,950,759). Below are some common errors which might be visible in bor. Note that these errors are from a background process and will not affect the general operations of node like syncing, rpc calls, etc.

  1. Retrying to fetch milestone. Bor queries heimdall in a background process to fetch the latest milestones but will fail to do so until the hard fork.

    INFO [09-20|03:17:25.102] Retrying again in 5 seconds to fetch data from Heimdall path=/milestone/latest attempt=1

  2. Error while trying to fetch milestone from heimdall. As heimdall isn’t producing milestones (until HF), the API call to fetch milestone will respond with a 404. If heimdall is not upgraded, it will return status code 500 as no such endpoint is registered.

    WARN [09-20|03:17:25.102] an error while trying fetching from Heimdall attempt=1 error="error while fetching data from Heimdall: response code 404"

  3. With no milestone in heimdall, bor will fail to whitelist them with the below log.

    ERROR [09-21|08:11:22.744] Failed to fetch latest milestone for whitelisting err="context deadline exceeded"

  4. As milestone behaves similar to checkpoint, it also has a concept of acknowledgment. Bor also tries to fetch the no-ack messages from heimdall for some operations but will also fail to do so until HF. Errors similar to the above error will be visible for no-ack-milestone as well.

Steps for Upgrading Heimdall

Before proceeding, please create a backup of your heimdall config file whose default location is: /var/lib/heimdall/config/config.toml. This might differ if you have setup heimdall at a different location.

  1. Stop heimdalld service

    sudo service heimdalld stop

  2. Install Heimdall with a version tag, network name (mainnet), and node type (sentry or validator).

    # Replace the network and node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/heimdall.sh | bash -s -- v1.0.1 mainnet <node_type>

  3. Check heimdall version

    /usr/bin/heimdalld version

# It should print
# 1.0.1

  4. Ensure that heimdalld service file contains the —-chain=mainnet flag. The heimdalld service file is available at /etc/systemd/system/heimdalld.service
    You can use the following as reference:

    1. For validator nodes: https://github.com/maticnetwork/heimdall/blob/develop/packaging/templates/systemd/heimdalld-mainnet-validator.service
    2. For non validator nodes: https://github.com/maticnetwork/heimdall/blob/develop/packaging/templates/systemd/heimdalld-mainnet-sentry.service

  5. Restart heimdall service

    sudo service heimdalld start

  6. Restart the telemetry services

    sudo service telemetry restart

Docker Images

You can find the latest docker images here:

Bor: https://hub.docker.com/r/0xpolygon/bor/tags

Heimdall: https://hub.docker.com/r/0xpolygon/heimdall/tags

Bor Changelog

Compared to version v0.5.0, the following features and improvements have been implemented. Checkout the v1.0.4 tag on GitHub for more information.

Heimdall Changelog

Compared to version v0.3.5, the following features and improvements have been implemented. Checkout the v1.0.1 tag on GitHub for more information.

Thanks,
PolygonLabs Team

2

Thank you for the information!

What is the proper way to prepare Erigon archive nodes for hard fork?

4

So the HF is technically in Heimdall nodes, Erigon nodes will continue working the same even without upgrading but to make it compliant with milestones feature just like bor, We have worked with erigon team and recently released v2.50.0 which has milestones in it, It’s currently in testing phase and should be announced soon.

1 Like
5

After upgrading to bor 1.0.4 (unsure if related or not), retrieving certain transactions result in a chainId of value 0x7fffffffffffffee, which causes the ethers library to crash (as the value overflows javascript numeric).

Example transaction hashes include:
0xc1be342a619e4d769b8e461eb796d9503787ab4097f7121191974e819baf6154
0x79de824175540cdf375beef2c6cc2575fdc60d9908677df932f9007990b74d27

Attempting to retrieve the transaction via erigon or bor 0.3.4 seems to result in a chainId of 0.

Do you have any guidance here?

6

@wwhchung , thanks for reporting this. The fix is available in Bor v1.0.5.

7

For erigon, please use release from yesterday : 2.54.2 (git tag) which should have been v2.52.1