Bor & Heimdall Mainnet Upgrade: v0.3.0 [Action Required]

Polygon Mainnet (PoS) Announcements
1

Hello All,

After the upgrade on Mumbai Testnet, we are now rolling out the new version of Bor and Heimdall - v0.3.0 on Mainnet. We have a new CLI in Bor and have also changed the way Bor and Heimdall are deployed to follow DevOps best practices and make it easier to manage the processes. This upgrade is backwards incompatible. Please ensure that all Mainnet nodes are upgraded soon, before the release of the next version that will contain a hardfork which is scheduled to kick in after mid-Jan (more details including the new version, block number for the hardfork will be made available early Jan 2023).

Important Note: For our next releases v0.3.x, upgrading to version v0.3.0 is necessary.

The new command line interface (cli)

The new-cli in this version of Bor aims to give users more control over the codebase when interacting with and starting a node. We have made every effort to keep most of the flags similar to the old CLI, except for a few notable changes. One major change is the use of the --config flag, which previously represented fields without available flags. It now represents all flags available to the user, and will overwrite any other flags if provided. As a node operator, you still have the flexibility to modify flags as needed. Please note that this change does not affect the internal functionality of the node, and it remains compatible with Geth and the Ethereum Virtual Machine (EVM). Please refer to this link for additional details about flags, defaults, and example config files for reference. ****

Please read the major changes in Bor and Heimdall from deployment perspective below before upgrading your node.

Deployment Changes

Changes in Bor:

  1. Provide a new approach to install bor binary with a shell script: GitHub - maticnetwork/install.
  2. Recommended default bor home is changed from ~/.bor to /var/lib/bor.
  3. bor/start.sh will be retired. We recommend moving flags from start.sh to /var/lib/bor/config.toml , which will be the single config file for all bor configurations. Example config toml files could be found in GitHub - maticnetwork/launch: Matic network mainnet v1 launch.
  4. A new user named bor will be created during package installation if it doesn’t exist. This user will be running bor service.
  5. Move bor.service file from /etc/systemd/system to /lib/systemd/system
  6. CLI flag changes. For validators and node operators who want to continue setting bor configs through CLI, see appendix for detailed changes.
  7. Bor profiles can be used with new hosts only, please note existing installations will not be able to take advantages of the profiles.
  8. Bor packaging installation is now simplified to sudo dpkg -i bor-$version-$arch.deb
  9. Bor profile installation for new hosts is accomplished by running sudo dpkg -i bor-$network-$nodetype-config_$version-$arch.deb

Changes in Heimdall:

  1. Provide a new approach to install heimdall binary with a shell script: GitHub - maticnetwork/install.
  2. Remove bridge binary and add it as a subcommand in heimdalld.
  3. Remove heimdalld-rest-server.service and heimdalld-bridge.service, and move heimdalld.service file from /etc/systemd/system to /lib/systemd/system.
  4. Change default heimdall home from ~/.heimdalld to /var/lib/heimdall.
  5. A new user named heimdall will be created during package installation if it doesn’t exist. This user will be running heimdall service.
  6. Heimdall packaging installation is now simplified to sudo dpkg -i heimdalld-$version-$arch.deb
  7. Heimdall profile installation for new hosts is accomplished by running sudo dpkg -i heimdalld-$network-$nodetype-config_$version-$arch.deb

Note: Please find more details about CLI and binary changes in the Appendix.

Upgrade/Migration Instructions

Steps to Migrate to the new version:

  1. Stop all the services

    sudo service bor stop
sudo service heimdalld stop
sudo service heimdalld-bridge stop
sudo service heimdalld-rest-server stop

  2. Install Bor with a version tag, network name (mainnet or mumbai), and node type (sentry, validator, or archive).

    # Replace the network and node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/bor.sh | bash -s -- v0.3.0 mainnet <node_type>

  3. Install Heimdall with a version tag, network name (mainnet or mumbai), and node type (sentry, validator). Note: There is no archive binary for Heimdall. For Heimdall running on archive nodes, simply use sentry binary.

    # Replace the network and node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/heimdall.sh | bash -s -- v0.3.0 mainnet <node_type>

  4. Set the old bor home directory path, and export it. This will be used, whenever needed, moving forward.

    # Note: in this case, the old bor directory is `~/.bor`.
#       Modify it according to your setup.
export OLD_BOR_HOME=~/.bor

  5. Link the Bor Data folder and change the owner of files to user bor, which is created in step(2) above

    # Note: The commands given below are under the assumption that
#       $OLD_BOR_HOME contains the correct bor home directory (BOR_DIR).
#       Modify it according to your setup as mentioned in the previous step.

sudo chown -R bor $OLD_BOR_HOME
sudo ln -nfs $OLD_BOR_HOME/data /var/lib/bor/data
sudo ln -nfs $OLD_BOR_HOME/keystore /var/lib/bor/keystore # only for validator node
sudo ln -nfs $OLD_BOR_HOME/password.txt /var/lib/bor/password.txt # only for validator node
sudo ln -nfs $OLD_BOR_HOME/address.txt /var/lib/bor/address.txt # only for validator node
sudo chown bor /var/lib/bor

  6. Set the old heimdall home directory path, and export it. This will be used, whenever needed, moving forward.

    # Note: in this case, the old heimdall directory is `~/.heimdalld`.
#       Modify it according to your setup.
export OLD_HEIMDALL_HOME=~/.heimdalld

  7. Link the Heimdall data folder and change the owner of files to user heimdall, which is created in step 3

    # Note: The commands given below are under the assumption that
#       $OLD_HEIMDALL_HOME contains the correct heimdalld home directory.
#       Modify it according to your setup as mentioned in the previous point.

sudo chown -R heimdall $OLD_HEIMDALL_HOME
sudo rm -rf /var/lib/heimdall
sudo ln -nfs $OLD_HEIMDALL_HOME /var/lib/heimdall
sudo chown heimdall /var/lib/heimdall

  8. Check Bor Version

    /usr/bin/bor version
# Output - v0.3.0-stable

  9. Check Heimdall Version

    /usr/bin/heimdalld version
# Output - v0.3.0

  10. Create backup for Bor

    cd ~
mkdir backup

# Use the path of your bor.service file
sudo mv /etc/systemd/system/bor.service ./backup

  11. Create backup for Heimdall

    cd ~

# Use the path of your heimdall service files
sudo mv /etc/systemd/system/heimdalld.service ./backup
sudo mv /etc/systemd/system/heimdalld-rest-server.service ./backup
sudo mv /etc/systemd/system/heimdalld-bridge.service ./backup

  12. Check and edit bor.service file to make any changes if required. You can skip this step if bor package is installed with correct network and node type arguments in step 2.

    sudo vi /lib/systemd/system/bor.service

    Example of bor.service file:

    [Unit]
  Description=bor
  StartLimitIntervalSec=500
  StartLimitBurst=5

[Service]
  Restart=on-failure
  RestartSec=5s
  ExecStart=/usr/bin/bor server -config "/var/lib/bor/config.toml"
  Type=simple
  KillSignal=SIGINT
  User=bor
  TimeoutStopSec=120

[Install]
  WantedBy=multi-user.target

  13. Check and edit heimdalld.service file to make any changes if required (like adding bridge flag --bridge --all for validators). You can skip this step if heimdall package is installed with correct network and node type arguments in step 3.

    sudo vi /lib/systemd/system/heimdalld.service

    Example of heimdald.service file:

    #Example heimdalld.service file for reference
[Unit]
  Description=heimdalld
  StartLimitIntervalSec=500
  StartLimitBurst=5
[Service]
  Restart=on-failure
  RestartSec=5s
  WorkingDirectory=/usr/bin
  ExecStart=/usr/bin/heimdalld start --home /var/lib/heimdall \
    --chain=mainnet \
    --rest-server
  Type=simple
  LimitNOFILE=65536
  User=heimdall
[Install]
  WantedBy=multi-user.target

  14. Migrating to config.toml from start.sh:

    To migrate the configuration/flags from start.sh to a config.toml, you can use a utility script as described below.
    Example usage:

    $ git clone https://github.com/maticnetwork/bor.git # If not already cloned
$ cd bor
$ git fetch
$ git checkout tags/v0.3.0
$ make bor
$ cd scripts

# Note: Running this command will output a config.toml which will have
#       values populated from the start.sh. Also, the script demands
#       for path to start.sh, validator address (if required) and path
#       to static-nodes.json if not found in the default location.
$ BOR_DIR=/var/lib/bor ./getconfig.sh
...
* Path to start.sh: /home/ubuntu/node/bor/start.sh
* Your validator address (e.g. 0xca67a8D767e45056DC92384b488E9Af654d78DE2), or press Enter to skip if running a sentry node: 0xca67a8D767e45056DC92384b488E9Af654d78DE2
* You dont have '~/.bor/data/bor/static-nodes.json' file. If you want to use static nodes, enter the path to 'static-nodes.json' here (press Enter to skip): /home/ubuntu/my/path/static-nodes.json

Thank you, your inputs are:
Path to start.sh: /home/ubuntu/node/bor/start.sh
Address: 0xca67a8D767e45056DC92384b488E9Af654d78DE2
Path to the config file: **/home/ubuntu/node/start-config.toml**
...

# Note: We're considering /home/ubuntu as our home directory. Please
#       update the commands to use your home directory before proceeding. 
#       Copy the newly created file from the output above
#       (Path to the config file) to /var/lib/bor/config.toml
$ sudo cp **/home/ubuntu/node/bor/start-config.toml** /var/lib/bor/config.toml
$ sudo chown bor /var/lib/bor/config.toml

  15. For better monitoring of overall network and conditions of node, we have hosted ethstats dashboard (https://bor-mainnet.vitwit.com/). We would advise all the nodes to use the public endpoint for sending data (if you’re not sending it to some other internal endpoint). The endpoint can be configured in the config file itself and is of the format: <node_identifier>:[email protected]:3000. We would encourage you to use a non-existing and understandable identifier. This dashboard would also help us in alerting incase the nodes are not working properly (e.g. if they’re constantly out of sync).

    Open the file /var/lib/bor/config.toml in your favourite text editor and edit the value for ethstats:

    ethstats = "<node_identifier>:[email protected]:3000"

  16. Ensure that all the values have been populated correctly in bor config file /var/lib/bor/config.toml. If you want to change any config options, you can edit the file.

    For your reference, an example config for mainnet can be found here. All Config flags could be found here and documentation for each bor subcommand can be found here.

    If you are not migrating from an old start.sh, the default values populated should work out of the box for a sentry node. For validator nodes, you might want to review and update some of the flags like: etherbase, unlock, password, keystore, maxpeers, static-nodes, ethstats, and additional telemetry configuration (from older start.sh based config).

  17. Restart system processes:

    sudo systemctl daemon-reload
sudo systemctl start heimdalld.service
journalctl -u heimdalld.service -f # Monitor heimdall service
sudo systemctl restart telemetry.service
sudo systemctl start bor
journalctl -u bor.service -f # Monitor bor service

Docker Images

You can find the docker images in the links below. Look for v0.3.0 tags.

Bor: https://hub.docker.com/r/0xpolygon/bor/tags

Heimdall: https://hub.docker.com/r/0xpolygon/heimdall/tags

Troubleshooting

Could not unlock account "0x…”

There are several potential reasons for this issue:

  • Verify that keystore, miner.etherbase, and accounts.unlock are configured with the correct values.
  • Make sure that only the correct key is present in the keystore directory. If multiple private keys are present in the keystore directory, remove any extra keys.

Invalid bootstrap url

If the node encounters an “invalid bootstrap url” error, this may be due to one of the following issues:

  • There may be invalid or malformed bootstrap URLs (enodes) present in static-nodes.json or trusted-nodes.json.
  • The conversion script may not be able to validate the enode URLs by itself, leading to invalid URLs in config.toml.

To fix this issue, check config.toml and remove any invalid URLs from the file.

datadir is not pointing to /var/lib/bor/data

In some older versions of start.sh, the BOR_DIR is not inherited from the environment by default. This can cause the conversion script to fail to automatically set the datadir to /var/lib/bor/data.

When this happens, you will need to manually modify the datadir, keystore, password, and ipcpath flags to point to the new locations prefixed with /var/lib/bor.

Config file not found

In some cases, you may encounter issues where heimdall or bor cannot find or access config files. This is often due to improper permissions on the old home directories of heimdall and bor. For example, if the old heimdall home was /home/ubuntu/.heimdalld, both /home and /home/ubuntu would need to have execute and read permissions for other users. Without these permissions, heimdall and bor will not have permission to list files under these directories, resulting in the error messages shown below:

ERROR: Config File "app" Not Found in "[/var/lib/heimdall /var/lib/heimdall/config]"

2022/12/12 15:18:06 Config File "heimdall-config" Not Found in "/var/lib/heimdall/config"

To fix this problem, make sure the permissions of the parent directories of bor and heimdall
are set to 755. For the example above where the old heimdall home is located at /home/ubuntu/.heimdalld, you can fix the issue by running the following command:

sudo chmod 755 /home/ubuntu /home

Bor error messages

With some features in this new version, bor might throw some specific error/warning messages. This section is just to clarify the reasoning behind those messages.

  • Unable to whitelist checkpoint
    • Bor leverages the checkpointing mechanism to stay connect to honest peers and it won’t connect to peers which are on wrong fork for large range of blocks (technically beyond the last checkpoint sent). For this, it will reach out to heimdall periodically to ask for latest checkpoints. Before it whitelists any checkpoint, it will perform some checks like do we have blocks for the checkpoint which we’re about to whitelist or not? If any of them fails, it won’t be able to whitelist them. A common scenario where this error is expected is when you’re syncing and have not yet reached the last check-pointed block. Once done, bor will eventually stop throwing this error.
  • Dropping peer - checkpoint mismatch
    • You might see an error for dropping peer which states “retrieved hash chain is invalid: checkpoint mismatch”. This is because before bor starts importing new blocks from a remote peer it will try to compare the blocks of that peer with it’s local copy of last whitelisted checkpoint. If there’s a mismatch it will drop the peer before even importing those false/wrong blocks with the stated error message.

Appendix

Bor Detailed Change log

  • New command line interface related changes
    • Documentation of all flags, defaults, and example configs for the new-cli for reference are available here.
    • In v0.2.x, bor and bootnode are two separate binaries. In v0.3.0, bootnode will become a subcommand of bor.
    • bor init is removed in v0.3.0. Instead, on starting, bor will automatically initialize a bor home directory if not found.
    • bor ’s default behavior (running bor client) is moved to bor server as a subcommand.
    • GRPC: On starting bor using the server subcommand, it will start a GRPC server on a separate port which will be used for cli related communication. Here, the user using the cli, acts as a client and interacts with the node using GRPC. It is used for operations like adding and removing peers, getting status of node (sync status, forks, chain head, etc), getting debug and pprof traces, etc.
      pprof options are merged to a new component, grpc. e.g. --pprof.addr 0.0.0.0 and --pprof.port 1234 should be passed as one argument to flag --grpc.addr 0.0.0.0:1234. If not set, default value will be 0.0.0.0:3131.
    • Validator must provide its public address to flag --miner.etherbase in order to mine blocks.
    • The format of toml passed to --config is changed. Notice that, in v0.2.x, only fields which did not have a flag like static or trusted nodes (inside Node.P2P tag) were mentioned in the toml file. With v0.3.x, you can pass the whole cli config using this flag. For more details and examples, visit the documentation link posted above.
    • --networkid is replaced by --chain, whose value is either mainnet or mumbai. The flag also accepts the path to a custom genesis file, e.g. --chain path/to/custom_genesis.json.
    • --verbosity is replaced by --log-level and instead of numeric values, string denoting the type of log should be passed.
    • --metrics.addr and --metrics.port are merged to --metrics.prometheus-addr. e.g. --metrics.addr 0.0.0.0 and --metics.port 1234 should be passed as one argument to flag --metrics.prometheus-addr 0.0.0.0:1234 . Notice that, in v0.2.x, --metrics.addr and --metrics.port , if not explicitly set, the metrics will be exported on pprof.addr:pprof.port. If no flag is provided, the prometheus endpoint is registered on a default host and port.
    • A new flag named disable-bor-wallet is introduced which disables the personal wallet endpoints in the node. This prevents misuse of the endpoints if exposed. It will only register and use the key-store for mining blocks (if mining is enabled through --unlock and --miner.etherbase). It’s set to true by default.
    • Tracing of blocks which contain state sync transactions can be enabled by setting borTraceEnabled to true in the TraceConfig. Usage :
      debug.traceBlockByNumber("0x20FE900",{"borTraceEnabled": true})
  • Config related changes
    • The default value of gpo.ignoreprice flag has been aligned with the value of txpool.pricelimit i.e. 30 Gwei for mainnet (which is also the value for miner.pricelimit). More context here.
    • The default cache value (set by --cache) is set to 1024 which is suitable for mumbai network or devnet. For mainnet, you’d need to explicitly specify the value. The suggested value for the same is 4096.
  • Checkpoint whitelisting feature
    • Bor now leverages the checkpointing mechanism (of heimdall) to whitelist checkpoints locally and prevent itself from connecting to wrong peers.
    • Before starting to import blocks from any remote peer, it will check the integrity of blocks (to be imported) with it’s local copy of latest checkpoint.
    • If they do not match, it will prevent downloading blocks from that peer by disconnecting with it. If they match, it’ll proceed to import blocks normally.
    • See this PR for more reference.

Heimdall Detailed Change Log

  • In v0.2.x, heimdalld and bridge are two separate binaries. In v0.3.0, bridge will become a subcommand of heimdalld.
  • In v0.2.x, heimdalld, bridge, and rest-server are run in three different processes. In v0.3.0, they could be launched together in a single process with one CLI command.
  • In v0.3.0, building different heimdalld binary for different network is no longer needed. heimdalld binary will support both mainnet and mumbai testnet.
  • Log level of heimdall is changed from a single letter to a full word. Example:
    Before I[2022-09-22|19:11:42.896] Served RPC HTTP response
    After INFO [2022-09-22|19:11:42.896] Served RPC HTTP response
  • Heimdall no longer needs a genesis file. Instead, the genesis file will be embedded in the binary of heimdalld.
  • Provide possibility to log in json format by setting the param logs_type = "json"
    in the heimdall-config.toml file.
  • Change TaskDelayBetweenEachVal from 24s to 10s to reduce StateSynced event delays
  • Log elapsed time for each method involved in the StateSynced flow.
  • Add benchmark tests for the most time consuming functions involved in the StateSynced flow.

Edits

Edit 1: The command for pruning was missed out in the new CLI but will be included in the next release (due early-mid Jan). If you want to prune your node, please do so before upgrading to v0.3.0.

Thanks,
Polygon Team

2 Likes