Bor & Heimdall Mumbai Upgrade: v0.3.0 [Action Required]

Polygon Testnets Mumbai Testnet
1

We have released a new version of Bor and Heimdall - v0.3.0 with an incremented minor version. As explained in the previous post, we have added a new CLI in Bor and have also changed the way Bor and Heimdall are deployed to follow DevOps best practices and make it easier to manage the processes. This upgrade is backwards incompatible. Please ensure that all Mumbai nodes are upgraded soon, before the release of the next version v0.3.1 that will contain a hardfork which is scheduled to kick in on 7th Dec 2022 (tentative).

Important Note: For our next releases, v0.3.1 and onward, upgrading to version 0.3.0 is necessary.

Please read the major changes in Bor and Heimdall from deployment perspective below before upgrading your node.

Deployment Changes

Changes in Bor:

  1. Provide a new approach to install bor binary with a shell script: GitHub - maticnetwork/install.
  2. Recommended default bor home is changed from ~/.bor to /var/lib/bor.
  3. bor/start.sh will be retired. We recommend moving flags from start.sh to /var/lib/bor/config.toml , which will be the single config file for all bor configurations. Example config toml files could be found in GitHub - maticnetwork/launch: Matic network mainnet v1 launch.
  4. A new user named bor will be created during package installation if it doesn’t exist. This user will be running bor service.
  5. Move bor.service file from /etc/systemd/system to /lib/systemd/system
  6. CLI flag changes. For validators who wants to continue setting bor configs through CLI, see appendix for detailed changes.
  7. Bor profiles can be used with new hosts only, please note existing installations will not be able to take advantages of the profiles.
  8. Bor packaging installation is now simplified to sudo dpkg -i bor-$version-$arch.deb
  9. Bor profile installation for new hosts is accomplished by running sudo dpkg -i bor-$network-$nodetype-config_$version-$arch.deb

Changes in Heimdall:

  1. Provide a new approach to install heimdall binary with a shell script: GitHub - maticnetwork/install.
  2. Remove bridge binary and add it as a subcommand in heimdalld.
  3. Remove heimdalld-rest-server.service and heimdalld-bridge.service, and move heimdalld.service file from /etc/systemd/system to /lib/systemd/system.
  4. Change default heimdall home from ~/.heimdalld to /var/lib/heimdall.
  5. A new user named heimdall will be created during package installation if it doesn’t exist. This user will be running heimdall service.
  6. Heimdall packaging installation is now simplified to sudo dpkg -i heimdalld-$version-$arch.deb
  7. Heimdall profile installation for new hosts is accomplished by running sudo dpkg -i heimdalld-$network-$nodetype-config_$version-$arch.deb

Note: Please find more details about CLI and binary changes in the Appendix.

Upgrade/Migration Instructions

Steps to Migrate to the new version:

  1. Stop all the services

    sudo service bor stop
sudo service heimdalld stop
sudo service heimdalld-bridge stop
sudo service heimdalld-rest-server stop

  2. Install Bor with a version tag, network name (mainnet or mumbai), and node type (sentry, validator, or archive).

    # Replace the network and node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/bor.sh | bash -s -- v0.3.0-beta <network> <node_type>

  3. Install Heimdall with a version tag, network name (mainnet or mumbai), and node type (sentry, validator, or archive).

    # Replace the network and node type
curl -L https://raw.githubusercontent.com/maticnetwork/install/main/heimdall.sh | bash -s -- v0.3.0-beta <network> <node_type>

  4. Set the old bor home directory path, and export it. This will be used, whenever needed, moving forward.

    # Note: in this case, the old bor directory is `~/.bor`.
#       Modify it according to your setup.
export OLD_BOR_HOME=~/.bor

  5. Link the Bor Data folder and change the owner of files to user bor, which is created in step 2

    # Note: The commands given below are under the assumption that
#       $OLD_BOR_HOME contains the correct bor home directory (BOR_DIR).
#       Modify it according to your setup as mentioned in the previous point.

sudo chown -R bor $OLD_BOR_HOME
sudo ln -nfs $OLD_BOR_HOME/data /var/lib/bor/data
sudo ln -nfs $OLD_BOR_HOME/keystore /var/lib/bor/keystore # only for validator node
sudo ln -nfs $OLD_BOR_HOME/password.txt /var/lib/bor/password.txt # only for validator node
sudo ln -nfs $OLD_BOR_HOME/address.txt /var/lib/bor/address.txt # only for validator node
sudo chown bor /var/lib/bor

  6. Set the old heimdall home directory path, and export it. This will be used, whenever needed, moving forward.

    # Note: in this case, the old heimdall directory is `~/.heimdalld`.
#       Modify it according to your setup.
export OLD_HEIMDALL_HOME=~/.heimdalld

  7. Link the Heimdall data folder and change the owner of files to user heimdall, which is created in step 3

    # Note: The commands given below are under the assumption that
#       $OLD_HEIMDALL_HOME contains the correct heimdalld home directory.
#       Modify it according to your setup as mentioned in the previous point.

sudo chown -R heimdall $OLD_HEIMDALL_HOME
sudo rm -rf /var/lib/heimdall
sudo ln -nfs $OLD_HEIMDALL_HOME /var/lib/heimdall
sudo chown heimdall /var/lib/heimdall

  8. Check Bor Version

    /usr/bin/bor version
# Output - 0.3.0-beta

  9. Check Heimdall Version

    /usr/bin/heimdalld version
# Output - 0.3.0-beta

  10. Create backup for Bor

    cd ~
mkdir backup

# Use the path of your bor.service file
sudo mv /etc/systemd/system/bor.service ./backup

  11. Create backup for Heimdall

    cd ~

# Use the path of your heimdall service files
sudo mv /etc/systemd/system/heimdalld.service ./backup
sudo mv /etc/systemd/system/heimdalld-rest-server.service ./backup
sudo mv /etc/systemd/system/heimdalld-bridge.service ./backup

  12. Check and edit bor.service file to make any changes if required. You can skip this step if bor package is installed with correct network and node type arguments in step 2.

    sudo vi /lib/systemd/system/bor.service

    Example of bor.service file:

    [Unit]
  Description=bor
  StartLimitIntervalSec=500
  StartLimitBurst=5

[Service]
  Restart=on-failure
  RestartSec=5s
  ExecStart=/usr/bin/bor server -config "/var/lib/bor/config.toml"
  Type=simple
  KillSignal=SIGINT
  User=bor
  TimeoutStopSec=120

[Install]
  WantedBy=multi-user.target

  13. Check and edit heimdalld.service file to make any changes if required (like adding bridge flag --bridge --all for validators). You can skip this step if heimdall package is installed with correct network and node type arguments in step 3.

    sudo vi /lib/systemd/system/heimdalld.service

    Example of heimdald.service file:

    #Example heimdalld.service file for reference
[Unit]
  Description=heimdalld
  StartLimitIntervalSec=500
  StartLimitBurst=5
[Service]
  Restart=on-failure
  RestartSec=5s
  WorkingDirectory=/usr/bin
  ExecStart=/usr/bin/heimdalld start --home /var/lib/heimdall \
    --chain=mumbai \
    --rest-server
  Type=simple
  LimitNOFILE=65536
  User=heimdall
[Install]
  WantedBy=multi-user.target

  14. Migrating to config.toml from start.sh:

    To migrate the configuration/flags from start.sh to a config.toml, you can use a utility script as described below.
    Example usage:

    $ git clone https://github.com/maticnetwork/bor.git # If not already cloned
$ cd bor
$ git fetch
$ git checkout tags/v0.3.0-beta
$ make bor
$ cd scripts

# Note: Running this command will output a config.toml which will have
#       values populated from the start.sh. Also, the script demands
#       for path to start.sh, validator address (if required) and path
#       to static-nodes.json if not found in the default location.
$ BOR_DIR=/var/lib/bor ./getconfig.sh
...
* Path to start.sh: /home/ubuntu/node/bor/start.sh
* Your validator address (e.g. 0xca67a8D767e45056DC92384b488E9Af654d78DE2), or press Enter to skip if running a sentry node: 0xca67a8D767e45056DC92384b488E9Af654d78DE2
* You dont have '~/.bor/data/bor/static-nodes.json' file. If you want to use static nodes, enter the path to 'static-nodes.json' here (press Enter to skip): /home/ubuntu/my/path/static-nodes.json

Thank you, your inputs are:
Path to start.sh: /home/ubuntu/node/bor/start.sh
Address: 0xca67a8D767e45056DC92384b488E9Af654d78DE2
Path to the config file: **/home/ubuntu/node/bor-start-config.toml**
...

# Note: We're considering /home/ubuntu as our home directory. Please
#       update the commands to use your home directory before proceeding. 
#       Copy the newly created file from the output above
#       (Path to the config file) to /var/lib/bor/config.toml
$ sudo cp **/home/ubuntu/node/bor-start-config.toml** /var/lib/bor/config.toml
$ sudo chown bor /var/lib/bor/config.toml

  15. With v0.3.0, we are updating the gas limit in Mumbai network to 30M from 20M to match the value in Mainnet. For this, miner.gaslimit field needs to be updated in the config file. Open the file /var/lib/bor/config.toml in your favourite text editor and edit the value for gaslimit under miner section:

    [miner]
		...
    gaslimit = 30000000
		...

  16. For better monitoring of overall network and conditions of node, we have hosted ethstats dashboard (https://bor-mumbai.vitwit.com/). We would advise all the nodes to use the public endpoint for sending data (if you’re not sending it to some other internal endpoint). The endpoint can be configured in the config file itself and is of the format: <node_identifier>:[email protected]:3000. We would encourage you to use a non-existing and understandable identifier. This dashboard would also help us in alerting incase the nodes are not working properly (e.g. if they’re constantly out of sync).

    Open the file /var/lib/bor/config.toml in your favourite text editor and edit the value for ethstats:

    ethstats = "<node_identifier>:[email protected]:3000"

  17. Ensure that all the values have been populated correctly in bor config file /var/lib/bor/config.toml. If you want to change any config options, you can edit the file.

    For your reference, an example config for mumbai can be found here. All Config flags could be found here and documentation for each bor subcommand can be found here.

    If you are not migrating from an old start.sh, the default values populated should work out of the box for a sentry node. For validator nodes, you might want to update some of the flags like: etherbase, unlock, password, keystore, maxpeers, static-nodes, ethstats, and additional telemetry configuration (from older start.sh based config).

  18. Restart system processes:

    sudo systemctl daemon-reload
sudo systemctl start heimdalld.service
journalctl -u heimdalld.service -f # Monitor heimdall service
sudo systemctl restart telemetry.service
sudo systemctl start bor
journalctl -u bor.service -f # Monitor bor service

Docker Images

You can find the latest docker images here:

Bor: https://hub.docker.com/r/0xpolygon/bor/tags

Heimdall: https://hub.docker.com/r/0xpolygon/heimdall/tags

Appendix

Bor Detailed Change log

  • In v0.2.x, bor and bootnode are two separate binaries. In v0.3.0, bootnode will become a subcommand of bor.
  • bor init is removed in v0.3.0. Instead, on starting, bor will automatically initialize a bor home directory if not found.
  • bor ’s default behavior (running bor client) is moved to bor server as a subcommand.
  • GRPC: On starting bor using the server subcommand, it will start a GRPC server on a separate port which will be used for cli related communication. Here, the user using the cli, acts as a client and interacts with the node using GRPC. It is used for operations like adding and removing peers, getting status of node (sync status, forks, chain head, etc), getting debug and pprof traces, etc.
  • Validator must provide its public address to flag --miner.etherbase in order to mint blocks.
  • The format of toml passed to --config is changed. Notice that, in v0.2.x, only P2P related info like static or trusted nodes were mentioned in the toml file. With v0.3.x, you can pass the whole cli config using this flag. For more details and examples, see https://github.com/maticnetwork/bor/blob/develop/docs/config.md.
  • --networkid is replaced by --chain, whose value is either mainnet or mumbai. The flag also accepts the path to a custom genesis file, e.g. --chain path/to/custom_genesis.json.
  • --verbosity is replaced by --log-level and instead of numeric values, string denoting the type of log should be passed.
  • --whitelist is now changed to --requiredblocks
  • pprof options are merged to a new component, grpc. e.g. --pprof.addr 0.0.0.0 and --pprof.port 1234 should be passed as one argument to flag --grpc.addr 0.0.0.0:1234. If not set, default value will be 0.0.0.0:3131.
  • --metrics.addr and --metrics.port are merged to --metrics.prometheus-addr. e.g. --metrics.addr 0.0.0.0 and --metics.port 1234 should be passed as one argument to flag --metrics.prometheus-addr 0.0.0.0:1234 . Notice that, in v0.2.x, --metrics.addr and --metrics.port , if not explicitly set, the metrics will be exported on pprof.addr:pprof.port. If no flag is provided, the prometheus endpoint is registered on a default host and port.
  • A new flag named disable-bor-wallet is introduced which disables the personal wallet endpoints in the node. This prevents misuse of the endpoints if exposed. It will only register and use the key-store for mining blocks (if mining is enabled through --unlock and --miner.etherbase). It’s set to true by default.
  • borTraceEnabled added in TraceConfig. When set true, it will return state-sync traces for a block if present. Usage :
    debug.traceBlockByNumber("0x20FE900",{"borTraceEnabled": true})
  • The miner.gaslimit value has been updated for mumbai nodes from 20M to 30M.

Heimdall Detailed Change Log

  • In v0.2.x, heimdalld and bridge are two separate binaries. In v0.3.0, bridge will become a subcommand of heimdalld.
  • In v0.2.x, heimdalld, bridge, and rest-server are run in three different processes. In v0.3.0, they could be launched together in a single process with one CLI command.
  • In v0.3.0, building different heimdalld binary for different network is no longer needed. heimdalld binary will support both mainnet and mumbai testnet.
  • Log level of heimdall is changed from a single letter to a full word. Example:
    Before I[2022-09-22|19:11:42.896] Served RPC HTTP response
    After INFO [2022-09-22|19:11:42.896] Served RPC HTTP response
  • heimdall no longer needs a genesis file. Instead, the genesis file will be embedded in the binary of heimdalld.
  • Provide possibility to log in json format by setting the param logs_type = "json"
    in the heimdall-config.toml file.
  • Change TaskDelayBetweenEachVal from 24s to 10s to reduce StateSynced event delays
  • Log elapsed time for each method involved in the StateSynced flow.
  • Add benchmark tests for the most time consuming functions involved in the StateSynced flow.

Thanks,
Polygon Team

3 Likes