After the upgrade on Mumbai Testnet, we are now rolling out the new version of Bor and Heimdall -
v0.3.0 on Mainnet. We have a new CLI in Bor and have also changed the way Bor and Heimdall are deployed to follow DevOps best practices and make it easier to manage the processes. This upgrade is backwards incompatible. Please ensure that all Mainnet nodes are upgraded soon, before the release of the next version that will contain a hardfork which is scheduled to kick in after mid-Jan (more details including the new version, block number for the hardfork will be made available early Jan 2023).
Important Note: For our next releases
v0.3.x, upgrading to version
v0.3.0 is necessary.
The new-cli in this version of Bor aims to give users more control over the codebase when interacting with and starting a node. We have made every effort to keep most of the flags similar to the old CLI, except for a few notable changes. One major change is the use of the --config flag, which previously represented fields without available flags. It now represents all flags available to the user, and will overwrite any other flags if provided. As a node operator, you still have the flexibility to modify flags as needed. Please note that this change does not affect the internal functionality of the node, and it remains compatible with Geth and the Ethereum Virtual Machine (EVM). Please refer to this link for additional details about flags, defaults, and example config files for reference. ****
Please read the major changes in Bor and Heimdall from deployment perspective below before upgrading your node.
Changes in Bor:
- Provide a new approach to install bor binary with a shell script: GitHub - maticnetwork/install.
- Recommended default bor home is changed from
bor/start.shwill be retired. We recommend moving flags from
/var/lib/bor/config.toml, which will be the single config file for all bor configurations. Example config toml files could be found in GitHub - maticnetwork/launch: Matic network mainnet v1 launch.
- A new user named
borwill be created during package installation if it doesn’t exist. This user will be running bor service.
- CLI flag changes. For validators and node operators who want to continue setting bor configs through CLI, see appendix for detailed changes.
- Bor profiles can be used with new hosts only, please note existing installations will not be able to take advantages of the profiles.
- Bor packaging installation is now simplified to
sudo dpkg -i bor-$version-$arch.deb
- Bor profile installation for new hosts is accomplished by running
sudo dpkg -i bor-$network-$nodetype-config_$version-$arch.deb
Changes in Heimdall:
- Provide a new approach to install heimdall binary with a shell script: GitHub - maticnetwork/install.
bridgebinary and add it as a subcommand in
heimdalld-bridge.service, and move
- Change default heimdall home from
- A new user named
heimdallwill be created during package installation if it doesn’t exist. This user will be running heimdall service.
- Heimdall packaging installation is now simplified to
sudo dpkg -i heimdalld-$version-$arch.deb
- Heimdall profile installation for new hosts is accomplished by running
sudo dpkg -i heimdalld-$network-$nodetype-config_$version-$arch.deb
Note: Please find more details about CLI and binary changes in the Appendix.
Steps to Migrate to the new version:
Stop all the services
sudo service bor stop sudo service heimdalld stop sudo service heimdalld-bridge stop sudo service heimdalld-rest-server stop
Install Bor with a version tag, network name (mainnet or mumbai), and node type (sentry, validator, or archive).
# Replace the network and node type curl -L https://raw.githubusercontent.com/maticnetwork/install/main/bor.sh | bash -s -- v0.3.0 mainnet <node_type>
Install Heimdall with a version tag, network name (mainnet or mumbai), and node type (sentry, validator). Note: There is no archive binary for Heimdall. For Heimdall running on archive nodes, simply use sentry binary.
# Replace the network and node type curl -L https://raw.githubusercontent.com/maticnetwork/install/main/heimdall.sh | bash -s -- v0.3.0 mainnet <node_type>
Set the old bor home directory path, and export it. This will be used, whenever needed, moving forward.
# Note: in this case, the old bor directory is `~/.bor`. # Modify it according to your setup. export OLD_BOR_HOME=~/.bor
Link the Bor Data folder and change the owner of files to user
bor, which is created in step(2) above
# Note: The commands given below are under the assumption that # $OLD_BOR_HOME contains the correct bor home directory (BOR_DIR). # Modify it according to your setup as mentioned in the previous step. sudo chown -R bor $OLD_BOR_HOME sudo ln -nfs $OLD_BOR_HOME/data /var/lib/bor/data sudo ln -nfs $OLD_BOR_HOME/keystore /var/lib/bor/keystore # only for validator node sudo ln -nfs $OLD_BOR_HOME/password.txt /var/lib/bor/password.txt # only for validator node sudo ln -nfs $OLD_BOR_HOME/address.txt /var/lib/bor/address.txt # only for validator node sudo chown bor /var/lib/bor
Set the old heimdall home directory path, and export it. This will be used, whenever needed, moving forward.
# Note: in this case, the old heimdall directory is `~/.heimdalld`. # Modify it according to your setup. export OLD_HEIMDALL_HOME=~/.heimdalld
Link the Heimdall data folder and change the owner of files to user
heimdall, which is created in step 3
# Note: The commands given below are under the assumption that # $OLD_HEIMDALL_HOME contains the correct heimdalld home directory. # Modify it according to your setup as mentioned in the previous point. sudo chown -R heimdall $OLD_HEIMDALL_HOME sudo rm -rf /var/lib/heimdall sudo ln -nfs $OLD_HEIMDALL_HOME /var/lib/heimdall sudo chown heimdall /var/lib/heimdall
Check Bor Version
/usr/bin/bor version # Output - v0.3.0-stable
Check Heimdall Version
/usr/bin/heimdalld version # Output - v0.3.0
Create backup for Bor
cd ~ mkdir backup # Use the path of your bor.service file sudo mv /etc/systemd/system/bor.service ./backup
Create backup for Heimdall
cd ~ # Use the path of your heimdall service files sudo mv /etc/systemd/system/heimdalld.service ./backup sudo mv /etc/systemd/system/heimdalld-rest-server.service ./backup sudo mv /etc/systemd/system/heimdalld-bridge.service ./backup
Check and edit bor.service file to make any changes if required. You can skip this step if bor package is installed with correct network and node type arguments in step 2.
sudo vi /lib/systemd/system/bor.service
[Unit] Description=bor StartLimitIntervalSec=500 StartLimitBurst=5 [Service] Restart=on-failure RestartSec=5s ExecStart=/usr/bin/bor server -config "/var/lib/bor/config.toml" Type=simple KillSignal=SIGINT User=bor TimeoutStopSec=120 [Install] WantedBy=multi-user.target
Check and edit heimdalld.service file to make any changes if required (like adding bridge flag
--bridge --allfor validators). You can skip this step if heimdall package is installed with correct network and node type arguments in step 3.
sudo vi /lib/systemd/system/heimdalld.service
#Example heimdalld.service file for reference [Unit] Description=heimdalld StartLimitIntervalSec=500 StartLimitBurst=5 [Service] Restart=on-failure RestartSec=5s WorkingDirectory=/usr/bin ExecStart=/usr/bin/heimdalld start --home /var/lib/heimdall \ --chain=mainnet \ --rest-server Type=simple LimitNOFILE=65536 User=heimdall [Install] WantedBy=multi-user.target
Migrating to config.toml from start.sh:
To migrate the configuration/flags from
config.toml, you can use a utility script as described below.
$ git clone https://github.com/maticnetwork/bor.git # If not already cloned $ cd bor $ git fetch $ git checkout tags/v0.3.0 $ make bor $ cd scripts # Note: Running this command will output a config.toml which will have # values populated from the start.sh. Also, the script demands # for path to start.sh, validator address (if required) and path # to static-nodes.json if not found in the default location. $ BOR_DIR=/var/lib/bor ./getconfig.sh ... * Path to start.sh: /home/ubuntu/node/bor/start.sh * Your validator address (e.g. 0xca67a8D767e45056DC92384b488E9Af654d78DE2), or press Enter to skip if running a sentry node: 0xca67a8D767e45056DC92384b488E9Af654d78DE2 * You dont have '~/.bor/data/bor/static-nodes.json' file. If you want to use static nodes, enter the path to 'static-nodes.json' here (press Enter to skip): /home/ubuntu/my/path/static-nodes.json Thank you, your inputs are: Path to start.sh: /home/ubuntu/node/bor/start.sh Address: 0xca67a8D767e45056DC92384b488E9Af654d78DE2 Path to the config file: **/home/ubuntu/node/start-config.toml** ... # Note: We're considering /home/ubuntu as our home directory. Please # update the commands to use your home directory before proceeding. # Copy the newly created file from the output above # (Path to the config file) to /var/lib/bor/config.toml $ sudo cp **/home/ubuntu/node/bor/start-config.toml** /var/lib/bor/config.toml $ sudo chown bor /var/lib/bor/config.toml
For better monitoring of overall network and conditions of node, we have hosted ethstats dashboard (https://bor-mainnet.vitwit.com/). We would advise all the nodes to use the public endpoint for sending data (if you’re not sending it to some other internal endpoint). The endpoint can be configured in the config file itself and is of the format:
<node_identifier>:firstname.lastname@example.org:3000. We would encourage you to use a non-existing and understandable identifier. This dashboard would also help us in alerting incase the nodes are not working properly (e.g. if they’re constantly out of sync).
Open the file
/var/lib/bor/config.tomlin your favourite text editor and edit the value for
ethstats = "<node_identifier>:email@example.com:3000"
Ensure that all the values have been populated correctly in bor config file
/var/lib/bor/config.toml. If you want to change any config options, you can edit the file.
If you are not migrating from an old start.sh, the default values populated should work out of the box for a sentry node. For validator nodes, you might want to review and update some of the flags like:
ethstats, and additional telemetry configuration (from older start.sh based config).
Restart system processes:
sudo systemctl daemon-reload sudo systemctl start heimdalld.service journalctl -u heimdalld.service -f # Monitor heimdall service sudo systemctl restart telemetry.service sudo systemctl start bor journalctl -u bor.service -f # Monitor bor service
You can find the docker images in the links below. Look for
There are several potential reasons for this issue:
- Verify that
accounts.unlockare configured with the correct values.
- Make sure that only the correct key is present in the keystore directory. If multiple private keys are present in the keystore directory, remove any extra keys.
If the node encounters an “invalid bootstrap url” error, this may be due to one of the following issues:
- There may be invalid or malformed bootstrap URLs (enodes) present in
- The conversion script may not be able to validate the enode URLs by itself, leading to invalid URLs in
To fix this issue, check
config.toml and remove any invalid URLs from the file.
In some older versions of
BOR_DIR is not inherited from the environment by default. This can cause the conversion script to fail to automatically set the
When this happens, you will need to manually modify the
ipcpath flags to point to the new locations prefixed with
In some cases, you may encounter issues where
bor cannot find or access config files. This is often due to improper permissions on the old home directories of
bor. For example, if the old
heimdall home was
/home/ubuntu would need to have execute and read permissions for other users. Without these permissions,
bor will not have permission to list files under these directories, resulting in the error messages shown below:
ERROR: Config File "app" Not Found in "[/var/lib/heimdall /var/lib/heimdall/config]"
2022/12/12 15:18:06 Config File "heimdall-config" Not Found in "/var/lib/heimdall/config"
To fix this problem, make sure the permissions of the parent directories of
are set to 755. For the example above where the old
heimdall home is located at
/home/ubuntu/.heimdalld, you can fix the issue by running the following command:
sudo chmod 755 /home/ubuntu /home
With some features in this new version, bor might throw some specific error/warning messages. This section is just to clarify the reasoning behind those messages.
- Unable to whitelist checkpoint
- Bor leverages the checkpointing mechanism to stay connect to honest peers and it won’t connect to peers which are on wrong fork for large range of blocks (technically beyond the last checkpoint sent). For this, it will reach out to heimdall periodically to ask for latest checkpoints. Before it whitelists any checkpoint, it will perform some checks like do we have blocks for the checkpoint which we’re about to whitelist or not? If any of them fails, it won’t be able to whitelist them. A common scenario where this error is expected is when you’re syncing and have not yet reached the last check-pointed block. Once done, bor will eventually stop throwing this error.
- Dropping peer - checkpoint mismatch
- You might see an error for dropping peer which states “retrieved hash chain is invalid: checkpoint mismatch”. This is because before bor starts importing new blocks from a remote peer it will try to compare the blocks of that peer with it’s local copy of last whitelisted checkpoint. If there’s a mismatch it will drop the peer before even importing those false/wrong blocks with the stated error message.
- New command line interface related changes
- Documentation of all flags, defaults, and example configs for the new-cli for reference are available here.
- In v0.2.x,
bootnodeare two separate binaries. In v0.3.0,
bootnodewill become a subcommand of
bor initis removed in v0.3.0. Instead, on starting,
borwill automatically initialize a bor home directory if not found.
bor’s default behavior (running bor client) is moved to
bor serveras a subcommand.
GRPC: On starting bor using the
serversubcommand, it will start a GRPC server on a separate port which will be used for cli related communication. Here, the user using the cli, acts as a client and interacts with the node using GRPC. It is used for operations like adding and removing peers, getting status of node (sync status, forks, chain head, etc), getting debug and pprof traces, etc.
pprof options are merged to a new component,
--pprof.port 1234should be passed as one argument to flag
--grpc.addr 0.0.0.0:1234. If not set, default value will be
- Validator must provide its public address to flag
--miner.etherbasein order to mine blocks.
- The format of toml passed to
--configis changed. Notice that, in v0.2.x, only fields which did not have a flag like static or trusted nodes (inside Node.P2P tag) were mentioned in the toml file. With v0.3.x, you can pass the whole cli config using this flag. For more details and examples, visit the documentation link posted above.
--networkidis replaced by
--chain, whose value is either
mumbai. The flag also accepts the path to a custom genesis file, e.g.
--verbosityis replaced by
--log-leveland instead of numeric values, string denoting the type of log should be passed.
--metrics.portare merged to
--metics.port 1234should be passed as one argument to flag
--metrics.prometheus-addr 0.0.0.0:1234. Notice that, in v0.2.x,
--metrics.port, if not explicitly set, the metrics will be exported on
pprof.addr:pprof.port. If no flag is provided, the prometheus endpoint is registered on a default host and port.
- A new flag named
disable-bor-walletis introduced which disables the personal wallet endpoints in the node. This prevents misuse of the endpoints if exposed. It will only register and use the key-store for mining blocks (if mining is enabled through
--miner.etherbase). It’s set to
- Tracing of blocks which contain state sync transactions can be enabled by setting borTraceEnabled to true in the TraceConfig. Usage :
- Config related changes
- The default value of
gpo.ignorepriceflag has been aligned with the value of
txpool.pricelimiti.e. 30 Gwei for mainnet (which is also the value for
miner.pricelimit). More context here.
- The default cache value (set by
--cache) is set to 1024 which is suitable for mumbai network or devnet. For mainnet, you’d need to explicitly specify the value. The suggested value for the same is 4096.
- The default value of
- Checkpoint whitelisting feature
- Bor now leverages the checkpointing mechanism (of heimdall) to whitelist checkpoints locally and prevent itself from connecting to wrong peers.
- Before starting to import blocks from any remote peer, it will check the integrity of blocks (to be imported) with it’s local copy of latest checkpoint.
- If they do not match, it will prevent downloading blocks from that peer by disconnecting with it. If they match, it’ll proceed to import blocks normally.
- See this PR for more reference.
- In v0.2.x,
bridgeare two separate binaries. In v0.3.0,
bridgewill become a subcommand of
- In v0.2.x,
rest-serverare run in three different processes. In v0.3.0, they could be launched together in a single process with one CLI command.
- In v0.3.0, building different
heimdalldbinary for different network is no longer needed.
heimdalldbinary will support both mainnet and mumbai testnet.
- Log level of heimdall is changed from a single letter to a full word. Example:
I[2022-09-22|19:11:42.896] Served RPC HTTP response
INFO [2022-09-22|19:11:42.896] Served RPC HTTP response
- Heimdall no longer needs a genesis file. Instead, the genesis file will be embedded in the binary of heimdalld.
- Provide possibility to log in
jsonformat by setting the param
logs_type = "json"
TaskDelayBetweenEachValfrom 24s to 10s to reduce
- Log elapsed time for each method involved in the
- Add benchmark tests for the most time consuming functions involved in the
Edit 1: The command for pruning was missed out in the new CLI but will be included in the next release (due early-mid Jan). If you want to prune your node, please do so before upgrading to