Slow syncing after reinitialization

Good evening, i’m having some trouble on the node since the BOR update from a few days ago

The step i took was:

  • updating BOR to latest version according to the instruction in this forum
  • running ./clean.sh
  • running ./setup.sh
  • Downloading latest snapshot bor-fullnode-snapshot-2021-06-22.tar.gz (is this the latest?)
  • Unzipping the content in the data folder
  • Starting the node back with the following:
#!/usr/bin/env sh

set -x #echo on

BOR_DIR=${BOR_DIR:-~/.bor}
DATA_DIR=$BOR_DIR/data

bor --datadir $DATA_DIR \
  --port 30303 \
  --http --http.addr '0.0.0.0' \
  --http.vhosts '*' \
  --http.corsdomain '*' \
  --http.port 8545 \
  --ipcpath $DATA_DIR/bor.ipc \
  --http.api 'eth,net,web3,txpool,bor' \
  --ws --ws.port 8546 --ws.addr '0.0.0.0' \
  --ws.api 'eth,net,web3,admin,debug,bor' \
  --rpc.txfeecap 50 \
  --syncmode 'full' \
  --networkid '137' \
  --snapshot=false \
  --miner.gaslimit '20000000' \
  --miner.gastarget '20000000' \
  --txpool.nolocals \
  --txpool.accountslots 16 \
  --txpool.globalslots 131072 \
  --txpool.accountqueue 64 \
  --txpool.globalqueue 131072 \
  --txpool.lifetime '1h30m0s' \
  --maxpeers 200 \
  --metrics \
  --pprof --pprof.port 7071 --pprof.addr '0.0.0.0' \
  --bootnodes "enode://0cb82b395094ee4a2915e9714894627de9ed8498fb881cec6db7c65e8b9a5bd7f2f25cc84e71e89d0947e51c76e85d0847
[email protected]:30303,enode://88116f4295f5a31538ae409e4d44ad40d22e44ee9342869e7d68bdec55b0f83
c1530355ce8b41fbec0928a7d75a5745d528450d30aec92066ab6ba1ee351d710@159.203.9.164:30303

Now the issue is that i’m syncing new block really really slowly (around 360 blocks every minutes)
BOR attach

> web3.net.peerCount
200
> eth.syncing
{
  currentBlock: 16221090,
  highestBlock: 18488166,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 16149020
}

journalctl -u bor.service -f
I constantly receive the following message

Aug 28 18:09:58 Pos-Full-Node bash[15951]: INFO [08-28|18:09:58.409] → committing new state                   eventRecord="id 492444, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x00000000000000000000000073174e6da2e1aed7fdf504cb2a94184729f07d730000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb00000000000000000000000000000000000000000000000008ac7230489e800000000000000000000000000000000000000000000000000000000000008d58154, txHash: 0x5bfcfb56ac3ec30c4ad9ddae3e96b1d04d7011389146914b98c625b76a237318, logIndex: 386, chainId: 137, time 2021-06-27T21:56:31Z"
Aug 28 18:09:59 Pos-Full-Node bash[15951]: INFO [08-28|18:09:59.168] Imported new chain segment               blocks=116  txs=13914   mgas=1394.992 elapsed=8.032s      mgasps=173.676 number=16,224,394 hash=cb641b..94faf9 age=2mo1d18h dirty=249.39MiB
Aug 28 18:10:02 Pos-Full-Node bash[15951]: WARN [08-28|18:10:02.569] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:02 Pos-Full-Node bash[15951]: WARN [08-28|18:10:02.570] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:02 Pos-Full-Node bash[15951]: INFO [08-28|18:10:02.570] Fetching state updates from Heimdall     fromID=492,445 to=2021-06-27T23:59:02+02:00
Aug 28 18:10:02 Pos-Full-Node bash[15951]: INFO [08-28|18:10:02.570] Fetching state sync events               queryParams="from-id=492445&to-time=1624831142&limit=50"
Aug 28 18:10:02 Pos-Full-Node bash[15951]: INFO [08-28|18:10:02.573] → committing new state                   eventRecord="id 492445, contract 0x8397259c983751DAf40400790063935a11afa28a, data: 0x0000000000000000000000000a0bbc022542ebe87ab4f58b3960e7b6176f704d00000000000000000000000091ec4095c74669958ae42bb7ff7925c3668384a2000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000019000000000000000000000000f751f674843f5bf8802c6e373a2fd2fc0027cd85, txHash: 0x3b775485171081992be9ebea29ba4404aca5f965942c5f861da4f55772b389d0, logIndex: 143, chainId: 137, time 2021-06-27T21:57:32Z"
Aug 28 18:10:02 Pos-Full-Node bash[15951]: INFO [08-28|18:10:02.584] → committing new state                   eventRecord="id 492446, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000a041de78be445480fa111e85fb4511a6c471e5f8000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000429d069189e0000, txHash: 0x517fd74a1fe4d68f03241ac944caf6584fccca3e99a5066ddbdef705aeff1796, logIndex: 172, chainId: 137, time 2021-06-27T21:57:32Z"
Aug 28 18:10:02 Pos-Full-Node bash[15951]: INFO [08-28|18:10:02.584] → committing new state                   eventRecord="id 492447, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x000000000000000000000000de0b4f4ddb6e024cc0224f12c7ffa4f3f28c96720000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb000000000000000000000000000000000000000000000000164a8bdd5e78a00000000000000000000000000000000000000000000000000000000000008d58155, txHash: 0xc7fc6e8a9e9f7a044d7ca4ae688bbc6559242e867cad700eb4a1ef99ed2ce1f3, logIndex: 259, chainId: 137, time 2021-06-27T21:57:32Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.211] Imported new chain segment               blocks=110  txs=10404   mgas=1316.582 elapsed=8.042s      mgasps=163.703 number=16,224,504 hash=4529d8..95164f age=2mo1d18h dirty=254.74MiB
Aug 28 18:10:07 Pos-Full-Node bash[15951]: WARN [08-28|18:10:07.961] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:07 Pos-Full-Node bash[15951]: WARN [08-28|18:10:07.961] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.961] Fetching state updates from Heimdall     fromID=492,448 to=2021-06-28T00:01:14+02:00
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.961] Fetching state sync events               queryParams="from-id=492448&to-time=1624831274&limit=50"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.966] → committing new state                   eventRecord="id 492448, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000051e6ceb5a318b1e288b26ef9479cd2f845e2efe8000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000570a9ec4ff40000, txHash: 0xc1415800696a08235fd8c826d54e261527beb65de3fae6cc87027de34eaceec6, logIndex: 398, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.967] → committing new state                   eventRecord="id 492449, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000178f41987d8bb0476e14824c166226ae65beffc4000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000016345785d8a0000, txHash: 0x22111b04c22c11184c015daae93f0384194b3ce8dfb2d806ee95f32da81f4cae, logIndex: 387, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.968] → committing new state                   eventRecord="id 492450, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000b3e29a8702519ef2d6dc0ed9b06b3f8112a82345000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000002386f26fc10000, txHash: 0x254571124c14207d85eba20f43442fe22e2ba31f00fba20770691606d8b490f5, logIndex: 219, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.968] → committing new state                   eventRecord="id 492451, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000707db8e9fabfbb9139ee4abed9458c2d5ce10e98000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee0000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000029a2241af62c0000, txHash: 0x143032927c84da204aae52275de2ef2995135529414df5dbae282c71add24f39, logIndex: 224, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.968] → committing new state                   eventRecord="id 492452, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000be52aa2f4ecb48df5bce64d920cf11e37275afa4000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000002e2f6e5e148000, txHash: 0xb1c2f9183328d731e651ea06c99499e6ebb83888e3bd4a6c87f331beb47aa4fb, logIndex: 374, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.969] → committing new state                   eventRecord="id 492453, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000fdb9e47ead181849161e5e720d770bf25a3efb0300000000000000000000000040986a85b4cfcdb054a6cbfb1210194fee51af88000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000005f6e0c907dfa91102ab, txHash: 0x885711b01a9a4afbe43d68819d73cb3ecc83d7b801e0cbaa384ed68b82e4b165, logIndex: 75, chainId: 137, time 2021-06-27T21:59:31Z"
Aug 28 18:10:07 Pos-Full-Node bash[15951]: INFO [08-28|18:10:07.970] → committing new state                   eventRecord="id 492454, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000da8575475c702d398c8bf8b9cea2576a935ae704000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb480000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000001dcd65000, txHash: 0x9746714c7cc60108bd1081d861f9feacdf813fe780668b2795ae337377acbff0, logIndex: 183, chainId: 137, time 2021-06-27T22:00:32Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: WARN [08-28|18:10:13.502] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:13 Pos-Full-Node bash[15951]: WARN [08-28|18:10:13.502] Caller gas above allowance, capping      requested=9,223,372,036,854,775,807 cap=50,000,000
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.502] Fetching state updates from Heimdall     fromID=492,455 to=2021-06-28T00:03:26+02:00
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.502] Fetching state sync events               queryParams="from-id=492455&to-time=1624831406&limit=50"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.508] → committing new state                   eventRecord="id 492455, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000d589e370c423b8185b57aea51dbeeaa3f889a624000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee0000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000001bc16d674ec8000, txHash: 0x03c8266ae863b30f4d9fbc076b654482f26ec128c17a2fb0285c8cf6ef7092b7, logIndex: 60, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.509] → committing new state                   eventRecord="id 492456, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000e1e7ba7f42a262069b8f296be7cbf9bff96f4ad8000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000186cc6acd4b0000, txHash: 0x8b58552056e5661d1ea8d9fea86a19a53ec32638a130849851e13afb1a56bd2b, logIndex: 146, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.510] → committing new state                   eventRecord="id 492457, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000fdb7eec5ebf4c4ac7734748474123ac25c6edcc800000000000000000000000033d0568941c0c64ff7e0fb4fba0b11bd37deed9f00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000ff8035888357a8d5ed2, txHash: 0x7f9abd6d12c19702b973d54a1a9abbe901ab71fd432fd7859db6ed428d7d345b, logIndex: 368, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.511] → committing new state                   eventRecord="id 492458, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000065c0a23b94f4c263f735f0184944b0c0ce7a3529000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb4800000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000003b9aca00, txHash: 0xf74803d25954f28e03e5ebbc7698c3884a04ea7eec898f6459ef383ca14e92e2, logIndex: 189, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.511] → committing new state                   eventRecord="id 492459, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000ace39a7bc696edf13a6b473ee3ad52bb16f998b9000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000886c98b760000, txHash: 0xf151f44af8a808ee7491f2fd240638d3a22b9fd1f165b1dd86f7ab49cc968162, logIndex: 236, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.512] → committing new state                   eventRecord="id 492460, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x00000000000000000000000040c77eeb999e05388d31803d57c3f1147a8e3be00000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb0000000000000000000000000000000000000000000000002b5e3af16b18800000000000000000000000000000000000000000000000000000000000008d58156, txHash: 0x122274db84be9998c6bf4deaedaba66d90f3d0f1431e8286e538f601f40db382, logIndex: 99, chainId: 137, time 2021-06-27T22:01:36Z"
Aug 28 18:10:13 Pos-Full-Node bash[15951]: INFO [08-28|18:10:13.512] → committing new state                   eventRecord="id 492461, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x0000000000000000000000000d8b49be1176b7c9436167a4faa2c0f8547aa7e70000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb000000000000000000000000000000000000000000000000c95fe13b0b39a1a760000000000000000000000000000000000000000000000000000000008d58157, txHash: 0x8a480813228a62db375821877da58cbc2d9a14a93eb740fda66f2e7fbd73b5dd, logIndex: 179, chainId: 137, time 2021-06-27T22:02:37Z"

I’m out of ideas to make it work

Can anybody point me to the right direction?

Thanks

Check your CPU, RAM usage, and disk IO (make sure on SSD). We were having same issues until upgrade of resources of our nodes to double what it was before the geth update… The latest geth security update seems to suck up a ton more resources (IMHO).

Thanks for the reply, i checked RAM, Cpu and Disk. my server don’t seem to be overloaded
I have an Octacore with 64Gb RAM and NVMe disks…
image

That should be enough resources, I believe, although I am currently running 16 vCPU and 64GB of RAM (not all used on RAM, but CPU is pretty high). Come join the Discord channel validator-queries and see if anyone else has suggestions for you. Great group of people there.

Today i started from scratch deleting node-ansible, heimdall and bor
Then i made the whole installation from the beginning using the latest snapshot available for Heimdall and Bor, but still i sync very slowly, it will take weeks for Bor to sync at this speed

Have any of you succeeded to get the node started? Mine is not working and I cant get it to sync.