Slow catching up blocks & dropping blocks

Hello.
I have been unable to catch up blocks with my bor for 1 month.
This both to slow syncing, dropping blocks and starting over with new snapshots.

My process has been like following.

I was running on a machine with 2x m.2 2tb drives for storage.
One was storing ancient data and second one was storing chain data.
After upgrading to v0.3.3 permissions got messed up and I had to change those for bor & heimdall to even run (Nothing mentioned in upgrade instructions at the time). But it was running fine after that.

  1. Everything was fine except that I was running out of space and I couldn’t prune
    I got an error that said: Failed to open snapshot tree
    err="head doesn’t match snapshot: have 0x…, want 0x…

  2. I was recommended to change multiple settings and nothing worked.
    I’m guessing that by upgrading from v0.3.0 to v0.3.3 something got messed up.
    Running /usr/bin/bor version Shows up as 0.3.3 Running bor version Shows up as 0.3.0 at this point
    I did → rm /home/polygon/go/bin/bor to solve this issue

  3. I was recommended to start over with new snapshot from https://monitor.stakepool.dev.br/snapshots
    Doing this did not help. It gave me new error that I didn’t have with my chaindata.
    Error in block freeze operation err="block receipts missing, can't freeze block 37661243"

  4. I was recommended to do everything from beginning on new machine.
    Hardware spec that I’m using now:

  • CPU: Ryzen 5700g 8c/16t (Up to 4.6GHz)
  • SSD: 4TB M.2 ( Up to 3500/mb/s ) +500gb sata SSD with ubutnu 20.04.6LTS
  • RAM: 32gb (3200 mhz)
  • 100mb/s network (1gb/s network instead of 100mb/s does not make a diference. Bor & heimdall together are only using max 1.2mb/s download )
    image

    I have installed bor & heimdall on 500gb sata SSD & I’m storing both heimdall & bor data on my 4tb m.2 ssd.

    Following official installation guide gave problems with permissions.
    I had so much problems with this that I on the end had to give run heimdall and bor as root user by editing /lib/systemd/system/heimdalld.service & bor.service
    Whatever else I did resulted in heimdall & bor users not having permission in new or old data.
  1. Having problems with catching up blocks
  • I still have error from downloading snapshot: Failed to open snapshot tree err="head doesn't match snapshot: have 0x..., want 0x....
  • Catching up blocks is slow & Bor is dropping blocks
    image
    image
  1. I was now recommended to start over with new snapshot from https://snapshots.polygon.technology/
    I have downloaded this snapshot but I’m now almost there with catching up blocks on previous snapshot with the error
    image
    Would love to solve this error instead: Failed to open snapshot tree err="head doesn't match snapshot: have 0x..., want 0x....
    & to increase speed off catching up blocks.

Anyone that can help me stop losing/dropping blocks & sync normally as I did before upgrading above v0.3.0?

I’m using bor v0.3.7-stable & heimdalld 0.3.0
My bor config looks like this:

chain = "mainnet"
# identity = "node_name"
# log-level = "INFO"
#datadir = "/var/lib/bor/data"
datadir = "/media/polygon/4tbm2/data/bor/data"
# ancient = ""
# keystore = ""
# "rpc.batchlimit" = 100
# "rpc.returndatalimit" = 100000
syncmode = "full"
# gcmode = "full"
snapshot = false
# ethstats = ""

# ["eth.requiredblocks"]

[p2p]
    maxpeers = 200
    port = 30303
    # maxpendpeers = 50
    # bind = "0.0.0.0"
    # nodiscover = false
    # nat = "any"
  [p2p.discovery] 
        # v5disc = false
        bootnodes = ["enode://0cb82b395094ee4a2915e9714894627de9ed8498fb881cec6db7c65e8b9a5bd7f2f25cc84e71e89d0947e51c76e85d0847de848c7782b13c0255247a6758178c@44.232.55.71:30303", "enode://88116f4295f5a31538ae409e4d44ad40d22e44ee9342869e7d68bdec55b0f83c1530355ce8b41fbec0928a7d75a5745d528450d30aec92066ab6ba1ee351d710@159.203.9.164:30303", "enode://4be7248c3a12c5f95d4ef5fff37f7c44ad1072fdb59701b2e5987c5f3846ef448ce7eabc941c5575b13db0fb016552c1fa5cca0dda1a8008cf6d63874c0f3eb7@3.93.224.197:30303", "enode://32dd20eaf75513cf84ffc9940972ab17a62e88ea753b0780ea5eca9f40f9254064dacb99508337043d944c2a41b561a17deaad45c53ea0be02663e55e6a302b2@3.212.183.151:30303", "enode://3178257cd1e1ab8f95eeb7cc45e28b6047a0432b2f9412cff1db9bb31426eac30edeb81fedc30b7cd3059f0902b5350f75d1b376d2c632e1b375af0553813e6f@35.221.13.28:30303", "enode://16d9a28eadbd247a09ff53b7b1f22231f6deaf10b86d4b23924023aea49bfdd51465b36d79d29be46a5497a96151a1a1ea448f8a8666266284e004306b2afb6e@35.199.4.13:30303", "enode://555b6acbe834d6e27e78ff50bb069324e8c5fb6fe661b5e100f988546da45ed7dda81359a9abf88bf894c19f15b41bba0c81d9d20ce35c212816808143e196c7@3.209.128.173:17377", "enode://defb0f28acf8a2d7416e7a271987520133ef866a2d444daa76d3e1f7beadcf60ba99081499f80d0385bc4ce7ecc7e364f2d5d0d73c8e448bf2b1e09a3f1cd26f@168.119.56.213:30303", "enode://de622c12037d5d61255fbd2c84ca9d72028c6c7b6a0fac350cc59b8a76e5371c10eb584d5d6303a7271fb1dcfd946bec1bb07677f58323f1980a7ef3f46ce09f@89.187.168.120:30010", "enode://3ac374904abfd3cfdd3357cc6bd0a668657c1b5280be72b9332aac2183eefced88638fb997b056995398486d7e8073e5d79140b84fcafd6ed4a84378656b277c@15.204.161.236:58326", "enode://f5c402acf331f9c7aa4ba1018b7dea601b5be776760c7bef27ebd3eedf99cc596589828811a8bbe8271488f9c2f42e1f1f199f0ea1dc1e75e28a4fbbf36eb097@52.48.97.75:30311", "enode://186ca4b2a502f75becc3c1f10765e2393f3a45fef0d673577d8093ed873fb7f9e68ce28d6ca4e89f546f4ad3cdcc8c399ddbadfe22d82b7706db5836dca94c19@65.21.75.50:30303", "enode://bcbbd590b7eb93e2f0973e676d16f2da719baca00c7a75684cd7e4755d49ad1599cc7f994c6319fcc7c36c7ce457349693b6843bbf7ca9d13d3f7b90b839a039@88.99.84.228:43684", "enode://7a09aaaa3e0a0338d9e0efa3157302ed9197c3eafa3de6b1cf858f319e265a3a03c0ecad87e493b1fa2941f4acd9731eb89ca0db0346a5602896f24268c30870@130.61.114.35:30303", "enode://f4b228759dac942bd79c374f5ea53b9304c3679d534ee3544c1f6d31b65f6fac4b639bceb1ced3f6e8521b40e41547517ea21e168a993abf2628fec8107e66f7@64.46.111.190:31410", "enode://89e78d926e04e13d6916bbd1161dbdb30fddb992339fd6bbdad078e2d0084cf6c553d03bd82f70b50c872e02430e9f417b89eaa6c89d92c2562ec22bc8df4711@3.239.49.184:35672", "enode://377d8004dcd1d971b9741a487b19f7bc9061989dccbb1ce22b2808ee947c95580d5b7bef93c6e24b077b3ce7a6448985b178c5ad764dee4b4d5c591be6d48d10@18.211.38.239:41815", "enode://567a92b00bef7e961b8dea91457a28a907a5856d3189ff94fd4cfd02e1121cabd688819406d1bdc0260b7556fe4e9b9b436ad3f05228289539fc9fde4d6627e4@65.109.111.198:30303", "enode://9e8620d654f508fb1a0c5e6f4572f8df0c827539cd20592dc307f6c2268f03734426f83c34cce9febe53aa6c89621f1b53cae47a64e853a3b9274897e7316d5e@34.213.79.29:15158", "enode://f5bbc8ff8e893fc0b2e7e8e8d8771084ccd4e3c2dd6438ddfdc530a1d1f89c5008135fc441e3edda21e2a9836e2f571782efda2da9d837678a4497254c46bad3@89.187.161.243:30010", "enode://a89498d11eb1466806347e5e0878c68cf4a2de2cbcb671a95ab1ce32831e6b457802eace43dd707ea42839eda3f563d081d387be1b057e0446e515a06c8be6eb@138.2.90.210:31954", "enode://3533205e803e98eb64e03b9dfc9924060a91fa8d91a7baa342db1fd328e354c0fb208bddba3fd0b5cd56269ed58d8f890bd37afa34c77fa23b26c9a2e3ef31a4@65.109.231.197:30303", "enode://61f65fadd8b2b28d11aefddb1be5915bfbea2816630de83060a4bb29d560625724ab5160371e0ef1a8c171ee89ad9330732fde4223a48cbc224f7b6345d21053@63.251.87.39:11410", "enode://703b9207366b277e519a605444f7e03e971b9a16bfd497fc71c078581b4e5bde2520ead47f49206b42e5a7a6f3ae1ed886c3f0182c4791364a53cd4766998355@91.107.128.243:30303", "enode://7a8ada240e032006d2282d37579a44f8ea427d7911b918708a8923377310c34cef32cdef138a464e8377ed701e0b91b58cb3b9ef84a60e7843a488ad1ef93418@89.187.185.158:30305", "enode://abfaf1c4628728f1e160a9685c6965bb22b2ea184e9241d8f345438db3010fb33c7ca410dfc1efcd4d7ef8334645805cd48cca49f390c3d472a605c00242e682@192.248.162.140:30303", "enode://317c96722317608638a19d032a56712f34f4391fcd859b6be17841f1a0bebf453630de82b403799f628e255ca8e86fb6bbe95f1e47b0076869dcbced627001fb@8.219.137.193:26105", "enode://219c3848da97335817a1944dbb50877927cdf77262e201c3fa453b3257f33682e90bae04207ecfd79a1e33e94171aa099b9a921cd9156960f6259fb024ab3c78@137.220.50.231:30304", "enode://4bf93e09f29480e895ce42c64b8cfac2de47b45f04e16b7ee447cc88aeaf051eecf7cd083b07820dfb95c37f0ec491afe043f420a4e7f79587064e65705f32de@62.12.164.254:43204", "enode://56f16ff1b7ecdf4df2f1ba24b816fefb6e27207397e54d780eafd848fdb9ccb3e66b3ff1749f773eced273777d8fb26e2cba70802e83e91b3762db0d2f6ca371@5.161.112.114:30303", "enode://d5f5a109bf8077aff5d773fe23d58a8ad7926f6fec74990d4fd4bca371ad40e357ad61aa9598e65b88b73acc89f99f6cd8d6a23e358538b00f3f25840292ec27@54.67.68.44:53624", "enode://6866b65cabab127fba0fb3af29de2e97fa8f5a9c636d34c1f8d7539209aa79fa0d62b1ed6ce06cfc639e99550422bdc7f01a98a58b3e82cfe9fa0f67fb0bc46e@162.19.136.124:44580", "enode://dc24c818466b1f24c1a374f037e173e84c0260a876d260722bbf0c1c0e339f62a98108f96c4cf66a4392b4536968cf9ec4327f965d355d5a79ec7651a40d3f5f@5.161.190.161:30303", "enode://5f1bf46cdc0e35d5733905c0a7c3bcbdda20cecc3feb937999c78b95e1f71c287344e170ce960feb9d5a859bb5bea68fe2eb8c26ad16e7bcc67e765ed7326051@185.229.191.148:30305", "enode://e8c3fd203864c078cf1f48f3543d679e3e0535585f3cca101500539cf5a3074dbd7331caafcacb99441230c0c2068908c560fab351394f125a606795e4bee79e@3.143.17.2:58664", "enode://c80d83b52bd126a39fc097e3c080233d13045c4dbc7d7de0dea0c9064199d6759d897695b12b1f01d2122e004d5c3e33a9849feaca6e0608cf1fe32afa954880@18.232.102.12:30303", "enode://12966a659d0658842b4abfb194aa368b8fe3826feced761991b55d09d014476a2ff989806a76d59ccaa7e3859d01509a6afb483e10bfa8dbe3e0688cf2db200d@37.19.216.43:30305", "enode://48a6bd11d75471e049da4876afc02e2d2a2ec754adc2852711593af3998c7b0b0ce6fe1d9977353910ab26a2419938325c3591874749744039b37d82633cb5c8@65.108.67.90:32303", "enode://2cb979d2451f3e883ae7ea47b2f633a4d63f3e2e088cce6d126faf3f8da2834bbba9c0c953353756bb277ee5aed18259e6ba845d58acd0095280383bd15ad110@45.250.254.64:30303", "enode://88bf7e1965d03e9786b5be053b762db8c59a8d91c6ace73c7707327bd01dd83bbc5c27beb6a4347438b096c1f71455bf8b594c1afe3a96baecf56f946b877b4f@57.128.92.32:30304", "enode://a447c5ab44fad2b92fd3335a4afea4a7b4cf9b40dc342e763d2d134681bf64154aa0a3ef7954476a2199e007a392b4a631df44e61040946cfa269ce258fb7c09@198.244.202.206:30303", "enode://b550180557ca898cdd57607e6e82699183660100e65e02346974020eddd4d0ca0721093d9599f4d62b17d7f6565cfac93c14ad1788dd751744c0b3ef921c18c8@54.92.126.8:9937", "enode://b3049a55def9c5fd49bf47f57f04780d060d4bf38b354dfdaa9b3bbc49501666c868de2e84eea44b79aaa234db6154e440e40d8b0513e586e59a406524fd0237@107.6.91.37:11410", "enode://f88b64da3b7d6bd01bd8b0d22c89fbbd2f29b8b5bbd49caec57abb6182d8288e09ae1a3d46e57de06d607c493c6b8c77c99893ddafee6b87819c70914715ba2f@123.58.220.36:42862", "enode://bed1f3995b81e385cfe0a7e24fed1c77816d811f6840d97bb615423d8f05f1dd8633c41e37901518132bb2de656bff3b94916b2e946493f4f0ff4fb41b2e4da2@78.46.39.55:6050", "enode://cb32f4136aa1fc63d99e3ebb0d15d1384f08da0f9fc5b7935d9f2efb42f2bf16de72e8d47da3bf719b82803e1431a5f1015d3f36917e82aa4996f0f67a8c279f@35.196.69.96:45650", "enode://ee2a06e8a6d92d91993068d9d7d4ab5adf2e8d2c26d3fda5bb7bd107f9159a3d92c12c4e845394fbaddf8eaba2ae1dea1587a2e0a48e4cb2af6b896225323f82@167.235.96.211:30303", "enode://3cd348c888fab8a8e3796f3a0eb00714798d4783524637e9cc9e558cde9ac659eedbd2c04439ad100901b616642e34431e4af216ffc6dfa5100325d3f07ab2b2@5.9.101.101:41230", "enode://0d80864c9e0742c0d530af8dda2d4f11a6678a36d815de80e38e7e9ad4958f09ce4ccfa17c4b6f71953f3b5a98944c71629cbc70c3c667855de8f8d8c4708342@3.213.8.33:24895", "enode://8ec74849bff54917e25a4a31cdb465dba3004ba7c48979cce0724d255cdabe3786d7775bd23d89927137e0203f5901b2e11dd9cc372de9d095664497128eb94e@3.213.8.33:49393", "enode://60ac80cb34e3120fd941e5b3ec390af6ebd0c766c6a73c837b450e68469ca4b7fc3ef90db421ca14a7d8a9d855727b407ec32844ba8d52ae27f79dbeae36a9fe@198.244.228.25:30303?discport=16751", "enode://f5b4bf1a5214bd91e63c341b78376fe5bb6873888a2ea1c5999819fa337a950a5dc997163b34f6b7f9066fc668d4d4de6940ce64651243919167ed2b56368e87@174.170.251.85:30304", "enode://cf5d37c8fcee624aaf41d5111dbeaba25aa2e99a626a79454278bcbe3bfadcf07bcc3036eda901c1bd3a1eb31066d3e128fdd981d8c65f5b9c32280ff53312b7@66.45.226.70:55790", "enode://ecdfd400c2a3048054c34da7486da1f342983dca504d0e7a9972c15651ee9ce04d51ff09b0210a51827706c646d0c4b944c338ddf57d87bd28dea2763e2c721a@143.244.63.70:30305", "enode://b9a8ced7aeb94d89dcbf11d3d9912960e4ddeb28f379fd63734756f95d6af4bef079bde4142f81d9b9ff92cff2a5ae8646b4b958e2d7ca1275c94344c06a6556@52.48.97.75:58686", "enode://6cdc2e4294e3b5138366c4ca45d0d6fde287ab10a7e8b704290f09413f8043171892291b4a98fce77eb0984194f3d0425673294e1715d4a398f73c1256ae5b59@157.90.134.197:30303", "enode://ffb03bf2ff102f925872a23ae6f92aab452d2d82db1022d9e6c054670359fcc1b9a3366af42da8e07b3f7a4d70026f77bb8c2cee3bcceecdf97cf87b510f746d@185.216.215.11:30303", "enode://7d68f217c9286693bd9ca63c8ee0398395427129d8b79be06fb498a964ea35e1df98bd4815e96f98fa39825caea312792b07550046521084b83326f6e829fa5b@5.161.157.33:51526", "enode://8fb412734b80e699765c37625f06764069595b9c69134268cf67e0f417e7dc495a9c502357cca664728bfbe558eea5c8f14fa78f1b82d56842d95b7a40cb86e0@13.125.5.42:30303", "enode://e4feae412ab1a9d59416b81ad1eb679e481ba0c6518dc5bb3ebbe4f75d2d50819db5c79eafebaa0d0760cd93f7f921991ca8750b319e6c644db4f136ad8d41d1@206.246.93.163:31410", "enode://602ff6ab87d1827de5ce3e4b6a23ea883a545f5a199613da1fb11e075831fcef6859fcb2224422b72fc9e0a47fc58544d80829c37c8d3d75ace77def8252d8c1@61.129.251.197:36120", "enode://921b8f248828cbe61980a964f8288139c1a426cd654fbd3f3bc89b1296303caf603069bbeca591942bfb3a6fe16c73955c93d68b3d53232b73e299ff48eb6cab@192.9.158.240:30303"]
        # bootnodesv4 = []
        # bootnodesv5 = []
        static-nodes = ["enode://3178257cd1e1ab8f95eeb7cc45e28b6047a0432b2f9412cff1db9bb31426eac30edeb81fedc30b7cd3059f0902b5350f75d1b376d2c632e1b375af0553813e6f@35.221.13.28:30303","enode://16d9a28eadbd247a09ff53b7b1f22231f6deaf10b86d4b23924023aea49bfdd51465b36d79d29be46a5497a96151a1a1ea448f8a8666266284e004306b2afb6e@35.199.4.13:30303"]
        trusted-nodes = ["enode://3178257cd1e1ab8f95eeb7cc45e28b6047a0432b2f9412cff1db9bb31426eac30edeb81fedc30b7cd3059f0902b5350f75d1b376d2c632e1b375af0553813e6f@35.221.13.28:30303","enode://16d9a28eadbd247a09ff53b7b1f22231f6deaf10b86d4b23924023aea49bfdd51465b36d79d29be46a5497a96151a1a1ea448f8a8666266284e004306b2afb6e@35.199.4.13:30303"]
        # dns = []

# [heimdall]
    # url = "http://localhost:1317"
    # "bor.without" = false
    # grpc-address = ""

[txpool]
    nolocals = true
    pricelimit = 30000000000
    accountslots = 16
    globalslots = 32768
    accountqueue = 16
    globalqueue = 32768
    lifetime = "1h30m0s"
    # locals = []
    # journal = ""
    # rejournal = "1h0m0s"
    # pricebump = 10

[miner]
    gaslimit = 30000000
    gasprice = "30000000000"
    # mine = false
    # etherbase = ""
    # extradata = ""

[jsonrpc]
    ipcpath = "/var/lib/bor/bor.ipc"
    # ipcdisable = false
    # gascap = 50000000
    # txfeecap = 5.0
    [jsonrpc.http]
        enabled = true
        port = 8545
        host = "127.0.0.1"
        api = ["eth", "net", "web3", "txpool", "bor"]
        vhosts = ["*"]
        corsdomain = ["*"]
        # prefix = ""
        #     ep-size = 40
#     ep-requesttimeout = "0s"
    # [jsonrpc.ws]
        # enabled = false
        # port = 8546
        # prefix = ""
        # host = "localhost"
        # api = ["web3", "net"]
        # origins = ["*"]
        #     ep-size = 40
#     ep-requesttimeout = "0s"
    # [jsonrpc.graphql]
        # enabled = false
        # port = 0
        # prefix = ""
        # host = ""
        # vhosts = ["*"]
        # corsdomain = ["*"]
    # [jsonrpc.timeouts]
        # read = "10s"
        # write = "30s"
        # idle = "2m0s"

[gpo]
    # blocks = 20
    # percentile = 60
    # maxprice = "5000000000000"
    ignoreprice = "30000000000"

[telemetry]
    metrics = true
    # expensive = false
    # prometheus-addr = ""
    # opencollector-endpoint = ""
    # [telemetry.influx]
        # influxdb = false
        # endpoint = ""
        # database = ""
        # username = ""
        # password = ""
        # influxdbv2 = false
        # token = ""
        # bucket = ""
        # organization = ""
    # [telemetry.influx.tags]

[cache]
    cache = 8192
    # gc = 25
    # snapshot = 10
    # database = 50
    # trie = 15
    # journal = "triecache"
    # rejournal = "1h0m0s"
    # noprefetch = false
    # preimages = false
    # txlookuplimit = 2350000
    # timeout = "1h0m0s"

# [accounts]
    # unlock = []
    # password = ""
    # allow-insecure-unlock = false
    # lightkdf = false
    # disable-bor-wallet = false

# [grpc]
    # addr = ":3131"

# [developer]
    # dev = false
    # period = 0

UPDATED INFO:

  1. I have noticed that catching up blocks is much faster when storing heimdall data & bor data on separate drives
  • seems that it does not matter that much that it is m.2 drive when storing both heimdall and bor data
  1. I have downloaded new snapshot and I’m syncing from that one
  • I do not have problems with this error anymore: Failed to open snapshot tree err="head doesn't match snapshot: have 0x..., want 0x....
1 Like

I’m up to date with block syncing.
Now my problem is that block syncing is to slow to catch up blocks from time to time.

Whatever the problem is. It came with versions above 0.3.0