BAD BLOCK error on Bor and snapshot downloads

Archive Network
1

Our Matic node crashed this weekend We are trying to restart it from the latest snapshot from https://snapshot.matic.today, to ensure the node follows the correct chain tip. However, after few attempts we are getting the same BAD BLOCK error.

Based on some Github research, this is usually an indication of Heimdall being down. However, in our case, catching_up looks correct.

We would like to close out the possibility the snapshot itself is corrupted. Thus, we would like to ask

  • How snapshot.matic.today service is being run? Is it run by the official Polygon team?

  • Would it be possible to get the MD5 file hashes of the downloads there to ensure there is nothing wrong with the download itself? Currently the snapshot is 1.1TB - it is a painful to redownload if you suspect the download is corrupted.

    "sync_info": {
      "latest_block_hash": "64A1EAEDC7C16E991770A33FF01B6F28CA0F9BA0CCFECB6BA1463824C0FCBA31",
      "latest_app_hash": "B052699D7A2D80E83D38B31E683BCE312C0A7815F90735D319C3A1B936749AF5",
      "latest_block_height": "7459532",
      "latest_block_time": "2021-12-07T10:27:52.848145397Z",
      "catching_up": false
    },

INFO [12-07|11:34:44.336] Looking for peers                        peercount=4 tried=239 static=0
INFO [12-07|11:34:49.965] Downloader queue stats                   receiptTasks=0 blockTasks=18319 itemSize=12.36KiB throttle=8192
INFO [12-07|11:34:50.027] Skip duplicated bad block                number=22,156,660 hash=7922f0..5fa061
ERROR[12-07|11:34:50.028]
########## BAD BLOCK #########
Chain config: {ChainID: 137 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: <nil>, Engine: bor}

Number: 22156660
Hash: 0x7922f041cf1f33d3947ce2629a9a9c3004a6130f1c7b29c4852b6207515fa061
         0: cumulative: 327191 gas: 327191 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7eb1e495f067d98f32285bfd7620345d80a40383874c7b5a4a75e198adb399e8 logs: [0xc0016d6630 0xc0016d66e0 0xc0016d6790 0xc0016d6840 0xc0016d68f0 0xc0016d69a0 0xc0016d6a50 0xc0016d6b00 0xc0016d6bb0 0xc0016d6c60 0xc0016d6d10 0xc0016d6dc0 0xc0016d6e70 0xc0016d6f20] bloom: 00000000001000000000200000020000400021000800000000000000000000000000000000000008000000100008000000008000104000000000000800202200000000000000000000010008000001800000000000002000001100000008000000020000020000000000000004000020004000000000000080000030010100001000000000000040011000000000000000010100011000100000040000000000220000000000000000000040000000000000000000000000000000000010004000000002400000000001000000000000000000000008800000108000000000020010000000000000000008000000000084042028000000000000080000300000 state:
         1: cumulative: 348191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x721d9ce635e9f1c66b7cf555c95a930cebd46e90123e174150d5f8ecb4b41aa0 logs: [0xc0016d7080] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         2: cumulative: 369191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x5a5712a15ed830ff3cb8624a064857beed25f1683238c385eaa5fe6533e77339 logs: [0xc0016d7130] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         3: cumulative: 390191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc25d54d6d1e747c2c58e66fb68dfdad62a03970665308e00d45d3850aabcd54a logs: [0xc0016d71e0] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         4: cumulative: 411191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6c7f1015575cc28bf21c1f59441cb11561ccc80556346363ebff0c66160fca1f logs: [0xc0016d7290] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         5: cumulative: 432191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x102acfa09d7f0beebf09293b13ba85fb5e7873825af98d4e8c60fcc04ace219e logs: [0xc0016d7340] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         6: cumulative: 453191 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xbbe903e6651c0048c883ed045a13d0c468dd2aad09e615ba9f539d489db61f02 logs: [0xc0016d73f0] bloom: 00000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000008000000020000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000002000000000000000000000000000000000000000000000000100000 state:
         7: cumulative: 512281 gas: 59090 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0xa18c1eb740c4ae345087a8991e6f7e63c91c5280d0534618f980e4a9ee001dca logs: [0xc0016d74a0] bloom: 00000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000008004000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000100000 state:
         8: cumulative: 607215 gas: 94934 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0xced737ea218d9f9888faa253538d7ec703117ca623deb8499a3d8469b564d507 logs: [0xc0016d7550] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000800000000000000000000000000000000000040000000000000001800000000000000000000100000000000000000000000000000004000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000100000 state:
         9: cumulative: 711432 gas: 104217 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x424d37a50c33179d26cc142915d91936b49dc34fa98f37e7ec738c5b7c1ea5c4 logs: [0xc0016d7600] bloom: 00000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000008000000000000000000002000000000000000000000000000000000001800000000000000000000100000000000000000000000000000000000000000000000000000000000080000000000000000000000000200000000000000000000000000000000000100000000000000000200000000000000000000040000000000000000000000000000000000000004000000000000000000001000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000100000 state:
         10: cumulative: 854088 gas: 142656 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x0a49b64bf3b98749c63b78a4464fd835461ff3fe551a50309229830093d6e81c logs: [0xc0016d76b0 0xc0016d7760 0xc0016d7810 0xc0016d78c0] bloom: 00000000000000000000200000040000000000000000000020000000480000000000000000000000000000000000000000008000008000000000000000000000000000200000000000400008000001800000000000000000000100000000000000000000001000000000000000000000000000000000000080040010000000000000000000000000000000400000000000000000000000100000004000000800200000020000000000080040000000000000020000000000000000000000004000000002000000000001000000000000000000000010000020110100000000000000000000000000000000000000000100000000000000000000000000100000 state:
         11: cumulative: 998700 gas: 144612 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x65f0180bb25960a4c228366defdf70fe28c169488c96d824edf1dba9b01c41d6 logs: [0xc0016d7970 0xc0016d7a20] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000001000000000000008000000000000000001800000000000000000010100000000000000000000000000000000800000000000000000000000000080000000000000400000000000008000000000000000000000800000000000100000000000000000200000000000000000000040000000000000000008000000000000000000004000000000000000000001000000000000000000000000000000100000002000000000000000000000000000000000000000000000000000000000000000100000 state:
         12: cumulative: 1175197 gas: 176497 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x600f8f4708c5c00ee68fa67e6287ee277333a1ba491488f9abe1511b4332d413 logs: [0xc0016d7ad0 0xc0016d7b80 0xc0016d7c30 0xc0016d7ce0] bloom: 00000020001000000000000000000000000100000000000000000000000000000000100000000000000000002000000000008000000200000000000000000000000000000000000000000001002001840000004001000000000100000000000000000000020000000000000000000800000000000000000080000002048000000000000000000000000000000000000000000480000000100000000000000000240000001000000000000040000020000000000000000000000000000000004000000000000000000001000000000000004000000800000000100000000020000000000000000000000000000000000000000000000000000000000000100000 state:
         13: cumulative: 1351814 gas: 176617 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x283c76a123f34e73e23c091e1f3d6106a10a651142e7970427e42ab4f0dc9e5e logs: [0xc0016d7d90 0xc0016d7e40 0xc0016d7ef0 0xc019884bb0] bloom: 00000000000000000008000000000800000000000000000000000100000200000000100000000000040000000000000000008000000200000000000000000000000000000000000000000001002001800000004001000000000100000000000000000000020000000000000000000800000000000000000080000000008000000000000000000000000000400000000000000480000000100000000000000000200000001000000000000040000020001000000002000000000000000000004000000000000000000001000000000000004000000000000000100000000020000000000000000000000000001000000000000000000000000000000000100000 state:
         14: cumulative: 1528289 gas: 176475 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7f286c9151374cf19f1ab6659341cb22f5a21c0d8a75408485b21c569c5c39a1 logs: [0xc019884c60 0xc019884d10 0xc019884dc0 0xc019884e70] bloom: 00000000000000000000002001000000004000000000000000000000000000000000100000000000000000000000000000008000000200000002000000000000000000000000000000000001002001800000004001000000000100000000000000000000020000000000000000000800000000000000000080000000008100000000000000000000000000000000000000000480000000100000000000000000200000001000000001000040000020000000000200000000000000000000004020000000000000000001000000000000004000000000000000100000000020000000000004000000000000000000000000000000000000000000000000100000 state:
         15: cumulative: 1741130 gas: 212841 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x52c4a184f448a104c78597338dcbeaf59a859ab50dec3ff7e9c9b30cfef07f86 logs: [0xc019884f20 0xc019884fd0 0xc019885080 0xc019885130 0xc0198851e0 0xc019885290 0xc019885340 0xc0198853f0 0xc0198854a0 0xc019885550 0xc019885600 0xc0198856b0 0xc019885760 0xc019885810 0xc0198858c0 0xc019885970] bloom: 002000000000000200080000a0000000000000000000000840000000000000000000000000000000001010100000000000018000000000000010000000000000100000000000400000020008000401a0000004800200000200010000800002000000000000000000000000000000040000000000000440008000001000000000000080000