Polygon edge node was runnign fine till block 428137 after that it is not producing any block unsure about the root cause restart the edge service and still the block is not progressing
Running Polygon edge version
/app $ polygon-edge version
[VERSION INFO]
Release version = v0.8.1
Git branch = <none>
Commit hash = <none>
Build time = <none>
server-3
+ export 'orgname=myorg'
+ export 'appchain=myappchain'
+ count=3
+ exec polygon-edge server --data-dir /polygon/data/myorg/myappchain/polygon-edge-3 --chain /polygon/data/myorg/myappchain/genesis.json --grpc-address 0.0.0.0:9632 --libp2p 0.0.0.0:1478 --jsonrpc 0.0.0.0:8545 --prometheus 0.0.0.0:5001 --seal
2023-05-30T20:44:35.133Z [INFO] polygon.server: Data dir: path=/polygon/data/myorg/myappchain/polygon-edge-3
2023-05-30T20:44:35.140Z [INFO] polygon.server: Prometheus server started: addr==0.0.0.0:5001
2023-05-30T20:44:35.716Z [INFO] polygon.blockchain: Current header: hash=0xfc554153d57591bf71f493778f614e6b4d5ab3335a890fb1dbbf4a36ba7a99a2 number=428138
2023-05-30T20:44:35.716Z [INFO] polygon.blockchain: genesis: hash=0x843fb40b72f796b1a581d4e84611b850a571db0da5806c4b0756a7b00fe1f41a
2023-05-30T20:44:35.751Z [INFO] polygon.server.ibft: validator key: addr=0xa7e7919Ce94E49629413cCD3fc1C4A7932B6D941
2023-05-30T20:44:35.751Z [INFO] polygon.server: GRPC server running: addr=0.0.0.0:9632
2023-05-30T20:44:35.751Z [INFO] polygon.network: LibP2P server running: addr=/ip4/10.100.132.190/tcp/1478/p2p/16Uiu2HAmFV7AxZZCmtq2JYiboxYKTipsguB6BcQCkr1NmT4Z8ut8
2023-05-30T20:44:35.763Z [INFO] polygon.server.jsonrpc: http server started: addr=0.0.0.0:8545
2023-05-30T20:44:35.765Z [INFO] polygon.server.ibft.consensus: sequence started: height=428139
2023-05-30T20:44:35.765Z [INFO] polygon.server.ibft.consensus: round started: round=0
2023-05-30T20:44:35.825Z [INFO] polygon.network: Peer connected: id=16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA
2023-05-30T20:44:35.833Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw
2023-05-30T20:44:35.841Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm2Y722cjQ1hQa3LBXVVTpHxYHYPh5iSW2oWsJVkeKYCef
2023-05-30T20:44:47.770Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=0
2023-05-30T20:44:47.771Z [INFO] polygon.server.ibft.consensus: round started: round=1
2023-05-30T20:45:09.773Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=1
2023-05-30T20:45:09.773Z [INFO] polygon.server.ibft.consensus: round started: round=2
2023-05-30T20:45:51.775Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=2
2023-05-30T20:45:51.777Z [INFO] polygon.server.ibft.consensus: round started: round=3
2023-05-30T20:45:51.777Z [INFO] polygon.server.ibft.consensus: we are the proposer
2023-05-30T20:47:13.778Z [ERROR] polygon.server.ibft.consensus: unable to build proposal
2023-05-30T20:47:13.778Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=3
2023-05-30T20:47:13.779Z [INFO] polygon.server.ibft.consensus: round started: round=4
server -0
2023-05-30T20:23:01.176Z [INFO] polygon.network: Omitting bootnode with same ID as host: id=16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA
2023-05-30T20:23:01.183Z [INFO] polygon.server.jsonrpc: http server started: addr=0.0.0.0:8545
2023-05-30T20:23:01.184Z [INFO] polygon.server.ibft.consensus: sequence started: height=428139
2023-05-30T20:23:01.185Z [INFO] polygon.server.ibft.consensus: round started: round=0
2023-05-30T20:23:01.195Z [INFO] polygon.server.ibft.consensus: we are the proposer
2023-05-30T20:23:02.000Z [INFO] polygon.server.ibft: executed txs: successful=0 failed=0 skipped=0 remaining=0
2023-05-30T20:23:02.001Z [INFO] polygon.server.ibft: build block: number=428139 txs=0
2023-05-30T20:23:13.196Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=0
2023-05-30T20:23:13.197Z [INFO] polygon.server.ibft.consensus: round started: round=1
2023-05-30T20:23:19.430Z [INFO] polygon.server.jsonrpc: Websocket connection established
2023-05-30T20:23:21.104Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm2Y722cjQ1hQa3LBXVVTpHxYHYPh5iSW2oWsJVkeKYCef
2023-05-30T20:23:35.199Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=1
2023-05-30T20:23:35.200Z [INFO] polygon.server.ibft.consensus: round started: round=2
2023-05-30T20:24:01.204Z [ERROR] polygon.network.discovery: Unable to execute bootnode peer discovery: bootnode=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw err="unable to create new discovery client connection, no good addresses"
2023-05-30T20:24:17.202Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=2
2023-05-30T20:24:17.204Z [INFO] polygon.server.ibft.consensus: round started: round=3
2023-05-30T20:24:41.604Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw
2023-05-30T20:25:39.205Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=3
2023-05-30T20:25:39.209Z [INFO] polygon.server.ibft.consensus: round started: round=4
2023-05-30T20:25:39.210Z [INFO] polygon.server.ibft.consensus: we are the proposer
2023-05-30T20:28:21.211Z [ERROR] polygon.server.ibft.consensus: unable to build proposal
2023-05-30T20:28:21.212Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=4
2023-05-30T20:28:21.213Z [INFO] polygon.server.ibft.consensus: round started: round=5
2023-05-30T20:33:43.215Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=5
2023-05-30T20:33:43.218Z [INFO] polygon.server.ibft.consensus: round started: round=6
2023-05-30T20:44:25.222Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=6
2023-05-30T20:44:25.223Z [INFO] polygon.server.ibft.consensus: round started: round=7
2023-05-30T20:44:35.831Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmFV7AxZZCmtq2JYiboxYKTipsguB6BcQCkr1NmT4Z8ut8
Server-1
2023-05-30T20:23:21.104Z [INFO] polygon.network: Peer connected: id=16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA
2023-05-30T20:23:21.191Z [WARN] polygon.server.syncer: failed to complete bulk sync with peer, try to next one: peer ID=error 16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA="unable to verify block, failed to verify the header: header 0x3b4f8f124f149982d9c4ff348cef478a434e9aab29b9478ce297ed7b68c232fd not found"
2023-05-30T20:23:21.191Z [ERROR] polygon.server.sync-peer-client: failed to get block from gRPC stream: peer=16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA err="rpc error: code = Canceled desc = grpc: the client connection is closing"
2023-05-30T20:23:22.000Z [INFO] polygon.server.ibft: executed txs: successful=0 failed=0 skipped=0 remaining=0
2023-05-30T20:23:22.000Z [INFO] polygon.server.ibft: build block: number=70569 txs=0
2023-05-30T20:23:33.083Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=0
2023-05-30T20:23:33.085Z [INFO] polygon.server.ibft.consensus: round started: round=1
2023-05-30T20:23:55.085Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=1
2023-05-30T20:23:55.086Z [INFO] polygon.server.ibft.consensus: round started: round=2
2023-05-30T20:24:21.064Z [ERROR] polygon.network.discovery: Unable to execute bootnode peer discovery: bootnode=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw err="unable to create new discovery client connection, no good addresses"
2023-05-30T20:24:37.087Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=2
2023-05-30T20:24:37.088Z [INFO] polygon.server.ibft.consensus: round started: round=3
2023-05-30T20:24:41.621Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw
2023-05-30T20:24:41.693Z [WARN] polygon.server.syncer: failed to complete bulk sync with peer, try to next one: peer ID=error 16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw="unable to verify block, failed to verify the header: header 0x3b4f8f124f149982d9c4ff348cef478a434e9aab29b9478ce297ed7b68c232fd not found"
2023-05-30T20:24:41.693Z [ERROR] polygon.server.sync-peer-client: failed to get block from gRPC stream: peer=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw err="rpc error: code = Canceled desc = grpc: the client connection is closing"
2023-05-30T20:25:59.089Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=3
2023-05-30T20:25:59.090Z [INFO] polygon.server.ibft.consensus: round started: round=4
2023-05-30T20:25:59.091Z [INFO] polygon.server.ibft.consensus: we are the proposer
2023-05-30T20:28:41.092Z [ERROR] polygon.server.ibft.consensus: unable to build proposal
2023-05-30T20:28:41.092Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=4
2023-05-30T20:28:41.093Z [INFO] polygon.server.ibft.consensus: round started: round=5
2023-05-30T20:34:03.095Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=5
2023-05-30T20:34:03.096Z [INFO] polygon.server.ibft.consensus: round started: round=6
2023-05-30T20:44:35.841Z [INFO] polygon.network: Peer connected: id=16Uiu2HAmFV7AxZZCmtq2JYiboxYKTipsguB6BcQCkr1NmT4Z8ut8
2023-05-30T20:44:35.876Z [WARN] polygon.server.syncer: failed to complete bulk sync with peer, try to next one: peer ID=error 16Uiu2HAmFV7AxZZCmtq2JYiboxYKTipsguB6BcQCkr1NmT4Z8ut8="unable to verify block, failed to verify the header: header 0x3b4f8f124f149982d9c4ff348cef478a434e9aab29b9478ce297ed7b68c232fd not found"
2023-05-30T20:44:35.876Z [ERROR] polygon.server.sync-peer-client: failed to get block from gRPC stream: peer=16Uiu2HAmFV7AxZZCmtq2JYiboxYKTipsguB6BcQCkr1NmT4Z8ut8 err="rpc error: code = Canceled desc = grpc: the client connection is closing"
2023-05-30T20:44:45.097Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=6
2023-05-30T20:44:45.098Z [INFO] polygon.server.ibft.consensus: round started: round=7
Server-2
+ exec polygon-edge server --data-dir /polygon/data/myorg/myappchain/polygon-edge-2 --chain /polygon/data/myorg/myappchain/genesis.json --grpc-address 0.0.0.0:9632 --libp2p 0.0.0.0:1478 --jsonrpc 0.0.0.0:8545 --prometheus 0.0.0.0:5001 --seal
2023-05-30T20:24:40.893Z [INFO] polygon.server: Data dir: path=/polygon/data/myorg/myappchain/polygon-edge-2
2023-05-30T20:24:40.922Z [INFO] polygon.server: Prometheus server started: addr==0.0.0.0:5001
2023-05-30T20:24:41.431Z [INFO] polygon.blockchain: Current header: hash=0x9837fd0506e9c20b290e11ec40bd746e2a5d261dd07eedb8737737bbbe8e2847 number=428138
2023-05-30T20:24:41.431Z [INFO] polygon.blockchain: genesis: hash=0x843fb40b72f796b1a581d4e84611b850a571db0da5806c4b0756a7b00fe1f41a
2023-05-30T20:24:41.455Z [INFO] polygon.server.ibft: validator key: addr=0xf750B25E44B2D88D50F0c627af7730a30E361169
2023-05-30T20:24:41.456Z [INFO] polygon.server: GRPC server running: addr=0.0.0.0:9632
2023-05-30T20:24:41.456Z [INFO] polygon.network: LibP2P server running: addr=/ip4/10.100.156.243/tcp/1478/p2p/16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw
2023-05-30T20:24:41.456Z [INFO] polygon.network: Omitting bootnode with same ID as host: id=16Uiu2HAm3gHuiS1KupDpG7neGhssqPaA5wntr4NDpKH2espL9TBw
2023-05-30T20:24:41.470Z [INFO] polygon.server.jsonrpc: http server started: addr=0.0.0.0:8545
2023-05-30T20:24:41.471Z [INFO] polygon.server.ibft.consensus: sequence started: height=428139
2023-05-30T20:24:41.471Z [INFO] polygon.server.ibft.consensus: round started: round=0
2023-05-30T20:24:41.600Z [INFO] polygon.network: Peer connected: id=16Uiu2HAkxQL7K7bj3shYJHLeDL6TpPHxsNHphMzV1Q1oQJAtSjnA
2023-05-30T20:24:41.622Z [INFO] polygon.network: Peer connected: id=16Uiu2HAm2Y722cjQ1hQa3LBXVVTpHxYHYPh5iSW2oWsJVkeKYCef
2023-05-30T20:24:53.473Z [INFO] polygon.server.ibft.consensus: round timeout expired: round=0