Small subset of Bor Mainnet nodes were stuck for a brief period.
Issue Summary
Few node operators reported that their RPC nodes were stuck at block #58,906,207. Some nodes on https://bor-mainnet.vitwit.com/ also were seen stuck at the same block.
Timeline
July 3rd 2024 (all times in IST)
- Few nodes in the network stuck at block 58,906,207 whose timestamp is 4.40 PM
- Some node operators reported the issue at 5.38 PM
- Team got on a call at 5.45 PM to observe internal nodes
- Infura mentioned that the issue got auto resolved at 5.42 PM
- Other node operators also mentioned that the issue was auto resolved around the same time
Root Cause
The root cause for some nodes to get stuck at a particular block height in this case is network partition between clusters of nodes. This was concluded by inspecting the logs and metrics such as ingress and sync rate for the nodes for which this data was available.