PoS Chain Downtime and Heimdall Issue

Polygon Mainnet (PoS)
1

Update: Temporary hotfix deployed to resume the Polygon PoS chain

See the new forum post with the details here.

Summary

  • Polygon PoS users will likely experience downtime starting at 5:50 PM UTC on March 10th due to an issue with the Heimdall implementation.
  • Note that all user funds and data remain fully secure.
  • The Polygon team is actively working on a solution and will share an update on ETA as soon as we can.

Details

The Polygon PoS team is currently looking into an issue with the Heimdall implementation which is used by one of the two layers of the Polygon PoS chain. This means the network is currently experiencing downtime which began at 5:50 PM UTC on March 10th.

Here are some further technical details:

  • Due to a recent upgrade, the Heimdall node, which is part of the dual-node architecture of Polygon PoS, has halted
  • All user funds and data are absolutely safe
  • While we’re working on identifying the definitive cause, it seems to have originated from an earlier upgrade consisting of a minor parameterization fix to the Ethereum to Polygon PoS state sync/bridging module
  • As part of the earlier upgrade, no consensus/state modules were supposed to be affected, the change what is technically called a side-handler is for the state sync mechanism
  • Although still under investigation by the team, we suspect, there may have been a bug in the upgrade which affected consensus, and caused different Heimdall validators to be on different versions of the chain, thereby not reaching 2/3 consensus. When using Tendermint consensus, this situation will cause the Heimdall chain to halt
  • Note that Heimdall does not handle user transactions. It is used for validator related transactions and bridging
  • The Bor chain relies on Heimdall for block proposer committee selection, specifically span creation. So once the last span that Heimdall specifies passes, the Bor chain also halts.
  • At that time, the Bor chain (or the user-facing Polygon PoS chain) will then only be halted, and while there is a liveness issue, there are no security or safety issues. Meaning that while there will be downtime, state and funds are not affected.

Mitigation

We’re currently working on identifying the definitive causes and preparing mitigation solutions that will resume operation as soon as possible. More details will be shared soon.

Please rest assured these issues are of the highest priority for the Polygon PoS team and we’ll keep you updated as soon as we resolve the issues. You can also follow us on our Twitter handle at 0xPolygonDevs for updates.

Thanks for your support,

Team Polygon

11 Likes
3

wen will it be back :upside_down_face:

1 Like
4

Please keep us posted on Twitter :+1:

6

Thanks for your hard working. :+1:t2: cheer up.
art.army team

1 Like
9

I have had a transaction hung up in the system for 17 hours. Was trying to bridge ETH to MATIC and the PoS shows the deposit as pending and the wheel is just spinning. Etherscan shows the transaction as successful. I know very little about this and do not know what to do. I opened a ticket on the PoS chain several hours ago with absolutely zero response. Does anyone have an idea what I can do?

1 Like
10

I see where they are saying a hotfix was deployed, but it has not fixed my issue. Still spinning and pending. Please help!!

2 Likes
11

same problem same time i stay on bridge withdraw and no answer support !

2 Likes
12

Have a withdrawal still pending 24hrs ??

2 Likes
13

bro como estas yo el dia de ayer compre una nft y la transaccion salio exitosa pero aun no se me refleja la nft yo tambien ando como preocupado

1 Like
15

Bruh you guys warned of deposits not withdrawals, if website would have said withdrawal would be affected I wouldn’t have done it. Got me thinking something else is wrong I had to literally come find all this information, why are u guys sooooo quiet about this ? Do we not matter ? These are thousands of dollars of peoples money which means their time, get it together gang.

2 Likes
16

your problem has been solved me it will be 3 days that my withdrawal is pending. I am asked for a qr via wallet and seed sentence something that I did despite reluctance on my part apparently it should solve my problem but the transaction does not move the problem is still present on my side and you?

17

There’s a scammer lurking around the different forums related to this issue asking people to visit a live chat. They are trying to get wallet passphrases. They created a fake forum forummaticnetwork.org/# and they ask you to click on a link that will ask you for the passphrase of your wallet. Their site have only been online since March 11 2022 (Whois forummaticnetwork.org).

18

Thanks for the update.