Late Thursday evening, the Ethereum blockchain encountered an issue temporarily preventing blocks from being validated correctlyFor 30 minutes, the Ethereum (ETH) blockchain stopped finalizing blocks.
Thursday late evening, the Ethereum blockchain encountered a problem that temporarily prevented blocks from being validated correctly. While everything has since returned to normal, here’s what we know about this episode for now.
Blocks have temporarily stopped being finalised on Ethereum
At the time of writing, the exact cause had not yet been communicated, but for about thirty minutes the Ethereum blockchain (ETH) encountered a problem preventing blocks from being finalized.
The beacon chain stopped finalizing about thirty minutes ago. I don’t know why yet, but in general the chain is designed to be resilient against this, transactions will continue as usual and finalization will kick in when the problem is resolved. pic.twitter.com/utAS0uAWpG
– superphiz.eth ️ (@superphiz) May 11, 2023
It’s important to point out that the blockchain hasn’t stopped. However, during 3 epochs, the consensus layer suffered a bug similar to what is known as an “inactivity leak”.
In simpler terms, the blockchain behaved as if many validators were offline.
This was not the case, but the fact is that during this period, a limited number of certificates were issued by validators:
Figure 1 – Epochs that encountered a consensus bug on Ethereum
A problem with a piece of software being used
To function, the Ethereum blockchain is divided into 2 pillars: the execution layer, which allows transactions to take place, and the consensus layer, which ensures that blocks are consistent. For these two pillars, validators can use different software called clients.
The Ethereum Foundation encourages validators to diversify the clients they use, so that if a bug occurs, the blockchain will continue to function normally.
In the illustration below, we can see that the software used for the consensus layer is correctly diversified, while for the execution layer, the Geth client is far too represented:
Figure 2 – Diversity of clients used by validators on Ethereum
As no consensus layer client is used by more than 50% of validators, last night’s bug could not have caused a blockchain shutdown or fork. That’s assuming that the problem did indeed originate in one of the software packages used.
In this eventuality, core developer Superphiz points out that this inactivity leakage problem could even have been avoided if no software represented more than 33% of validators:
If a consensus layer client caused the loss of finality:
* We averted a fork by not having a supermajority client
* We could have avoided the loss of finality entirely if no client had more than 33%.
I’m proud of our diversity work, but we’re not done. https://t.co/TUtXnQu5hD
– superphiz.eth ️ (@superphiz) May 11, 2023
For his part, Terence Tsao has indicated that an issue has been identified with the Prysm client, although we cannot say at this time whether this is the source of the block validation stoppage :
Update from Prysm. We have identified areas where state caching could be improved. With the improved cache, the node should perform better during a degraded time like before.
Given the chain has stabilized, as a staker / node operator, there’s no action required moment.
– terence.eth (@terencechain) May 11, 2023
The most important thing right now is that everything quickly returned to normal, and no validators were slashed during this episode. In fact, slashing is used to siphon off some of the ETH stashed when validators don’t do their job properly.
A detailed report on what happened should be provided soon by the development team, and we can come back to this information at that time. While everything has since returned to normal, here’s what we know about this episode for now.
Blocks have temporarily stopped being finalised on Ethereum
At the time of writing, the exact cause had not yet been communicated, but for about thirty minutes the Ethereum blockchain (ETH) encountered a problem preventing blocks from being finalized.
The beacon chain stopped finalizing about thirty minutes ago. I don’t know why yet, but in general the chain is designed to be resilient against this, transactions will continue as usual and finalization will kick in when the problem is resolved. pic.twitter.com/utAS0uAWpG
– superphiz.eth ️ (@superphiz) May 11, 2023
It’s important to point out that the blockchain hasn’t stopped. However, during 3 epochs, the consensus layer suffered a bug similar to what is known as an “inactivity leak”.
In simpler terms, the blockchain behaved as if many validators were offline.
This was not the case, but the fact is that during this period, a limited number of certificates were issued by validators:
Figure 1 – Epochs that encountered a consensus bug on Ethereum
A problem with a piece of software being used
To function, the Ethereum blockchain is divided into 2 pillars: the execution layer, which allows transactions to take place, and the consensus layer, which ensures that blocks are consistent. For these two pillars, validators can use different software called clients.
The Ethereum Foundation encourages validators to diversify the clients they use, so that if a bug occurs, the blockchain will continue to function normally.
In the illustration below, we can see that the software used for the consensus layer is correctly diversified, while for the execution layer, the Geth client is far too represented:
Figure 2 – Diversity of clients used by validators on Ethereum
As no consensus layer client is used by more than 50% of validators, last night’s bug could not have caused a blockchain shutdown or fork. That’s assuming that the problem did indeed originate in one of the software packages used.
In this eventuality, core developer Superphiz points out that this inactivity leakage problem could even have been avoided if no software represented more than 33% of validators:
If a consensus layer client caused the loss of finality:
* We averted a fork by not having a supermajority client
* We could have avoided the loss of finality entirely if no client had more than 33%.
I’m proud of our diversity work, but we’re not done. https://t.co/TUtXnQu5hD
– superphiz.eth ️ (@superphiz) May 11, 2023
For his part, Terence Tsao has indicated that an issue has been identified with the Prysm client, although we cannot say at this time whether this is the source of the block validation stoppage :
Update from Prysm. We have identified areas where state caching could be improved. With the improved cache, the node should perform better during a degraded time like before.
Given the chain has stabilized, as a staker / node operator, there’s no action required moment.
– terence.eth (@terencechain) May 11, 2023
The most important thing right now is that everything quickly returned to normal, and no validators were slashed during this episode. In fact, slashing is used to siphon off some of the ETH stashed when validators don’t do their job properly.
A detailed report on what happened should be provided by the development team shortly, and we can come back to this information at that time.
