On Tuesday, May 30th, 2023 a slashing event happened in the Ethereum Prater network. Around 584 validators were slashed during 1008 epochs (179520 - 180528). These validators were part of an experiment carried out by the Migalabs team for testing purposes with ObolNetwork about the Charon software. This post explains the events related to this incident.
Timeline
March 18th: The Charon experiment started
May 29th at 17:27 CET: Migalabs team shuts down all the Charon nodes as the experiment finished.
After the closure of the experiment, we intended to move these validators to a single machine for maintenance and future testing purposes. From this moment onwards, there were no Charon nodes running on our machines.
May 30th at 08:30 CET: Migalabs team sets up a full Ethereum node in the Prater network with the 3000 keys. We had some issues with this new node and we had to sync from Genesis, so the validators were offline for around 15 hours.
As an example, we can observe one of the validators was not performing any duties from epoch 179338 (when we shut down the Charon setup) until epoch 179481 (when we started running the solo instance).
Due to some issues with the machine, it had to be shut down and we started a process to move the validator keys to another machine.
May 30th at 12:24 CET: Migalabs team shuts down the previous solo node and we set up another full Ethereum node in the Prater network with the same 3000 keys. We waited 30 minutes between these two steps.
During this transition, we accidentally left the first node running. We understand the importance of slashing and double attestation (we did wait 30 minutes before running the second instance). However, the shutdown of the first instance was not successful.
This accident is due to a human mistake.
At the time of the slashing event, the Obol experiment had already finished. Thus, the slashing event did not concern the Charon software or the ObolNetwork team.
May 30th at 13:20 CET: The slashing event began, which resulted in 584 slashed validators (at the time of writing this).
We can observe that one of the validators was slashed due to proposing two different blocks at slot 5745527 (on May 30th at 15:45 CET). The graffiti on both blocks is “Teku”, which was configured upon the validators' migration. During the experiment with Charon, the configured graffiti on the proposed blocks was “charon/v0.15.0-41e05c0”, find an example here.
May 30th at 16:00 CET: We noticed some of the validators used for the experiment were getting slashed. After confirming the two nodes were running the same keys, both nodes were shut down immediately.
May 31st at 00:57 CET: 548 slashings had been registered in the network.
Few more slashings were registered between May 31st and June 3rd.
As a result, the slashed validators have a reduced balance, they have already been exited from the network and will not be withdrawable until, at least, epoch 187821 (on July 6th at 10:14 UTC).
Migalabs does not run validators on Mainnet at the moment, and we only use these validators in the Prater network for research and testing purposes with multiple research collaborators. The impact of this incident is limited. Nonetheless, we understand the importance of maintaining the test networks stable and we will make sure we take all precautions to avoid similar incidents in the future.
