Conduit releases Degen Chain outage report: Alerting and monitoring of Orbit chain has been improved

On May 22nd, Conduit released a post-mortem analysis report on the previous downtime of Degen Chain. On May 10th, Conduit increased the batch size of Degen and Proof of Play Apex to 10MB to reduce costs, which delayed the time for data to be batched and released to its parent chain. On May 12th, this configuration was restored to fix batch publishing. This resulted in a reorganization on two networks because the batch was released after a 24-hour forced inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay these transactions with a new timestamp. After the reorganization, nodes returned with a corrupted database because their depth was not handled well by geth. This requires resyncing the data directory from genesis. The synchronization time for each network exceeded 40 hours, and the replay rate was about 100M gas/s. Once the nodes were resynchronized, Conduit attempted various transaction replay schemes, although not all transactions could be recovered because some transactions depended on precise timestamps. After consulting with each rollup team, Conduit discussed and parallelly attempted various strategies to keep the network online and restore it to its pre-reorganization state. The Degen Chain network was restored online 54 hours after the outage. Proof of Play's Apex chain was restored around the same time but was only available to the public after implementing another recovery plan.