Stacks blockchain testnet slow block progression

Incident Report for Hiro Systems

Postmortem

Impact

All clients that relied on the testnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the testnet API were still being served, but we showing the same block and transactions for some time.

Root cause

This was determined to be caused by a Testnet Bitcoin block storm. See a writeup on what these are here:

https://blog.lopp.net/the-block-storms-of-bitcoins-testnet/

After the storm subsided, testnet mining resumed and blocks started progressing as normal.

Action Items

There are a couple issues open on the stacks-blockchain repository aimed at reducing the impact of Testnet Bitcoin block storms on the Stacks testnet network
- https://github.com/stacks-network/stacks-blockchain/issues/3173
- https://github.com/stacks-network/stacks-blockchain/issues/3252

Posted Nov 21, 2022 - 12:16 EST

Resolved

This incident has been resolved, testnet Stacks blocks are progressing as expected.

Posted Oct 31, 2022 - 22:45 EDT

Monitoring

The Stacks blockchain testnet is progressing at a decreased rate due to a testnet Bitcoin block storm. Once the storm passes, testnet Stacks block progression should increase soon after.

You can follow the testnet Bitcoin block progression here: https://www.blockchain.com/explorer/blocks/btc-testnet

Posted Oct 31, 2022 - 22:36 EDT

This incident affected: Services for Testnet (Stacks Blockchain API) and explorer.hiro.so.