Testnet progress paused

Incident Report for Hiro Systems

Postmortem

Summary

The Stacks testnet network stalled at BTC block 107099 and STX block 3631980 at approximately 6:08PM ET Oct 29th 2025, affecting all services and APIs across the ecosystem which relied on the testnet network. The network was restored at roughly 8:22PM ET Oct 29th 2025 starting at BTC block 106884 and STX block 3630559, initiating a regtest Bitcoin and testnet Stacks network chain split. Remaining impacted Hiro services and APIs were gradually restored by 12:04PM ET Oct 30th 2025.

Impact

The network stall impacted all users and services which use the Stacks testnet network. This includes Hiro's testnet infrastructure and services, such as the Stacks Blockchain API, Stacks Core RPC pools, Stacks testnet Seeds, Token Metadata API, Signer Metrics API, Chainhooks, Archival services for the Stacks Blockchain API, Stacks miner, regtest Bitcoin seed, regtest Bitcoin miner, regtest Bitcoin Explorer, and other internal services.

Root cause

The testnet stall occurred due to signers not be able to issue stack-extend contract function calls for their associated Stacks addresses. The addresses they used did not have enough unlocked STX to complete the contract function call, and thus were not eligible for stacking in the upcoming cycle 119. A requirement for a signer to actively sign newly proposed blocks is they must be stacking STX.

Once the Bitcoin block boundary was passed to enter cycle 119, the network did not have enough signing power to continue since the signers were no longer stacked, at which point the network entered an unrecoverable state.

Resolution

Action was immediately taken to communicate, diagnosis, and rectify the network stall through restoration to an earlier point before the chain stall. The testnet network was restored at roughly 8:22PM ET Oct 29th 2025 using disk snapshots taken earlier that same day at BTC block 106884 and STX block 3630559.

The order in which services were restored were as follows:

  • Regtest Bitcoin Miner & Regtest Bitcoin Seed
  • Regtest Bitcoin Explorer
  • Testnet Stacks Miner & Testnet Stacks Seed
  • Testnet Stacks RPC Pools
  • Testnet Stacks Blockchain API
  • Testnet Stacks Blockchain API Archive
  • Testnet Chainhooks
  • Testnet Signer Metrics API
  • Testnet Token Metadata API

In addition to Hiro, anyone running a Testnet Stacks Core node, Regtest Bitcoin node, Stacks signer, sBTC signer, or other API service on testnet will have to restore to a point at or before BTC block 106884 and STX block 3630559.

Further action will be taken to prevent testnet network stalls as a result of stacking problems, and improvements will be made to ensure network restores are quicker and easier if needed again.

Action Items

Below are some items Hiro will complete to prevent another testnet network stall under similar conditions, and to improve the speed at which the network is recovered:

  • Automate restores for all services involved in this event.
  • Update and/or add any necessary runbook entries to document and clarify the Testnet restoration process.
  • Review internal incident response plan.
  • Review why a StatusPage incident was not automatically created.
  • Add monitoring and alert for Testnet and Mainnet signer addresses that have run out of funds.
  • Add monitoring and alert for Testnet and Mainnet signer addresses that are unable to stack.
Posted Oct 30, 2025 - 16:31 EDT

Resolved

This incident has been resolved.
Posted Oct 30, 2025 - 08:23 EDT

Update

Testnet stacks network has been restored. We are continuing to work on restoring other testnet services and API's. Data reflected in APIs may not be accurate until their restore is complete.
Posted Oct 29, 2025 - 21:01 EDT

Identified

We've identified an issue with Testnet block progression and are working to resolve it.
Posted Oct 29, 2025 - 18:47 EDT
This incident affected: Services for Testnet (Stacks Blockchain API, Token Metadata API, Regtest Bitcoin, Chainhook, Signer Metrics API).