Mainnet API Block Progression Stalled

Incident Report for Hiro Systems

Postmortem

Summary

On December 19, 2025, at approximately 8:47 AM ET, the mainnet Stacks Blockchain API stopped advancing at Bitcoin block 928,558 and Stacks block 5,385,227. This stall impacted the Explorer and any services relying on the mainnet Stacks Blockchain API.

Block ingestion was restored at approximately 9:46 AM ET, after which the API resumed syncing. The API fully caught up to chain tip by 10:11 AM ET.

Impact

The stall affected all consumers of the mainnet Stacks Blockchain API, including:

  • The Stacks Explorer
  • Chainhooks v2
  • Any downstream services dependent on API block progression

Detection and mitigation were delayed due to a misconfigured silenced alert in Hiro’s alerting system. As a result, engineers were notified later than expected, extending the duration of user impact.

Root cause

The definitive root cause is still under investigation. However, initial analysis indicates that the Stacks Blockchain API entered an unexpected internal state regarding an internal block-synchronization system it depends on, preventing further block progression.

Recovery required manual intervention at the database level to reset the API’s block sync state.

For context, Hiro operates a custom internal block-sync system to improve sync performance and recovery behavior for our API services. We do not believe this condition would affect users syncing directly from a standard Stacks Core node.

Resolution

Once discovered, action was immediately taken to communicate, diagnose, and rectify the mainnet Stacks Blockchain API stall via manual database intervention.

Additionally, the silenced alert that should have notified Hiro engineers has been corrected. Should this happen again before a permanent fix is in place, our engineers will be notified appropriately and able to restore block progression much faster.

Further action will be taken to verify any issues in how the Stacks Blockchain API interacts with our block-sync system and implement a permanent fix.

Action Items

Below are some items Hiro will complete to prevent another mainnet Stacks Blockchain API stall under similar conditions, and to improve the speed at which it was recovered:

  • Investigate any issues in the Stacks Blockchain API that would lead to an unexpected state with our internal block-sync system, and implement a permanent fix.
  • Update any critical alerts to reduce the need for silencing specific instances of them, reducing the possibility for human error.
  • Add a new alert(s) for any gaps found.
Posted Dec 19, 2025 - 12:48 EST

Resolved

This incident has been resolved, and a post-mortem will be added later today.
Posted Dec 19, 2025 - 10:16 EST

Identified

The issue has been identified and a fix is being implemented.
Posted Dec 19, 2025 - 09:50 EST

Investigating

We're investigating a stall in block progression for the Mainnet Stacks Blockchain API.
Posted Dec 19, 2025 - 09:44 EST
This incident affected: Services for Mainnet (Stacks Blockchain API) and explorer.hiro.so.