API Outage
Incident Report for Hiro Systems
Postmortem

Impact

All clients that relied on the mainnet Hiro API were impacted. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

Multiple instances of the Stacks Blockchain API were crashing due to out-of-memory errors. In the past we’ve found this to be related to certain unoptimized queries in the database, and the amount of data the service stores locally for specific request types.

Action Items

  • We’ve temporarily increased the amount of memory available to each API instance
  • We’ve created an issue in the Stacks Blockchain API repo to investigate further and make performance improvements where relevant.
Posted Feb 25, 2023 - 11:40 CST

Resolved
This incident has been resolved.
Posted Feb 25, 2023 - 11:32 CST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 25, 2023 - 11:24 CST
Investigating
We're experiencing a major outage in the Hiro mainnet API and are currently looking into the issue.
Posted Feb 25, 2023 - 11:20 CST
This incident affected: Services for Mainnet (Stacks Blockchain API), Sites (explorer.hiro.so), and Apps (Leather Desktop Wallet, Leather Web Wallet).