API Outage

Incident Report for Hiro Systems

Postmortem

Impact

All clients that relied on the mainnet Hiro API were impacted. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

Multiple instances of the Stacks Blockchain API were crashing due to out-of-memory errors. In the past we’ve found this to be related to certain unoptimized queries in the database, and the amount of data the service stores locally for specific request types.

Action Items

  • We’ve temporarily increased the amount of memory available to each API instance
  • We’ve created an issue in the Stacks Blockchain API repo to investigate further and make performance improvements where relevant.
Posted Feb 25, 2023 - 12:40 EST

Resolved

This incident has been resolved.
Posted Feb 25, 2023 - 12:32 EST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 25, 2023 - 12:24 EST

Investigating

We're experiencing a major outage in the Hiro mainnet API and are currently looking into the issue.
Posted Feb 25, 2023 - 12:20 EST
This incident affected: Services for Mainnet (Stacks Blockchain API) and explorer.hiro.so.