API Outage
Incident Report for Hiro Systems
Postmortem

Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

An unoptimized and deprecated endpoint on the API saw a spike in traffic over Sept 20th, causing our APIs to consume all available memory and crash, resulting in one very short moment of unavailability (~5 seconds).

Action Items

  • A hotfix for the API was released and deployed to optimize data retrieval for this endpoint
  • A rate-limit was put in place for this endpoint for the Hiro API
  • Memory settings were adjusted for our API deployments to garbage-collect more frequently
Posted Sep 21, 2022 - 21:20 UTC

Resolved
This incident has been resolved.
Posted Sep 20, 2022 - 20:54 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 20, 2022 - 18:39 UTC
Investigating
We're experiencing a major outage in the Hiro mainnet API and are currently looking into the issue.
Posted Sep 20, 2022 - 18:25 UTC
This incident affected: Sites (explorer.stacks.co), Services for Mainnet (Stacks Blockchain API), and Apps (Hiro Desktop Wallet, Hiro Web Wallet).