Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

An unoptimized and deprecated endpoint on the API saw a spike in traffic over Sept 20th, causing our APIs to consume all available memory and crash, resulting in one very short moment of unavailability (~5 seconds).

Action Items

A hotfix for the API was released and deployed to optimize data retrieval for this endpoint
A rate-limit was put in place for this endpoint for the Hiro API
Memory settings were adjusted for our API deployments to garbage-collect more frequently

Posted Sep 21, 2022 - 16:20 CDT

Resolved

This incident has been resolved.

Posted Sep 20, 2022 - 13:26 CDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 20, 2022 - 13:26 CDT

Investigating

We're experiencing a major outage in the Hiro mainnet API and are currently looking into the issue.

Posted Sep 20, 2022 - 13:25 CDT

This incident affected: Services for Mainnet (Stacks Blockchain API), Sites (explorer.hiro.so), and Apps (Leather Desktop Wallet, Leather Web Wallet).