API Outage
Incident Report for Hiro Systems
Postmortem

Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

After scouring our infrastructure, events, metrics, and third party services, we believe brief GCP network instability is what caused this short outage based on general issues on their status page around the affected time: https://status.cloud.google.com/incidents/7BkNzwMsik35A2V5j6vM#FxgErUaBEZS6pEeX6yiz

https://status.cloud.google.com/incidents/qv2VvxMoAzY8xgH4bsLQ

Action Items

  • We’ve updated our internal alerting to notify us of cloud-provider issues
Posted Nov 21, 2022 - 17:10 UTC

Resolved
This incident has been resolved.
Posted Nov 04, 2022 - 05:46 UTC
Investigating
We're experiencing a major outage in the Hiro testnet API and are currently looking into the issue.
Posted Nov 04, 2022 - 05:45 UTC
This incident affected: Services for Testnet (Stacks Blockchain API).