API Outage

Incident Report for Hiro Systems

Postmortem

Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

After scouring our infrastructure, events, metrics, and third party services, we believe brief GCP network instability is what caused this short outage based on general issues on their status page around the affected time: https://status.cloud.google.com/incidents/7BkNzwMsik35A2V5j6vM#FxgErUaBEZS6pEeX6yiz

https://status.cloud.google.com/incidents/qv2VvxMoAzY8xgH4bsLQ

Action Items

  • We’ve updated our internal alerting to notify us of cloud-provider issues
Posted Nov 21, 2022 - 12:10 EST

Resolved

This incident has been resolved.
Posted Nov 04, 2022 - 01:46 EDT

Investigating

We're experiencing a major outage in the Hiro testnet API and are currently looking into the issue.
Posted Nov 04, 2022 - 01:45 EDT
This incident affected: Services for Testnet (Stacks Blockchain API).