Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were unable to be served.

Root cause

After scouring our infrastructure, events, metrics, and third party services, we believe brief GCP network instability is what caused this short outage based on general issues on their status page around the affected time: https://status.cloud.google.com/incidents/7BkNzwMsik35A2V5j6vM#FxgErUaBEZS6pEeX6yiz

https://status.cloud.google.com/incidents/qv2VvxMoAzY8xgH4bsLQ

Action Items

We’ve updated our internal alerting to notify us of cloud-provider issues

Posted Nov 21, 2022 - 11:11 CST

Resolved

This incident has been identified and appears to be related to a broad service disruption Google Cloud Platform is experiencing. While availability is currently restored, we'll continue monitoring the situation for any impacts to Hiro services.

More details here: https://status.cloud.google.com/incidents/7BkNzwMsik35A2V5j6vM#FxgErUaBEZS6pEeX6yiz

Posted Nov 02, 2022 - 08:11 CDT

Investigating

We're experiencing a major outage in the Hiro testnet API and are currently looking into the issue.

Posted Nov 02, 2022 - 01:46 CDT

This incident affected: Services for Testnet (Stacks Blockchain API).