Impact

All clients that relied on the mainnet Hiro API. This includes the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to the mainnet API were still being served, but with a slower response time.

Root cause

Some of our Stacks Blockchain API deployments running v6.2.1 are exhibiting out of memory (OOM) errors under certain situations. When an OOM occurs, traffic is diverted to other API deployments to serve traffic, but sometimes multiple deployments run out of memory at the same time, causing requests to hang for a noticeably longer time period.

Action Items

We’re currently investigating memory heap dumps to understand what scenarios cause an API deployment to run out of memory, and ways to mitigate high memory consumption in the service.
Traffic has been temporarily split to some API deployments running the previous version of software (v5), which seems to handle these scenarios with better memory management.

Posted Nov 23, 2022 - 16:32 CST

Resolved

The degraded performance was stabilized within a few minutes. We've been monitoring the situation and it does not appear to be persistent.

Posted Nov 23, 2022 - 15:05 CST

Investigating

We're experiencing degraded performance in the Hiro mainnet API and are currently looking into the issue.

Posted Nov 23, 2022 - 12:43 CST

This incident affected: Services for Mainnet (Stacks Blockchain API), Sites (explorer.hiro.so), and Apps (Leather Desktop Wallet, Leather Web Wallet).