Impact

Multiple Hiro services we impacted, including the Explorer, APIs, the Platform, and Platform VSCode environments. All clients that relied on the mainnet and testnet Hiro APIs were impacted, including the Explorer, Stacks desktop and web wallets, the stacks CLI, and others. Requests on all endpoints to these affected services were not being served.

Root cause

The rollout of a new internal service for the Platform VSCode environments led to our gateway proxies to suddenly become inundated with additional workloads, resulting in repeated out-of-memory errors (OOMs) for each instance of the gateway proxy. Once memory was increased for the gateway proxies, service was immediately restored to the Explorer, APIs, Platform, and any clients relying on the APIs.

Service to the Platform VSCode environments was slowly restored afterwards, as the new internal service completed rolling out.

Action Items

• We’re assessing how we can more accurately mirror our production traffic for the Platform VSCode deployments in a separate staging environment to better prepare for large infrastructure changes.