Explorer Intermittent Failure

Incident Report for Hiro Systems

Postmortem

Impact

Between Jan 27, 2023 10:50 UTC and  Jan 29, 2023 9:16 UTC, all requests to explorer.stacks.co intermittently displayed a blank screen.

Root Cause

It was discovered JS bundles were missing in a number of replica pods resulting in failed requests.  Tracking back to the root cause, a version of the Explorer utilizing new JS bundles was inadvertently deployed into production.  The Explorer recovered soon after the correct version was re-deployed.

Action Items

  • Update CI/CD to identify and prevent deploying non-production ready versions of the Explorer
  • Update monitoring tool to more accurately detect request failures.
Posted Feb 09, 2023 - 11:44 EST

Resolved

The Hiro Explorer is experiencing intermittent failures.
Posted Jan 31, 2023 - 15:49 EST