Explorer Intermittent Failure

Impact

Between Jan 27, 2023 10:50 UTC and  Jan 29, 2023 9:16 UTC, all requests to explorer.stacks.co intermittently displayed a blank screen.

Root Cause

It was discovered JS bundles were missing in a number of replica pods resulting in failed requests.  Tracking back to the root cause, a version of the Explorer utilizing new JS bundles was inadvertently deployed into production.  The Explorer recovered soon after the correct version was re-deployed.

Action Items

• Update CI/CD to identify and prevent deploying non-production ready versions of the Explorer
• Update monitoring tool to more accurately detect request failures.