Mainnet Ordhooks Outage

Incident Report for Hiro Systems

Postmortem

The Ordhook Mainnet service experienced downtime due to a corrupted or outdated hord.sqlite database, critical for chainhook operations. This issue caused the affected node to fall behind the chain tip, disrupting normal functionality.

Recovery steps included placing the node in maintenance mode and patching the hord.sqlite file manually. Once the node caught up to the chain tip, service functionality was restored. The final step involved syncing the Ordinals API, with updates added to the runbook to improve handling of similar incidents in the future.

Posted Feb 04, 2025 - 10:07 EST

Resolved

This incident has been resolved.
Posted Jan 23, 2025 - 03:49 EST

Investigating

We're experiencing degraded performance in the Hiro Mainnet Ordhooks and are currently looking into the issue.
Posted Jan 23, 2025 - 03:44 EST
This incident affected: platform.hiro.so.