Mainnet Ordinals API Outage

Incident Report for Hiro Systems

Postmortem

The Ordhook Mainnet service experienced downtime due to a corrupted or outdated hord.sqlite database, critical for chainhook operations. This issue caused the affected node to fall behind the chain tip, disrupting normal functionality.

Recovery steps included placing the node in maintenance mode and patching the hord.sqlite file manually. Once the node caught up to the chain tip, service functionality was restored. The final step involved syncing the Ordinals API, with updates added to the runbook to improve handling of similar incidents in the future.

Posted Feb 04, 2025 - 10:08 EST

Resolved

This incident has been resolved.
Posted Jan 28, 2025 - 01:22 EST

Investigating

We're experiencing degraded performance in the Hiro Mainnet Ordinals API and are currently looking into the issue.
Posted Jan 28, 2025 - 01:17 EST
This incident affected: Services for Mainnet (Ordinals API).