Crashes Dashboard malfunctioning
Incident Report for Instabug
Postmortem

Summary

On April 18th, 2022, between 13:36 UTC and 14:30 UTC, Instabug experienced an operational incident that caused some crashes to not be accessible.  

During this time, no crashes or occurrences were lost or dropped.  This issue only impacted viewing of crashes, and our APIs were not impacted in any way.

What Happened

Amazon Web Services (AWS) was performing a blue/green deployment while services underlying Instabug’s infrastructure.  During this deployment, several of the indices associated with our Elasticsearch database were incorrectly deleted.  Once the issue was discovered, Instabug engineers contacted AWS support who acknowledged the issue and restored the indices from a snapshot.  Upon restoration, services were fully functional.

What We Are Doing About This

AWS has committed to notifying us before any release or migration that may impact our services.

Posted May 23, 2022 - 21:40 UTC

Resolved
This incident has been resolved.
Posted Apr 18, 2022 - 15:02 UTC
Monitoring
Restore is done. Issue should be now resolved. We're monitoring to make sure everything is working as expected
Posted Apr 18, 2022 - 14:49 UTC
Identified
This issue has been identified and confirmed to be caused by a deployment by our cloud provider. A restore is currently in progress
Posted Apr 18, 2022 - 14:15 UTC
Investigating
The Crashes dashboard is not rendering properly for customers. We are aware of the issue and working to promptly resolve it. This does not impact any private-cluster customers.
Posted Apr 18, 2022 - 14:00 UTC
This incident affected: Web Dashboard (Web App).