Crashes Dashboard malfunctioning

Incident Report for Instabug

Postmortem

Summary

On April 18th, 2022, between 13:36 UTC and 14:30 UTC, Instabug experienced an operational incident that caused some crashes to not be accessible.

During this time, no crashes or occurrences were lost or dropped. This issue only impacted viewing of crashes, and our APIs were not impacted in any way.

What Happened

Amazon Web Services (AWS) was performing a blue/green deployment while services underlying Instabug’s infrastructure. During this deployment, several of the indices associated with our Elasticsearch database were incorrectly deleted. Once the issue was discovered, Instabug engineers contacted AWS support who acknowledged the issue and restored the indices from a snapshot. Upon restoration, services were fully functional.

What We Are Doing About This

AWS has committed to notifying us before any release or migration that may impact our services.

Posted May 23, 2022 - 21:40 UTC

Resolved

This incident has been resolved.

Posted Apr 18, 2022 - 15:02 UTC

Monitoring

Restore is done. Issue should be now resolved. We're monitoring to make sure everything is working as expected

Posted Apr 18, 2022 - 14:49 UTC

Identified

This issue has been identified and confirmed to be caused by a deployment by our cloud provider. A restore is currently in progress

Posted Apr 18, 2022 - 14:15 UTC

Investigating

The Crashes dashboard is not rendering properly for customers. We are aware of the issue and working to promptly resolve it. This does not impact any private-cluster customers.

Posted Apr 18, 2022 - 14:00 UTC

This incident affected: Web Dashboard (Web App).