At 16:26 UTC our monitoring system indicated elevated load on our API and database infrastructure. At 16:28 UTC this issue escalated further and caused an outage in our API services. As the alarm went out, our 24/7 operations team started the recovery and all systems were operational by 16:39 UTC. We had two brief performance degradations during the recovery process (16:48-16:50 and 17:06-17:08) that affected a small number of our customers.

The issue was caused by a misconfiguration in our Sense Platform that led to increased database load which caused transactions to time out in our API layer. Our engineering team is working on performance improvements to prevent such issues from happening again.