Aircall is a leading global cloud-based phone provider. Our customers rely on us to deliver the best experience to their customers and to run their daily operations. That’s why we’re building distributed phone systems in order to sustain their growth by being more productive with their phone calls.

Tuesday between 7AM UTC and 9:30AM UTC, we had our most critical failure since Aircall was launched last year. The root cause behind this major failure was an overload of our primary database. This event affected all of our main services: calls, dashboard, apps, and website. The main consequences of this overload on our primary database were huge slowdowns in web requests responses - that’s why you had difficulties to load our apps, jump in calls or access your analytics.

The first indication of such problem was an automatic alert sent after 1 minute of high load on our primary database. As it’s really unusual, we started to investigate and tried to find out the slowest requests made to our database. In the same time, our sales team (front line users of Aircall!) sent us bad reports on call drops and slow page loading on our website.

After a few minutes we noticed that, although the load on the database was rising, our « business » load (number of calls, number of users, or number of customers) didn’t increase during the previous hours. After our first investigation, we found out that a « snowball effect » overloaded our database. The root cause was the high number of queries made to the entire call list, which is used by most of our systems. Once our database was overloaded, each new query was slower than the previous one and the database was more and more overloaded minute after minute.

The entire tech team scrambled into firefighting mode to reduce the load on our primary database and speed up all our services again. We shut down some services (like analytics view) temporarily and added hardware capacities in order to connect as most calls as possible. Even if we didn’t lose data from calls (they appeared as missed calls on your apps), an unacceptable number of calls didn’t ring at all. The complete decrease of the load took an hour. This duration reflects the one of the Aircall usage peaks we generally have during this timeframe. After the database load became lower, we continued to monitor it until 9:30AM UTC.

Since the beginning of the week, we made all the fixes and upgrades in order to avoid the recurrence of such failure. It has been more than two months now; however, that we started redesigning a totally new architecture devoted to the pure telephony side of Aircall. This architecture allows us to sustain our growth (x30 in volume over the past 12 months) and has a major purpose of separating all our services so to make them totally independant. We’ll start to deploy these new systems in June 2016 in order to continue to deliver the highest reliability our customers expect from their phone system.

If you have more questions, feel free to send us an email at support@aircall.io or call us: (+1) 888-240-6923