Dec. 4th server problems

Dec. 4th 2012 | by Stefan Schuster

Some of you have probably experienced some problems with Mind42 today. The reason for this is that we've been fighting with our servers for about the last 12 hours. It all started around 2AM GMT +1 time, and nobody from our team was awake to do something against it until the morning (we are not a big company with 24 hour stand by support staff). The error itself was shown as a "504 Gateway Timeout", and Mind42 was unreachable. After restarting the servers, this happened again twice today.

So what happened? Basically, the connections from our application server to our database were overloaded. After the restart, the same thing happened again, and again. So we know what happened, but we are not quite sure about the reason yet. The one thing we do know is that immediately before the first outage there was a huge spike in the number of connections our server had to handle (as you can see in the image to the left). This, of course, could have lead to the overload. What we don't know is whether this was a "natural" spike or some kind of attack.

So since the problems started, we are constantly monitoring, analyzing, optimizing and configuring to deal with the issues. Whether it was an attack or not, we have now tweaked our system to handle such peaks in a much better way, and hope that this also puts an end to todays' issues. Right now everything has been running smoothly for the last two hours, and we will do our best to keep it that way.

Having said this, we are sorry for any inconvenience todays problems caused you, and ask for your understanding.