Postini email outage (Google’s SPAM/Virus Filtering Service)
13 October 2009 was a *bad* email day for us at work. We use Google’s SPAM/Virus filtering service, which was purchased from Postini. Up until the 13th of October, it has served us well. I started to get reports of people getting rejection notices that our email server was “timed out”. I even had someone email me the rejection notice to another account, which was not going through Postini. I then started to look around to see what was happening, because I was getting some emails. I found this thread on Google, which chronicles the outage, as viewed by many other people affected by the issue:
http://www.google.com/support/forum/p/AppSecurity/thread?tid=43203b3c6982650f&hl=en
If you browse through the thread, you will find the common denominator:
* Not knowing what is going on
Being mostly technical people on the thread, I think most people would have appreciated a simple non-laymens explanation of what was happening. Just something like, “We are experiencing a heavy increase in traffic” or “We seem to be suffering a DOS”. But all we got was “We are aware of the issue, and engineers are working on it.” Why not tell us what the issue is?
Another huge frustration, is that the redundancy for this was absolutely horrible. Google is a huge company with some of the brightest people in the world working for them. Overlooking redundancy for increases in traffic, and not having a contingency plan is unacceptable. The only work-around, was for customers to point their MX records back to their internal SMTP servers, and hopefully cobble something up for the SPAM and viruses that would inevitably get through. But, even that work-around was not entirely possible, because it takes time for DNS propagation to occur sometimes. With no ETA on a fix, it was a tough call to make: do we wait for the problem to resolve, or institute some kind of work-around?
I certainly understand that sometimes there can be no ETA, because you don’t have all the facts as to what the exact problem is. So, tell us that! I would rather know the complete truth about a situation, then make assumptions based on incorrect facts. Google is a huge player in today’s communications on the Internet, and they need to start playing that part by fully communicating with their customers fully.