Be Careful What You Slash

Facing malware is a way of life in the digital world, and the pressure on everyone from home users to corporate giants to do all they can to minimize its damaging effects is immense. Just how much pressure there is was made quite clear on Saturday morning, as a routine update to Google's list of malware sites went horribly wrong, labeling every site in the company's database as dangerous and rendering the service unusable.

The glitch, which appeared just before 9:30AM Eastern and had been resolved by 10:30AM, was caused by an update to the list of "badware" sites the company uses to warn searchers about the possibility of becoming infected. A stray / was introduced to the index, causing Google's search system to return a positive match for every URL in the search database. Google Vice President Marissa Mayer originally suggested the error arose from a list provided by the nonprofit StopBadware.org, but later clarified that the erroneous wildcard was introduced by Google personnel in a simple case of "human error." "What happened? Very simply, human error."

Google technicians quickly discovered the source of the issue and reverted the malware list, restoring service in less than an hour. Unfortunately, searchers who attempted to click through the warning generated while the bug was active were forcibly redirected to StopBadware.org, causing a denial-of-service to the organization's website as millions of Google users converged. Service was restored later in the day, though it continued to run slowly.

While the effect on Google's search engine results was widely reported by both technology publications and mainstream media, a concurrent bug was largely overlooked. The same filters used to flag search results are used by Google's Gmail service, and according to Google Software Engineer Brad Taylor, during the time the glitch was active, and for some time after, legitimate messages sent to some Gmail accounts were marked as spam and delivered to the user's junk mail folder. A fix was implemented on Sunday to restore erroneously marked messages to users' inboxes, though Taylor recommends users check their spam folders anyway, especially if they were expecting any messages from 9:00-11:00AM Eastern on Saturday morning.

Google vowed to look into the glitch and implement procedures to avoid a repeat of the glitch. Said Google's Mayer: "We will carefully investigate this incident and put more robust file checks in place to prevent it from happening again."