Google vs. AllTheWeb
There used to be a debate about which search engine was best. And maybe there still is, but we haven't been hearing much about it because Google is pretty much it. Even Yahoo uses Google. The situation is typified by these remarks posted by Jason Kottke the other day at Kottke.org: "Google has been down for most of the day (for me, at least), so I had to use, ugh, Altavista to search for something earlier. It's the first time I'd used something other than Google in more than a year, and it took me about 3 times as long as normal to find what I was looking for. Google is useful enough that I would pay a $5-8 subscription fee per month for access to it. Google is the default command-line interface to the Web...and well worth paying for."
Now there's a pull-quote for you: "default command-line interface for the Web". And maybe that's what we should expect from a well-funded runaway hack by Linux weenies (who nonetheless have a policy of patenting their software).
When you're the default de facto portal for searching everything on the Web, you don't need to do a lot of PR. So Google doesn't. But they're certainly glad to share info when they're asked, which is what happened when I asked Google's VP Corporate Communications, Cindy McCaffrey, to share a few up-to-date facts about the company. Here's some of what she gave me:
Data centers: 4
Linux computers: >10,000
Searches per day: >150 million
Index of Web pages: >1.6 billion
Image base: >330 million
Usenet messages: >650 million (going back >5yrs)
Language subsets in the index: 28
International domain sites: 23
PDFs: >22 million
Included in searches by file type: wk1,wk2, wk3, wk4, wk5, wki, wks, wku, mw, xls, ppt, doc, wks, wps, wdb, wr, irtf, ans, txt
They also have maps, phone directories, dictionary definitions, Web page translation... the list just keeps growing.
Fast Search and Transfer ASA is a Norwegian company with offices in the US and elsewhere. Their original and persistent goal has been to build the world's largest and deepest search engine. Early on they partnered with Dell and Lycos, which ultimately employed FAST engines for searching the Web, images, multimedia and everything else.
And now FAST has rebranded its site as "AllTheWeb", with the tagline "all the web. all the time". And they're doing some aggressive PR. Normally I resist that kind of thing, but I've been warming to these Norwegian guys ever since I started hearing from them, mostly because they felt that they should be no less legit to the community than Google. Their engines run on FreeBSD and were developed on FreeBSD and Linux machines. In fact, FAST's first engine, FTPsearch, was developed under the GPL. You can still download the GPL version of that software at ftp://ftpsearch.ntnu.no/pub/ftpsearch/. Search results are also presented by Apache and PHP.
I was also told that some of the same folks were involved in PHP's development for a long time, and that many of FAST's R&D people in Norway come from one UNIX-oriented computer club at the university in Trodheim. It's called "Programvareverkstedet," or PVV.
Whether it's merit, PR or both, AllTheWeb.com is clearly getting some mojo going. A few days ago Kevin Elliot at About.com wrote, "for searches related to news and current events, it blows the conventional wisdom about Google right out of the water". There's more positive spin at SearchDay, Pandia, Research Buzz and the company's own press release list.
I just ran a quick test of the two services. Here's how they did, at least in terms of returning raw numbers:
"Geeks on the Half Shell":
That last one was a real test, because it referred to a real piece that's been up on both the old and the new LJ site since November 7.
So here's a PR lesson for the AllTheWeb folks. If you're going to send out press releases to editors bragging about how fast you crawl news sites, at least crawl the ones you're pitching.
That said, I've been an AllTheWeb user since it started, and I still use their image searches as much as I use Google's. If you're in heavy search mode, it's better to choose between them with AND logic, not OR.
Doc Searls is Senior Editor of Linux Journal.
Doc Searls is Senior Editor of Linux Journal
Fast/Flexible Linux OS Recovery
On Demand Now
In this live one-hour webinar, learn how to enhance your existing backup strategies for complete disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible full-system recovery solution for UNIX and Linux systems.
Join Linux Journal's Shawn Powers and David Huffman, President/CEO, Storix, Inc.
Free to Linux Journal readers.Register Now!
- The Qt Company's Qt Start-Up
- Devuan Beta Release
- May 2016 Issue of Linux Journal
- EnterpriseDB's EDB Postgres Advanced Server and EDB Postgres Enterprise Manager
- The US Government and Open-Source Software
- Open-Source Project Secretly Funded by CIA
- The Humble Hacker?
- The Death of RoboVM
- BitTorrent Inc.'s Sync
- New Container Image Standard Promises More Portable Apps
In modern computer systems, privacy and security are mandatory. However, connections from the outside over public networks automatically imply risks. One easily available solution to avoid eavesdroppers’ attempts is SSH. But, its wide adoption during the past 21 years has made it a target for attackers, so hardening your system properly is a must.
Additionally, in highly regulated markets, you must comply with specific operational requirements, proving that you conform to standards and even that you have included new mandatory authentication methods, such as two-factor authentication. In this ebook, I discuss SSH and how to configure and manage it to guarantee that your network is safe, your data is secure and that you comply with relevant regulations.Get the Guide