Search Engines

To start with, it has to be said that the term search engine is often too loosely used. It might seem pedantic, but sometimes it can be really quite important for the results you get back. There is in fact a crucial difference between a search engine and a directory.

A directory is list of sites organised by subject. The key point is that this categorisation is done by humans, who respond to suggestions on what sites to include. A bit like library cataloguers with a trolley full of new books.

The most famous of Internet directories is Yahoo!, which has been providing its Internet navigational guide since April 1994. Yahoo! today employs over 2000 people to compile its entries.

An important feature of directories is the fact that, as well as being able to initiate a computer search, you can browse them yourself, by category and subcategory.

Although this wasn’t the case when it began, these days a query initiated at Yahoo! is also simultaneously served by a true search engine. (Until recently it relied on Inktomi, but since July 2000 it has changed to Google, more of which anon.)

A search engine is a robot (i.e. a computer program) that travels around the Internet, capturing every word on every page it visits. This completely automated travelling around (or “spidering”, as it is often called), takes place continuously, with every Web page (billions of them now) at present being revisited by each such engine once every two months on the average.

There are actually three separate parts to a search engine. First there’s the spider, which goes on its merry way following links. Then there’s the cataloguing robot which actually builds the index. Finally there’s the searching software, which sifts through the millions of references in the index in response to your query.

Note that phrase “capturing every word on every page”, above. Herein lies the power of true search engines. For, while a magazine like Bikwil, say, may get classified by a Yahoo! editor into this and/or that category, this is only done for the magazine as a whole. The search engine, on the other hand, will index deep down in the magazine, making entries, perhaps, for Bet Briggs, Miles Davis, Harlish Goop, Land ‘o’ Useless Facts, Edith Sitwell, among others (and they’re just from our first issue).

So, on the face of it, why not just use Yahoo!? After all, it has human-classified and robot-indexed entries.

Fair question. Indeed it was Yahoo! that I myself used almost exclusively for about two years, but gradually I became aware of and tried out other indexes, such as Alta Vista, HotBot, Lycos, etc.

I stayed longest with HotBot. Then I discovered meta-search engines.

As you might expect, your chances of finding what you want are both increased and speeded up if you use a meta-searcher

That said, then, which is the best of the meta-search engines? In my opinion, it’s got to be CNet Search. No doubt about it.

Why? Because it has the ability to send your query to 800 engines. The complete list can be dug out at the CNet Search site, but these are some of the major ones likely to be used for an average query:

What I particularly like about CNet Search is how rapidly it integrates all the responses it finds into one list. Just remember to click on the “See more Web Pages” and “All sources” links.

As good as CNet Search is, usually I rush straight to the always impressive Google. Elegantly simple to look at, it has by far the finest deep indexing capability of all search tools I know for finding that elusive reference — over 3.7 billion pages indexed to date (the largest database of any search tool). Indeed, the more obscure your need, the more it seems to excel — a fact that certain other Bikwil readers are sure to applaud.

And it’s fast, fast, fast — taking less than half a second to respond with its ranked list of results. Every time. All this, mind you, while servicing 100 million searches a day. Fantastic!

Apart from its speed, Google has two other features that set it apart. Firstly, it makes much use of “link popularity” to rank its findings. In other words, the more links a Web page has to it from other well-known pages, the more relevant Google considers it. Its other valuable feature is that for each result returned it shows an extract from the page found, and even highlights your search words.