Search Engine Optimization Book: The Search Engine Year book


How Search Engines Work

Let’s start by distinguishing between search engines and directories.

Search Engines (like

The main characteristic of search engines is that they rely on spiders to crawl the web, indexing pages as they go. Spiders are browser-like programs that follow links from page to page and from site to site, indexing everything it finds.

When you submit a web page to a search engine, all you really do is tell the spider about the page.Your page does not get added to the search engine’s database immediately – that only happens once the spider gets around to visiting and indexing the page.

There are very few real search engines.

You'll often see claims like "We'll submit your site to 10,000 search engines". That's nonsense. There simply are not 10,000 or even 1,000 real search engines. There are maybe 100 worth knowing about and no more than 10 worth submitting to.

People who make claims like the one above often submit your site to thousands of FFA pages and obscure directories. Submissions to FFA pages could actually hurt your site.

For more specific information on Google, please refer to the Google section of this site.

Directories (like

Directories do not use spiders.

Instead, they use real people (editors) who visit and evaluate sites – and add them only if they meet the directory’s minimum quality requirement.

This is an important difference:

  • Search engine spiders can index thousands of pages a day.
  • Directory editors cannot.

So why do we have directories if they can’t compete?

The answer is quality.

Editors are considerably harder to impress than spiders. The page has to offer unique information or a unique product. When you submit a page to a specific category in a directory, the editor of that category will visit your page and decide if it’s good enough to add to the directory.

Editors usually reject pages with typos, broken links, unclear navigation etc.

For more on submitting to DMOZ specifically, refer to Getting Listed at DMOZ (ODP)

The Components Of Search Engines

Search engines and directories all consist of 5 major components:
1. The spider (or editor in the case of directories)
2. The indexer (again the editor in the case of directories)
3. The database
4. The search software
5. The interface

For a more detailed explanation of what each component does as well as how they fit together, please refer to the current version of the Search Engine Yearbook.

Search Engine Book: Search Engine Year BookThis page is based on information contained in the Search Engine Yearbook 2003. For more detailed search engine information & help, please refer to the current version of the book.

EnginePaper Newsletter for Search Engine NewsStay up to date on changes in the search engine world with the EnginePaper Newsletter. It goes out only when something important changes in the search engine world. Subscribe now with a blank email to . It's 100% free and safe. View our full privacy policy here.

© Copyright 2003, Pandecta Magazine. All Rights Reserved. Use of this site constitutes acceptance of the disclaimer.