THE CURRENT VERSION
IS SEY 2003
How Search Engines
Lets start by distinguishing between search
engines and directories.
The main characteristic of search engines
is that they rely on spiders to crawl the web, indexing pages as they
go. Spiders are browser-like programs that follow links from page to
page and from site to site, indexing everything it finds.
When you submit a web page to a search engine,
all you really do is tell the spider about the page.Your page does not
get added to the search engines database immediately that
only happens once the spider gets around to visiting and indexing the
There are very few real search engines.
You'll often see claims like "We'll submit
your site to 10,000 search engines". That's nonsense. There
simply are not 10,000 or even 1,000 real search engines. There are maybe
100 worth knowing about and no more than 10 worth submitting to.
People who make claims like the one above often
submit your site to thousands of FFA pages and obscure directories.
Submissions to FFA pages could actually hurt your site.
For more specific information on Google, please
refer to the Google section of this site.
Directories do not use spiders.
Instead, they use real people (editors) who visit
and evaluate sites and add them only if they meet the directorys
minimum quality requirement.
This is an important difference:
- Search engine spiders can index thousands of
pages a day.
- Directory editors cannot.
So why do we have directories if they cant
The answer is quality.
Editors are considerably harder to impress than
spiders. The page has to offer unique information or a unique product.
When you submit a page to a specific category in a directory, the editor
of that category will visit your page and decide if its good enough
to add to the directory.
Editors usually reject pages with typos, broken
links, unclear navigation etc.
For more on submitting to DMOZ specifically, refer
Listed at DMOZ (ODP)
The Components Of
Search engines and directories all consist of 5
1. The spider (or editor in the case of directories)
2. The indexer (again the editor in the case of directories)
3. The database
4. The search software
5. The interface
For a more detailed explanation of what each component
does as well as how they fit together, please refer to the current
version of the Search Engine Yearbook.
page is based on information contained in the Search Engine Yearbook 2003.
For more detailed search engine information & help, please refer to the
current version of the book.
up to date on changes in the search engine world with the EnginePaper
Newsletter. It goes out only when something important changes in
the search engine world. Subscribe now with a blank email to