Search Engine Optimization Book: The Search Engine Year book

THE CURRENT VERSION IS SEY 2003


All About Robots & Spiders
by André le Roux
Dec. 2002

What's a "Robot" And A "Spider"? What's The Difference?

Robots - in the context of the Internet - are browser-like programs that automatically surf the web. They do this by following links from one page to the next and from one site to the next.

Robots have many different uses. Some collect information on web sites, some harvest e-mail addresses, some check link validity etc.

Spiders are also robots, but the term is used to specifically refer to robots used by search engines to collect information about web sites. Spiders "index" pages they find - in other words they record the information on these pages. This information is dumped in a database. When you do a search at a search engine, it queries the database (also called the index) rather than really "searching the web".

Spider Spotting

For various reasons, webmasters often track spider activity on their sites. You may have noticed names like "Scooter" and "Googlebot" in your server log files. These are spider names. Here are a couple of names you are likely to encounter:

Search engine Spider name
Abacho AbachoBOT
Aesop AESOP_com_SpiderMan
Ah-ha ah-ha.com crawler
Alexa ia_archiver
AltaVista Scooter
AlltheWeb FAST-WebCrawler
Atomz Atomz
Excite ArchitextSpider
Euroseek Arachnoidea
EZResults EZResult
Google Googlebot
Inktomi Slurp
Lexis-Nexis LNSpiderguy
LookSmart MantraAgent
Lycos Lycos_Spider_(T-Rex)
Mirago HenryTheMiragoRobot
Northernlight Gulliver
National Directory NationalDirectory-SuperSpider
Openfind Openfind piranha,Shark
SearchHippo Fluffy the spider
Teoma teoma_agent1
Ttravel Finder ESISmartSpider
UKSearcher UK Searcher Spider
Walhello appie
Websmostlinked Nazilla
Wisenut ZyBorg

For more information on controlling spider activity on your web site, please refer to the Robots.txt section of this site.

For additional spider names and links to spider resources, check out the Search Engine Dictionary's spider page at http://www.searchenginedictionary.com/spider-names.shtml

Search Engine Book: Search Engine Year BookThis page is based on information contained in the Search Engine Yearbook 2003. For more detailed search engine information & help, please refer to the current version of the book.

EnginePaper Newsletter for Search Engine NewsStay up to date on changes in the search engine world with the EnginePaper Newsletter. It goes out only when something important changes in the search engine world. Subscribe now with a blank email to send-ep-subscribe@topica.com . It's 100% free and safe. View our full privacy policy here.


© Copyright 2003, Pandecta Magazine. All Rights Reserved. Use of this site constitutes acceptance of the disclaimer.
HOME --- DISCLAIMER --- CONTACT

BUY THE BOOK
MORE INFORMATION ABOUT THE BOOK (INCLUDING THE FULL TOC)
DOWNLOAD THE FREE (LIMITED) VERSION