Spiderline
custom search engine solutions
Your Own Search Engine.
Just seconds after registering, your web site can be searchable with the features you want and reliability you need. No software to install or maintenance required. Search results can match your website design seamlessly.

Site Search Knowledge Base

Search  
   
Browse by Category
Site Search Knowledge Base .: Crawl Questions .: Does Spiderline honor the robot exclusion protocol?

Does Spiderline honor the robot exclusion protocol?

Yes, Spiderline does honor the robot exclusion protocol. Our spiders will not index directories or follow links that have been disallowed in the robots.txt configuration file located on your server. Learn more about Robot Exclusion and robots.txt

Spiderline also honors META tags that designate "noindex" and/or "nofollow".

If you have a robot.txt file or Robot META tags for other robots and want Spiderline to ignore either, you can change this in crawl settings. Default is to honor all robot exculsions.

How helpful was this article to you?

Related Articles

article Robot Exclusion Guide
The robots.txt file and robot META tags are methods used to allow and disallow crawling portions of your site by robots (web robots, spiders). Website administrators and content providers can...
  2005-01-20    Views: 9328   
article Robot META tags Tutorial.
The Robots META tag is another method that may be used to indicate to visiting robots whether a page should be indexed (crawled), or links on the page should be followed. It differs from the...
  2005-04-27    Views: 4094   
article How do robot meta tags work?
The Robots META tag is another method that may be used to indicate to visiting robots whether a page should be indexed (crawled), or links on the page should be followed. It differs from the...
(No rating)  2005-04-27    Views: 4636   


.: Powered by Lore 1.5.3

Spiderline LLC © 2000-2007 | Knowledge Base | Privacy Policy | Terms Of Use | Resellers | Affiliate Program
Powered by Lucene