Spiderline
custom search engine solutions
Your Own Search Engine.
Just seconds after registering, your web site can be searchable with the features you want and reliability you need. No software to install or maintenance required. Search results can match your website design seamlessly.

Site Search Knowledge Base

Search  
   
Browse by Category
Site Search Knowledge Base .: Account Questions .: Why didn't Spiderline find all of my pages?

Why didn't Spiderline find all of my pages?

There are several reasons that all of your web site pages may not have been crawled.

  1. Account Document Limit: The default Document Limit is 100 pages. In order for Spiderline to crawl over this limit you must activate your account. If you are managing an active account, you will need to upgrade your service plan to increase the document limit.
  2. URL Configuration:
    • -Check the URL configuration page for spelling errors.
    • -Redirected URL: If you use a free web hosting service, you may have entered a virtual URL. Make sure that you have entered a URL that does not redirect to another site.
    • -Starting URLs: These should be complete URLs that point to actual web pages. Enter one URL per line. You may use the noindex option in this field. However, entering the nofollow option in this field negates the purpose of the starting URL.
    • -Patterns: Entries in the Patterns field should follow the format: Pattern Index_Option Follow_Option More FAQs about configuring URLs.
  3. Robot Exclusion: Spiderline supports robot exclusion standards. Check your robots.txt file and robot meta tags. Learn more about Robot Exclusion and robots.txt

How helpful was this article to you?

Related Articles

article Why didn't Spiderline find all of my pages?
There are several reasons that all of your web site pages may not have been crawled. Account Document Limit: The default Document Limit is 100 pages. In order for Spiderline to crawl over this...

(No rating)  2005-01-19    Views: 10297   
article Why didn't Spiderline crawl documents on other sites that I linked to?
Check your URL Configuration Patterns. In order to make documents on other websites searchable, but only the documents you link to and not the entire other website, enter "/  INDEX  NOFOLLOW" on...

  2005-01-20    Views: 7657   
article Can I control the layout of the search results page?
Yes. Spiderline provides you with a general HTML template divided by Header, Results Line (repeated for each search result), and Footer. These templates offer you control of the complete HTML page....

  2005-01-20    Views: 31742   


.: Powered by Lore 1.5.3

Powered by Lucene