There are several reasons that all of your web site pages may not have been crawled.
- Account Document Limit: The default Document Limit is 100 pages. In order for Spiderline to crawl over this limit you must activate your account. If you are managing an active account, you will need to upgrade your service plan to increase the document limit.
- URL Configuration:
- -Check the URL configuration page for spelling errors.
- -Redirected URL: If you use a free web hosting service, you may have entered a virtual URL. Make sure that you have entered a URL that does not redirect to another site.
- -Starting URLs: These should be complete URLs that point to actual web pages. Enter one URL per line. You may use the noindex option in this field. However, entering the nofollow option in this field negates the purpose of the starting URL.
- -Patterns: Entries in the Patterns field should follow the format:
Pattern Index_Option Follow_Option
More FAQs about configuring URLs.
- Robot Exclusion: Spiderline supports robot exclusion standards. Check your robots.txt file and robot meta tags. Learn more about Robot Exclusion and robots.txt
Print
