Spiderline News

We're adding features left and right! Spiderline developers have a master list of every feature imaginable - and they are building in the order of customer demand. If you don't see what you want at Spiderline, let us know and we'll be sure to let you know when it becomes available.

July 29, 2009 -- All New Pricing on our Service Plans!
We have reviewed our service plan pricing model and significantly reduced prices across the board. Sign up today to lock in a low price on your service plan.

December 18, 2005 -- Product Website Redesign
Following the major architectural improvements made earlier this year, Spiderline is upgrading its website design and product information to better reflect the high quality service we provide. We welcome your feedback or improvement ideas!

May 14, 2005 -- New Engine Behind Spiderline Search
We have exciting news to bring you! On May 14th Spiderline installed a major new service release into production. This release was in development for over a year and includes many significant features that we are sure you will find valuable. Most notably, Spiderline is now powered by Apache Lucene software to provide powerful search and platform independence.

• Faster Searches -- Searches are between 10x and 100x faster. On average, search results are provided in 0.014 seconds, making Spiderline one of the fastest search services on the market.
• Exact/Phrase, wild-card, and substring searches are now supported.
• Index redundancy and efficiency is greatly improved. When Spiderline performs an update of your index, it will use a previously saved copy of your document if it is not able to retrieve it from your website due to network or system errors.
• More detailed crawl logs -- identifying problems is much easier.

February 9, 2005 -- Additional Service Plans Added
In response to the many requests we've received for other service plan configurations, Spiderline has released 16 new plans providing more cost effective options for clients - especially those who have large websites but do not need frequent updates.

February 1, 2005 -- Site Uptime Monitoring
Spiderline has always had significantly high uptime and reliability. Now we have the metrics to prove it. Spiderline provides access to reports from an independent reporting service SiteUptime.Com from our corporate website. Since inception of the service (January 7th, 2005) we have had 100% uptime.

January 5, 2005 -- Reporting Architecture Replaced
Our reporting service had become painfully slow due to volume of data. We have entirely replaced the reporting architecture with a much faster system which will provide ability to store monthly summary detail for indefinite periods. Furthermore, we guarantee there will be no more data corruption problems preventing clients from accessing their reports.

February 3, 2004 -- Service Mark Registration Awarded
United States Patent and Trademark Office awarded Spiderline "e" service mark, Reg. No. 2,810,738

February 1, 2004 -- Migrated from Sun/Solaris to Linux Systems
We have migrated our database and application servers to Linux running on Intel processors. This has allowed us to significantly increase performance and at the same time lower operational costs. In some areas we have found a full four times (4x) speed increase over sun/solaris servers of comprable price.

With this change we have also added two new datacenter sites to support Spiderline crawl and search services.

October 26, 2003 -- Moved to New Datacenters
The key features of the new datacenters are:

• More space for expansion and growth
• 14 more electrical circuits with room for even more
• larger power feed (3 phase)
• Closer to the Central Office for internet connections
• Several T1 connections currently
• The new datacenter also offers easier control of more T1's and other dedicated lines
• Much larger AC unit.
• More UPS's which took up critical space in the old datacenter
• Additional 14 racks added

February 19, 2003 -- New Corporate Image
The Spiderline service mark and related corporate ID has been rebranded. We think the new image is a better reflection of the high quality service we provide. You can find new logo material under the account design section to place on your web site.

September 11, 2002 -- Pass-thru Query Data
For websites which encode their links with a session ID or other state information, Spiderline now offers the ability to take in external query parameters, preserve them, and pass them back when the customer selects a link back to the originating website. This is an exciting new feature which extends Spiderine site search capability to even more websites!

August 19, 2002 -- Document-type Icons
A design enhancement many of our customers have been asking for has been added. Document type icons can be placed beside each of your search result listings to bring further attention to Adobe PDF or Microsoft Word documents which may appear in the index.

July 22, 2002 -- Dynamically Included URLs
In response to our customer feedback, we have added support to our search result templates for dynamic inclusion of content from external URLs. This will allow you to insert a header/footer from an HTML document from your website automatically so you don't have to maintain the HTML at Spiderline.

July 16, 2002 -- Robot Section Exclusion Tag
Spiderline supports the proprietary "robots" comment tag. This tag allows a web author to apply robots exclusion rules to arbitrary sections of a document. The tag has one attribute, content, with the following possible values:

• noindex - the text enclosed in the tag is not saved in the index
• nofollow - links are not extracted from the text enclosed
• none - enclosed text is not indexed nor searched for links

This feature is expected to fit the customer need of preventing certain parts of a document - such as a navigational sidebar - from being included in the search.

June 25, 2002 -- Document Title Preference
Now Spiderline offers complete control over how matching documents are titled. An order of preference is selected so that the best title is used when available. Options include: Document Title, Document URL, Link Title, and Default Text.

April 16, 2002 -- Initial Web Posts
To aid in establishing secure sessions, Spiderline now offers the ability to automatically post data to a client's website before each crawl. Our web crawler will actually login just as a person using a web browser would to gain access to controlled content areas.

March 26, 2002 -- Custom Cookie Power!
Spiderline has supported cookies in its web crawler for some time, but now our clients can define custom cookies to be defined in the crawler before it visits their sites. Among other benefits, this feature makes it easy for webmasters to integrate Spiderline with complex website authentication systems.

Cookies are defined on the Crawl Settings page as URL, KEY, and VALUE. If the URL supplied uses SSL (https), the cookie will be treated as secure and will not be transmitted over unencrypted lines.

January 16, 2002 -- Microsoft Word 6, 7, 97, 2000 and 2002
The next in line of many document types to be supported by Spiderline in the months to come. Now Spiderline will index HTML, text, PDF, and all versions of Microsoft Word.

January 2, 2002 -- Spiderline Selects PayPal
For more security and international customer support, Spiderline has migrated its credit card processing services to PayPal (http://www.paypal.com). We welcome those of you living outside the US to register for Spiderline!

December 12, 2001 -- Shift to Full Commercial Status
Clients who participated in our development phase with free accounts have been given an additional year of free service in appreciation of their patronage and feedback contributions. We are now in the process of developing an associate program whereby users may continue to obtain free Spiderline service by referring new business to us. We'll keep you informed!

December 3, 2001 -- Broken Link Reporting
As an extended service of Spiderline's indexing process, customers now have access to a report of all broken links found on their website. This is another easy way Spiderline can help you build a better website and keep your website happy with valuable content. Link reports will soon be integrated into crawl notice emailings - so you won't have do anything to learn about potential problems on your website.

November 29, 2001 -- Hop Count Relevency Factor
Spiderline offers a large number of document relevancy controls to better control what rank documents are ordered in. We have added yet another weighting control called "Hop Factor" which gives higher weighting to words found in documents closer to the root (homepage) of a website.

November 26, 2001 -- Subsection/Partition Searches
Now users can refine their search to specific areas of content on your website. You can configure as many URL patterns to include and exclude in each category as you like, then add a $(FORM_CATEGORIES) parameter to your search page template where you want the category list to appear.

It's easy to do! Start by visiting the URL Settings page and defining your website categories. Then request a recrawling of your site. As always you can contact Spiderline support if you have questions.

November 13, 2001 -- Password Protected Site Support
Many websites secure document areas using what is known as HTAccess. Normally, this security would prohibit Spiderline from including these documents in the index. Now Spiderline customers can provide credentials for password protected areas of their website specifically for the Spiderline web crawler.

October 26, 2001 -- Search Results in XML Format
In support of the growing number of web platforms which use XML to acquire, manipulate, and pass data, Spiderline now offers output of search results in XML format. Just pass another value through the HTTP post ("f=xml") and an XML document will be produced. It's just that simple.

XML data feeds allow systems and developers of systems to do exactly what they want with the data Spiderline provides. Using Spiderline as a back-end search engine, users would perform searches against client applications, those applications would make queries to Spiderline and use the data to produce results in whatever form would best benefit their service.

October 23, 2001 -- Remote Crawl Requests
For websites which automate the addition of new content, Spiderline now offers support for remote crawl requests. Applications within your website environment can send automatically send a request to update your site search index. Securing this request is also supported by sending a cryptographic checksum instead of the remote crawl password.

HTTP Post to http://www.spiderline.com/exec/crawl
"a" = Your account ID

Non-Secure Request:
"p" = Your Remote Crawl Pasword

Secure Request:
"s" = A seed value (should be different for every request)
"h" = MD5 hex hash value from (Remote Crawl Password + Seed)

See your crawl configuration for more detail.

October 21, 2001 -- Crawl Throttling Control
For websites with an already too heavy load of traffic, or those with bandwidth/cpu resource constraints, Spiderline now offers crawl throttling control. Customers can identify exactly how many seconds or microseconds to wait between requests to their server.

October 17, 2001 -- Private Domain Name Labeling
Though spiderline already offers total customization of search result pages, users will still see "www.spiderline.com" in the browser location bar. Now Spiderline offers private domain labeling allowing our customers to use Spiderline without their customers knowing.

Depending on what domain name you want to use, implementing this feature may or may not require domain name registration changes. Only two steps are required:

1. The domain of your choice must point to the same address as www.spiderline.com.
2. You must indicate your domain name change in your design settings configuration.

October 17, 2001 -- Support for SSL (HTTPS) Crawling
Spiderline can now crawl all sites which require secure website connections using secure socket layer (SSL). Indicate the appropriate URL (either "http://www.acme.com" or "https://www.acme.com") anywhere in Spiderline site search configuration.

October 1, 2001 -- One-click Mailing List Opt-Out
Now Spiderline places a link at the bottom of all automated and announcement mailings for simple one-click opt-out. Also included in every link is a cryptographic digest to prevent changes to your account profile from those who did not receive the email.