We're adding features left and right! Spiderline developers have a master list
of every feature imaginable - and they are building in the order of customer
demand. If you don't see what you want at Spiderline, let us know and we'll be
sure to let you know when it becomes available.
July 29, 2009 -- All New Pricing on our Service Plans!
We have reviewed our service plan pricing model and significantly reduced prices across the board.
Sign up today to lock in a low price on your service plan.
December 18, 2005 -- Product Website Redesign
Following the major architectural improvements made earlier this year, Spiderline is upgrading its
website design and product information to better reflect the high quality service we provide. We
welcome your feedback or improvement ideas!
May 14, 2005 -- New Engine Behind Spiderline Search
We have exciting news to bring you! On May 14th Spiderline installed a major new service release into
This release was in development for over a year and includes many significant features that we are sure
you will find valuable. Most notably, Spiderline is now powered by Apache Lucene software to provide
powerful search and platform independence.
- Faster Searches -- Searches are between 10x and 100x faster. On average, search results are provided
in 0.014 seconds,
making Spiderline one of the fastest search services on the market.
- Exact/Phrase, wild-card, and substring searches are now supported.
- Index redundancy and efficiency is greatly improved. When Spiderline performs an update of your
index, it will use a previously saved copy of your
document if it is not able to retrieve it from your website due to network or system errors.
- More detailed crawl logs -- identifying problems is much easier.
February 9, 2005 -- Additional Service Plans Added
In response to the many requests we've received for other service plan configurations, Spiderline
has released 16 new plans providing more cost effective options for clients - especially those who
have large websites but do not need frequent updates.
February 1, 2005 -- Site Uptime Monitoring
Spiderline has always had significantly high uptime and reliability. Now we have the metrics to
prove it. Spiderline provides access to reports from an independent reporting service
SiteUptime.Com from our corporate website. Since inception of the service (January 7th, 2005)
we have had 100% uptime.
January 5, 2005 -- Reporting Architecture Replaced
Our reporting service had become painfully slow due to volume of data. We have entirely replaced
the reporting architecture with a much faster system which will provide ability to store monthly
summary detail for indefinite periods. Furthermore, we guarantee there will be no more data corruption
problems preventing clients from accessing their reports.
February 3, 2004 -- Service Mark Registration Awarded
United States Patent and Trademark Office awarded Spiderline "e" service mark,
Reg. No. 2,810,738
February 1, 2004 -- Migrated from Sun/Solaris to Linux Systems
We have migrated our database and application servers to Linux running on
Intel processors. This has allowed us to significantly increase performance
and at the same time lower operational costs. In some areas we have found a
full four times (4x) speed increase over sun/solaris servers of comprable
With this change we have also added two new datacenter sites to support
Spiderline crawl and search services.
October 26, 2003 -- Moved to New Datacenters
The key features of the new datacenters are:
- More space for expansion and growth
- 14 more electrical circuits with room for even more
- larger power feed (3 phase)
- Closer to the Central Office for internet connections
- Several T1 connections currently
- The new datacenter also offers easier control of more T1's and other
- Much larger AC unit.
- More UPS's which took up critical space in the old datacenter
- Additional 14 racks added
February 19, 2003 -- New Corporate Image
The Spiderline service mark and related corporate ID has been rebranded. We
think the new image is a better reflection of the high quality service we
provide. You can find new logo material under the account design section
to place on your web site.
September 11, 2002 -- Pass-thru Query Data
For websites which encode their links with a session ID or other state information,
Spiderline now offers the ability to take in external query parameters, preserve them,
and pass them back when the customer selects a link back to the originating website.
This is an exciting new feature which extends Spiderine site search capability to
even more websites!
August 19, 2002 -- Document-type Icons
A design enhancement many of our customers have been asking for has been added.
Document type icons can be placed beside each of your search result listings
to bring further attention to Adobe PDF or Microsoft Word documents which
may appear in the index.
July 22, 2002 -- Dynamically Included URLs
In response to our customer feedback, we have added support to our search
result templates for dynamic inclusion of content from external URLs. This will
allow you to insert a header/footer from an HTML document from your website
automatically so you don't have to maintain the HTML at Spiderline.
July 16, 2002 -- Robot Section Exclusion Tag
Spiderline supports the
proprietary "robots" comment tag.
This tag allows a web
author to apply robots exclusion rules to arbitrary sections of a document. The
tag has one attribute, content, with the following possible values:
- noindex - the text enclosed in the tag is not saved in the index
- nofollow - links are not extracted from the text enclosed
- none - enclosed text is not indexed nor searched for links
This feature is expected to fit the customer need of preventing certain parts of
a document - such as a navigational sidebar - from being included in the search.
June 25, 2002 -- Document Title Preference
Now Spiderline offers complete control over how matching documents are
titled. An order of preference is selected so that the best title is used
when available. Options include: Document Title, Document URL, Link Title,
and Default Text.
April 16, 2002 -- Initial Web Posts
To aid in establishing secure sessions, Spiderline now offers the ability to
automatically post data to a client's website before each crawl. Our web
crawler will actually login just as a person using a web browser would to
gain access to controlled content areas.
March 26, 2002 -- Custom Cookie Power!
Spiderline has supported cookies in its web crawler for some time, but now
our clients can define custom cookies to be defined in the crawler before it
visits their sites. Among other benefits, this feature makes it easy for
webmasters to integrate Spiderline with complex website authentication systems.
Cookies are defined on the Crawl Settings page as URL, KEY, and VALUE. If
the URL supplied uses SSL (https), the cookie will be treated
as secure and will not be transmitted over unencrypted lines.
January 16, 2002 -- Microsoft Word 6, 7, 97, 2000 and 2002
The next in line of many document types to be supported by Spiderline in the
months to come. Now Spiderline will index HTML, text, PDF, and all versions
of Microsoft Word.
January 2, 2002 -- Spiderline Selects PayPal
For more security and international customer support, Spiderline has
migrated its credit card processing services to PayPal
(http://www.paypal.com). We welcome
those of you living outside the US to register for Spiderline!
December 12, 2001 -- Shift to Full Commercial Status
Clients who participated in our development phase with free accounts have been given
an additional year of free service in appreciation of their patronage and feedback
contributions. We are now in the process of developing an associate program
whereby users may continue to obtain free Spiderline service by
referring new business to us. We'll keep you informed!
December 3, 2001 -- Broken Link Reporting
As an extended service of Spiderline's indexing process, customers
now have access to a report of all broken links found on their
website. This is another easy way Spiderline can help you build a
better website and keep your website happy with valuable content.
Link reports will soon be integrated into crawl notice emailings -
so you won't have do anything to learn about potential problems on
November 29, 2001 -- Hop Count Relevency Factor
Spiderline offers a large number of document relevancy controls
to better control what rank documents are ordered in. We have
added yet another weighting control called "Hop Factor" which
gives higher weighting to words found in documents closer to
the root (homepage) of a website.
November 26, 2001 -- Subsection/Partition Searches
Now users can refine their search to specific areas of content
on your website.
You can configure as many URL patterns to include and exclude in
each category as you like, then add a $(FORM_CATEGORIES) parameter
to your search page template where you want the category list to
It's easy to do! Start by visiting the URL Settings page and
defining your website categories. Then request a recrawling of your
site. As always you can contact Spiderline support if you have questions.
November 13, 2001 -- Password Protected Site Support
Many websites secure document areas using what is known as HTAccess.
Normally, this security would prohibit Spiderline from including these
documents in the index. Now Spiderline customers can provide
credentials for password protected areas of their website specifically for
the Spiderline web crawler.
October 26, 2001 -- Search Results in XML Format
In support of the growing number of web platforms which use XML to
acquire, manipulate, and pass data, Spiderline now offers output of
search results in XML format. Just pass another value through the
HTTP post ("f=xml") and an XML document will be produced. It's just
XML data feeds allow systems and developers of systems to do exactly
what they want with the data Spiderline provides. Using Spiderline
as a back-end search engine, users would perform searches against
client applications, those applications would make queries to Spiderline
and use the data to produce results in whatever form would best
benefit their service.
October 23, 2001 -- Remote Crawl Requests
For websites which automate the addition of new content, Spiderline now
offers support for remote crawl requests. Applications within your
website environment can send automatically send a request to update
your site search index. Securing this request is also supported by
sending a cryptographic checksum instead of the remote crawl password.
HTTP Post to http://www.spiderline.com/exec/crawl
"a" = Your account ID
"p" = Your Remote Crawl Pasword
"s" = A seed value (should be different for every request)
"h" = MD5 hex hash value from (Remote Crawl Password + Seed)
See your crawl configuration for more detail.
October 21, 2001 -- Crawl Throttling Control
For websites with an already too heavy load of traffic, or those
with bandwidth/cpu resource constraints, Spiderline now offers
crawl throttling control. Customers can identify exactly how many
seconds or microseconds to wait between requests to their server.
October 17, 2001 -- Private Domain Name Labeling
Though spiderline already offers total customization of search
result pages, users will still see "www.spiderline.com" in the browser
location bar. Now Spiderline offers private domain labeling
allowing our customers to use Spiderline without their customers knowing.
Depending on what domain name you want
to use, implementing this feature may or may not require domain name
registration changes. Only two steps are required:
- The domain of your choice must point to the same address as
- You must indicate your domain name change in your design settings
October 17, 2001 -- Support for SSL (HTTPS) Crawling
Spiderline can now crawl all sites which require secure website
connections using secure socket layer (SSL). Indicate the appropriate
URL (either "http://www.acme.com" or "https://www.acme.com") anywhere
in Spiderline site search configuration.
October 1, 2001 -- One-click Mailing List Opt-Out
Now Spiderline places a link at the bottom of all automated and
announcement mailings for simple one-click opt-out.
Also included in every link is a cryptographic digest to prevent changes
to your account profile from those who did not receive the email.