The Robots META tag is another method that may be used to indicate to visiting robots whether a page should
be indexed (crawled), or links on the page should be followed. It differs from the Protocol for Robots
Exclusion in that you need no effort or permission from your Web Server Administrator.
The content of the robots META tag contains directives separated by commas. You can define [no]index, [no]follow,
all, or none. The INDEX directive specifies if an indexing robot should index the page. While a robot crawls
around your web site, it collects information about the words and links on each page; this is the process of
indexing. The FOLLOW directive specifies if a robot is to follow links on the page. The defaults are INDEX and
FOLLOW. The values ALL and NONE set all directives on or off: all=index,follow and none=noindex,nofollow.
NOTE: The "robots" name of the tag and the content are case insensitive.
Like any META tag it should be placed beteen the <head></head> tags of an HTML page:
| |
<html>
<head>
<meta name="robots" content="none">
<meta name="description" content="This page ....">
<title>...</title>
</head>
<body>
...
|
Examples:
| HTML page you do not want crawled/indexed: |
| |
<meta name="robots" content="noindex">
|
 |
| HTML page you want crawled, but do not want the robot to
follow the links on that page: |
| |
<meta name="robots" content="nofollow">
|
|
| HTML page you do not want crawled AND do not want the robot
to follow the links on that page: |
| |
<meta name="robots" content="none">
|
Print
