No I haven't got arachnophobia.

There are many good reason why you would want to hide some of your website from the search engine spiders. Among them:

• The content is private
• It is duplicate (Wordpress produces multiple URLs for the same content).
• You don't want it to leak page rank. (contact us , privacy policy pages etc)
• It is not worth being indexed

How To Hide

There are three techniques you can use to stop your page(s) from appearing in the search engine index. They are:

1. Robots Tag and the Noindex and Nofollow Meta tags
2. No Follow tag on links
3. The Robots.txt file

Robots Tag

The robots tag is not normally used. It is used to tell search engines how to index the Web Page and follow and links contained on the page. Web pages will be indexed by search engines and links will be followed unless explicitly being told not to.

Normally you will want your page to be indexed by the search engines and hence you will not need to include this tag. Include the tag only when you want to stop a Web page being indexed.

Use:

<META NAME="ROBOTS"  CONTENTS=" index or noindex ,follow or nofollow">

Examples:

This tag will stop your web page being indexed and links on the page being followed.

<META NAME="ROBOTS"  CONTENTS=" noindex ,nofollow">

Here you can find out more  about meta tags.

NoFollow Link Attribute

This is relatively new and somewhat controversial; being originally used in an attempt to block comment spam in blogs. It seems that Google  obeys this tag whereas yahoo and Microsoft don't.

Therefore you should not rely on this tag to keep your pages out of the search engine index.

I would recommend that you reserve the use of this tag to linking to external sites. In fact Google have recommended that paid directories use this tag.

Here is the syntax for the link:

 <a href="http://www.build-your-website.co.uk" rel="nofollow">making a website</a>

Robots.txt File

This is the best method but also potentially the most dangerous one. You need to understand this file before you start using it. You can find more details in the robots.txt article.

Summary

For less technical and inexperienced web masters I would recommend you use the meta tags and the nofollow link attribute.

For the more experienced and technical savvy the robots.txt file is preferred in conjunction with the nofollow link attribute is preferred.

Comments?- you can comment on the article here

Web www.build-your-website.co.uk