Controlling a Search Engine
By using robots.txt files, Meta code and the no index and no follow tags, you can control what a search engine will find and what it will record. Doing this helps to prevent a search engine from indexing old or unsuitable content (like a log in screen) and will help to improve rankings and aid the user experience from the Search engine end.
NoFollow tags [used in href tag]
These tags are used within links to prevent a link from passing on “link credit” to another. A link will pass on a certain amount of credit depending on the total number of links on that page. So one link on a page of ten links will pass on 1/10th of the link credit associated with that page. By using the “NoFollow“ tag, more links can be used, but if they are less important, can be prevented from getting any link credit. This is useful for links pointing to pages about generic information like terms and conditions, this will limit the ranking potential of the “terms and conditions” page but since it is a page not likely to generate traffic, what is the point in helping it the rank at the expense of another, more valuable page?
Because a search engine will follow links to find and index more pages, using this will prevent a search engine from finding a link to another page that is only accessible from the “NoFollow-ed” page.
NoIndex tags [used in main body]
These are tags used to control what a search engine will record in its index. It should be used for pages that contain the same information. So for example, if you had a multilingual site with a USA/UK page with the same info on it, you could leave one page as normal so the info would be indexed, but then use the “NoIndex” tag to prevent a search engine from indexing this duplicate info. Preventing duplicate data will benefit your site in the long run, so it is important to consider the pages a search engine will see as duplicated information and then use the “NoIndex” tag to prevent this from being indexed. This will help a search engine to index a site accurately.
NoFollow tags [used in main body]
These tags are used to prevent a search engine from following any of the links on a page.