FRIHOSTFORUMSSEARCHFAQTOSBLOGSCOMPETITIONS
You are invited to Log in or Register a free Frihost Account!


Question about robots.txt





sonam
I need little help for crating robots.txt because I would like to prevent indexing of one page. This is easy to do if page have fixed url all the time. For example:

Code:
User-agent: *
Disallow: /name.php


But I am not sure are this work if this page have dynamic url because this name.php sometimes looking like:
/name.php?id=111
or
/name.php?id=222
etc.

Are crawlers disallow all dynamic urls, too, or only plain name.php and how I can stop to crawl any of dynamic url. Note: I cannot use:
Code:
Disallow: /*?

because I have other dynamic pages. Maybe I can write something like
Code:
Disallow: /name.php*


Sonam
Bondings
You can also use a meta tag for it. Of course the search engines have to crawl the file, but they should not index it.
Code:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
sonam
Bondings wrote:
You can also use a meta tag for it. Of course the search engines have to crawl the file, but they should not index it.
Code:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


This is good solution, but I would like to use boot meta tags and robots.txt because some search engines ignore neta tags.

Sonam
Related topics
Robots.txt
robots.txt
Robots.txt.... how important is it?
Robots.txt
Flash Site or No?!
get to top in SEO ranking
Robot.txt and Meta tags
about robots.txt
How to prepare a "robots.txt" to get crawled by Se
Google released robots.txt generator
Please check this robots.txt file its getting complex for me
Evitar conteúdo duplicado com robots.txt
Robots.txt
Robots.txt and Meta Robots
Reply to topic    Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.