FRIHOST FORUMS SEARCH FAQ TOS BLOGS COMPETITIONS
You are invited to Log in or Register a free Frihost Account!


Please check this robots.txt file its getting complex for me





imagefree
i have a short URL service, where i want to disallow bots' access to all the Short URLs, and at the same time i want to keep bots visiting other pages.

My allowed pages are:
/ #the index page
/m page for mobile
/n/ where the whole site is kept.

i want to block the rest.

here is what i am using in robots.txt please check and tell should it work as i expected it to do?


Code:
User-agent: *
Crawl-delay: 50

Allow: /n/
Allow: /m$ # allow /m only and not /m*****
Allow: /$   #force allow the index page
Allow: /    #alternative of above (i think)

Disallow: /*   #disallow any page that has even a single character in URL after /, so index page is allowed always
Disallow: /m*  #this keeps /m always allowed
Disallow: /?   #disallow urls with ? in it on index page

#Disallow: /*  #commented
Peterssidan
It's looks correct but I think some of the lines are redundant.
imagefree
Peterssidan wrote:
It's looks correct but I think some of the lines are redundant.


can you please pointout?

thanks
pradeepc
How do you use this " $ " and " * " in Robots.txt
imagefree
pradeepc wrote:
How do you use this " $ " and " * " in Robots.txt


http://en.wikipedia.org/wiki/Robots_exclusion_standard
inphurno
isn't:

Disallow: /* #disallow any page that has even a single character in URL after /, so index page is allowed always
Disallow: /m* #this keeps /m always allowed

going to conflict with:

Allow: /m$ # allow /m only and not /m*****

since the rules are read in order?
Related topics
Robots.txt
robots.txt
google search engine??
Robots.txt.... how important is it?
Hide Part of Your Web Site from Yahoo!
get to top in SEO ranking
Brett Tabke's robot.txt blog
about robots.txt
Zero $$ Clicks! :(
Saving a .txt file to a ftp using vb6. i need help please
how to check if a file exists. please help (visual basics 6)
How to prepare a "robots.txt" to get crawled by Se
Google released robots.txt generator
Robots.txt and Meta Robots
Reply to topic    Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.