FRIHOSTFORUMSSEARCHFAQTOSBLOGSCOMPETITIONS
You are invited to Log in or Register a free Frihost Account!


about robots.txt





aningbo
I'm sorry if this is the wrong forum, please move it if it is. apologise.

anyways my website is getting a lot of hits and i seriously dont know where on earth they come from. i guess its those robots and something to do with placing robots.txt in my base directory but i haven't got much time and i just can't go thru the tutorials. is there anyone around to simplified the tutorial here pliz. thank you.

http://aningbo.frih.net
hexkid
http://www.robotstxt.org/
http://en.wikipedia.org/wiki/Robots.txt
aningbo
hmmm... let me see the links. thanx for the reply. any other help
sonam
You can see all your traffic in CPanel. You don't need robots.txt.

Sonam
aningbo
i have to set up some third party script! we dont use cpnel here. thank you.

i just installed robots.txt at my site: http://aningbo.frih.net/robots.txt i really dont know if this is gonna work or not though.
hexkid
There are a few robots missing from your robots.txt file Smile

Maybe you'd like to write it as follows to only let the "Mediapartners-Google" robot in and disallow all other robots which follow robots.txt guidelines.
Code:
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /
aningbo
dont you think that might create a problem later on? is it safe. are you using it?
hexkid
I don't really know. I don't want one of my sites to be found/indexed so the robots.txt file I use is
Code:
User-agent: *
Disallow: /


I haven't checked the logs recently, but it looked like it was working when I first worried about it Smile
I just searched for my page with Google and couldn't find it -- well, ... almost couldn't find it: there are other sites which link to my site Confused
aningbo
oh well thanx. but why are you so secretive about your website? any particular reasons? or is classified Smile
Related topics
Robots.txt
robots.txt
Robots.txt.... how important is it?
Robots.txt
get to top in SEO ranking
Brett Tabke's robot.txt blog
How to prepare a "robots.txt" to get crawled by Se
Google released robots.txt generator
Sitemaps, Robots and Pageranking
Please check this robots.txt file its getting complex for me
Evitar conteúdo duplicado com robots.txt
Question about robots.txt
Robots.txt
Robots.txt and Meta Robots
Reply to topic    Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.