about robots.txt

aningbo

I'm sorry if this is the wrong forum, please move it if it is. apologise.

anyways my website is getting a lot of hits and i seriously dont know where on earth they come from. i guess its those robots and something to do with placing robots.txt in my base directory but i haven't got much time and i just can't go thru the tutorials. is there anyone around to simplified the tutorial here pliz. thank you.

http://aningbo.frih.net

hexkid

http://www.robotstxt.org/
http://en.wikipedia.org/wiki/Robots.txt

aningbo

hmmm... let me see the links. thanx for the reply. any other help

sonam

You can see all your traffic in CPanel. You don't need robots.txt.

Sonam

aningbo

i have to set up some third party script! we dont use cpnel here. thank you.

i just installed robots.txt at my site: http://aningbo.frih.net/robots.txt i really dont know if this is gonna work or not though.

hexkid

There are a few robots missing from your robots.txt file Smile

Maybe you'd like to write it as follows to only let the "Mediapartners-Google" robot in and disallow all other robots which follow robots.txt guidelines.

aningbo

dont you think that might create a problem later on? is it safe. are you using it?

hexkid

I don't really know. I don't want one of my sites to be found/indexed so the robots.txt file I use is

aningbo

oh well thanx. but why are you so secretive about your website? any particular reasons? or is classified Smile