FRIHOST • FORUMS • SEARCH • FAQ • TOS • BLOGS • DIRECTORY
You are invited to Log in or Register a Frihost Account!

NO Search Engine Robots Allowed

   Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines
 


Possum
Hi

I have pages that I don't want indexed by search engines

Can I put something in the page to stop the search engine Robots doing this...
For Example in the meta tag or header


Cheers Possum
Peterssidan
Meta tag that tells the bots not to index the page:
Code:
<meta name="robots" content="noindex, nofollow">

If you don't want the page indexed but still want the bot to follow links change nofollow to follow.

It's also possible to use the robots.txt file to prevent crawling. http://www.robotstxt.org/
LostOverThere
I use the robots.txt method. This way you can specify what files/directories you want to disallow robots from entering in a single text file. Here's how to do it:

Create a robots.txt file in your top level directory (in www/)

Code:
User-agent: *
Disallow: /phpBB/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/


That will stop all robots entering, to disallow a single Search engine robot, do this:

Code:
User-agent: BadBot
Disallow: /random/


And so forth. Good Luck.
paskall
yeah you guys right, but if I were you, I wouldnot use

Quote:
User-agent: *
Disallow: /phpBB/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/


in that case if ppl use this link to your robots.txt "http://yoursite.com/robots.txt", they would see your folder names and in that case they may use this direct link to for example your images "http://yoursite.com/images/".

I have written a topic, how to prepare a good robots.txt file, read it, it be helpful I believe.

On the other hand, don't forget only good robots would obey the commands in your robots.txt file. If you are storing e-mail addresses or private info about your clients or users etc. you should be more careful, robots.txt file may not be helpful for you. then I recommend you to make a new folder system, put all the folders inside a new one and keep the rest which you want to hide out of it, with a different name, private or imagfes are easy to guess, and insert the robots.txt inside the new folder, it only works for the same and sub level folders and files, if you keep your private files above it, they be safer.

Hope this be helpful.
Related topics

~~total SEARCH ENGINE OPTIMIZATION for your site!~~
Yahoo developing an audio search engine!
Basic Search Engine Optimizing
Make search engine With PHP and mySQL, for your site
Search Engine Scrips
submit to search Engine
What Did We Learn From The Great Search Engine Experiment
Google's Search Engine Performance Drop
Search Engine optimisation tutorial.
Integrate Personal search engine
Is Google really a great search engine?
role on advertisements on the search engine rankings
search engine?
Search Engine Optimization some Mistakes
Search Engine Hits
Reply to topic    Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2007 Frihost, forums powered by phpBB.