I've found an interested article from Google Webmaster Central Blog it's about Speaking the language of robots written by Sean Harding the Webmaster Tools Team on Thursday, March 27, 2008
They released and introduced you to your personal robot translator: the Robots.txt Generator in Webmaster Tools. It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots and you can create fine-grained rules for specific robots and areas of your site as well if you need.
You can test your new robots.txt file with their robots.txt analysis tool after finished with the generator to see its effect. After you're done, just save your file to the top level (root) directory of your site, and you're good to go.
This new tool will helps you inform what you want to Googlebot and all other robots whose visit your site.
Hope this useful.
They released and introduced you to your personal robot translator: the Robots.txt Generator in Webmaster Tools. It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots and you can create fine-grained rules for specific robots and areas of your site as well if you need.
You can test your new robots.txt file with their robots.txt analysis tool after finished with the generator to see its effect. After you're done, just save your file to the top level (root) directory of your site, and you're good to go.
This new tool will helps you inform what you want to Googlebot and all other robots whose visit your site.
Hope this useful.
