FRIHOST • FORUMS • FAQ • TOS • BLOGS • DIRECTORY
You are invited to Log in or Register a Frihost Account!

Googlebot banned from server 2?

   Frihost Forum Index -> Support and Web Hosting -> Web Hosting Support
 


Peterssidan
I believe Google have problems to crawl sites on server 2.
It all started with that Google webmaster tools began to show up urls under "Unreachable URL" with the explanation "robots.txt unreachable". I first believed that it was because the server had some downtime when the googlebot was on visit but it has come more reports like this with different dates.

I began to worry when I saw that one of my sites have lost it's startpage from the google index. But now my other site also got the startpage out of google index and other pages(doesn't shows on "site:" search. No cache).
I have tried to figure out why it happens. It looks like the robots.txt is alright on both the domains so that is not the problem.

I looked on other sites hosted on server 2 and it looks like googlebot last crawl the sites the 9th October. No cache show a newer date except one that I found.

Can Googlebot by accident have been banned from accessing the server 2?
bloodrider
I doubt that Confused
Googlebot doesn't visit your site every day... I think that if the site it isn't actualized too often (like a blog), the bot only comes from 2 in 2 months.
Also, the site must has some text (at least some metas or divs, something like that) to the bot read, for example, if your site is in flash the bot can't read it as it is a text browser it only can read text Exclamation

I'm not 100% sure about this but for certain someone better informed will correct me Wink
Bondings
I found a cached page from 28 October on server 2, so I don't think this is a problem. Maybe it was just a glitch.
Peterssidan
Yes but googlebot are using various IP numbers so maybe there is some of them that are not banned. I don't know how the server handle automatic ban if there is very fast crawl.

If there is non other than me that have this problem it's maybe my fault.
Bondings
Peterssidan wrote:
Yes but googlebot are using various IP numbers so maybe there is some of them that are not banned. I don't know how the server handle automatic ban if there is very fast crawl.

If there is non other than me that have this problem it's maybe my fault.

This could be the case, but I doubt it. It might also have been the case that the server was busy or down when the googlebot tried to fetch the robots.txt page.

There are also a lot of other reasons that your pages don't show up anymore in Google. And I'm not sure if an unreachable robots.txt page has that effect.
Peterssidan
http://www.google.com/support/webmasters/bin/answer.py?answer=35154&hl=en
Quote:

URL unreachable /robots.txt unreachable

Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your robots.txt file was unreachable. To make sure we didn't crawl any pages listed in that file, we postponed our crawl. When this happens, we return to your site later and crawl it once we can reach your robots.txt file. Note that this is different from a 404 response when looking for a robots.txt file. If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl.


So that is the reason why it stops on the robots.txt file. Has it been very much downtime on the server the last time that I havn't notice? I have removed the robots.txt file to see if that makes any change but I now that it returned 200 status before.

googlebot have had at least one succesful crawl but something looks wrong. I will start a new thread in the SEO and Search Engines section to ask the other frihost members if they have the same problem or if they know what it is about.
Peterssidan
I don't know if you, Bondings have seen the thread I created at SEO and Search Engines forum. Here it is: http://www.frihost.com/forums/vt-83228.html

I want to know if you still don't belive that googlebot is banned or have problems with server 2 for some reason? And can we expect it to be solved in a soon future? I know you are busy so I understand that it can take some time.
Reply to topic    Frihost Forum Index -> Support and Web Hosting -> Web Hosting Support

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2007 Frihost, forums powered by phpBB.