trafic and robot

yvanmarl

Hi all!

I'm new to web hosting. What is the trafic (percentage) generated by robot (Yahoo, Googlebot and such) on a "regular" web site?

Any tips to reduce this?

I'm setting a web site about trains. Is there any website subject more "appreciated" by robot than other?

I'm asking the question because I heard some people telling me that some subjects may "attract" robot more easily... True?
Martin. Cool

n0obie4life

The first one has something to do with robots.txt. Google that.

The second on subject, err, no, google/any other bots does not favour anything.

yvanmarl

Thank for the reply.

I think I was not clear with the "generated trafic" statement.
I'll try to be more specific!! Wink

Some website allow a specific bandwidth. If, for example, bots and spiders count for 500 Mb/month (I know this number is unreal, but it is only for "educational" pripose!! Laughing

), so a 1Gb/mo bandwidth will be use more quickly than one may think... (50% by bots, 50% by real visitors)

So, is there statistics showing how much bandwitch for a typical website is eated by bots and other "unsollicited" trafic?

Martin Very Happy

n0obie4life

Very little.

It depends on your site.

First, the robot will check for a file called "robots.txt". If its there, I'll download it (normally less than 2kb). Googlebot only downloads it once per week or day, for the others, I'm not sure.

Then, it'll broswer your homepage. If your homepage takes up 500kb of bandwidth / real visitor visit. Googlebot/otherbots would take up 500kb (Bondings will have to confirm that).

Bondings