FRIHOST FORUMS SEARCH FAQ TOS BLOGS COMPETITIONS
You are invited to Log in or Register a free Frihost Account!


how often does spiders crawl your pages





vignesh_natraj
google crawls my pages once in 10-15 days and updates the results but i am not in top 10 the keyword is my domain name
any idea to decrease this time period.
manumiglani
well it depends on how often you update your website and how often new lnbound links point to your website.
Indyan
Google last crawled my site 2 hours ago.
I have a sitemap.So it does crawl my blog pretty often.
Marston
vignesh_natraj wrote:
google crawls my pages once in 10-15 days and updates the results but i am not in top 10 the keyword is my domain name
any idea to decrease this time period.
Get a google sitemap.
erlendhg
This might help (maybe... I'm not sure it always does...)

Add in the <head> of your index page, this line:
<Meta name="Revisit-After" content="X days">

where X is number of days between each spider or robot visit.

Hope it helps.
Donutey
as said above use google sitemaps, where you can specify the period between crawls

but also, google will crawl your site more frequently if you update often... I usually get MSNbot, googlebot, and inktomi slurp every day to every couple of days.
Scorpio
Donutey wrote:
as said above use google sitemaps, where you can specify the period between crawls

but also, google will crawl your site more frequently if you update often... I usually get MSNbot, googlebot, and inktomi slurp every day to every couple of days.


I have a xml sitemap which I have submitted to google.

Not exactly the python one which they offer.

Another one but it was the correct one.

My site was crawled in 5 days after I submitted it.
SoftStag
Something will crawl my site every day; google will most days crawl at least one page. Having a google sitemap does help google to search.

Make sure you submit your site to a number of directories so the spiders can find your site.

Be aware also that google can take quite some time to display your site in it's results, even if you are being crawled regularily. This is because Google wants to be sure new sites are not "spam sites" before using them in it's results. Spam sites do not tend to be arround for many months, so it waits to see how the site evolves and decides whether or not to use it in it's results. MSN and Yahoo do not do this, your results should appear within a couple of days of your site being crawled.
Scorpio
Just the opposite happened here, Google crawled my site. but yahoo! has not crawled it in about 4 weeks.
Mumpay
i get all crawlers often, google seems to have set up camp in my forum and just munches away at it all day and all night. the others swing by at least a few times a week. damn crawlers are killing my bandwidth, heh.
NjRocket
is there anyway to know if any of the search engines have sent spiders to your site?
Scorpio
NjRocket wrote:
is there anyway to know if any of the search engines have sent spiders to your site?


Yes, you have to check google for your site

Check the date on which they have taken a cache of your site.

Also, check if the new contents you added are present in the search results

You can make an approximation
SoftStag
NjRocket wrote:
is there anyway to know if any of the search engines have sent spiders to your site?

View the stats of your page, or look in the logs. Google's spider is called Googlebot, Yahoo's is Yahoo Slurp!
dacode
Hi all,

I also have some Google sitemaps for most of my websites.
I've noticed that, no matter the "changefreq" parameter is set to, Google bot will come often if your pages are being updated frequently.

Ie. I've got one site that has almost daily changes. Everytime I re-submit the googlemap, it just takes a couple of hours to process the file, inlike whenever I update some sitemaps for other not-so-often-updated sites, it can take days for Google to crawl it anew.
SoftStag
dacode wrote:
Hi all,

I also have some Google sitemaps for most of my websites.
I've noticed that, no matter the "changefreq" parameter is set to, Google bot will come often if your pages are being updated frequently.

Ie. I've got one site that has almost daily changes. Everytime I re-submit the googlemap, it just takes a couple of hours to process the file, inlike whenever I update some sitemaps for other not-so-often-updated sites, it can take days for Google to crawl it anew.

There is no need to resubmit the sitemap. Google will come back automatically. I think Google uses the "changefreq" parameter as a guide, not a rule; they also periodically check for updates and the spiders will return when they anticipate that it will have been updated.
Jamatu


My website has been up (officially) for about 6 days i've had 2 visits each from MSNBot, Googlebot and Yahoo Slurp. So at the minute they visit me about twice a week.

What's annoying me is the bots seem to crawl the Index page and maybe one random page then leave, wtf is up with that?
Scorpio
Ummm. Jamatu, could you tell me how you get that list of bots and the visits/bandwidth etc.??
Jamatu
scorpio wrote:
Ummm. Jamatu, could you tell me how you get that list of bots and the visits/bandwidth etc.??

It's Awstats, I installed it on my server. Frihost has it, along with most other servers you just need to enable it Smile
Scorpio
How does one enable it? Very Happy
Jamatu
In cPanel you click on the "Statistics Software Configuration" link or the link "frontend/x/statmanager/index.html". All you do is enable Awstats and wait for the cronjob to build your site data (which is hourly most of the time). The logs are deleted almost daily so your first set of data will be less than a day old.

There are 3 different statistic programs available Analog, Awstats and Webalizer. You should only enable one as they are quite resource intensive.
Scorpio
I have only DirectAdmin Here Shocked
Mr. McCoy
I get crawled every day, and i get even more spam bots on my forum

but i think its okay!.. i even got 4 googlee visits in UNDER 10min after jeg upgraded my forum too Plus 1.53
Donutey
scorpio wrote:
I have only DirectAdmin Here Shocked


It will still work, it might be already enabled, just go to website statistics or something similiar and you should see awstats.

Google used to have my site mapped out pretty well, then one day I found that pretty much all of my pages no longer showed up on a site search, but then they reappear, I'm guessing I'm just getting a different datacenter depending on when I search.
Related topics
how often does tidruG smile?
The exploition of children by the ACLU
HUGE FREE LINK EXCHANGE!!
Has anyone solved the Awstats problem...?
God a superstition?
I have some error to post topic in forum!
About Adsense Click Fraud
How often does Google Update PR?
Googlebot problems on server 2?
Awesome macros: >>New ones added<<
Does Google crawl Frihost?
How is this still possible - dolphins butchered for bait?
Math in webpages
VPS1 problems
Reply to topic    Frihost Forum Index -> Webmaster and Internet -> SEO and Search Engines

FRIHOST HOME | FAQ | TOS | ABOUT US | CONTACT US | SITE MAP
© 2005-2011 Frihost, forums powered by phpBB.