google crawls my pages once in 10-15 days and updates the results but i am not in top 10 the keyword is my domain name
any idea to decrease this time period.
any idea to decrease this time period.
how often does spiders crawl your pages
well it depends on how often you update your website and how often new lnbound links point to your website.
Google last crawled my site 2 hours ago.
I have a sitemap.So it does crawl my blog pretty often.
I have a sitemap.So it does crawl my blog pretty often.
| vignesh_natraj wrote: |
| google crawls my pages once in 10-15 days and updates the results but i am not in top 10 the keyword is my domain name
any idea to decrease this time period. |
This might help (maybe... I'm not sure it always does...)
Add in the <head> of your index page, this line:
<Meta name="Revisit-After" content="X days">
where X is number of days between each spider or robot visit.
Hope it helps.
Add in the <head> of your index page, this line:
<Meta name="Revisit-After" content="X days">
where X is number of days between each spider or robot visit.
Hope it helps.
as said above use google sitemaps, where you can specify the period between crawls
but also, google will crawl your site more frequently if you update often... I usually get MSNbot, googlebot, and inktomi slurp every day to every couple of days.
but also, google will crawl your site more frequently if you update often... I usually get MSNbot, googlebot, and inktomi slurp every day to every couple of days.
| Donutey wrote: |
| as said above use google sitemaps, where you can specify the period between crawls
but also, google will crawl your site more frequently if you update often... I usually get MSNbot, googlebot, and inktomi slurp every day to every couple of days. |
I have a xml sitemap which I have submitted to google.
Not exactly the python one which they offer.
Another one but it was the correct one.
My site was crawled in 5 days after I submitted it.
Something will crawl my site every day; google will most days crawl at least one page. Having a google sitemap does help google to search.
Make sure you submit your site to a number of directories so the spiders can find your site.
Be aware also that google can take quite some time to display your site in it's results, even if you are being crawled regularily. This is because Google wants to be sure new sites are not "spam sites" before using them in it's results. Spam sites do not tend to be arround for many months, so it waits to see how the site evolves and decides whether or not to use it in it's results. MSN and Yahoo do not do this, your results should appear within a couple of days of your site being crawled.
Make sure you submit your site to a number of directories so the spiders can find your site.
Be aware also that google can take quite some time to display your site in it's results, even if you are being crawled regularily. This is because Google wants to be sure new sites are not "spam sites" before using them in it's results. Spam sites do not tend to be arround for many months, so it waits to see how the site evolves and decides whether or not to use it in it's results. MSN and Yahoo do not do this, your results should appear within a couple of days of your site being crawled.
Just the opposite happened here, Google crawled my site. but yahoo! has not crawled it in about 4 weeks.
i get all crawlers often, google seems to have set up camp in my forum and just munches away at it all day and all night. the others swing by at least a few times a week. damn crawlers are killing my bandwidth, heh.
is there anyway to know if any of the search engines have sent spiders to your site?
| NjRocket wrote: |
| is there anyway to know if any of the search engines have sent spiders to your site? |
Yes, you have to check google for your site
Check the date on which they have taken a cache of your site.
Also, check if the new contents you added are present in the search results
You can make an approximation
| NjRocket wrote: |
| is there anyway to know if any of the search engines have sent spiders to your site? |
View the stats of your page, or look in the logs. Google's spider is called Googlebot, Yahoo's is Yahoo Slurp!
Hi all,
I also have some Google sitemaps for most of my websites.
I've noticed that, no matter the "changefreq" parameter is set to, Google bot will come often if your pages are being updated frequently.
Ie. I've got one site that has almost daily changes. Everytime I re-submit the googlemap, it just takes a couple of hours to process the file, inlike whenever I update some sitemaps for other not-so-often-updated sites, it can take days for Google to crawl it anew.
I also have some Google sitemaps for most of my websites.
I've noticed that, no matter the "changefreq" parameter is set to, Google bot will come often if your pages are being updated frequently.
Ie. I've got one site that has almost daily changes. Everytime I re-submit the googlemap, it just takes a couple of hours to process the file, inlike whenever I update some sitemaps for other not-so-often-updated sites, it can take days for Google to crawl it anew.
| dacode wrote: |
| Hi all,
I also have some Google sitemaps for most of my websites. I've noticed that, no matter the "changefreq" parameter is set to, Google bot will come often if your pages are being updated frequently. Ie. I've got one site that has almost daily changes. Everytime I re-submit the googlemap, it just takes a couple of hours to process the file, inlike whenever I update some sitemaps for other not-so-often-updated sites, it can take days for Google to crawl it anew. |
There is no need to resubmit the sitemap. Google will come back automatically. I think Google uses the "changefreq" parameter as a guide, not a rule; they also periodically check for updates and the spiders will return when they anticipate that it will have been updated.
My website has been up (officially) for about 6 days i've had 2 visits each from MSNBot, Googlebot and Yahoo Slurp. So at the minute they visit me about twice a week.
What's annoying me is the bots seem to crawl the Index page and maybe one random page then leave, wtf is up with that?
Ummm. Jamatu, could you tell me how you get that list of bots and the visits/bandwidth etc.??
| scorpio wrote: |
| Ummm. Jamatu, could you tell me how you get that list of bots and the visits/bandwidth etc.?? |
It's Awstats, I installed it on my server. Frihost has it, along with most other servers you just need to enable it
How does one enable it? 
In cPanel you click on the "Statistics Software Configuration" link or the link "frontend/x/statmanager/index.html". All you do is enable Awstats and wait for the cronjob to build your site data (which is hourly most of the time). The logs are deleted almost daily so your first set of data will be less than a day old.
There are 3 different statistic programs available Analog, Awstats and Webalizer. You should only enable one as they are quite resource intensive.
There are 3 different statistic programs available Analog, Awstats and Webalizer. You should only enable one as they are quite resource intensive.
I have only DirectAdmin Here 
I get crawled every day, and i get even more spam bots on my forum
but i think its okay!.. i even got 4 googlee visits in UNDER 10min after jeg upgraded my forum too Plus 1.53
but i think its okay!.. i even got 4 googlee visits in UNDER 10min after jeg upgraded my forum too Plus 1.53
| scorpio wrote: |
| I have only DirectAdmin Here |
It will still work, it might be already enabled, just go to website statistics or something similiar and you should see awstats.
Google used to have my site mapped out pretty well, then one day I found that pretty much all of my pages no longer showed up on a site search, but then they reappear, I'm guessing I'm just getting a different datacenter depending on when I search.
 |