Yahoo slurp is acting weird whilst Indexing my website. I get two hits from it every couple of hours, one on robots.txt then another straight after on a random page. How come it's doing this? Why doesn't it just index the whole site all in one go?
It did not even touch my index page
About half of my website was indexed by Yahoo Slurp last night over the course of 8 hours.
Edit: It's still crawling, i'm getting about 6 hits an hour from it.
That's quick! What page are you on? I submitted it when the link was posted and I'm on the 9th page if you search for 'Band Of The Week.'
You can only see the first few pages on my website currently, there is a delay between a website getting crawled and the pages showing up in search results.