Understanding GoogleBot Crawl Strategy and Server Logs
One important facet of Search Engine Optimization is understanding the depth and frequency of the GoogleBot crawl. Said another way, there are strategies for maximizing the number of pages indexed or crawled by the search engines. This is particularly important for sites that are not frequently updated, that do not focus on recent news or those that are 1,000 pages or more.
Here are some helpful resources to understand the many sides of this issue: