Googlebots Crawling Around Everywhere…

Google wants to know what the millions of sites on the internet are about, in order to categorise them correctly and deliver the best search results experience to end users. Although Google has never revealed how many computers they’re running, a report last year estimated they are using around 900,000 servers based on their energy use.

Googlebots are programmes governed by algorithms to fetch websites and go by a few different names – spiders, robots and bots. They also decide how often your site is crawled and which pages are fetched. Although Google provides some insight within Google Webmaster Tools as to how these algorithms work, no one except for Google (this includes Search Engine Optimisation firms) knows specifically how they work. 

Google states that “Google’s crawl process begins with a list of Web page URLs generated from previous crawl processes and augmented with Sitemap data provided by webmasters,”. Basically the bots begin crawling URL’s on their lists and when a link is found it’s placed on the “to be crawled” list. Dead links are updated in the Google index.

Each word on your site counts, as Googlebot goes through each line of it and even notes where on the page words are found. This is why it helps to have the most valuable keywords listed sooner rather than later. Google also processes information included in key content tags and attributes, such as Title tags and ALT tags. Therefore, your HTML should contain keywords in these sections. Making sure your ALT attributes (which relate to images on your site) and Title tags include relevant words can be a beneficial SEO technique.

According to Google, most websites in their database have actually been found by automatic crawling, rather than being added manually. There are some sites which Google misses, but these are only sites which don’t have links to them. It’s important to keep up your number of inbound links and ensure there are no dead or bad links.

You can confirm whether Google has indexed your site by entering “site:yoursite.com” into your Google search. If your site appears, then it’s been indexed by Google. You can supply a sitemap of your website to help encourage Google to crawl and index your site. Also think about asking Google to crawl your site after any major page updates or additions of new pages to let them know it’s time to be crawled again.