Different Types of Google Robots, Crawlers, Spiders

By Posted 2010 Updated   GoogleGuest PostsSEO

Do you know how many types of Google robots, bots there are? Google indexes millions of pages each and every day. These pages are crawled by a special Google algorithm called Google Bot, before each and every page in its database is indexed. In fact, Google uses a number of spiders to crawl the web. This article reveals the most important Google spiders and their functionality.

Google Robots, Crawlers, Spiders Types

google botGoogle Bot

This is the Google robot which crawls the web to identify new pages and add to Google index. It has two different versions. Deep bot and Fresh bot. As the name suggests, Deep bot tries to follow every link and downloads as many pages as possible. This google web crawler tries to read minute details of a site including the internal structure.

Fresh bot crawls the web in search of fresh content. The Fresh bot crawls the already indexed pages in search of newly updated content. This Google spider tries to give fresh and updated information in the search results. Hence it is important to have quality inbound links if you’d like Fresh bot crawling your site more and more often and updating it in Google Index.

Media Bot

If you are wondering how Google places different relevant ads in your different pages, here is the answer. Media bot analyzes AdSense pages and decides which ads to be placed on different pages.

So if you want to allow Media bot to access your entire site put this code in robot.txt file.
User-agent: Mediapartners-Google*

Remember that ads will still be shown in your pages even if Media bot has not visited your site yet. In such a case, ads will be chosen based on the overall theme of your site.

Image Bot

Image bot is another google robot used to crawl the images and place them in Google’s image search. Images are ranked based on their file name, alt text, surrounding text and page title.  If your website is primarily image based, then you’d definitely want to tweak and optimize your images to receive extra traffic from Google image search results.

However, all the Google traffic may not convert into buyers as many people are just looking for images. So if you want to block Image Bot from accessing your images and save bandwidth, you can do it in your robot.txt file.

Ads Bot

It crawls and analyzes the advertising landing pages and determines the quality score that will be assigned to your ads. Along with this score, Google uses your bid amount to determine the position of your ads. Hence it is important to have quality content on the landing page of your advertisement. Sometimes Google positions high quality ads first even when others are willing to pay more. It is advised to give AdsBot complete access to your site.

By now you must have understood that it is important to give complete access to Google if you want to increase your search engine visibility. One way is to create a sitemap for your site. A sitemap tells Google about your most important, new and updated pages. Google in turn tells you the pages it was unable to crawl. This helps you pinpoint the problems and fix them as soon as possible with which you can gain increased traffic and exposure from search engine results.

Google is ever-changing and Google spiders are becoming more advanced all the time. However, relevant, updated and quality content along with good inbound links make your site stand front in this every changing search engine world.

This article is written by guest author Indu Priya, an online marketer from theideashome.com. She is a traffic expert and offers traffic building services to small business owners. Get her step-by-step guide on how to drive 100+ traffic per day to a new site

This is a guest article and represents opinions of the guest author

5 comments on “Different Types of Google Robots, Crawlers, Spiders

  1. Nimit Kashyap says:

    useful post, and i didn’t know about the fresh bot before.

  2. Giorgio says:

    Very interesting article!! Well done! ;)

    Has ever happened to anyone to see a different response (status HTTP) from different Google bots that crawled the same resource at the same time? The right status code is 200 because the page is online and there are no redirections.

    In other words:

    66.249.71.153 – – [07/Feb/2010:02:44:11 -0600] “GET /mypage.htm HTTP/1.1” 301 465 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

    66.249.71.42 – – [07/Feb/2010:02:44:11 -0600] “GET /mypage.htm HTTP/1.1” 200 13942 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

  3. antheay says:

    Its a good article..If you add more details about how google count fresh backlinks through algoritham then it will be more useful..thank you

  4. R says:

    Thanks for explaining the different Google Bots, do you have any information about creating robots.txt files for these bots?

  5. Harrison says:

    My website is on forumotion.com, has often had googlebots on it. how do i let all the different types crawl it? where would i put the commands to let them on it?

Leave a Reply

Your email address will not be published. Required fields are marked *




Next Article »
css.php