Types of Google Crawlers and Bots: Googlebot, AdsBot, Google-Extended (2026)
Do you know how many types of Google robots visit your website? Google doesn’t use just one “Googlebot”. It runs a whole family of crawlers, spiders and fetchers: some build the search index, others check ads, images, videos, shopping pages or content for AI training. Here’s a 2026 guide to the different types of Google crawlers and bots, what each one does, and how to control them with robots.txt.
Quick answer: The main Google crawler is Googlebot, which crawls mostly as a smartphone (mobile-first indexing). Others include Googlebot Image, Googlebot Video, Googlebot News, Google StoreBot, AdsBot, Mediapartners-Google (AdSense) and GoogleOther. The Google-Extended token lets you control whether your content is used to train Google’s Gemini AI models.
Table of Contents
How Google crawlers work
Crawlers (also called robots, bots or spiders) are programs that visit web pages, follow links and download content. Googlebot fetches your pages, renders them like a browser, and sends them to be indexed so they can appear in search results. Google lists every crawler and its user-agent on its Google crawlers overview.
Main Google crawlers
| Crawler | User-agent token | What it does |
|---|---|---|
| Googlebot Smartphone | Googlebot | Main crawler for Google Search; crawls as a mobile phone |
| Googlebot Desktop | Googlebot | Crawls a smaller share of pages as a desktop browser |
| Googlebot Image | Googlebot-Image | Crawls images for Google Images and Search |
| Googlebot Video | Googlebot-Video | Crawls video files and video pages |
| Googlebot News | Googlebot-News | Uses Googlebot to crawl news articles for Google News |
| Google StoreBot | Storebot-Google | Crawls product, cart and checkout pages for Google Shopping |
| Google-InspectionTool | Google-InspectionTool | Used by Search Console’s URL Inspection and the Rich Results Test |
| GoogleOther | GoogleOther | General crawler for Google product teams and research |
Special-case Google bots
- AdsBot-Google: checks the quality of Google Ads landing pages, which affects your Quality Score. It ignores the general
User-agent: *rule, so you must name it to block it. - Mediapartners-Google: the AdSense crawler. It reads your pages to show relevant ads, so don’t block it if you use AdSense.
- APIs-Google: delivers push notifications for Google APIs.
- Google-Safety: checks for malware and abuse, and ignores robots.txt.
Google-Extended: control AI training
Google-Extended isn’t a separate crawler but a robots.txt token. Blocking it tells Google not to use your content to train or improve its Gemini AI models, without affecting how your site appears in Google Search:
User-agent: Google-Extended
Disallow: /How to control Google crawlers with robots.txt
You can allow or block specific Google bots in your robots.txt file. For example, to keep your images out of Google Images while allowing normal search crawling:
User-agent: Googlebot-Image
Disallow: /Be careful: blocking the main Googlebot removes your site from Google Search. Robots.txt controls crawling, not indexing; to keep a page out of search results, use a noindex tag instead. Also make sure robots.txt doesn’t block the CSS and JavaScript Google needs to render your pages.
How to verify a real Googlebot
Fake bots often pretend to be Googlebot. To check, run a reverse DNS lookup on the visitor’s IP address: real Google crawlers resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup should return the same IP. Google also publishes its crawler IP ranges.
Myths from the old days: Deep bot and Fresh bot
When this guest post was first published in 2010, SEO forums talked about a “Deep bot” that crawled whole sites and a “Fresh bot” that checked for new content. Google never officially named these. Today one Googlebot decides how often to crawl each page based on its importance and how often it changes. The best way to get crawled quickly is still fresh, useful content, good internal links and an up-to-date XML sitemap. If you publish news, see how to create a Google News sitemap.
Frequently asked questions
How often does Googlebot crawl my site?
It varies by site. Popular, frequently updated sites are crawled many times a day; small sites less often. The Crawl Stats report in Search Console shows Googlebot’s activity on your site.
Can I ask Google to crawl a page?
Yes. Use the URL Inspection tool in Google Search Console and click Request indexing, or submit an updated sitemap.
Does blocking Google-Extended hurt my rankings?
No. Google says Google-Extended doesn’t affect inclusion or ranking in Google Search.
This article was first written by guest author Indu Priya and has been updated for 2026.
Get our free weekly email: new tech guides and tools, no spam.
New and updated tech guides, a free tool worth bookmarking and one useful pick, once a week. No spam, unsubscribe anytime.
One email a week. Unsubscribe anytime. See our privacy policy.
My website is on forumotion.com, has often had googlebots on it. how do i let all the different types crawl it? where would i put the commands to let them on it?
Thanks for explaining the different Google Bots, do you have any information about creating robots.txt files for these bots?
Its a good article..If you add more details about how google count fresh backlinks through algoritham then it will be more useful..thank you
Very interesting article!! Well done! ;)
Has ever happened to anyone to see a different response (status HTTP) from different Google bots that crawled the same resource at the same time? The right status code is 200 because the page is online and there are no redirections.
In other words:
66.249.71.153 – – [07/Feb/2010:02:44:11 -0600] “GET /mypage.htm HTTP/1.1” 301 465 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.71.42 – – [07/Feb/2010:02:44:11 -0600] “GET /mypage.htm HTTP/1.1” 200 13942 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”