Official Google Webmaster Central Blog: January 2014

A new Googlebot user-agent for crawling smartphone content

Thursday, January 23, 2014

Webmaster level: AdvancedOver the years, Google has used different crawlers to crawl and index content for feature phones and smartphones. These mobile-specific crawlers have all been referred to as Googlebot-Mobile. However, feature phones and smartphones have considerably different device capabilities, and we've seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing. This ambiguity made it impossible for Google to index smartphone content of some sites, or for Google to recognize that these sites are smartphone-optimized.A new Googlebot for smartphonesTo clarify the situation and to give webmasters greater control, we'll be retiring "Googlebot-Mobile" for smartphones as a user agent starting in 3-4 weeks' time. From then on, the user-agent for smartphones will identify itself simply as "Googlebot" but will still list "mobile" elsewhere in the user-agent string. Here are the new and old user-agents:The new Googlebot for smartphones user-agent:

Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

(updated August 2015)

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The Googlebot-Mobile for smartphones user-agent we will be retiring soon:

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

This change affects only Googlebot-Mobile for smartphones. The user-agent of the regular Googlebot does not change, and the remaining two Googlebot-Mobile crawlers will continue to refer to feature phone devices in their user-agent strings; for reference, these are:Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)The two Googlebot-Mobile user-agents for feature phones:

SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

You can test your site using the Fetch as Google feature in Webmaster Tools, and you can see a full list of our existing crawlers in the Help Center.Crawling and indexingPlease note this important implication of the user-agent update: The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile. For example, when the new crawler is deployed, this robots.txt directive will block all crawling by the new Googlebot for smartphones user-agent, and also the regular Googlebot:User-agent: Googlebot Disallow: / This robots.txt directive will block crawling by Google’s feature phone crawlers:User-agent: Googlebot-Mobile Disallow: / Based on our internal analyses, this update affects less than 0.001% of URLs while giving webmasters greater control over the crawling and indexing of their content. As always, if you have any questions, you can:

Read our recommendations for building smartphone-optimized sites

Learn more about controlling Googlebot crawling and indexing

Ask in our Webmaster help forums or visit one of our Webmaster Central office hours hangouts.

Posted by Zhijian He, Smartphone search engineer

Changes in crawl error reporting for redirects

Wednesday, January 22, 2014

Webmaster level: intermediate-advanced

In the past, we have seen occasional confusion by webmasters regarding how crawl errors on redirecting pages were shown in Webmaster Tools. It's time to make this a bit clearer and easier to diagnose! While it used to be that we would report the error on the original - redirecting - URL, we'll now show the error on the final URL - the one that actually returns the error code.

Let's look at an example:

URL A redirects to URL B, which in turn returns an error. The type of redirect, and type of error is unimportant here.

In the past, we would have reported the error observed at the end under URL A. Now, we'll instead report it as URL B. This makes it much easier to diagnose the crawl errors as they're shown in Webmaster Tools. Using tools like cURL or your favorite online server header checker, you can now easily confirm that this error is actually taking place on URL B.

This change may also be visible in the total error counts for some websites. For example, if your site is moving to a new domain, you'll only see these errors for the new domain (assuming the old domain redirects correctly), which might result in noticeable changes in the total error counts for those sites.

Note that this change only affects how these crawl errors are shown in Webmaster Tools. Also, remember that having crawl errors for URLs that should be returning errors (e.g. they don't exist) does not negatively affect the rest of the website's indexing or ranking (also as discussed on Google+).

We hope this change makes it a bit easier to track down crawl errors, and to clean up the accidental ones that you weren't aware of! If you have any questions, feel free to post here, or drop by in the Google Webmaster Help Forum.

Posted by John Mueller, Website Error Analyst

Google Publisher Plugin beta: Bringing our publisher products to WordPress

Wednesday, January 15, 2014

Cross-posted from the Inside AdSense blog.

We’ve heard from many publishers using WordPress that they’re looking for an easier way to work with Google products within the platform. Today, we’re excited to share the beta release of our official Google Publisher Plugin, which adds new functionality to publishers’ WordPress websites. If you own your own domain and power it with WordPress, this new plugin will give you access to a few Google services — and all within WordPress.

Please keep in mind that because this is a beta release, we’re still fine-tuning the plugin to make sure it works well on the many WordPress sites out there. We’d love for you to try it now and share your feedback on how it works for your site.

This first version of the Google Publisher Plugin currently supports two Google products:

Visit the WordPress.org plugin directory to download the new plugin and give it a try. For more information about the plugin and how to use it, please visit our Help Center. We look forward to hearing your feedback!

Posted by Michael Smith - Product Manager

More detailed search queries in Webmaster Tools

Tuesday, January 07, 2014

We hope this makes it easier for you to see the finer details of how users are finding your website, and when they're clicking through. Should you have any questions, feel free to visit our help forum.

Posted by John Mueller, frequent searcher & Webmaster Trends Analyst, Google Switzerland

Improved Search Queries stats for separate mobile sites

Tuesday, January 07, 2014

Prior to this Search Queries improvement, Webmaster Tools reported Skip Redirect impressions with the desktop URL. Now we’ve consolidated information when Skip Redirect is triggered, so that impressions, clicks, and CTR are calculated solely with the verified m. site, making your mobile statistics more understandable.

Best practices if you have a separate m. site

Here are a few search-friendly recommendations for those publishing content on a separate m. site:

Follow our advice on Building Smartphone-Optimized Websites

On the desktop page, add a special link rel="alternate" tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site's mobile pages.
On the mobile page, add a link rel="canonical" tag pointing to the corresponding desktop URL.
Use the HTTP Vary: User-Agent header if your servers automatically redirect users based on their user agent/device.

Verify ownership of both the desktop (www) and mobile (m.) sites in Webmaster Tools for improved communication and troubleshooting information specific to each site.

* Be sure you've verified ownership for your mobile site!

Written by Maile Ohye, Developer Programs Tech Lead

Webmaster ToolsSearch Queries

Queries where your m. pages appeared in search results for mobile browsers

Queries where Google applied Skip Redirect. This means that, while search results displayed the desktop URL, the user was automatically directed to the corresponding m. version of the URL (thus saving the user from latency of a server-side redirect).

Skip Redirect information (impressions, clicks, etc.) calculated with mobile site.Best practices if you have a separate m. site

Follow our advice on Building Smartphone-Optimized Websites

On the desktop page, add a special link rel="alternate" tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site's mobile pages.
On the mobile page, add a link rel="canonical" tag pointing to the corresponding desktop URL.
Use the HTTP Vary: User-Agent header if your servers automatically redirect users based on their user agent/device.

Verify ownership of both the desktop (www) and mobile (m.) sites in Webmaster Tools for improved communication and troubleshooting information specific to each site.

Written by Maile Ohye, Developer Programs Tech Lead

Webmaster Central Blog

Affiliate programs and added value

A new Googlebot user-agent for crawling smartphone content

Changes in crawl error reporting for redirects

Google Publisher Plugin beta: Bringing our publisher products to WordPress

More detailed search queries in Webmaster Tools

Improved Search Queries stats for separate mobile sites

Labels

Archive

Feed

Subscribe via email