Googlebot Starting to Crawl Sites Over HTTP/2

Starting in November, Googlebot will support crawling over HTTP/2 for some websites. HTTP/2 is the next major revision of HTTP.

Google confirmed this forthcoming update in an announcement made today on the company blog.

What most people need to know about HTTP/2 is: it’s more robust, efficient, and faster than HTTP.

Google says this change is being made to make crawling more efficient with respect to server and resource usage.

“With h2, Googlebot is able to open a single TCP connection to the server and efficiently transfer multiple files over it in parallel, instead of requiring multiple connections.

The fewer connections open, the fewer resources the server and Googlebot have to spend on crawling.”

Here’s how the implementation of HTTP/2 crawling will work.

Googlebot Crawling Over HTTP/2

The switch to crawling some sites over HTTP/2 will happen in phases.

The first phase, Googlebot will crawl a small number of sites over HTTP/2, and gradually expand to more sites that many benefit from the supported features.

Googlebot decides which sites to crawl over HTTP/2 asked on whether the site supports it and whether Googlebot would benefit from crawling over HTTP/2.

Sites may be eligible for the connection upgrade if:

• The site’s server supports HTTP/2
• Googlebot already crawls the site a lot

There is nothing site owners need to do in order to prepare for the change.

Google notes that it’s fine if your site’s server still only talks HTTP/1.1.

There’s no drawback to crawling over HTTP, and there will be no disruption to Googlebot’s regular crawling patterns for your site.

Despite no negative impact being shown in preliminary tests, Google is offering a way for sites to opt-out of this change if they wish.

You can opt out of Google crawling your site over HTTP/2 by instructing the server to respond with a 421 HTTP status code when Googlebot attempts to crawl your site over h2.

Google addresses other concerns about HTTP/2 in a set of answers to frequently asked questions.

FAQs: Googlebot & HTTP/2

Why is Googlebot being upgraded now?

The software that allows Googlebot to crawl over HTTP/2 has finally matured enough that it can be used in production.

Do site owners need to upgrade their servers to prepare for this change?

Google says it’s up to the individual site owner to decide.

Googlebot will only switch to crawling over HTTP/2 for sites that “support it and will clearly benefit from it.”

That means, if there’s no clear benefit to crawling over HTTP/2, Googlebot will continue to crawl over HTTP.

How can I tell if my site supports HTTP/2?

Google points to this blog post from Cloudflare which has different methods to test whether a site supports HTTP/2.

How do I upgrade my site to HTTP/2?

Google advises talking to your server administrator or hosting provider.

How can I force Googlebot to crawl over HTTP/2?

This is not possible, Google says.

As mentioned, a site will be crawled over HTTP/2 if the server supports it, and both the site and Googlebot will benefit from it.

If crawling over HTTP/2 would not result in noticeable resource savings, for example, then Googlebot will continue to crawl the site over HTTP/1.1.

What are the benefits of Googlebot crawling over HTTP/2?

The primary benefit for both the server and Googlebot is resource savings.

Crawling over HTTP/2 does not affect site indexing or crawl frequency.

Is there a ranking benefit for a site being crawled over HTTP/2?

No, there is absolutely no ranking benefit.