Google: Disallowed URLs Through Robots.txt Does Not Affect Crawl Budget

  •   June 21, 2019

Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.

He added this question and answer:

Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.

He posted last night that he added the Q&A for a former Googler, Pierre Far:

Forum discussion at Twitter.

This marketing news is not the copyright of Scott.Services – please click here to see the original source of this article. Author: barry@rustybrick.com (Barry Schwartz)

For more SEO, PPC, internet marketing news please check out https://news.scott.services

Why not check out our SEO, PPC marketing services at https://www.scott.services

We’re also on:
https://www.facebook.com/scottdotservices/
https://twitter.com/scottdsmith
https://plus.google.com/112865305341039147737