Google: Disallowed URLs Through Robots.txt Does Not Affect Crawl Budget

  •   November 14, 2019

Gary Illyes from Google added to the big crawl budget article he published on the webmaster blog in January 2017 a new question and answer. It basically says that if you disallow a URL in your robots.txt file, then those URLs do not affect your crawl budget.

He added this question and answer:

Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?
A: No, disallowed URLs do not affect the crawl budget.

He posted last night that he added the Q&A for a former Googler, Pierre Far:

Forum discussion at Twitter.

This marketing news is not the copyright of Scott.Services – please click here to see the original source of this article. Author: (Barry Schwartz)

For more SEO, PPC, internet marketing news please check out

Why not check out our SEO, PPC marketing services at

We’re also on: