Search Console’s Crawl Limit Changed By Google (Removed)
Choose Your Language:
The crawl rate tool in Google Search Console has officially been retired. Google had announced last November that this feature would be deprecated as of January 8, 2024. That day has now passed, and the tool is no longer available.
Google explained that advancements in its crawling processes and other available tools made the crawl rate limiter obsolete. If you attempt to access this tool in Search Console, you will now see a notice indicating the feature has been taken offline.
0.1 What Was the Crawl Rate Limiter?
The crawl rate limiter was part of the legacy version of Google Search Console. It allowed site owners to request that Googlebot reduce the frequency at which it crawled their sites. This tool was particularly useful in cases where heavy crawling caused excessive server strain.
Google, however, has long advised against restricting crawl rates unless there was clear evidence that Googlebot was directly causing server overload.
0.2 Why the Tool Was Removed
Google’s Gary Illyes shared additional context about this decision. He stated that updates to Google’s crawling technology made the tool redundant. Googlebot now adjusts crawling rates automatically based on how a site’s server responds.
For example, if Googlebot encounters consistent HTTP 500 errors or if server response times increase significantly, it slows down its crawling activity quickly and automatically. This eliminates the need for manual adjustments.
Illyes pointed out that the crawl rate tool in Search Console had slower results compared to these automated adjustments. It could take over a day for changes made using the tool to take effect. Additionally, the tool was rarely used, and when it was, many users set crawl speeds unnecessarily low.
0.3 What Has Changed?
Even though the crawl rate tool is gone, Google has adjusted its system to mimic the effects of the old settings. The minimum crawl speed is now set at a lower level, which should help reduce server load in cases of low search interest. If you previously set limits using this tool, Google’s current system should still accommodate your site’s needs without requiring further action.
0.4 What to Do If Issues Arise
If you experience problems with Googlebot’s crawling activity, you can refer to Google’s help documentation for guidance. There is also a report form you can use to notify Google of any issues.
0.5 Why This Matters
For those who relied on the crawl rate tool, it’s important to know that it’s no longer an option. Take some time to monitor your site’s crawl rate and server performance over the next few weeks to ensure everything continues to run smoothly.
While this change may seem significant, Google’s automated crawling adjustments are designed to address the same concerns that the tool once handled, likely rendering it unnecessary.
