{"id":24423,"date":"2026-01-08T07:27:04","date_gmt":"2026-01-08T07:27:04","guid":{"rendered":"https:\/\/kwebby.com\/blog\/?p=24423"},"modified":"2026-01-08T07:27:07","modified_gmt":"2026-01-08T07:27:07","slug":"how-to-fix-cloudflare-blocking-googlebot-firewall-waf-bots","status":"publish","type":"post","link":"https:\/\/kwebby.com\/blog\/how-to-fix-cloudflare-blocking-googlebot-firewall-waf-bots\/","title":{"rendered":"How to Fix Cloudflare Blocking Googlebot: Firewall, WAF, Bots"},"content":{"rendered":"\n<p>Google stops indexing pages when Cloudflare blocks Googlebot. Your rankings drop because Google cannot crawl, render, or fetch key assets. You can fix this fast if you identify the exact Cloudflare feature that triggers the block, then add a safe allow rule for verified <a href=\"https:\/\/kwebby.com\/blog\/seo-trends-in-2026\/\" data-type=\"post\" data-id=\"24337\">Googlebot<\/a>, and then confirm the fix in <a title=\"How to Clear Storage on Google Drive (8 Methods)\" href=\"https:\/\/kwebby.com\/blog\/how-to-clear-storage-on-google-drive\/\">logs and<\/a> Google Search Console.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Key Takeaways<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Find the exact block source first by using Cloudflare Security Events and matching the request to a rule ID.<\/li>\n\n\n\n<li>Verify real Googlebot before you allow it by using reverse DNS and forward DNS checks.<\/li>\n\n\n\n<li>Fix common causes by adjusting WAF Managed Rules, custom Firewall rules, and Bot Management settings.<\/li>\n\n\n\n<li>Disable or tune Bot Fight Mode if it blocks verified bots or creates false positives.<\/li>\n\n\n\n<li>Allow Googlebot with narrow rules (ASN, verified bot, URI scope) instead of wide IP whitelists.<\/li>\n\n\n\n<li>Validate the fix with Crawl Stats, URL Inspection, and Cloudflare logs to confirm 200 responses for Googlebot.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Understand why Cloudflare blocks Googlebot<\/h2>\n\n\n\n<p><a href=\"https:\/\/kwebby.com\/blog\/how-to-stop-targeted-scraping\/\" data-type=\"post\" data-id=\"23178\">Cloudflare<\/a> blocks Googlebot when a security control treats the request as suspicious. The most common triggers are WAF Managed Rules, Bot Management signals, rate limiting, and custom Firewall rules. You need to map the block to the product that fired the action. That step prevents guesswork and prevents you from weakening security.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Common block symptoms you will see<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Google Search Console shows <strong>Blocked due to access forbidden (403)<\/strong> or <strong>Blocked due to other 4xx issue<\/strong>.<\/li>\n\n\n\n<li>URL Inspection shows <strong>Crawl allowed? Yes<\/strong> but <strong>Page fetch failed<\/strong>.<\/li>\n\n\n\n<li>Cloudflare returns <strong>1020 Access Denied<\/strong> (often a Firewall rule) or <strong>403<\/strong> (often WAF\/Bot).<\/li>\n\n\n\n<li>Google cannot fetch CSS\/JS, so rendering fails and indexing quality drops.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Does Cloudflare block bots?<\/h3>\n\n\n\n<p>Yes. <a href=\"https:\/\/kwebby.com\/blog\/how-to-use-cdn-for-images-for-seo\/\" data-type=\"post\" data-id=\"23163\">Cloudflare blocks bots<\/a> by design when you enable WAF rules, Bot Management, Bot Fight Mode, Super Bot Fight Mode, rate limiting, or strict custom Firewall rules. Cloudflare also allows many verified bots automatically, but misconfigurations and aggressive rules can still block Googlebot.<\/p>\n\n\n\n<p>Next, you need proof of what blocked the request.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1: Confirm the block and identify the exact rule<\/h2>\n\n\n\n<p>You fix Cloudflare blocking only after you identify the blocking control. Use Cloudflare Security Events first. Then confirm with logs if you have them.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Use Cloudflare Security Events to find the rule<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open your Cloudflare dashboard.<\/li>\n\n\n\n<li>Select your site.<\/li>\n\n\n\n<li>Go to <strong>Security<\/strong> \u2192 <strong>Overview<\/strong> \u2192 <strong>Bot traffic<\/strong> \u2192 <strong>View bot events<\/strong><\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"478\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-1024x478.png\" alt=\"\" class=\"wp-image-24426\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-1024x478.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-300x140.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-768x358.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-1536x717.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.41.12@2x-1-2048x955.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Filter by <strong>Action: Block<\/strong> (and also check <strong>Managed Challenge<\/strong> if Googlebot gets challenged).<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"609\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x-1024x609.png\" alt=\"\" class=\"wp-image-24427\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x-1024x609.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x-300x178.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x-768x457.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x-1536x913.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.42.39@2x.png 2012w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Filter by <strong>User Agent<\/strong> contains <strong>Googlebot<\/strong>.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"609\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x-1024x609.png\" alt=\"\" class=\"wp-image-24428\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x-1024x609.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x-300x178.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x-768x457.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x-1536x914.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.43.49@2x.png 2014w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Click an event and note:\n<ul class=\"wp-block-list\">\n<li><strong>Service<\/strong> (WAF, Firewall rules, Bot, Rate limiting)<\/li>\n\n\n\n<li><strong>Rule ID \/ Rule name<\/strong><\/li>\n\n\n\n<li><strong>Host<\/strong>, <strong>URI<\/strong>, <strong>Country<\/strong>, <strong>ASN<\/strong><\/li>\n\n\n\n<li><strong>Response code<\/strong> (403, 1020)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p><strong>Goal:<\/strong> You want one clear answer like \u201cWAF Managed Rules: Cloudflare Specials\u201d or \u201cFirewall rule: Block bad bots\u201d or \u201cBot Fight Mode.\u201d<\/p>\n\n\n\n<p>Alternatively Go to <strong>AI Crawl Control <\/strong>and search for Googlebot from the list and click on three dots icon on last column and click on <strong>View metrics<\/strong>;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"464\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-1024x464.png\" alt=\"\" class=\"wp-image-24429\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-1024x464.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-300x136.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-768x348.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-1536x696.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.45.33@2x-2048x928.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Here you will find allowed requests and blocked requests;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"464\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-1024x464.png\" alt=\"\" class=\"wp-image-24430\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-1024x464.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-300x136.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-768x348.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-1536x696.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.47.12@2x-2048x928.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Check Google Search Console for the exact URL and time<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open <strong>Google Search Console<\/strong> \u2192 <strong>Pages<\/strong> report.<\/li>\n\n\n\n<li>Open a blocked example URL.<\/li>\n\n\n\n<li>Check the last crawl time and the crawl result.<\/li>\n\n\n\n<li>Run <strong>URL Inspection<\/strong> \u2192 <strong>Test Live URL<\/strong>.<\/li>\n<\/ul>\n\n\n\n<p>Match the timestamp with Cloudflare events. That match helps you avoid chasing unrelated blocks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Use Cloudflare logs if available (best proof)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you use <strong>Logpush<\/strong>, query for the blocked request by URI and user agent.<\/li>\n\n\n\n<li>Look for fields like <strong>ClientRequestUserAgent<\/strong>, <strong>EdgeResponseStatus<\/strong>, <strong>WAFAction<\/strong>, <strong>FirewallMatchesActions<\/strong>, and <strong>FirewallMatchesRuleIDs<\/strong>.<\/li>\n<\/ul>\n\n\n\n<p>Now you know what caused the block. Next, confirm the crawler is real Googlebot.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2: Verify that the request is real Googlebot (avoid spoofing)<\/h2>\n\n\n\n<p>Attackers often spoof the Googlebot user agent. If you allow by user agent alone, you can open your site to scraping and attacks. Google provides a verification method based on DNS. Use it before you whitelist a Google bot.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to verify Googlebot with reverse DNS and forward DNS<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Get the IP address from Cloudflare Security Events or logs.<\/li>\n\n\n\n<li>Run a reverse DNS lookup (PTR) for the IP.<\/li>\n\n\n\n<li>Confirm the hostname ends with <strong>.googlebot.com<\/strong> or <strong>.google.com<\/strong>.<\/li>\n\n\n\n<li>Run a forward DNS lookup (A\/AAAA) for that hostname.<\/li>\n\n\n\n<li>Confirm the forward lookup returns the same IP address.<\/li>\n<\/ul>\n\n\n\n<p><strong>Example commands:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>host 66.249.66.1<\/code> (reverse lookup)<\/li>\n\n\n\n<li><code>host crawl-66-249-66-1.googlebot.com<\/code> (forward lookup)<\/li>\n<\/ul>\n\n\n\n<p>If the DNS checks fail, do not allow the request as Googlebot. Fix the block only for verified Googlebot.<\/p>\n\n\n\n<p>Next, apply the right fix based on the product that blocked the request.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fix 1: Firewall rules that block Googlebot (custom rules and IP rules)<\/h2>\n\n\n<div class=\"wp-block-image wp-block-image aligncenter\">\n<figure ><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"896\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-1-1.png\" alt=\"Cloudflare Security Events: blocked Googlebot request, rule ID, WAF\/Bot\/Firewall action, 403\u2014How to Fix Googlebot Blocking in\" class=\"wp-image-24433\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-1-1.png 1200w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-1-1-300x224.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-1-1-1024x765.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-1-1-768x573.png 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n<\/div>\n\n\n<p>Custom Firewall rules often cause 1020 errors. These rules can block by country, ASN, path, threat score, user agent, or a bot score. You need to find the rule and then add an allow exception above it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to fix Cloudflare blocking from a Firewall rule<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to <strong>Security<\/strong> \u2192 <strong>Security rules<\/strong> (or <strong>Security<\/strong> \u2192 <strong>Firewall rules<\/strong> depending on your plan\/UI).<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"464\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-1024x464.png\" alt=\"screenshot of Go to Security \u2192 Security rules\" class=\"wp-image-24431\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-1024x464.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-300x136.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-768x348.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-1536x696.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/CleanShot-2026-01-08-at-12.50.56@2x-2048x928.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Find the rule ID from Security Events.<\/li>\n\n\n\n<li>Open the rule and review the expression.<\/li>\n\n\n\n<li>Check for conditions like:\n<ul class=\"wp-block-list\">\n<li><code>(http.user_agent contains \"bot\")<\/code><\/li>\n\n\n\n<li><code>(cf.threat_score > 10)<\/code><\/li>\n\n\n\n<li><code>(ip.geoip.country in {\"CN\" \"RU\"})<\/code> with broad blocks<\/li>\n\n\n\n<li><code>(cf.client.bot)<\/code><br><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Add a safe allow rule for verified bots<\/h3>\n\n\n\n<p>Use Cloudflare\u2019s verified bot signals when available. Prefer a rule that allows verified Googlebot traffic and keeps other bots filtered.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create a new rule at the top with action <strong>Allow<\/strong>.<\/li>\n\n\n\n<li>Use one of these strategies (use what your account supports):\n<ul class=\"wp-block-list\">\n<li><strong>Allow Verified Bots<\/strong> if Cloudflare exposes a verified bot field in your plan\/UI.<\/li>\n\n\n\n<li><strong>Allow by ASN 15169<\/strong> (Google) plus a narrow URI scope.<\/li>\n\n\n\n<li><strong>Allow by IP list<\/strong> only if you maintain it and you also verify DNS (least preferred).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p><strong>Practical allow rule patterns (choose one):<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Option A (best if available):<\/strong> Allow if request is from a verified bot category and matches Google.<\/li>\n\n\n\n<li><strong>Option B (often available):<\/strong> Allow if <code>ip.geoip.asnum eq 15169<\/code> and <code>http.request.uri.path<\/code> matches crawl paths you need.<\/li>\n<\/ul>\n\n\n\n<p><strong>Important:<\/strong> Do not allow by user agent alone. Do not disable your blocking rule if it protects sensitive endpoints.<\/p>\n\n\n\n<p>After you fix custom rules, check WAF Managed Rules. Many sites get blocked there.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fix 2: WAF Managed Rules blocking Googlebot (Cloudflare Specials and OWASP)<\/h2>\n\n\n\n<p><a href=\"https:\/\/developers.cloudflare.com\/waf\/\" data-type=\"link\" data-id=\"https:\/\/developers.cloudflare.com\/waf\/\" target=\"_blank\">WAF Managed Rules<\/a> can block Googlebot if the crawler hits URLs that look like attacks. This happens often on search pages, filter pages, and URLs with many query parameters. You should tune the exact rule that fired. You should not disable the full WAF unless you have no other choice.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Find the Managed Rule that triggered the block<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to <strong>Security<\/strong> \u2192 <strong>Security rules<\/strong>.<\/li>\n\n\n\n<li>Use the rule ID from Security Events to locate the exact rule set and rule.<\/li>\n\n\n\n<li>Check the action: <strong>Block<\/strong>, <strong>Challenge<\/strong>, or <strong>Log<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Fix \u201cCloudflare Managed Special rules are blocking Googlebot\u201d<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open the <strong>Cloudflare Managed Rules<\/strong> section.<\/li>\n\n\n\n<li>Locate <strong>Cloudflare Specials<\/strong> (name can vary by UI version).<\/li>\n\n\n\n<li>Do one of the following:\n<ul class=\"wp-block-list\">\n<li>Set the specific rule to <strong>Log<\/strong> for Googlebot requests.<\/li>\n\n\n\n<li>Create an exception that skips the rule for verified Googlebot.<\/li>\n\n\n\n<li>Disable only the single rule that causes false positives on safe URLs.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Create a WAF exception for Googlebot on safe paths<\/h3>\n\n\n\n<p>Use exceptions to keep protection for users while letting Google crawl.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Scope the exception to:\n<ul class=\"wp-block-list\">\n<li>Verified Googlebot traffic (preferred)<\/li>\n\n\n\n<li>Or Google ASN 15169 plus DNS verification process<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Scope the exception to:\n<ul class=\"wp-block-list\">\n<li>Public pages only (for example: <code>\/blog\/<\/code>, <code>\/products\/<\/code>)<\/li>\n\n\n\n<li>Exclude admin paths (for example: <code>\/wp-admin<\/code>, <code>\/login<\/code>)<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Keep a log-only period first if you are unsure.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Fix OWASP false positives without weakening security<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If <a href=\"https:\/\/developers.cloudflare.com\/waf\/managed-rules\/reference\/owasp-core-ruleset\/\" data-type=\"link\" data-id=\"https:\/\/developers.cloudflare.com\/waf\/managed-rules\/reference\/owasp-core-ruleset\/\" target=\"_blank\">OWASP<\/a> rules block URLs with many parameters, reduce the sensitivity for that rule group.<\/li>\n\n\n\n<li>If a specific parameter triggers SQLi or XSS rules, add a targeted exception for that parameter on that endpoint.<\/li>\n\n\n\n<li>Keep blocking enabled for login, checkout, and API endpoints.<\/li>\n<\/ul>\n\n\n\n<p>Next, check Bot Management. It can block Googlebot even if WAF looks fine.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fix 3: Bot Management and Bot Fight Mode blocking Googlebot<\/h2>\n\n\n\n<p>Bot controls can block or challenge crawlers based on behavior signals. If your configuration treats automated traffic as hostile, Googlebot can get blocked. You should allow verified bots and reduce aggressive actions on crawl traffic.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to disable Bot Fight Mode in Cloudflare<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open Cloudflare dashboard and select your site.<\/li>\n\n\n\n<li>Go to <strong>Security<\/strong> \u2192 <strong>Bots<\/strong> (menu name can vary).<\/li>\n\n\n\n<li>Find <strong>Bot Fight Mode<\/strong> or <strong>Super Bot Fight Mode<\/strong>.<\/li>\n\n\n\n<li>Toggle it <strong>Off<\/strong> or switch to a less aggressive mode.<\/li>\n\n\n\n<li>Save changes and retest in Search Console.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Safer alternative: keep Bot Fight Mode, but allow verified bots<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable settings that allow <strong>Verified Bots<\/strong> if available.<\/li>\n\n\n\n<li>Create a custom rule that <strong>skips bot checks<\/strong> for verified Googlebot on public paths.<\/li>\n\n\n\n<li>Keep bot protections on high-risk paths like <code>\/wp-login.php<\/code>, <code>\/cart<\/code>, and <code>\/api<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Fix Bot Management blocks caused by score thresholds<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Review actions tied to bot score or bot categories.<\/li>\n\n\n\n<li>Change action from <strong>Block<\/strong> to <strong>Managed Challenge<\/strong> for general traffic if you see false positives.<\/li>\n\n\n\n<li>Add an allow exception for verified Googlebot so it never gets challenged.<\/li>\n<\/ul>\n\n\n\n<p>After Bot settings, review rate limiting. Rate limits can block crawlers on large sites.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fix 4: Rate limiting and crawl bursts that trigger blocks<\/h2>\n\n\n\n<p>Googlebot can crawl fast after site changes. A strict rate limit can block it. You should tune rate limits to protect login and API endpoints while allowing crawl traffic on public pages.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to check if rate limiting blocks Googlebot<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In Security Events, look for a service label that points to <strong>Rate limiting<\/strong>.<\/li>\n\n\n\n<li>Check if blocked requests cluster in a short time window.<\/li>\n\n\n\n<li>Check if the URI is a public page list or a sitemap fetch.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to fix rate limiting without removing protection<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Increase thresholds for public GET requests.<\/li>\n\n\n\n<li>Keep strict thresholds for POST requests and authentication endpoints.<\/li>\n\n\n\n<li>Add an allow rule for verified Googlebot on:\n<ul class=\"wp-block-list\">\n<li><code>\/sitemap.xml<\/code><\/li>\n\n\n\n<li><code>\/robots.txt<\/code><\/li>\n\n\n\n<li>Key category and product pages<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>Next, confirm that SSL\/TLS and firewall settings do not create crawl failures.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fix 5: SSL\/TLS, redirects, and edge settings that look like blocking<\/h2>\n\n\n\n<p>Some crawl failures look like blocking but come from redirect loops, TLS errors, or blocked resources. Googlebot needs consistent 200 responses and stable redirects.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Check for redirect loops and mixed schemes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Test the URL with and without <code>www<\/code>.<\/li>\n\n\n\n<li>Test <code>http<\/code> and <code>https<\/code> versions.<\/li>\n\n\n\n<li>Confirm you have a single canonical redirect path.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Check Cloudflare SSL mode<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to <strong>SSL\/TLS<\/strong> \u2192 <strong>Overview<\/strong>.<\/li>\n\n\n\n<li>Use <strong>Full (strict)<\/strong> if your origin has a valid certificate.<\/li>\n\n\n\n<li>Avoid <strong>Flexible<\/strong> if it causes redirect loops at the origin.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Check robots.txt and origin firewall rules<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Confirm <code>\/robots.txt<\/code> returns 200 and does not disallow important paths.<\/li>\n\n\n\n<li>Confirm your origin firewall (server, CDN, load balancer) does not block Google IPs.<\/li>\n\n\n\n<li>Confirm your hosting WAF does not block Googlebot while Cloudflare allows it.<\/li>\n<\/ul>\n\n\n\n<p>Now you can whitelist Googlebot safely with a clean rule design.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to whitelist a Google bot safely (recommended rule design)<\/h2>\n\n\n\n<p>A safe whitelist uses verification signals and narrow scope. A risky whitelist uses only a user agent match or a broad IP allow. Use the safe approach so you keep protection for real attacks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Rule design checklist (use this before you click \u201cAllow\u201d)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use verified bot detection if your plan supports it.<\/li>\n\n\n\n<li>Use DNS verification for any suspicious case.<\/li>\n\n\n\n<li>Scope the allow to public content paths only.<\/li>\n\n\n\n<li>Do not allow access to admin, login, or private API endpoints.<\/li>\n\n\n\n<li>Place the allow rule above block rules that catch bots.<\/li>\n\n\n\n<li>Log the allowed traffic for a week and review anomalies.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example allow scopes that work well<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Allow Googlebot to crawl content:<\/strong> public GET requests to <code>\/<\/code>, <code>\/blog\/<\/code>, <code>\/category\/<\/code>, <code>\/product\/<\/code>.<\/li>\n\n\n\n<li><strong>Allow Googlebot to fetch assets:<\/strong> <code>\/wp-content\/<\/code>, <code>\/static\/<\/code>, <code>\/assets\/<\/code>.<\/li>\n\n\n\n<li><strong>Always allow:<\/strong> <code>\/robots.txt<\/code> and <code>\/sitemap.xml<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What to avoid when you whitelist Googlebot<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Do not create a rule like <code>http.user_agent contains \"Googlebot\"<\/code> with action Allow.<\/li>\n\n\n\n<li>Do not disable the full WAF rule set to solve one false positive.<\/li>\n\n\n\n<li>Do not allow all of ASN 15169 to all endpoints if you run sensitive APIs.<\/li>\n<\/ul>\n\n\n\n<p>Next, validate the fix with a repeatable test plan.<\/p>\n\n\n<div class=\"wp-block-image wp-block-image aligncenter\">\n<figure ><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"896\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-2-1.png\" alt=\"How to Fix Googlebot Blocking in Cloudflare (Firewall, Bot Management, WAF Rules) flowchart: detect, verify DNS, adjust, vali\" class=\"wp-image-24432\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-2-1.png 1200w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-2-1-300x224.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-2-1-1024x765.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2026\/01\/inline-cf3c28d7-2-1-768x573.png 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\">Validate the fix (prove Googlebot can crawl and index)<\/h2>\n\n\n\n<p>You need confirmation from both Cloudflare and Google. Use a short test loop. Then watch crawl data for several days.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Validation checklist in Cloudflare<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Open <strong>Security Events<\/strong> and confirm no new blocks for Googlebot user agents.<\/li>\n\n\n\n<li>Confirm the action for Googlebot requests is <strong>Allow<\/strong> or <strong>Skip<\/strong> for the correct feature.<\/li>\n\n\n\n<li>Confirm response status is <strong>200<\/strong> for key URLs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Validation checklist in Google Search Console<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run <strong>URL Inspection<\/strong> \u2192 <strong>Test Live URL<\/strong> on a previously blocked page.<\/li>\n\n\n\n<li>Click <strong>View tested page<\/strong> and confirm resources load.<\/li>\n\n\n\n<li>Request indexing for a small set of important pages.<\/li>\n\n\n\n<li>Monitor <strong>Crawl stats<\/strong> for drops in 4xx responses.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Validation checklist on your origin<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Check origin access logs for Googlebot requests that reach the server.<\/li>\n\n\n\n<li>Confirm your origin security tools do not block the same IPs.<\/li>\n<\/ul>\n\n\n\n<p>Next, prevent the issue from returning with a simple monitoring routine.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prevent future Googlebot blocks (monitoring and safe defaults)<\/h2>\n\n\n\n<p>Most repeat incidents happen after a rule change. You can prevent them by keeping a small set of guardrails and by logging bot-related actions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Set safe defaults for WAF and bot controls<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keep WAF enabled, but tune false positives with exceptions.<\/li>\n\n\n\n<li>Keep bot protection enabled for login and API endpoints.<\/li>\n\n\n\n<li>Use <strong>Log<\/strong> mode first for new managed rules if your site has many query URLs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Change control checklist for Cloudflare security edits<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Record the rule change and the reason.<\/li>\n\n\n\n<li>Test with a staging hostname if you have one.<\/li>\n\n\n\n<li>After deployment, check Security Events for Googlebot within 30 minutes.<\/li>\n\n\n\n<li>Re-test 5 to 10 key URLs in Search Console within 24 hours.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Keep a \u201ccrawl allow\u201d policy that stays narrow<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Allow verified Googlebot on public content paths.<\/li>\n\n\n\n<li>Keep challenges for unknown bots.<\/li>\n\n\n\n<li>Keep strict rules for admin and payment paths.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently Asked Questions (FAQs)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How to fix Cloudflare blocking?<\/h3>\n\n\n\n<p>Open Cloudflare Security Events, find the blocked request, note the rule ID and service, then tune or add an allow exception for verified Googlebot. Retest <a href=\"https:\/\/kwebby.com\/blog\/does-facebook-marketplace-ad-show-in-google-search-engine\/\" title=\"Does Facebook Marketplace Ad Show in Google Search Engine?\">in Google<\/a> Search Console.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I whitelist a Google bot?<\/h3>\n\n\n\n<p>Verify the crawler with reverse DNS and forward DNS, then create an allow rule based on verified bot signals or Google ASN with a narrow path scope. Do not allow by user agent alone.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Does Cloudflare block bots?<\/h3>\n\n\n\n<p>Yes. WAF rules, Bot Management, Bot Fight Mode, and rate limits can block or challenge bots. Bad configuration can also block good bots like Googlebot.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to disable bot fight mode in Cloudflare?<\/h3>\n\n\n\n<p>Go to your site in Cloudflare, open Security \u2192 Bots, find Bot Fight Mode (or Super Bot Fight Mode), toggle it off, save, and then retest crawl in Search Console.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why does Google Search Console show 403 but I can load the page?<\/h3>\n\n\n\n<p>Cloudflare can block specific user agents or IP ranges while allowing browsers. Googlebot can also fetch assets and parameters that you do not test in a normal browser session.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should I turn off Cloudflare WAF to let Googlebot crawl?<\/h3>\n\n\n\n<p>No. Turn off only the specific rule that causes false positives, or add a targeted exception for verified Googlebot on public paths. Keep WAF protection for sensitive endpoints.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Final Thoughts<\/h2>\n\n\n\n<p><a href=\"https:\/\/kwebby.com\/blog\/spot-delete-malware-wordpress\/\" title=\"Is Your WordPress Site Under Attack? How to Spot &amp; Delete Malware Fast!\">How to<\/a> Fix Googlebot Blocking in Cloudflare (Firewall, Bot Management, WAF Rules) comes down to one workflow: identify the exact blocking feature in Security Events, verify real Googlebot with DNS checks, apply a narrow allow or exception rule, and then confirm the result in Search Console and logs. If you want a fast win, start by locating the rule ID that fired, then tune Managed Rules or Bot settings instead of disabling security. If you need help, export a blocked event from Cloudflare and share the rule ID with your team so you can fix the exact control without weakening your site.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Googlebot can lose access when Cloudflare blocks it with Firewall rules, WAF Managed Rules, or Bot Management. Learn how to identify the rule, verify Googlebot, whitelist safely, and validate in GSC.<\/p>\n","protected":false},"author":1,"featured_media":24421,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[277,700,3,423],"tags":[],"class_list":["post-24423","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-advanced-seo-techniques","category-security","category-seo","category-webmaster-tutorials"],"_links":{"self":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts\/24423","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/comments?post=24423"}],"version-history":[{"count":1,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts\/24423\/revisions"}],"predecessor-version":[{"id":24434,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts\/24423\/revisions\/24434"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/media\/24421"}],"wp:attachment":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/media?parent=24423"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/categories?post=24423"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/tags?post=24423"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}