{"id":23178,"date":"2024-08-28T07:51:29","date_gmt":"2024-08-28T07:51:29","guid":{"rendered":"https:\/\/kwebby.com\/blog\/?p=23178"},"modified":"2024-08-28T07:51:34","modified_gmt":"2024-08-28T07:51:34","slug":"how-to-stop-targeted-scraping","status":"publish","type":"post","link":"https:\/\/kwebby.com\/blog\/how-to-stop-targeted-scraping\/","title":{"rendered":"How to stop targeted scraping to improve performance? Google Answers"},"content":{"rendered":"\n<p>Automated scraping is a major performance killer as we saw during the Google Search Console <a href=\"https:\/\/www.youtube.com\/watch?v=Fj42gKDQxYI\" data-type=\"link\" data-id=\"https:\/\/www.youtube.com\/watch?v=Fj42gKDQxYI\" target=\"_blank\">SEO office hours<\/a> in August 2024.<\/p>\n\n\n\n<p>A site that\u2019s being scraped is hit with severe consequences including <a href=\"https:\/\/kwebby.com\/blog\/how-server-response-time-affects-googles-crawling\/\" data-type=\"post\" data-id=\"23074\">increased server load<\/a> and decreased responsiveness. When someone asked about persistent scraping issues even after implementing common prevention methods like IP blocking the answer was to provide the steps to mitigate.<\/p>\n\n\n\n<p>You need to consider both technical and external support to manage and reduce the impact of aggressive scraping.<\/p>\n\n\n\n<p>Here are the steps to deal with this, including finding the source of unwanted traffic, using traceroute tools and a reliable <a href=\"https:\/\/kwebby.com\/blog\/how-to-use-cdn-for-images-for-seo\/\" data-type=\"post\" data-id=\"23163\">Content Delivery Network (CDN)<\/a> to make your site more resilient to malicious bot traffic.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Google Answers: How to stop targeted scraping<\/h2>\n\n\n\n<iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/Fj42gKDQxYI?si=srr-CEHXmnp23lCx&amp;start=401\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n\n\n\n<p>During the Google Search Console office hours a question was asked about the ongoing scraping issue: <\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em><em>&#8220;Our website is experiencing significant disruptions due to targeted scraping by automated software leading to performance issues, increased server load, and despite IP blocking and other preventive measures, the problem persists. What can we do?&#8221;<\/em><\/em><\/p>\n<\/blockquote>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"493\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/English-Google-SEO-office-hours-from-August-2024-YouTube-5-1024x493.png\" alt=\"\" class=\"wp-image-23186\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/English-Google-SEO-office-hours-from-August-2024-YouTube-5-1024x493.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/English-Google-SEO-office-hours-from-August-2024-YouTube-5-300x144.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/English-Google-SEO-office-hours-from-August-2024-YouTube-5-768x370.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/English-Google-SEO-office-hours-from-August-2024-YouTube-5.png 1118w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>The answer was to provide a structured way to fight back against aggressive scraping. Here are the steps to reduce the impact:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Find the Source of Traffic:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>First, find the source of the traffic. As the Google&#8217;s Mueller says;<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>&#8220;This sounds like a distributed denial of service if the crawling is so aggressive that it causes performance degradation.&#8221;<\/em><\/p>\n<\/blockquote>\n\n\n\n<p>Use WHOIS to find the hosting provider of the suspicious traffic and send an abuse report.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Use Traceroute Tools:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>If you can\u2019t find the network owner, use <a href=\"https:\/\/tools.keycdn.com\/traceroute\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/tools.keycdn.com\/traceroute\" rel=\"noreferrer noopener\">traceroute tools<\/a>. This will allow you to trace the path of the packets to your server and see the route the scraper\u2019s requests are taking.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Use a Content Delivery Network (CDN):<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Another way to intervene is to use a CDN. The experts said;<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em><em>&#8220;CDNs often have features to detect this kind of bot traffic and block it.&#8221;<\/em><\/em><\/p>\n<\/blockquote>\n\n\n\n<p>By using a CDN you not only reduce server load (since it distributes traffic away from your main servers) but you also make your site more resilient.<\/p>\n\n\n\n<p>You can reach out to the CDN\u2019s customer support to make sure it won\u2019t block legitimate <a href=\"https:\/\/kwebby.com\/blog\/privacy-focused-search-engines\/\" data-wpil-monitor-id=\"1499\">search engine<\/a> bots as they are usually recognized and allowed.<\/p>\n\n\n\n<p>By following these steps suggested by Google&#8217;s Mueller, you can reduce the impact of targeted scraping and have a more stable and responsive site.<\/p>\n\n\n\n<p>Other Articles from Podcast;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/kwebby.com\/blog\/how-to-resolve-wrong-currency-shown-on-googles-rich-snippets-google-answers\/\">How to Resolve Wrong Currency Shown on Google\u2019s Rich Snippets? Google Answers<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kwebby.com\/blog\/remove-expired-domains-url-from-google-search-console\/\">How to remove Expired Domains URL From Google Search Console?<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kwebby.com\/blog\/how-to-use-cdn-for-images-for-seo\/\">Google Suggests How to Use CDN for Images for SEO<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kwebby.com\/blog\/how-to-handle-404s-for-seo\/\">How to handle 404s for SEO? Google Warns &amp; Solution<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/kwebby.com\/blog\/how-to-handle-multilingual-websites-in-seo-google-answers\/\">How to handle multilingual websites in SEO? Google Answers<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">How to Block Targeted Scraping on Your Site<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">#1 Block All Bad Bot Crawlers Using robots.txt (List)<\/h3>\n\n\n\n<p>To prevent scraping on your site one way is to use <a href=\"https:\/\/kwebby.com\/blog\/robotstxt-for-seo\/\" data-type=\"post\" data-id=\"12922\">robots.txt<\/a> to block known bad bot crawlers. This file tells compliant web crawlers which part of your site they can access.<\/p>\n\n\n\n<p>By blocking bad bots, you can <a href=\"https:\/\/kwebby.com\/blog\/google-core-web-vitals\/\" data-type=\"post\" data-id=\"14090\">improve your site\u2019s performance<\/a> and reduce disruption caused by malicious automated traffic.<\/p>\n\n\n\n<p>Here\u2019s how to do this step by step:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Go to Your Site\u2019s Root Directory:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Use your FTP client or web hosting file manager to go to your site\u2019s root directory where your existing robots.txt file is located.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Create or Open robots.txt:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>If you <a href=\"https:\/\/kwebby.com\/blog\/where-to-place-robots-txt-google-suggests-this\/\" data-type=\"post\" data-id=\"22090\">don\u2019t have a robots.txt<\/a> file, create one. If you already have one open it.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Add Bad Bot Blocks:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Go to <a href=\"https:\/\/github.com\/mitchellkrogza\/apache-ultimate-bad-bot-blocker\/blob\/master\/robots.txt\/robots.txt\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/github.com\/mitchellkrogza\/apache-ultimate-bad-bot-blocker\/blob\/master\/robots.txt\/robots.txt\" rel=\"noreferrer noopener\">Apache Ultimate Bad Bot Blocker.<\/a> There you\u2019ll find a list of user-agent strings for known bad bots. Copy the lines that contains these entries.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"521\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/apache-ultimate-bad-bot-blockerrobots.txtrobots.txt-at-master-\u00b7-mitchellkrogzaapache-ultimate-bad-bot-blocker-1024x521.png\" alt=\"\" class=\"wp-image-23179\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/apache-ultimate-bad-bot-blockerrobots.txtrobots.txt-at-master-\u00b7-mitchellkrogzaapache-ultimate-bad-bot-blocker-1024x521.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/apache-ultimate-bad-bot-blockerrobots.txtrobots.txt-at-master-\u00b7-mitchellkrogzaapache-ultimate-bad-bot-blocker-300x153.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/apache-ultimate-bad-bot-blockerrobots.txtrobots.txt-at-master-\u00b7-mitchellkrogzaapache-ultimate-bad-bot-blocker-768x391.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/apache-ultimate-bad-bot-blockerrobots.txtrobots.txt-at-master-\u00b7-mitchellkrogzaapache-ultimate-bad-bot-blocker.png 1463w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Paste the Entries into robots.txt:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>In your robots.txt paste the copied entries. Your file should look like this:<\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:.875rem;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:1.25rem;--cbp-tab-width:2;tab-size:var(--cbp-tab-width, 2)\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"\nUser-agent: BadBot\n\nDisallow: \/\n\nUser-agent: AnotherBadBot\n\nDisallow: \/\n\n\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9\">User<\/span><span style=\"color: #81A1C1\">-<\/span><span style=\"color: #D8DEE9FF\">agent<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">BadBot<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">Disallow<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">\/<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9\">User<\/span><span style=\"color: #81A1C1\">-<\/span><span style=\"color: #D8DEE9FF\">agent<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #D8DEE9\">AnotherBadBot<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">Disallow<\/span><span style=\"color: #ECEFF4\">:<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #81A1C1\">\/<\/span><\/span>\n<span class=\"line\"><\/span>\n<span class=\"line\"><\/span><\/code><\/pre><\/div>\n\n\n\n<p>Replace &#8220;BadBot&#8221; and &#8220;AnotherBadBot&#8221; with the actual bot names you want to block.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Save and Upload robots.txt:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Save the file and upload it back to your site\u2019s root directory. If prompted overwrite the existing file.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Test robots.txt:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Use <a href=\"https:\/\/search.google.com\/search-console\/settings\/robots-txt\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/search.google.com\/search-console\/settings\/robots-txt\" rel=\"noreferrer noopener\">Google\u2019s Robots Testing Tool <\/a>to check if your robots.txt is correct and the crawlers you blocked are indeed blocked. This is important to confirm your settings work.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"533\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/robots.txt-1024x533.png\" alt=\"\" class=\"wp-image-23180\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/robots.txt-1024x533.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/robots.txt-300x156.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/robots.txt-768x400.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/robots.txt.png 1535w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>By following these steps you can block unwanted crawlers from your site and improve your site\u2019s security and performance.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">#2 Cloudflare Super Bot Fight Mode<\/h3>\n\n\n\n<p>To add an extra layer of protection against scraping and malicious bot activities use <a href=\"https:\/\/developers.cloudflare.com\/bots\/get-started\/biz-and-ent\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/developers.cloudflare.com\/bots\/get-started\/biz-and-ent\" rel=\"noreferrer noopener\">Cloudflare\u2019s Super Bot Fight Mode<\/a>.<\/p>\n\n\n\n<p>This feature will automatically block bot traffic, allow legitimate search engine bots and block aggressive and harmful crawlers. Follow these steps to configure Super Bot Fight Mode:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Log in to Your Cloudflare Account:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Go to your Cloudflare dashboard with your credentials.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Select Your Domain:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Select the site you want to protect from the list of domains in your account.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Go to Security:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>In the dashboard go to the \u2018Security\u2019 on the left menu and select it.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Go to Bots:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>In the Security settings, go to the \u2018Bots\u2019 Option and select &#8220;Configure Super Bot Fight mode&#8221;. This is where you can manage bot traffic.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1024x518.png\" alt=\"\" class=\"wp-image-23181\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1024x518.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-300x152.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-768x388.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1536x777.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare.png 1663w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Allow Verified Bots:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Under Super Bot Fight Mode settings make sure to allow bots from Google, Bing and other legitimate search engines. This is important so your site\u2019s SEO isn\u2019t affected by blocking useful crawlers.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1-1024x518.png\" alt=\"\" class=\"wp-image-23182\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1-1024x518.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1-300x152.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1-768x389.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1-1536x778.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-1.png 1663w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Optimize for WordPress (if applicable):<\/strong><\/li>\n<\/ul>\n\n\n\n<p>If your <a href=\"https:\/\/kwebby.com\/blog\/wordpress-site-visible-on-google-search\/\" data-wpil-monitor-id=\"1500\">site is on WordPress<\/a> make sure to enable \u201cOptimize for WordPress\u201d. This optimizes <a href=\"https:\/\/kwebby.com\/blog\/properly-size-images\/\" data-wpil-monitor-id=\"1501\">performance and works with common WordPress plugins<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2-1024x518.png\" alt=\"\" class=\"wp-image-23183\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2-1024x518.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2-300x152.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2-768x389.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2-1536x778.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-2.png 1663w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Enable JavaScript Detection:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Turn on JavaScript detection if it\u2019s not already on. This will allow Cloudflare to detect if a visitor is a human or a bot and block bots effectively.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3-1024x518.png\" alt=\"\" class=\"wp-image-23184\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3-1024x518.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3-300x152.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3-768x389.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3-1536x778.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-3.png 1663w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Block AI Scrapers and Crawlers:<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Finally turn on the setting to block AI scrapers and crawlers. This is an extra layer of protection to block sophisticated scraping technologies that can bypass standard protections.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"518\" src=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4-1024x518.png\" alt=\"\" class=\"wp-image-23185\" title=\"\" srcset=\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4-1024x518.png 1024w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4-300x152.png 300w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4-768x389.png 768w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4-1536x778.png 1536w, https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2024\/08\/Security-Bots-kwebby.com-Raman@kwebby.coms-Account-Cloudflare-4.png 1663w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>By following these steps you can use Cloudflare\u2019s Super Bot Fight Mode to secure your site, reduce the risks of scraping and keep your legitimate users happy.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">All done!<\/h2>\n\n\n\n<p>In conclusion, protecting your site from targeted scraping requires a multi layered approach. By configuring your robots.txt to block known bad bots and using Cloudflare\u2019s Super Bot Fight Mode you can secure your site and keep it performing.<\/p>\n\n\n\n<p>This will block unwanted automated traffic and allow legitimate search engine bots to crawl without interference and protect your site\u2019s integrity and usability.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Automated scraping is a major performance killer as we saw during the Google Search Console SEO office hours in August 2024. A site that\u2019s being&hellip;<\/p>\n","protected":false},"author":1,"featured_media":23187,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[277,3],"tags":[],"class_list":["post-23178","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-advanced-seo-techniques","category-seo"],"_links":{"self":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts\/23178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/comments?post=23178"}],"version-history":[{"count":0,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/posts\/23178\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/media\/23187"}],"wp:attachment":[{"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/media?parent=23178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/categories?post=23178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kwebby.com\/blog\/wp-json\/wp\/v2\/tags?post=23178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}