Bad Bots and Scrapers: Protecting Your Server Without Blocking Google

To protect your server from bad bots without blocking Google, stop trusting the User-Agent string, since faking it takes one line. Verify the real Googlebot with reverse DNS followed by forward DNS: the IP address must resolve to a name ending in googlebot.com or google.com, and that name must resolve back to the same address. Then apply rate limits in Nginx (for example 10 requests per second per IP with a burst of 20), block the targeted paths such as xmlrpc.php and wp-login.php if you do not run WordPress at all, and move heavy blocking to a WAF at the edge so the request never reaches your server. Be careful about overdoing it. Blocking entire IP ranges or every AI crawler can hide your site from search results and from the answers AI assistants give, and a growing share of your customers ask those assistants before they buy.

How Much of Your Traffic Is Actually Human

Recent industry reports estimate that close to half of internet traffic is automated, and that more than a third of it comes from bad bots doing content and price scraping, password guessing, and vulnerability probing. A typical company website shows the same picture in its logs, where human requests often make up only 30% to 60% of the total. Do not take that number from anyone else, measure it in your own logs:

# Top 15 IP addresses by request count today
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -15
# Most frequent User-Agents
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -15
# How many hits on WordPress paths on a site that is not WordPress
grep -cE "wp-login|xmlrpc|wp-admin" /var/log/nginx/access.log

If one address shows ten thousand requests in a day, or thousands of requests hit wp-login.php on a Laravel system, that is traffic bringing in zero customers while eating CPU and database connections. On some sites it accounts for 20% to 40% of server load.

Telling the Real Googlebot From a Fake

Any script can send User-Agent: Googlebot, which is exactly why malicious crawlers use it, knowing most sites will not dare block it. Proper verification takes two steps:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The name ends in googlebot.com and resolves back to the same address, so it really is Google. Stopping after the first step is not enough, because anyone who controls an IP range can set a PTR record that says whatever they like. Google also publishes its crawler ranges as JSON, and Bing works the same way through search.msn.com. Avoid running this check on every request since DNS lookups are relatively slow. Run it on addresses that cross a threshold and cache the result for 24 hours.

Rate Limiting in Nginx

Rate limiting is the cheapest and most effective protection, because you do not need to know who the attacker is, only that their behavior exceeds what a human does. A person browsing your site generates 1 to 3 page requests per second at most, while a crawler fires off dozens.

http {
  # 10 requests per second per IP; 10MB of memory tracks about 160,000 addresses
  limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
  # A stricter limit for login pages
  limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
  limit_req_status 429;
}
server {
  location / {
    limit_req zone=perip burst=20 nodelay;
  }
  location = /login {
    limit_req zone=login burst=3;
  }
}

The burst=20 matters, since a single page can request 15 CSS files and images at once, and without that headroom you would block real visitors. Better still, exclude static files from the limit entirely or serve them from a CDN. Status 429 tells well-behaved search crawlers to slow down, which is different from 403, a flat refusal.

Blocking Paths That Are Not Yours

If your system runs on Laravel, Django, or any framework other than WordPress, every request for wp-login.php, xmlrpc.php, or .env is a guaranteed automated probe. Close them as cheaply as possible:

location ~* (wp-login\.php|xmlrpc\.php|wp-admin|/\.env|/\.git) {
  return 444;
}

Status 444 is specific to Nginx and closes the connection without a response, so the server wastes no time building an error page. If you do run WordPress, disable xmlrpc.php unless you use the mobile app, because it allows hundreds of password attempts in a single request, and limit wp-login.php to five attempts per minute. Add Fail2ban to read the 429 and 444 entries and ban repeat offenders at the firewall for an hour, so they stop reaching Nginx at all.

A WAF at the Edge: Block Before the Request Arrives

Every request you block on your own server still costs a connection and some CPU. When an attack comes from thousands of different addresses, per-IP rate limits are not enough, and that is where a web application firewall at the edge comes in, such as the CDN services that sit in front of a site. Practical rules to start with:

Rule	Action	Search engine impact
Verified known bots (Google, Bing)	Allow	None, indexing stays complete
User-Agent claims Googlebot from an unverified address	Block	None, since it is not Google
More than 100 requests in 10 seconds to one path	Challenge	None if verified bots are exempt
Admin paths from outside Saudi Arabia	Challenge or block	None, those paths are not indexed
Blocking whole countries	Avoid	May shut out crawlers operating from there

Watch your origin server. If its address is known, attackers bypass the edge and hit it directly, so configure the firewall to accept ports 80 and 443 only from your edge provider's addresses. For large network-level volumetric attacks, a cloud server can add DDoS protection for 1,200 SAR a year.

Content and Price Scrapers

Not every bad bot is hunting for a vulnerability. Another common type copies your pages wholesale to republish them, or pulls your price and product tables every hour so a competitor can track you. These are smarter than scanners: they slow down to one request every two or three seconds to stay under the radar and rotate through hundreds of addresses from residential proxy networks. Their signature in the logs is long sessions that walk through hundreds of pages in sequence without ever requesting CSS or fonts, executing JavaScript, or carrying cookies from an earlier visit.

Protection here works on behavior rather than address. Three approaches give good results at low cost:

• Put sensitive data endpoints, such as detailed pricing, behind a session or a short-lived token issued with the page.
• Cap pages per session per hour, for example at 300, since a human rarely goes past 50.
• Use an edge challenge for sessions that have not requested a single static file after 20 pages.

Do not expect to block this type completely. The goal is to raise the cost of scraping until your site is no longer worth the effort.

When Overblocking Hurts You

The worst outcome of enthusiastic protection is your pages quietly disappearing from Google. The signs in Search Console are rising "crawl failed" or 5xx errors and a drop in indexed pages within two weeks, so watch those numbers after every new rule.

The same applies to AI crawlers. Some collect training data, while others fetch a page at the moment a user asks a question so they can quote it with a link to your site. Blocking them all with one rule means AI assistants will not mention you when someone asks about your service. Decide bot by bot in robots.txt:

User-agent: GPTBot
Disallow: /portal/
Allow: /

User-agent: *
Disallow: /portal/
Disallow: /admin/

Respectable crawlers follow robots.txt and malicious ones ignore it, so treat it as a tool for managing good bots rather than a defense against bad ones.

A Rollout Plan for One Working Day

1. Analyze a full day of logs with the commands above, and write down the share of automated requests and the most targeted paths.
2. Enable a global limit_req of 10r/s with burst=20, plus a stricter limit on login pages.
3. Close WordPress paths and hidden files with 444 if you do not need them.
4. Add Fail2ban on the Nginx logs with a one-hour ban.
5. If you use an edge or CDN, allow verified bots, block impostors, and lock the origin to edge addresses only.
6. Watch Search Console for 14 days after the changes.

If your site still slows down under load after blocking, the problem may be resources rather than bots. Check pricing and plans to see what an extra vCPU at 500 SAR a year or 2GB of RAM at 600 SAR a year would cost.