Block Unwanted Bots: robots.txt, WAF & Rate Limits

Updated 14 November 2026 · Standard formula, shown below

This guide shows you how to block unwanted bots using three complementary layers: robots.txt to set polite rules, a Web Application Firewall (WAF) to filter malicious traffic, and rate limiting to slow down or stop abusive requests. You’ll learn exactly what to configure in each layer and how they work together to protect your site.

What You’ll Need

Access to your website’s root directory (to edit robots.txt).
A WAF service (e.g., Cloudflare, AWS WAF, or your hosting provider’s built-in WAF).
Access to your server configuration (e.g., Nginx, Apache) or a CDN/WAF dashboard to set rate limits.
Basic understanding of HTTP status codes (e.g., 403, 429).

Step-by-Step

Identify the bots you want to block. Check your server logs or analytics for user-agent strings, IP addresses, and request patterns. Look for high request rates, 404 errors on non-existent pages, or suspicious user agents like “curl”, “python-requests”, or known bad bots. Use a tool like a log analyzer or your WAF’s traffic logs to identify them.
Configure robots.txt. Create or edit the robots.txt file in your site’s root directory. Use the User-agent and Disallow directives to block specific bots. For example:
```
User-agent: bad-bot
Disallow: /

User-agent: *
Allow: /
    
```
You can list multiple user agents or use wildcards (if supported). Remember, robots.txt is a suggestion—cooperative bots will follow it, but malicious ones won’t. It’s your first, polite layer.
Set up a WAF rule to block known bad bots. In your WAF dashboard, create a rule that inspects the User-Agent header and blocks requests from known malicious bots. Many WAFs have managed rule sets for this. If not, create a custom rule that matches a list of user agents you’ve identified. For example, in AWS WAF, you can use a string match condition on the User-Agent header with a list of bad agents.
Add IP-based blocking in the WAF. If you see repeated requests from specific IP addresses or IP ranges, create a WAF rule to block them. Be careful with IP blocking—shared IPs (like those from cloud providers) may affect legitimate users. Use it for persistent offenders only.
Implement rate limiting. Rate limiting controls how many requests a client can make in a given time. Configure it at your server or WAF level. For example, in Nginx, you can use the limit_req directive:
```
limit_req_zone $binary_remote_addr zone=botlimit:10m rate=10r/s;

server {
    location / {
        limit_req zone=botlimit burst=20 nodelay;
    }
}
    
```
Or in Cloudflare, set a rate limiting rule that allows, say, 100 requests per minute per IP, and returns a 429 status when exceeded.
Test your configuration. After implementing each layer, test by making requests from a normal browser and from a simulated bot. Use tools like curl with a custom user agent to see if you get blocked. Verify that legitimate users are not affected.
Monitor and adjust. Watch your logs and WAF analytics. If you see legitimate traffic being blocked, adjust your rules. If bots are still getting through, tighten the rate limits or add more WAF rules. This is an ongoing process.

Common Mistakes

Relying only on robots.txt. Malicious bots ignore it. Always combine with WAF and rate limits.
Blocking by user agent alone. Bots can easily spoof user agents. Use multiple signals (IP, rate, behavior).
Rate limits too strict. Setting a very low request rate can block legitimate users, especially those behind a shared IP (like a corporate NAT).
Not whitelisting known good bots. Search engine crawlers like Googlebot should be allowed. Blocking them will hurt your SEO. Use their official verification methods (e.g., reverse DNS) to allow them.
Forgetting to test after changes. Always test after any change to ensure you haven’t broken something.

Tips

Use a layered approach. Robots.txt for politeness, WAF for malicious traffic, rate limits for abuse. Each layer covers what the other misses.
Start with logging, not blocking. First, log and analyze traffic for a few days to understand what’s hitting your site. Then block based on evidence.
Use challenge pages. Many WAFs offer a JavaScript challenge or CAPTCHA for suspicious requests. This can stop bots without blocking legitimate users.
Keep your WAF rules updated. Bot signatures change. Use managed rule sets that update automatically.
Return proper HTTP status codes. Use 403 (Forbidden) for blocked requests, and 429 (Too Many Requests) for rate-limited clients. This helps bots and tools understand why they’re blocked.

Troubleshooting

Legitimate users are being blocked. Check your rate limit thresholds—increase them if needed. Also, check if you’ve accidentally blocked a shared IP. Use WAF rules that are more specific, like combining user agent and IP.
Bots still getting through. Review your WAF logs to see if the bot is spoofing user agents or using a rotating IP. Consider adding behavioral analysis (e.g., request frequency, patterns) to your rules. Also, ensure your rate limiting is applied to the correct location (e.g., all URLs, not just the homepage).
Search engines are blocked. Verify that you haven’t blocked Googlebot or Bingbot. Check your robots.txt and WAF rules. Use the official verification methods (e.g., Google’s Search Console) to confirm that Googlebot is allowed.
Rate limiting not working. Ensure that your server or WAF is actually enforcing the limit. Check the configuration syntax and that the module is loaded. Test with a simple script that sends many requests to see if you get a 429.

Frequently Asked Questions

Does robots.txt block malicious bots? No, it only works for cooperative bots. Malicious bots ignore it. Use WAF and rate limits for real protection.
What is a good rate limit for a typical website? It depends on your traffic. A common starting point is 100 requests per minute per IP, but you should adjust based on your normal traffic patterns and your server’s capacity.
Can I block bots by their IP address? Yes, but be cautious. IPs can be shared or dynamic. Use IP blocking for persistent offenders, not as a first line of defense.
Will blocking bots affect my SEO? Only if you block legitimate search engine crawlers. Make sure to allow Googlebot, Bingbot, and other legitimate bots by verifying them properly.
What’s the difference between a WAF and rate limiting? A WAF filters based on content (e.g., user agent, IP reputation, SQL injection patterns), while rate limiting controls the number of requests over time. They complement each other.
How do I know if a bot is good or bad? Check the user agent and IP against known lists (e.g., Googlebot’s IP ranges). Good bots usually identify themselves clearly and follow robots.txt. Bad bots often have generic or random user agents and high request rates.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.