How Websites Detect and Block Web Scraping

Updated 14 November 2026 · Standard formula, shown below

Web scraping is the automated extraction of data from websites, and while it powers price comparison, research, and SEO tools, it can also overload servers, steal content, or violate terms of service. This guide explains the technical methods websites use to detect and block scrapers, from simple IP rate limiting to advanced browser fingerprinting, and how you can identify which technique a site is using.

What You’ll Need

A basic understanding of HTTP requests and headers
Familiarity with browser developer tools (Network tab)
Optional: access to a test server or a site you control to experiment
A web browser (Chrome, Firefox, or Edge) for testing

Step-by-Step

Check for basic rate limiting. Send multiple requests from the same IP address in a short time. If the server returns HTTP 429 (Too Many Requests) or 403 (Forbidden) after a threshold, the site is using IP-based rate limiting. Tools like curl or browser extensions can help you test.
Examine HTTP headers. Look at the User-Agent, Accept-Language, and Referer headers. Many scrapers use default libraries (e.g., Python-requests) that reveal themselves. Sites often block requests with empty or suspicious User-Agent strings. You can check this by sending a request with a missing User-Agent and seeing if you get blocked.
Test for JavaScript challenges. Load the page in a headless browser (like Puppeteer or Playwright) and see if the content appears. If the site uses a JavaScript challenge (e.g., a script that sets a cookie after a short delay), then the content is only served after the script runs. Scrapers that don’t execute JavaScript will see empty or placeholder content.
Look for honeypot traps. Inspect the HTML for hidden links or form fields (e.g., display:none or visibility:hidden). A real user never sees or clicks these, but a scraper might follow them. If a site blocks you after accessing a hidden link, it’s a honeypot trap.
Analyze browser fingerprinting. Use a tool like fingerprintjs or a test site to see what data your browser exposes (canvas, WebGL, fonts, plugins). Sites can compare this fingerprint across sessions. If you change your IP but the fingerprint stays the same, the site can link your requests. To detect this, try accessing the site from two different browsers and see if you get different content or challenges.
Check for behavioral analysis. Observe how the site responds to mouse movements, scroll events, and timing. Some sites use scripts that track human-like behavior. If you make requests at a constant rate with no randomness, the site may flag you. You can test this by using a tool that simulates human-like delays and see if the block rate drops.
Look for CAPTCHA or proof-of-work challenges. If you see a CAPTCHA (like reCAPTCHA) or a proof-of-work challenge (e.g., a JavaScript that computes a hash), the site is actively blocking automated access. These are often triggered after a certain number of requests or when the fingerprint looks suspicious.
Review robots.txt and terms of service. While not a technical detection method, robots.txt tells you what parts of the site are off-limits. Sites may also block based on the IP ranges of known cloud providers (AWS, Google Cloud) because those are common for scrapers. You can check if your IP is in such a range by using an IP lookup tool.
Test with a proxy or VPN. If you suspect IP-based blocking, try using a different IP (e.g., a residential proxy). If the block disappears, it confirms IP-based detection. If it persists, the site is using more advanced methods like fingerprinting.
Monitor response headers. Some sites add custom headers like X-Robots-Tag or X-Frame-Options to signal blocking. Also, look for Retry-After in 429 responses – it tells you how long to wait before retrying, which is a sign of rate limiting.

Common Mistakes

Ignoring the User-Agent. Sending requests with a default or empty User-Agent is the easiest way to get blocked. Always set a realistic one.
Not checking for JS-rendered content. If you only use simple HTTP requests, you might miss content that requires JavaScript. This leads to incomplete data extraction, not a block, but it’s a common mistake.
Assuming all blocks are IP-based. Many sites use multiple layers. If you change IP but still get blocked, you might be fingerprinted.
Overlooking honeypots. Hidden links are easy to miss, but following them can trigger an immediate ban.
Not respecting rate limits. Even if you don’t get an explicit 429, sending too many requests too fast can get you permanently banned.

Tips

Use a headless browser with human-like behavior. Tools like Playwright can simulate mouse movements and random delays.
Rotate IPs and User-Agents. Use a pool of residential IPs and change User-Agents periodically to avoid detection.
Parse robots.txt and respect it. It’s not just a guideline; it can also give you clues about what the site considers sensitive.
Implement exponential backoff. If you get a 429, wait longer before retrying, and gradually increase the wait time.
Use a session object. Maintain cookies and session data to appear more like a real user.
Monitor your own traffic. Use logging to see when you get blocked and adjust your strategy.

Troubleshooting

If you are getting blocked despite following best practices, try these steps:

Check if the block is temporary. Wait a few hours or a day. Some blocks are time-based.
Test with a different browser. If you’re using a headless browser, try a regular browser to see if the issue is with your automation setup.
Examine the response body. Sometimes the block page looks like a normal page. Look for hidden messages or unusual HTML.
Use a different network. If you’re on a shared IP (like a university or office), you might be blocked because of others’ activity.
Check your DNS and SSL settings. Some sites block requests with mismatched TLS fingerprints. Tools like curl with the --tls-max option can help.
Look at the site’s API. If the site has an official API, use it instead. It’s more reliable and less likely to get you blocked.

Frequently Asked Questions

Q: Is web scraping illegal?
A: Not inherently, but it can violate a site’s terms of service. Whether it’s illegal depends on the jurisdiction and the data being scraped. Always check the site’s terms and applicable laws.

Q: What is the difference between a bot and a scraper?
A: A bot is any automated program, while a scraper specifically extracts data. All scrapers are bots, but not all bots scrape.

Q: Can a site detect a headless browser?
A: Yes, many headless browsers have detectable traits like missing plugins, unusual window sizes, or different JavaScript execution. Sites can use these to block them.

Q: How do I avoid getting blocked when scraping?
A: Use a real User-Agent, rotate IPs, add random delays, and handle cookies and sessions. Also, respect robots.txt and the site’s rate limits.

Q: What is a honeypot trap?
A: A hidden link or form field that only a scraper would follow. Real users don’t see it, so if you access it, you’re clearly automated.

Q: Does using a VPN help avoid detection?
A: It can help with IP-based blocking, but if the site uses fingerprinting, a VPN alone won’t help. You also need to change your browser fingerprint.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.