Scraped Data Change Detection: Snapshots to Alerts

This article explains how to detect changes in scraped data by moving from simple snapshots to reliable alerts. You compare current data against a stored baseline, normalize it, and use a diffing algorithm to identify meaningful changes, then route those changes through a notification pipeline. The key is to distinguish trivial fluctuations from substantive updates so you only get alerted when it matters.

Formula

Checks per day = 1440 / t, where t is the time between checks in minutes. Data volume per day = checks per day × snapshot size (s) in items. These give you a sense of the operational load of your monitoring setup.

Worked Example

With t = 60 minutes and s = 1000 items: checks per day = 1440 / 60 = 24, data per day = 24 × 1000 = 24,000 items. So you’d run 24 checks daily and process 24,000 items per day.

Background: Why Snapshots Alone Fall Short

Web scraping is rarely a one-time fetch. Prices, stock levels, article text, and metadata change constantly. A snapshot is just a point-in-time copy of the target page. If you only store snapshots, you still have to manually compare them to see what changed. That works for a few pages, but fails at scale. You need automated change detection that turns raw snapshots into actionable alerts.

Early approaches used simple checksums or file hashes. If the hash changed, the page changed. But that gives no detail on what changed, and it triggers on every minor edit, including ads, timestamps, or dynamic counters. Modern change detection uses structured diffing—parsing the page into fields and comparing those fields individually.

How Change Detection Works

Reliable change detection follows a pipeline:

1. Fetch: Retrieve the page or API response at scheduled intervals.
2. Normalize: Strip dynamic noise (timestamps, session IDs, ad slots) and convert to a canonical format (JSON, XML, or a database row).
3. Diff: Compare the new normalized data against the last stored version. Use field-level comparison, not whole-page hashing.
4. Filter: Apply rules to ignore insignificant changes (e.g., price change under $0.01, stock count fluctuation).
5. Alert: Send a notification via email, Slack, webhook, or SMS when a meaningful change is detected.

Each step is crucial. Skipping normalization leads to false positives. Skipping filtering leads to alert fatigue. A well-designed system lets you configure thresholds per field.

Choosing the Right Diffing Strategy

Not all diffs are equal. The right strategy depends on your data type:

• Exact match: For IDs, SKUs, or status codes. Simple equality check.
• Numeric threshold: For prices, ratings, or counts. Alert only if the absolute or percentage change exceeds a limit.
• Text similarity: For descriptions or reviews. Use Levenshtein distance or cosine similarity to detect meaningful rewrites.
• Structural diff: For JSON/XML trees. Compare keys and values recursively, ignoring order where appropriate.

For most e-commerce scraping, a combination works best. For example, alert on price drop of 5% or more, but ignore stock count changes unless it hits zero.

Common Problems and How to Avoid Them

Even with a good pipeline, you’ll run into issues:

• Dynamic content: JavaScript-rendered pages need a headless browser. Use a tool that executes JS before snapshotting.
• Anti-bot measures: Rate limiting, CAPTCHAs, and IP blocks. Rotate user agents and proxies, but stay within the site’s terms of service.
• Data drift: The site’s HTML structure changes. Your parser breaks. Schedule periodic validation that the expected fields still exist.
• Time zone mismatches: Store timestamps in UTC and compare using the same timezone to avoid off-by-one errors.

A common misconception is that more frequent checks always mean faster alerts. In reality, many sites update only a few times a day. Checking every minute wastes resources and increases the chance of being blocked. Start with hourly checks and adjust based on observed update frequency.

Snapshot Storage and Versioning

You need to keep more than just the latest snapshot. Storing historical versions allows you to track trends, roll back to previous states, and debug alerts. A simple approach is to store each normalized snapshot in a database table with a timestamp. For large volumes, consider a time-series database or object storage with versioning.

Example: a product price tracker stores each day’s price in a price_history table. When a price change is detected, the alert includes the previous price and the new one. This gives context and helps the recipient decide if action is needed.

Alerting: From Raw Diff to Human-Readable Message

An alert that says “Field price changed from 10.00 to 9.50� is useful, but you can do better. Include the item name, a link to the page, and the time of change. For high-volume systems, group changes into a digest. For critical changes, send an immediate alert. Define severity levels:

• Info: Non-essential change, logged only.
• Warning: Change above threshold, but not urgent (e.g., price increase).
• Critical: Change that requires immediate action (e.g., stock out, price drop below cost).

Make sure alerts include enough detail to be actionable without opening the site. That means including the changed fields and their old/new values.

Cost and Scalability Considerations

Running change detection at scale has costs: compute for parsing and diffing, storage for snapshots, and network bandwidth. The formula above helps estimate data volume. If you check 1000 items every hour, that’s 24,000 items/day, which is trivial for most databases. But if you scrape 10,000 items every minute, that’s 14.4 million items/day—significant.

To reduce cost, use incremental fetching (only fetch changed pages) if the site supports it via APIs or ETags. Also, consider sampling: check a subset of items more frequently and the rest less often, based on how volatile they are.

Alternatives to Custom-Built Systems

You don’t have to build everything from scratch. Several open-source tools and services offer change detection out of the box. Some are visual and let you click on the part of the page you care about. Others are API-based and integrate with your existing stack. When evaluating, check whether they support normalization, field-level diffing, and custom alert rules. If you need full control, building your own pipeline with a scheduler (like cron), a scraper library, and a database is straightforward.

Frequently Asked Questions

How often should I check for changes?

It depends on how fast the source updates. For product prices, hourly is often enough. For news headlines, every few minutes may be needed. Start with a reasonable interval (e.g., every hour) and monitor how often changes actually occur. If you see that changes happen right after your check, you might shorten the interval. If you never see changes, lengthen it to save resources.

What is the best way to diff JSON data?

Use a library that does deep recursive comparison, like Python’s deepdiff or JavaScript’s fast-json-patch. These will return a list of changes with paths and old/new values. For large JSON, consider hashing each field individually and comparing hashes.

How do I handle pages that load content with JavaScript?

You need a headless browser like Puppeteer or Playwright to render the page before snapshotting. This is slower and more resource-intensive, but necessary for SPAs. Alternatively, look for a public API that the site uses internally and scrape that directly—it’s often faster and more stable.

Can I detect changes in PDFs or images?

Yes, but it’s harder. For PDFs, you can extract text and diff that. For images, you can compare perceptual hashes (pHash) to detect visual changes, but you won’t know what changed without further analysis. If you only need to know if something changed, a simple hash works.

How do I avoid being blocked by the website?

Respect robots.txt, set a reasonable crawl rate, and use a real user agent. Avoid hitting the site too frequently. If you must scrape at scale, consider using official APIs when available. Also, handle HTTP errors gracefully and back off on 429 or 503 responses.

Is it legal to scrape data and monitor changes?

It depends on the website’s terms of service and your jurisdiction. Many sites prohibit scraping in their ToS. However, scraping publicly available data for personal, non-commercial use is often tolerated. For commercial use, consult a legal professional. Always check the site’s robots.txt and terms before building a scraper.

Related

• Quilt Size Calculator: Find Your Perfect Dimensions
• Queens Step Test Calculator & VO2max Formula
• One Rep Max Calculator: Estimate Your 1RM

in t | Time between checks | min | 60 | 1
in s | Snapshot size (items) | items | 1000 | 1
out per_day | Checks per day | checks | 1440 / t | 0
out data_per_day | Data volume per day | items | (1440 / t) * s | 0