ThisCalculator

Everyday

Monitor Regulatory Websites for Changes

Updated · Standard formula, shown below

This guide shows you how to automatically track multiple regulatory websites for updates, so you never miss a new rule, guidance, or announcement. You’ll set up a free or low-cost monitoring system that checks pages for you and sends alerts when something changes. You’ll need a Google account (for Google Sheets) and access to a web browser; no coding experience is required.

What You’ll Need

  • A Google account (for Google Sheets and Google Apps Script)
  • A list of the specific regulatory pages you want to monitor (e.g., FDA, EPA, SEC, or local agency pages)
  • An email address to receive alerts (or a Slack/Discord webhook if you prefer)
  • Basic familiarity with copying and pasting text

Step-by-Step

  1. Identify the exact pages to monitor. List the URLs of the regulatory pages you care about. Focus on pages that actually change, like rulemaking dockets, press release feeds, or “What’s New” sections. Avoid monitoring the homepage of a large agency, as it changes frequently and will cause alert fatigue.
  2. Create a Google Sheet for tracking. Go to sheets.new and create a new spreadsheet. In the first row, create headers: URL, Last Checked Hash, Last Checked Time, Status, and Notes. This sheet will store the current state of each page.
  3. Open the Apps Script editor. In the Google Sheet, click Extensions > Apps Script. This opens a new tab where you can write a small script.
  4. Write the monitoring script. Copy and paste the following code into the Apps Script editor. This script fetches each URL, calculates a hash of the page content, and compares it to the stored hash. If it differs, it sends an email alert and updates the sheet. 
    function monitorPages() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet1');
  var data = sheet.getDataRange().getValues();
  var email = Session.getActiveUser().getEmail(); // or set a fixed email
  
  for (var i = 1; i < data.length; i++) {
    var url = data[i][0];
    if (!url) continue;
    
    var oldHash = data[i][1];
    var response = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
    var content = response.getContentText();
    var newHash = Utilities.computeDigest(Utilities.DigestAlgorithm.MD5, content).map(function(b) { return (b & 0xFF).toString(16); }).join('');
    
    if (oldHash && oldHash != newHash) {
      MailApp.sendEmail(email, 'Regulatory Update: ' + url, 'The page has changed. Check it now: ' + url);
      sheet.getRange(i+1, 3).setValue(new Date()); // Last Checked Time
      sheet.getRange(i+1, 4).setValue('Changed');
    } else {
      sheet.getRange(i+1, 3).setValue(new Date()); // Last Checked Time
      sheet.getRange(i+1, 4).setValue('No change');
    }
    sheet.getRange(i+1, 2).setValue(newHash); // Update hash
  }
}
  5. Run the script manually first. In the Apps Script editor, select the monitorPages function from the dropdown and click the Run button. Authorize the script when prompted (it needs access to your sheet and email). This initial run will populate the hash for each URL without sending alerts.
  6. Set up a time-based trigger. In the Apps Script editor, click the clock icon (Triggers). Click Add Trigger, choose monitorPages, select Time-driven, and choose an interval (e.g., every hour or daily). Save the trigger.
  7. Add your URLs to the sheet. Go back to your Google Sheet and paste your list of URLs in column A, starting from row 2. Leave column B empty for now (the script will fill it). Save the sheet.
  8. Test the monitoring. After the trigger runs once, check the sheet to see the “Last Checked Time” and “Status” columns. Then, manually change something on one of the pages (or add a test page that you control) to verify you receive an email alert.
  9. Optionally, add a “force check” button. To run the script on demand, add a drawing in the sheet (Insert > Drawing) and assign the script to it via the three-dot menu. This is handy for immediate checks.
  10. Maintain and refine. Review the alerts weekly. If a page changes too often or too rarely, adjust your URL list or trigger frequency. Remove pages that are no longer relevant.

Common Mistakes

  • Monitoring the wrong pages. Homepages and navigation pages change frequently (e.g., due to rotating banners), causing false alerts. Target specific subpages or feeds.
  • Not handling redirects. Some URLs redirect to a login page or a different domain. The script fetches the final content, so ensure your URLs are direct and accessible.
  • Ignoring dynamic content. Many regulatory sites use JavaScript to load content. The script fetches raw HTML, which may not include dynamic elements. For such sites, consider using the site’s RSS feed or a service like Visualping.
  • Forgetting to authorize the script. If you skip authorization, the script won’t run. Always run it manually once to trigger the authorization popup.
  • Using a shared email. If the script uses the active user’s email, it may not send to a team inbox. Set a fixed email address in the script (replace Session.getActiveUser().getEmail() with a string).

Tips

  • Use RSS feeds when available. Many regulatory agencies offer RSS feeds for press releases or rule changes. You can add these feeds to a feed reader (like Feedly) or use the script to monitor them—just fetch the feed URL and compare the XML.
  • Filter out noise. If a page changes but only in a minor way (e.g., a date), you might want to compare only the main content area. This requires more advanced parsing, but you can start with the full-page hash and refine later.
  • Set a reasonable check frequency. For most regulatory updates, checking once a day is sufficient. Hourly checks can trigger rate limits and clutter your inbox.
  • Use a dedicated email folder. Create a filter in your email client to automatically label or archive alerts from the script, keeping your inbox clean.
  • Back up your sheet. Periodically export your sheet to Excel or make a copy, so you don’t lose your tracking history.

Troubleshooting

  • Script runs but no alerts are sent. Check that the email address is correct and that the script has permission to send email. Try sending a test email via the script (add a line like MailApp.sendEmail('you@example.com', 'Test', 'Test')). Also, check your spam folder.
  • “Exception: Invalid email” error. The active user’s email might not be available in some contexts. Replace the email variable with a hardcoded address.
  • “Quota exceeded” error. Google Apps Script has daily quotas for UrlFetchApp and MailApp. If you have many URLs or check too frequently, you may hit the limit. Reduce the number of URLs or check less often.
  • Page content is too large. Some pages are huge, causing the script to slow down or time out. Use a shorter hash (e.g., only the first 1000 characters) or switch to an RSS feed.
  • Changes are missed. If a page updates between checks, you won’t see it. Increase the check frequency, or use a service that checks more often (like Visualping).
  • The script stops working after a while. Check the Apps Script execution log (View > Logs) for errors. Common causes include changed URL structure or authorization expiry. Re-run the script manually to refresh.

Frequently Asked Questions

Can I monitor multiple websites with this method?

Yes, you can add as many URLs as you want to the Google Sheet. Just make sure you stay within the daily quotas (typically 20,000 URL fetches per day for consumer accounts).

Will this work for sites that require login?

No, the script uses anonymous HTTP requests. For password-protected pages, you would need a more advanced solution like a headless browser or a paid monitoring service.

How often should I check for changes?

For most regulatory updates, daily checks are sufficient. If you need real-time alerts, consider using a dedicated service or increase the trigger to every hour, but be mindful of quotas.

Can I use this to monitor PDFs or documents?

The script fetches the raw content of a URL. If the URL points to a PDF, the content is binary, so the hash will change only if the file itself changes. This works, but you may want to also monitor the page that links to the PDF.

What if I get too many false alerts?

Refine your URL list to target specific subpages or use RSS feeds. Also, consider comparing only a portion of the page (e.g., the main content area) to reduce noise.

Is there a free alternative to Google Sheets?

Yes, you can use a similar script with Python and a free hosting service like PythonAnywhere, or use a no-code tool like IFTTT with a RSS feed. However, the Google Sheets method is the simplest for most users.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.