Dynamic Web Table Scraping After Rows Load

Updated 6 November 2026 · Standard formula, shown below

Yes, you can capture a dynamic web table after its rows load, but you cannot do it with a simple static HTML request. Dynamic tables load their data via JavaScript after the initial page response, so you need a tool that executes JavaScript, waits for the network request to finish, and then extracts the rendered DOM. This article explains the practical methods, the timing pitfalls, and how to choose the right approach for your specific situation.

Why Dynamic Tables Are Different

When you request a URL with a normal HTTP client (like Python’s requests or curl), the server returns raw HTML. In a dynamic table, that HTML contains a placeholder (often an empty <tbody> or a loading spinner) and a JavaScript file. The browser executes that script, which makes an AJAX call to an API endpoint, receives JSON or XML, and then builds the table rows in the DOM. The data never exists in the initial HTML.

Therefore, to capture the table you must either:

Execute the JavaScript in a browser-like environment and then read the rendered DOM.
Find the underlying API endpoint that the JavaScript calls and request it directly.

Method 1: Browser Automation (Selenium, Playwright, Puppeteer)

These tools control a real browser (Chrome, Firefox, etc.) and let you wait for elements to appear. The key is to wait for the table rows to be present, not just a fixed time. For example, in Playwright (Python or Node.js):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('https://example.com/dynamic-table')
    page.wait_for_selector('table tbody tr')  # waits for at least one row
    rows = page.locator('table tbody tr').all_text_contents()
    browser.close()

The wait_for_selector command polls the DOM until the selector matches, which is more reliable than a fixed sleep. If the table loads in chunks (e.g., infinite scroll), you may need to wait until a specific row count appears or until a network request completes.

Method 2: Intercepting the API Response

Many dynamic tables fetch data from a JSON endpoint. You can find that endpoint by opening your browser’s Developer Tools (F12), going to the Network tab, and reloading the page. Look for XHR or Fetch requests that return JSON. The URL often contains parameters like page, limit, or sort. You can then request that URL directly with any HTTP client and parse the JSON. This is faster and uses fewer resources than a full browser, but it requires you to understand the API structure and may need headers (like an API key) to work.

How to Choose the Right Approach

Consider these factors:

Table size – For a few hundred rows, browser automation is fine. For thousands, the API method is much faster.
Pagination – If the table has “Next� buttons or infinite scroll, you need to handle clicks or scroll events. Playwright and Puppeteer can simulate those.
Anti-bot protection – Some sites use Cloudflare or similar. Browser automation may still work if you use a real browser profile, but it’s not guaranteed. Direct API calls may be blocked.
Your programming environment – If you’re already in Python, Playwright or Selenium are natural. In Node.js, Puppeteer is common.

For a one-off capture, browser automation is simpler to set up. For recurring captures, the API approach is more maintainable.

Common Problems and How to Solve Them

Table loads after a delay – Use explicit waits (e.g., wait_for_selector) instead of time.sleep(). If the delay is long, increase the timeout.
Rows change or update – If the table updates periodically, you may need to capture multiple times and diff the results.
Shadow DOM – Some tables are inside a shadow root. You need to pierce the shadow with page.evaluate or use a selector that crosses the boundary.
Iframes – If the table is in an iframe, switch the context to that frame before querying.
Memory issues – Capturing a huge table can crash the browser. Use a headless browser and consider streaming the data to a file.

Comparing Tools: Playwright vs. Selenium vs. Puppeteer

All three can capture dynamic tables, but they differ in ease of use and speed.

Playwright – Modern, fast, supports multiple browsers, has auto-waiting (it waits for elements to be actionable). Good for both Python and Node.js.
Selenium – Older, widely documented, but slower and requires explicit waits more often. Works with many languages.
Puppeteer – Node.js only, but very fast and has a simple API. Great for Chrome-specific tasks.

For a beginner, Playwright is recommended because of its built-in waiting and clear error messages.

Safety and Legal Considerations

When capturing data from a website, always check the site’s robots.txt and Terms of Service. Some sites prohibit scraping, and repeated automated requests can overload a server. Use a reasonable delay between requests, and identify your bot with a user-agent string. Never bypass authentication or CAPTCHA to access data you are not authorized to see. Also, be mindful of personal data – if the table contains personal information, you may be subject to privacy laws like GDPR.

Frequently Asked Questions

Why does my table appear empty when I use requests.get()?

Because the data is loaded by JavaScript after the initial HTML is parsed. The requests library does not execute JavaScript, so the table is still empty. You need a browser automation tool or to find the API endpoint.

How do I find the API endpoint for a dynamic table?

Open Developer Tools (F12), go to the Network tab, reload the page, and filter by XHR or Fetch. Look for a request that returns JSON or XML with the table data. The URL of that request is your endpoint.

Can I use pandas.read_html() on a dynamic table?

No, pandas.read_html() only parses static HTML. It will not see rows added by JavaScript. You must first render the page with a browser tool, then pass the HTML to pandas.

What is the best wait time before capturing?

Instead of a fixed time, wait for a specific condition, like a row count or a loading spinner disappearing. If you must use a fixed time, start with 3 seconds and increase if the table is slow.

Is it legal to scrape a dynamic table?

It depends on the website’s terms and the nature of the data. Public data may be fine, but always respect robots.txt, rate limits, and copyright. For personal or educational use, it’s usually acceptable, but commercial use may require permission.

How can I handle infinite scroll tables?

Use a browser automation tool to scroll to the bottom repeatedly until no new rows appear. In Playwright, you can evaluate window.scrollTo(0, document.body.scrollHeight) in a loop, checking the row count after each scroll.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.