Skip to main content

Fallback Selector for Web Scraping

Web Scraper Extension Feature Ideas

When scraping data from a webpage, we often expect certain elements (selectors) to be present. However, there may be cases where the expected selector is missing. In such situations, the scraper should be able to look for an alternative selector to gather the data from.

How It Works:

  1. The scraper first looks for the primary selector.

  2. If the primary selector is not found, the scraper automatically looks for a fallback selector.

  3. If neither selector is found, the scraper could either stop or log an error.

This feature ensures that the web scraper can handle missing or changing page structures and still gather data efficiently.

1 comment

Log in to comment and vote

Comments1

  • Martins Balodis

    Team•

    Dec 29, 2024

    This already can be achieved with custom CSS selectors.

    Let’s say you have a page with two potential price css selectors:

    • div.price1 span

    • div.price2 span

    Solution 1:

    Combine both selectors:

    div.price1 span, div.price2 span

    If price1 doesn’t exist, price2 will be extracted. If both exist, then a Text selector with multiple option not checked would extract the first one.

    Solution 2:

    Combine both selectors but also only execute price2 if price1 isn’t found. This would add exclusivity in case when Element selector is used and you don’t want two records. CSS selectors :has() and :not() can be used.

    div.price1 span, html:not(:has(div.price1 span)) div.price2 span

    Solution 3:

    Create two separate selectors and combine data in post processing.

    Regarding the request for when neither selectors are found:


    You can create a wrapper Element selector like this:

    html:has(div.price1 span), html:has(div.price2 span)

    When neither selectors are found the page would be considered empty and all child selectors wouldn’t get executed. Currently Web Scraper browser extension would skip these pages while Web Scraper Cloud would skip them and also list them as empty for later review.

    We plan in a future release simplify this for Web Scraper Cloud users via data quality settings. And also we plan to add the option to review failed or empty pages review for the browser extension.

    If you feel that there should be a UI to solve this problem within the extension, up-vote this feature idea and comment on how it could look like.