DOCSEARCHbyAlgolia
Submit Docs

Import.io Documentation

website·Indexed 39 days ago

https://docs.import.io

Documentation for Import.io's web data extraction platform. Includes guides on creating and managing Data Reports and Change Reports, using the Editor, and configuring extractors with advanced features like regular expressions, webhooks, and interactive extractors for authentication, pagination, and infinite scroll. Provides integration examples for Node.js and Python 3, including writing data to MySQL databases.

Tokens
13K
Snippets
11
Records
119
Agent score
63%

What's inside Import.io

  1. Overview of Import.io web data extraction

    Import.io is a web scraping platform that uses a point-and-click interface to transform websites into structured tabular data. It supports complex extraction scenarios, including pages that require JavaScript execution, user interaction, or authentication (logins).

  2. Use Regular Expressions in Import.io

    Import.io provides a Set Regular Expression tool to find text matching a specific pattern and replace it with other strings. The platform uses native JavaScript regular expression functions. Note that some advanced regex features, such as lookbehind, are not supported.

  3. Navigate and manage the Import.io Dashboard

    The Import.io Dashboard serves as the central hub for managing data extraction workflows. It provides access to the following primary functional areas:

    • Extractors Tab: The default view where all created extractors are managed.
    • Home Tab: Provides an account overview, including data health, account usage, and scheduled extractors.
    • Reports Tab: Used to create Data, Change, and Comparison Reports.
    • Portal Tab: Allows sharing of extracted data with external users who do not have an Import.io login.
    • Insights Tab: Used to visualize extracted website data.
    • Account Tab: Used for managing account settings and billing information.
    • Plan Information: Displays current plan details and the real-time count of queries used.
    • Scheduled Extractors: A dedicated list of all extractors currently scheduled in the account.
    • Run History: Displays the last 30 crawl runs for a specific extractor, allowing users to view statistics and download or preview results for each run.

  4. Understand Import.io Extractor and Training concepts

    An Extractor is a configuration built to select specific data points (e.g., Product ID, Name, Price) from a specific type of web page. Training is the process of creating or editing an extractor by loading an example URL and selecting desired data using Point-and-Click, XPath, or Page Interaction tools. Once trained, the extractor uses these rules to identify data during future runs.

  5. Create and share Data Reports from Extractor crawls

    A Data Report allows you to select specific columns from an Extractor's crawl runs to include in a curated report with basic styling. These reports can be shared to the Import.io Portal, enabling stakeholders to view the data without requiring access to your Import.io account. The final output of a Data Report run is available for download as an Excel file.

  6. Monitor data changes using Change Reports

    Change Reports in Import.io allow you to monitor data changes that occur between the most recent crawl run of an Extractor and the previous crawl run. These reports can be shared to the Import.io Portal, enabling stakeholders to view the changes without requiring direct access to your account.

  7. Navigate the Import.io Dashboard

    The Import.io Dashboard uses a Left Navigation Panel to access the following core areas:

    • Home: The starting location of the dashboard.
    • Portal: Access to publicly published reports.
    • Extractors: The default view for managing and viewing your data extractors.
    • Transforms: Area to apply functions and formulas to modify extractor output data.
    • Reports: Dashboard for creating Data, Change, and Comparison Reports.
    • Account: Management of account information, subscriptions, and billing.
    • Help: Access to the Help Center, user guides, videos, and community resources.

  8. Schedule an extractor for automatic runs

    To automate data retrieval, you can schedule an extractor to run on a regular basis. To configure this, navigate to the extractor's Settings tab and locate the Schedule section. You can choose from the following run frequencies:

    • Hourly: Runs at a specific minute of every hour.
    • Daily: Runs every day at a specific time.
    • Weekly: Runs every week at a specific time on a chosen day.
    • Monthly: Runs once a month at a specific date and time.
    • Advanced: Allows for a fully customized schedule using cron-style expressions.

  9. Use Manual XPath to target non-selectable elements in Import.io

    Manual XPaths allow you to override the extractor's automatically generated selections by specifying the exact HTML path of the desired elements. This is specifically useful when data is not selectable within the Import.io editor because it is hidden in a dropdown list, located in an undisplayed tab, or otherwise not fully visible on the webpage.

  10. Manage multiple parameters in the URL Generator

    The URL generator supports multiple parameters per URL. To add more, repeat the highlighting process for different parts of the URL. The tool will automatically generate all possible combinations of the specified parameters (e.g., 3 cities × 3 cuisines × 5 pages = 45 URLs).

    • To remove a parameter: Click the X located to the left of the parameter definition.
    • To modify the base URL: Click the Edit button at any time.

  11. Set a default page value for single-page results in XPath

    Many websites hide pagination links when only one page of results exists, which would cause a standard XPath query to return a blank value. To ensure the extractor returns '1' when no pagination elements are found, use a concat and substring logic to provide a default value.
    concat((//button[contains(@aria-label,'Page')])[last()], substring('1', 1 div not((//button[contains(@aria-label,'Page')])[last()])))

  12. Train an Interactive Extractor with additional URLs

    To improve extractor accuracy, you can add more pages to the training set. While in the editor, click Train With Additional URLs to access the Inputs table. After adding or editing the desired inputs, click Train. This will replay the new set of inputs and add the corresponding pages to the training data.