Crawly Documentation

If your spider returns multiple types of items (e.g., blog posts and weather data), you can use pattern matching in your pipeline to target specific data types.

1. Struct-based pattern matching

Use this when you want to utilize existing Ecto schemas or pre-defined structs. Note: When using Ecto structs, you may need to convert the struct to a map before insertion to handle metadata.

2. Key-based pattern matching

Use this to process related items together or for bulk processing. This requires the spider's Crawly.Spider.parse_item/1 callback to return items with specific keys.

Example: Key-based matching for blog posts

If your spider returns:

%{parsed_items: [%{blog_post: post}, %{weather: data}]}

Your pipeline can target only the :blog_post key:

defmodule MyApp.BlogPostPipeline do
  @impl Crawly.Pipeline
  def run(%{blog_post: old_blog_post} = item, state, _opts \ []) do
    # process the blog post
    updated_item = Map.put(item, :blog_post, %{my: "data"})
    {updated_item, state}
  end

  # Fallback: do nothing if it does not match
  def run(item, state, _opts), do: {item, state}
end

A spider is a module that implements the Crawly.Spider behavior. You must implement the following callbacks:

• base_url/(): Returns the root URL of the site.
• init/(): Returns a keyword list of starting URLs (e.g., [start_urls: ["..."]]).
• parse_item(response): The core logic where you parse the HTML response, extract items, and return a %Crawly.ParsedItem{} struct containing extracted items and next_requests.

# lib/crawly_example/books_to_scrape.ex
defmodule BooksToScrape do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url(), do: "https://books.toscrape.com/"

  @impl Crawly.Spider
  def init() do
    [start_urls: ["https://books.toscrape.com/"]]
  end

  @impl Crawly.Spider
  def parse_item(response) do
    # Parse response body to document
    {:ok, document} = Floki.parse_document(response.body)

    # Create item (for pages where items exists)
    items =
      document
      |> Floki.find(".product_pod")
      |> Enum.map(fn x ->
        %{
          title: Floki.find(x, "h3 a") |> Floki.attribute("title") |> Floki.text(),
          price: Floki.find(x, ".product_price .price_color") |> Floki.text(),
          url: response.request_url
        }
      end)

    next_requests =
      document
      |> Floki.find(".next a")
      |> Floki.attribute("href")
      |> Enum.map(fn url ->
        Crawly.Utils.build_absolute_url(url, response.request.url)
        |> Crawly.Utils.request_from_url()
      end)

    %Crawly.ParsedItem{items: items, requests: next_requests}
  end
end

To create a custom crawler, you must implement the Crawly.Spider behaviour. Spiders define how to crawl a site (following links) and how to extract structured data (scraping items).

Required/Key behaviour functions:

• init() or init(options): Returns a keyword list containing start_urls (a list of URLs to begin crawling). Alternatively, use start_requests if you need to prepare specific initial requests (e.g., including session cookies). Note that start_requests are processed before start_urls.
• base_url(): Defines the base URL of the spider. This is used by Crawly.Middlewares.DomainFilter to prevent the crawler from leaving the target website.
• parse_item(response): The core logic that translates a Response into a %Crawly.ParsedItem{} struct. This function must return a struct containing both items (the scraped data) and requests (new URLs to visit).

Spiders run within Crawly.Worker processes. You can control concurrency using the concurrent_requests_per_domain setting.

defmodule MySpider do
  use Crawly.Spider

  def init, do: [start_urls: ["https://example.com"]]

  def base_url, do: "https://example.com"

  def parse_item(response) do
    # Use a library like Floki to parse response.body
    items = [%{title: "Example Item"}]
    requests = [%Crawly.Request{url: "https://example.com/next-page"}]
    
    %Crawly.ParsedItem{items: items, requests: requests}
  end
end

Starting from version 0.15.0, you can define spiders directly as YML files via the Crawly Management interface. This approach reduces boilerplate for simple spiders that only require CSS selectors for data extraction and link following.

To use this feature:

1. Start the Crawly application.
2. Open the Crawly Management interface at localhost:4001.
3. Define your spider using the required YML structure (see YML Spider Structure).
4. Use the Preview button in the interface to verify the extracted data format.
5. Save the spider to enable scheduling via the Management interface.

name: BooksSpiderForTest
base_url: "https://books.toscrape.com/"
start_urls:
    - "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
fields:
    - name: title
    selector: ".product_main"
    - name: price
    selector: ".product_main .price_color"
links_to_follow:
    - selector: "a"
    attribute: "href"