chromiumoxide

repository·main·Indexed 23 days ago

https://github.com/mattsse/chromiumoxide

A high-level, asynchronous Rust library for controlling Chrome or Chromium via the Chrome DevTools Protocol (CDP). It supports both headless and full-UI browser modes, provides a type-safe API for interacting with web pages, and includes a BrowserFetcher to automatically download and install compatible browser binaries.

Tokens
13.2K
Snippets
16
Records
89
Agent score
80%

What's inside chromiumoxide

  1. Launch and control a browser with chromiumoxide

    main

    chromiumoxide provides an async API to control Chrome or Chromium via the DevTools Protocol. To use it, you must launch a Browser and spawn a background task to continuously poll the handler which drives the websocket connection.

    Key steps:

    1. Create a BrowserConfig (e.g., using .with_head() for non-headless mode).
    2. Call Browser::launch(config) to get a Browser instance and a handler.
    3. Spawn a tokio task to poll handler.next().
    4. Use browser.new_page(url) to interact with pages and elements.
    use futures::StreamExt;
    use chromiumoxide::browser::{Browser, BrowserConfig};
    
    #[tokio::main]
    async fn main() -> Result<(), Box<dyn std::error::Error>> {
        // create a `Browser` that spawns a `chromium` process running with UI (`with_head()`, headless is default)
        // and the handler that drives the websocket etc.
        let (mut browser, mut handler) =
            Browser::launch(BrowserConfig::builder().with_head().build()?).await?;
    
        // spawn a new task that continuously polls the handler
        let handle = tokio::spawn(async move {
            while let Some(h) = handler.next().await {
                if h.is_err() {
                    break;
                }
            }
        });
    
        // create a new browser page and navigate to the url
        let page = browser.new_page("https://en.wikipedia.org").await?;
    
        // find and click the search toggle button to reveal the search bar
        page.find_element(".search-toggle").await?.click().await?;
    
        // find the search bar type into the search field and hit `Enter`,
        // this triggers a new navigation to the search result page
        page.find_element("input[name='search']")
            .await?
            .click()
            .await?
            .type_str("Rust programming language")
            .await?
            .press_key("Enter")
            .await?;
    
        let html = page.wait_for_navigation().await?.content().await?;
    
        browser.close().await?;
        handle.await?;
        Ok(())
    }
  2. Automatically download and install Chromium using the Fetcher

    main

    By default, chromiumoxide looks for an existing Chromium installation. You can enable the fetcher feature to download and install one automatically for supported platforms.

    To use the fetcher, you must also enable one of the following TLS features and one of the following zip features in your Cargo.toml:

    • TLS: rustls or native-tls
    • Zip: zip0 or zip8

    Use BrowserFetcher to download the binary and retrieve its path for BrowserConfig.

    use std::path::Path;
    
    use futures::StreamExt;
    use chromiumoxide::browser::{BrowserConfig};
    use chromiumoxide::fetcher::{BrowserFetcher, BrowserFetcherOptions};
    
    #[tokio::main]
    async fn main() -> Result<(), Box<dyn std::error::Error>> {
        let download_path = Path::new("./download");
        tokio::fs::create_dir_all(&download_path).await?;
        let fetcher = BrowserFetcher::new(
            BrowserFetcherOptions::builder()
                .with_path(&download_path)
                .build()?,
        );
        let info = fetcher.fetch().await?;
    
        let config = BrowserConfig::builder()
            .chrome_executable(info.executable_path)
            .build()?;
        // ...
    }
  3. Evaluate JavaScript in the page context

    main

    The Page API provides several ways to execute JavaScript:

    1. evaluate: The most flexible method. It attempts to detect if the input is a function or an expression. If eval_as_function_fallback is set to true in EvaluateParams, it will retry as a function if the initial expression evaluation returns a Function type.
    2. evaluate_expression: Strictly evaluates the input as an expression. Use this when you want to ensure no function detection occurs.
    3. evaluate_function: Strictly executes the input as a function. This is useful for passing arguments to a function declaration.

    To handle asynchronous code, you can pass an async function to evaluate_function, and the method will wait for the promise to resolve.

  4. Understand the Handler abstraction

    main

    The Handler is the central event loop of chromiumoxide. It monitors the state of the Chromium browser, manages the websocket connection, and drives all requests and events.

    It is responsible for:

    • Managing Targets (tabs, workers, etc.) and their associated Sessions.
    • Handling Page creation and navigation lifecycles.
    • Dispatching CDP (Chrome DevTools Protocol) events to registered event listeners.
    • Managing command timeouts to prevent hanging requests.
    • Coordinating between external commands (sent via channels) and internal CDP responses.

    Note: The Handler implements Stream. It must be polled (e.g., using tokio::spawn or within an async loop) to actually process messages and drive the browser.

  5. Inspect Protocol, Domain, and Type definitions in PDL

    main
    The chromiumoxide_pdl package provides core types for representing the Chrome DevTools Protocol (CDP). The primary entry point is the Protocol struct, which contains a version and a collection of Domain objects. Each Domain encapsulates the specific capabilities of a CDP module, including its commands, events, and types.
  6. Manage Browser Contexts and Incognito mode

    main

    Chromiumoxide supports browser contexts to isolate sessions.

    • Incognito Mode: If not already configured via BrowserConfig, you can call start_incognito_context() to create a new incognito session. This session will not share cookies or cache with other contexts. Use quit_incognito_context() to dispose of it.
    • Manual Contexts: You can manually create new contexts using create_browser_context(params) and dispose of them using dispose_browser_context(id). All new pages created while in an incognito context will run within that context.
  7. Understand DOMWorld and DOMWorldKind

    main

    In chromiumoxide, a Page can contain multiple execution contexts. DOMWorld represents a context for JavaScript execution within a Frame.

    There are two primary kinds of worlds tracked for each frame, represented by the DOMWorldKind enum:

    1. Main (DOMWorldKind::Main): The default execution context of a frame, created when the frame is attached to the DOM.
    2. Secondary (DOMWorldKind::Secondary): Additional execution contexts, such as those created by browser extensions' content scripts. These provide isolated worlds with universal access.

    Execution contexts can also be found in Web Workers.

  8. Understand CDP Message and Command types

    main

    The chromiumoxide_types crate defines the core data structures for communicating with Chromium via the Chrome DevTools Protocol (CDP).

    Key abstractions include:

    • MethodCall: A request sent by the client to the server, identified by a unique CallId.
    • Command: A trait for request types. It links a request to its expected Response type via an associated type.
    • CommandResponse<T>: A successful response containing the result of type T.
    • CommandResult<T>: A type alias for Result<CommandResponse<T>, Error>, representing either a successful command execution or a server-side error.
    • Message<T>: An enum representing an incoming WebSocket message, which can be either a Response to a previous request or an Event emitted by the server.
    • Method: A trait for types that identify themselves via a CDP method string (e.g., DOM.removeNode). It provides helpers to split the identifier into a domain_name and a method_name.
  9. Understand Chrome DevTools Protocol (CDP) revisions

    main

    The Chrome DevTools Protocol (CDP) is not a stable API and changes over time in ways that may be backward incompatible. chromiumoxide uses specific CDP revisions that correspond to Chromium master commit positions.

    To ensure compatibility with your specific Chromium browser version, you can map a revision to a Chromium version using Chromium Dash. While using the latest revision is an option, using an older, stable CDP revision is often recommended to avoid breaking changes.

    The current built-in revision in this crate is CURRENT_REVISION.

  10. Manage Chromium processes and avoid zombie processes

    main

    When using Browser::launch, the library spawns a child process. To ensure the process is cleaned up properly and to avoid 'zombie' processes, follow these patterns:

    1. Graceful Close: Call browser.close().await to request the browser to shut down.
    2. Wait for Exit: After closing, call browser.wait().await to asynchronously wait for the process to exit completely.
    3. Non-blocking Check: Use browser.try_wait() to check if the process has already exited without blocking.
    4. Force Kill: If the browser fails to close, use browser.kill().await to forcibly terminate the process.

    Note: If a Browser instance is dropped without being closed manually, it will attempt to kill the child process in the background.

  11. Troubleshoot Chromium timeout issues

    main

    If a new Chromium instance is launched but immediately times out, ensure that your system's Chromium language settings are set to English.

    chromiumoxide relies on parsing the debugging port from the Chromium process output, and this parsing logic is currently limited to English output.