scrapli Documentation

repository·main·Indexed 20 days ago

https://github.com/carlmontanari/scrapli

A fast, flexible, sync/async Python 3.10+ screen scraping client for network devices such as routers, switches, and firewalls. It supports connectivity via Telnet, SSH, and NETCONF, utilizing various transports including a pty-wrapper around binaries, a custom Zig telnet driver, and a libssh2 wrapper. Key features include support for input modes, interactive prompt handling via read_with_callbacks, NETCONF RPC operations, and integration with TextFSM for parsing unstructured text output.

Tokens
15.1K
Snippets
25
Records
47
Agent score
72%

What's inside scrapli

  1. Overview of scrapli

    main
    scrapli is a Python 3.10+ library designed for connecting to network devices (such as routers, switches, and firewalls). It provides mechanisms to interact with these devices using several protocols, including Telnet, SSH, and NETCONF.
  2. How to use Input Modes for varying privilege levels

    main

    Many network devices require interacting with different privilege levels or 'modes' (e.g., moving from user EXEC mode to global configuration mode). Scrapli allows you to define these modes within your device definitions (or custom definitions) so you can switch between them to send inputs at the appropriate level.

    To use this feature, ensure your device definition includes the necessary mode transitions, then use the CLI or API to target specific modes during your session.

  3. How libscrapli handles Async IO

    main

    libscrapli is designed for high efficiency across synchronous and asynchronous environments:

    • Internal Read Loop: libscrapli runs a dedicated pthread for a "read loop" that constantly consumes data from the transport and stages it into a queue.
    • Non-blocking Transports: Transports use epoll/kqueue to await readable data without blocking.
    • Consumption Patterns:
      • Python (async/await): Operations are queued and pollable via an operation ID. Python awaits these by selecting on a file descriptor that notifies when an operation is complete.
      • Go: Natively asynchronous via the Go runtime.
      • Python (sync): Operations are polled on an interval with a simple backoff timer.

    This architecture allows for cancellable reads and efficient resource usage regardless of whether the consumer is using synchronous or asynchronous patterns.

  4. Configure Platform Definitions

    main

    A platform definition is a YAML file that tells Scrapli how to interact with a specific CLI device. It defines how to match prompts, manage privilege levels (modes), and handle connection lifecycles.

    Key components include:

    • prompt_pattern: A PCRE2 regular expression that matches device prompts.
    • prompt_excludes: Substrings that, if present, prevent a match from being considered a valid prompt.
    • modes: An array defining different privilege levels (e.g., exec, configuration). Each mode specifies its own prompt_pattern and accessible_modes (how to move to other modes).
    • on_open_instructions: Commands executed immediately after connection (e.g., disabling pagination).
    • on_close_instructions: Commands executed before closing the connection (e.g., exit or quit).
    • failure_indicators: Strings that, if found in output, mark an operation as failed (annotated, not a hard error).
  5. Understand Scrapli Transports

    main

    Scrapli uses transports to handle the actual sending and receiving of data to/from a server. There are three supported transports:

    • telnet: A custom telnet driver (written in Zig) used for connecting to CLI devices via telnet.
    • bin: The default transport for non-telnet connections. It is a pty-wrapper around a binary (typically /bin/ssh). This provides native support for OpenSSH features like ProxyJump, ControlPersist, ciphers, and key exchanges. The binary can be swapped for other tools like docker exec or serial port binaries.
    • ssh2: A Zig wrapper around libssh2 with OpenSSL crypto. It does not require OpenSSH to be installed, making it ideal for containerized environments, though it only exposes minimum libssh2 features.
  6. Use read_with_callbacks for long-running outputs and event-driven interactions

    main

    The read_with_callbacks method allows you to handle device interactions by triggering specific function calls based on the content received during a read operation. This is particularly useful for:

    • Connecting to device consoles during boot up.
    • Zero-touch-provisioning (ZTP) workflows.
    • Tailing logs or other long-running outputs.
    • Triggering specific logic immediately when certain patterns appear in the stream.
  7. How the libscrapli FFI loader works

    main
    The scrapli FFI (Foreign Function Interface) loader is responsible for locating and loading the libscrapli shared object. By default, it attempts to load the shared object from the scrapli.lib package directory. If you need to use a different shared object, you can provide an override path (the specific mechanism for this override depends on the library's configuration settings).
  8. Send inputs to a device using the Cli object

    main

    The Cli object provides several methods to send inputs (data/commands) to a network device. Depending on your use case, you can send a single input, multiple inputs, or inputs loaded from a file.

    Available methods:

    • send_input: Sends a single input.
    • send_inputs: Sends multiple inputs.
    • send_inputs_from_file: Sends inputs read from a file.
  9. Install and build libscrapli shared objects

    main

    The scrapli.lib package contains the libscrapli shared objects required for operation. While these files are not stored in version control, they are populated during wheel installation or sdist installation.

    If you are developing scrapli or want to use it directly from source, you must build the shared object for your specific platform. Running pip install . or pip install -e . will build the shared object and place it in the scrapli.lib directory.

    # For standard installation
    pip install .
    
    # For editable development installation
    pip install -e .
  10. Migrate from legacy scrapli to libscrapli-based libraries

    main

    The modern scrapli and scrapligo libraries are thin wrappers around libscrapli (written in Zig). When migrating from legacy versions, note the following architectural shifts:

    • Core Engine: All core logic now resides in libscrapli. Python and Go packages act as idiomatic wrappers.
    • Driver Model: The distinction between generic and network drivers has been removed. There is no longer a concept of sending "configurations"; instead, you send inputs at any desired "mode".
    • Platform Selection: When using a Cli connection, you are no longer required to specify a platform. A generic default is auto-selected, though providing a specific platform is still recommended for proper pagination and device-specific behavior.
    • Privilege Levels to Modes: Privilege levels have been replaced by "modes". Most definitions will attempt to acquire a sane default/initial mode upon connection.
    • Authentication: The auth_secondary parameter (formerly used for "enable" passwords) has been replaced by lookups. Lookups consist of an array of lookup keys and values. You reference them in a definition using the syntax __lookup::key_name (e.g., __lookup::enable).
    • SSH Configuration: The bin transport now honors your local SSH config files by default (it no longer uses -F /dev/null).
  11. Handle simple interactive prompts in Scrapli

    main

    For simple semi-interactive device prompts—such as confirmation requests when writing a configuration or deleting a file—you can use the basic interaction handling capabilities provided by Scrapli. This approach is intended for straightforward scenarios where the device expects a simple response to a prompt.

    For more complex or elaborate interactive scenarios (e.g., multi-step dialogues or conditional logic), use the read_with_callbacks method instead.

  12. Install scrapligo for Go

    main

    To include scrapligo in your Go project, use the standard Go toolchain:

    go get github.com/scrapli/scrapligo/v2

    Or pin to a specific tag/commit:

    go get github.com/scrapli/scrapligo/v2@v2.0.0

    Important: This command fetches the source code but does not install libscrapli. You must manage libscrapli using one of the following methods:

    1. Automatic Fetching

    If you do nothing, the first time you run a program using scrapligo, libscrapli will be fetched and cached.

    • Cache Path: Controlled by the LIBSCRAPLI_CACHE_PATH environment variable. Defaults to XDG_CACHE_HOME (if set), $HOME/.cache/scrapli on Linux, or $HOME/Library/Caches/scrapli on Darwin.
    • Requirements: The host must have access to GitHub. If using a development version pinned to a commit hash, you must have Docker available as libscrapli will be built in a container.

    2. Manual Installation

    If you are building containers or working in offline environments, you should provide libscrapli yourself:

    • Using the helper script: If you have cloned the scrapligo project, run:
      go run build/write_libscrapli_to_cache/main.go
    • Manual download: Download the appropriate platform build from the libscrapli releases and place it in the cache path or set the override path via environment variables.