scrapli Documentation
repository·main·Indexed 20 days ago
https://github.com/carlmontanari/scrapliA fast, flexible, sync/async Python 3.10+ screen scraping client for network devices such as routers, switches, and firewalls. It supports connectivity via Telnet, SSH, and NETCONF, utilizing various transports including a pty-wrapper around binaries, a custom Zig telnet driver, and a libssh2 wrapper. Key features include support for input modes, interactive prompt handling via read_with_callbacks, NETCONF RPC operations, and integration with TextFSM for parsing unstructured text output.
What's inside scrapli
- scrapli is a Python 3.10+ library designed for connecting to network devices (such as routers, switches, and firewalls). It provides mechanisms to interact with these devices using several protocols, including Telnet, SSH, and NETCONF.
How to use Input Modes for varying privilege levels
mainMany network devices require interacting with different privilege levels or 'modes' (e.g., moving from user EXEC mode to global configuration mode). Scrapli allows you to define these modes within your device definitions (or custom definitions) so you can switch between them to send inputs at the appropriate level.
To use this feature, ensure your device definition includes the necessary mode transitions, then use the CLI or API to target specific modes during your session.
How libscrapli handles Async IO
mainlibscrapli is designed for high efficiency across synchronous and asynchronous environments:
- Internal Read Loop: libscrapli runs a dedicated pthread for a "read loop" that constantly consumes data from the transport and stages it into a queue.
- Non-blocking Transports: Transports use
epoll/kqueueto await readable data without blocking. - Consumption Patterns:
- Python (async/await): Operations are queued and pollable via an operation ID. Python awaits these by selecting on a file descriptor that notifies when an operation is complete.
- Go: Natively asynchronous via the Go runtime.
- Python (sync): Operations are polled on an interval with a simple backoff timer.
This architecture allows for cancellable reads and efficient resource usage regardless of whether the consumer is using synchronous or asynchronous patterns.
Configure Platform Definitions
mainA platform definition is a YAML file that tells Scrapli how to interact with a specific CLI device. It defines how to match prompts, manage privilege levels (modes), and handle connection lifecycles.
Key components include:
prompt_pattern: A PCRE2 regular expression that matches device prompts.prompt_excludes: Substrings that, if present, prevent a match from being considered a valid prompt.modes: An array defining different privilege levels (e.g.,exec,configuration). Each mode specifies its ownprompt_patternandaccessible_modes(how to move to other modes).on_open_instructions: Commands executed immediately after connection (e.g., disabling pagination).on_close_instructions: Commands executed before closing the connection (e.g.,exitorquit).failure_indicators: Strings that, if found in output, mark an operation as failed (annotated, not a hard error).
Understand Scrapli Transports
mainScrapli uses transports to handle the actual sending and receiving of data to/from a server. There are three supported transports:
telnet: A custom telnet driver (written in Zig) used for connecting to CLI devices via telnet.bin: The default transport for non-telnet connections. It is a pty-wrapper around a binary (typically/bin/ssh). This provides native support for OpenSSH features likeProxyJump,ControlPersist, ciphers, and key exchanges. The binary can be swapped for other tools likedocker execor serial port binaries.ssh2: A Zig wrapper aroundlibssh2with OpenSSL crypto. It does not require OpenSSH to be installed, making it ideal for containerized environments, though it only exposes minimumlibssh2features.
Use read_with_callbacks for long-running outputs and event-driven interactions
mainThe
read_with_callbacksmethod allows you to handle device interactions by triggering specific function calls based on the content received during a read operation. This is particularly useful for:- Connecting to device consoles during boot up.
- Zero-touch-provisioning (ZTP) workflows.
- Tailing logs or other long-running outputs.
- Triggering specific logic immediately when certain patterns appear in the stream.
How the libscrapli FFI loader works
mainThescrapliFFI (Foreign Function Interface) loader is responsible for locating and loading thelibscraplishared object. By default, it attempts to load the shared object from thescrapli.libpackage directory. If you need to use a different shared object, you can provide an override path (the specific mechanism for this override depends on the library's configuration settings).Send inputs to a device using the Cli object
mainThe
Cliobject provides several methods to send inputs (data/commands) to a network device. Depending on your use case, you can send a single input, multiple inputs, or inputs loaded from a file.Available methods:
send_input: Sends a single input.send_inputs: Sends multiple inputs.send_inputs_from_file: Sends inputs read from a file.
Install and build libscrapli shared objects
mainThe
scrapli.libpackage contains thelibscraplishared objects required for operation. While these files are not stored in version control, they are populated during wheel installation orsdistinstallation.If you are developing
scraplior want to use it directly from source, you must build the shared object for your specific platform. Runningpip install .orpip install -e .will build the shared object and place it in thescrapli.libdirectory.# For standard installation pip install . # For editable development installation pip install -e .Migrate from legacy scrapli to libscrapli-based libraries
mainThe modern
scrapliandscrapligolibraries are thin wrappers aroundlibscrapli(written in Zig). When migrating from legacy versions, note the following architectural shifts:- Core Engine: All core logic now resides in
libscrapli. Python and Go packages act as idiomatic wrappers. - Driver Model: The distinction between
genericandnetworkdrivers has been removed. There is no longer a concept of sending "configurations"; instead, you send inputs at any desired "mode". - Platform Selection: When using a
Cliconnection, you are no longer required to specify aplatform. A generic default is auto-selected, though providing a specific platform is still recommended for proper pagination and device-specific behavior. - Privilege Levels to Modes: Privilege levels have been replaced by "modes". Most definitions will attempt to acquire a sane default/initial mode upon connection.
- Authentication: The
auth_secondaryparameter (formerly used for "enable" passwords) has been replaced by lookups. Lookups consist of an array of lookup keys and values. You reference them in a definition using the syntax__lookup::key_name(e.g.,__lookup::enable). - SSH Configuration: The
bintransport now honors your local SSH config files by default (it no longer uses-F /dev/null).
- Core Engine: All core logic now resides in
Handle simple interactive prompts in Scrapli
mainFor simple semi-interactive device prompts—such as confirmation requests when writing a configuration or deleting a file—you can use the basic interaction handling capabilities provided by Scrapli. This approach is intended for straightforward scenarios where the device expects a simple response to a prompt.
For more complex or elaborate interactive scenarios (e.g., multi-step dialogues or conditional logic), use the
read_with_callbacksmethod instead.Install scrapligo for Go
mainTo include
scrapligoin your Go project, use the standard Go toolchain:go get github.com/scrapli/scrapligo/v2Or pin to a specific tag/commit:
go get github.com/scrapli/scrapligo/v2@v2.0.0Important: This command fetches the source code but does not install
libscrapli. You must managelibscrapliusing one of the following methods:1. Automatic Fetching
If you do nothing, the first time you run a program using
scrapligo,libscrapliwill be fetched and cached.- Cache Path: Controlled by the
LIBSCRAPLI_CACHE_PATHenvironment variable. Defaults toXDG_CACHE_HOME(if set),$HOME/.cache/scraplion Linux, or$HOME/Library/Caches/scraplion Darwin. - Requirements: The host must have access to GitHub. If using a development version pinned to a commit hash, you must have Docker available as
libscrapliwill be built in a container.
2. Manual Installation
If you are building containers or working in offline environments, you should provide
libscrapliyourself:- Using the helper script: If you have cloned the
scrapligoproject, run:go run build/write_libscrapli_to_cache/main.go - Manual download: Download the appropriate platform build from the libscrapli releases and place it in the cache path or set the override path via environment variables.
- Cache Path: Controlled by the