DOCSEARCHbyAlgolia
Submit Docs

Mitogen Documentation

repository·master·Indexed 25 days ago

https://github.com/mitogen-hq/mitogen

A high-performance Python library designed to improve the efficiency and speed of Python execution, specifically for optimizing SSH-based automation and parallel task execution. It provides a Router API to construct execution contexts via SSH, sudo, Docker, kubectl, podman, buildah, and local subprocesses, allowing users to execute Python functions and system commands on remote hosts without complex shell scripting.

Tokens
9.8K
Snippets
12
Records
62
Agent score
82%

What's inside Mitogen

  1. Extend Mitogen via Internal APIs

    master
    Mitogen provides internal APIs for users who need to modify or extend the library. Note that these APIs are subject to rapid change, even across minor releases. Use these components to build custom connection types, custom import mechanisms, or specialized process management logic.

  2. How disconnect propagation and error handling works

    master

    To prevent contexts from hanging indefinitely when a remote connection fails, Mitogen implements a disconnect propagation mechanism.

    The Disconnect Flow

    1. Tracking: mitogen.core.Router records the destination context ID of every message received on a stream.
    2. Detection: When a DEL_ROUTE message is generated or received, mitogen.parent.RouteMonitor identifies every stream that previously communicated with that route.
    3. Notification: The DEL_ROUTE handler finds any mitogen.core.Context in the local process corresponding to the disappearing route and fires a disconnected event on it.
    4. Recovery: Consumers (such as mitogen.core.Receiver) can subscribe to the disconnected event to abort threads that are waiting for replies from the disconnected context.

  3. Concurrency and deduplication in Mitogen imports

    master

    Mitogen ensures that duplicate GET_MODULE requests are never issued to a parent. Each layer in the context tree deduplicates requests and synchronizes local threads and descendants on LOAD_MODULE responses.

    • Local Threads: mitogen.core.Importer executes under the runtime importer lock. It checks for cached source or in-flight GET_MODULE requests. If none exist, it starts a request and waits via a callback.
    • Child Requests: When a GET_MODULE is received from a child, mitogen.master.ModuleForwarder checks for cached source or in-flight requests to the parent. Once the source arrives, it issues LOAD_MODULE messages to the child, including any required dependencies.

  4. How Mitogen ensures message authenticity and trust

    master

    Mitogen implements a trust chain to prevent unauthorized message injection (e.g., a downstream context attempting to impersonate the master).

    Source Verification

    Before dispatching a message, mitogen.core.Router performs a verification step:

    • It looks up the mitogen.core.Stream that should be used to send responses to the context ID listed in the message's auth_id field.
    • If the stream used to receive the message does not match the expected stream for that auth_id, the message is discarded and a warning is logged.

    auth_id vs src_id

    • auth_id: Used for security and trust. It allows privileged functionality like CALL_FUNCTION <mitogen.core.CALL_FUNCTION> to make trust decisions. It can be used to grant privileges to contexts that do not follow the natural tree hierarchy (e.g., siblings communicating or a mitogen.unix client connecting to an existing tree).
    • src_id: Represents the actual source ID of the message.

  5. How the Mitogen Module Importer works

    master
    The mitogen.core.Importer intercepts Python import statements via sys.meta_path. When a module is requested, the importer initiates an RPC to the parent context using a GET_MODULE request to fetch the source code. If the parent does not have the module, the request is forwarded upstream, avoiding duplicate requests for the same module across threads and child contexts.

  6. Design principles for Mitogen programs

    master

    When writing programs that span multiple hosts and privilege domains with Mitogen, follow these principles:

    1. Handle Asynchrony: Expect communication to fail at any moment due to network unreliability. Design for unexpected failures.
    2. Assume Untrusted Data: A parent must always assume data received from a child is suspect. Never base privileged control decisions (like forming a command to execute) on strings received from a child.
    3. Separate Program and Data: Avoid Python idioms that rely on manipulating functions or closures as data (e.g., passing a lambda that is closed over local program state). This is difficult and unsafe in a distributed environment.

  7. How message routing works in Mitogen

    master

    Mitogen uses a tree-based routing system to facilitate communication between contexts. Routing relies on the mitogen.core.Router to find paths between context IDs.

    Routing Logic

    When a mitogen.core.Router receives a message, it attempts to route it using the following priority:

    1. Direct Match: If a stream is directly connected to the target ID, the message is forwarded down that stream.
    2. Explicit Route: If the master has sent an ADD_ROUTE <mitogen.core.ADD_ROUTE> message associating a stream with a target ID, the message is forwarded down that stream.
    3. Upward Propagation: If no direct or explicit route is found, the message is forwarded up to the immediate parent. This continues recursively until a parent is reached that knows how to route the message down the tree.

    Route Management

    • Establishing Routes: When a parent creates a new child, it sends an ADD_ROUTE <mitogen.core.ADD_ROUTE> message towards its parent. This propagates recursively up to the root.
    • Route Cleanup: If a stream is disconnected, parents trigger DEL_ROUTE messages that propagate upstream for every route associated with that stream.
    • Ordering Guarantee: Because Mitogen streams are strictly ordered, a parent will always receive the ADD_ROUTE message describing how to reply to a new child before it receives any messages from that child.

  8. How Import Preloading reduces network round-trips

    master

    Mitogen optimizes imports by scanning module bytecode for IMPORT_NAME opcodes to identify dependencies. It then pre-loads these dependencies into the child context using LOAD_MODULE messages before sending the requested module, reducing latency.

    Optimization Scenarios:

    • Package Requests: If a child requests a package (e.g., django), the master identifies dependencies within that package (e.g., django.utils) that are likely missing in the child and pre-loads them. This can turn 4 round-trips into 1.
    • Sub-module Requests: If a child has already loaded a package, and subsequently requests a sub-module (e.g., django.db), the parent pre-loads all known missing dependencies of that sub-module. This can turn 17+ round-trips into 1.

  9. How Mitogen bootstraps remote Python processes

    master

    Mitogen achieves remote execution without pre-installed software by using a multi-stage bootstrapping process:

    1. The UNIX First Stage: A small, zlib-compressed, and base64-encoded Python command is sent to the host. This command executes mitogen.parent.Connection._first_stage to handle decompression and setup.
    2. Forking: The first stage forks. The parent re-executes Python with a clean argv[0] and connects its stdin to a UNIX pipe. The child handles the decompression of the actual Mitogen payload.
    3. Bootstrap Payload: The payload contains the source code for mitogen.core, optimized via mitogen.master.minimize_source (stripping docstrings and comments) to reduce size.
    4. Synthetic Package Generation: Once mitogen.core is loaded, Mitogen rearranges sys.modules to create a synthetic mitogen package. It also deletes sys.modules['__main__'] so that remote imports of the master's __main__ module are correctly satisfied.
    5. Environment Setup: The child configures logging (matching the master's level), installs a custom mitogen.core.Importer into sys.meta_path, and redirects stdin/stdout/stderr using mitogen.core.IoLogger to prevent subprocesses from corrupting the communication stream.

  10. Understand the Router class

    master

    Mitogen uses Router classes to manage communication paths. There are different implementations of the Router depending on the component's role in the architecture:

    • mitogen.core.Router: The base router implementation.
    • mitogen.parent.Router: A router used by parent processes to manage child connections.
    • mitogen.master.Router: A router used by the master process, which can be initialized with a broker (e.g., mitogen.master.Router(broker=None)).

  11. Use Mitogen signals for component decoupling

    master

    Mitogen uses a simplistic signal mechanism to decouple components. When an instance fires a signal, registered callback functions are executed.

    WARNING: Signals execute on the Broker thread and do not have exception handling. User-defined signal handlers should be extremely careful; bugs in these handlers can cause crashes or hangs that prevent the broker from forwarding logs or ensuring a clean shutdown.

  12. How Mitogen avoids negative imports

    master

    In Python 2.x, relative imports can cause many 'negative imports' (requests for modules that do not exist, e.g., mypkg.sys instead of sys), leading to unnecessary network round-trips.

    To solve this, when a package is imported, the master sends a list of child modules known to exist. The mitogen.core.Importer checks if a requested module belongs to a known package; if it does, it ignores the request if the module is not in the provided enumeration of child modules.