procspawn

repository·master·Indexed 41 days ago

https://github.com/mitsuhiko/procspawn

A Rust crate for spawning child processes using a pattern similar to `thread::spawn`. It utilizes `serde` to serialize data and return values across process boundaries and can serialize panics. It includes a `Pool` abstraction for reusing persistent processes to reduce overhead and isolate crashing code. Supports macOS and Linux.

Tokens
5.3K
Snippets
22
Records
27
Agent score
51%

What's inside procspawn

  1. Initialize procspawn in your application

    master

    To use procspawn, you must call procspawn::init() as the very first operation in your main function.

    Because a spawned process works by re-executing the current binary and re-entering main, any code placed before the init() call will run in every child process. To avoid unintended side effects or redundant application setup in child processes, ensure all setup logic occurs after procspawn::init().

    fn main() {
        procspawn::init();
        // Application setup starts here.
    }
  2. How process pools work for reuse

    master
    The default spawn() function starts and stops a new process for every call, which can be expensive. For high-frequency spawning, use a Pool. A Pool keeps processes alive for reuse, allowing them to maintain state between calls and reducing the overhead of process creation.
  3. Use the Process Pool for task isolation

    master

    The Pool abstraction allows you to run closures in a pool of persistent processes. This is useful for isolating potentially crashing code, as the pool automatically restarts worker processes if they fail or crash.

    Key behaviors:

    • Automatic Restarts: If a process in the pool breaks, it is automatically replaced.
    • Lifecycle: When the Pool is dropped, all managed processes are killed.
    • Feature Requirement: This requires the pool feature to be enabled in your Cargo.toml.
    • Limitations: You cannot intercept the raw streams (stdin/stdout/stderr) of individual processes managed by the pool.
    // Example usage pattern
    let pool = Pool::new(4)?;
    
    // Spawn a task into the pool
    let handle = pool.spawn(args, my_function);
    
    // Wait for the result
    let result = handle.join()?;
  4. Customize serialization behavior using `in_ipc_mode()`

    master

    When implementing Serialize or Deserialize for custom types, you can use in_ipc_mode() to detect if the current operation is part of a procspawn IPC (Inter-Process Communication) transaction.

    This allows you to provide optimized or specialized serialization logic for IPC. For example, instead of serializing a large memory-mapped file's raw bytes, you can choose to serialize only the file path when in_ipc_mode() is true, allowing the receiving process to reopen the file locally.

  5. Quickstart: Spawn a function using the top-level spawn API

    master

    For simple use cases, use the top-level procspawn::spawn function. This uses a default Builder and is the most direct way to run a function in a new process.

    Note: You must call procspawn::init() early in your main() function to allow spawned functions to be invoked correctly.

    // call this early in your main() function
    procspawn::init();
    
    let data = vec![1, 2, 3, 4];
    let handle = procspawn::spawn(data, |data| {
        println!("Received data {:?}", &data);
        data.into_iter().sum::<i64>()
    });
    
    let result = handle.join().unwrap();
    assert_eq!(result, 10);
  6. Set up testing with `enable_test_support!`

    master

    To use procspawn with rusttest, you must:

    1. Enable the test-support feature flag.
    2. Call the procspawn::enable_test_support!() macro once at the top level of your test suite.

    When enabled, tests behave as if init() was called with default arguments, and procspawn registers a dummy helper process to manage spawning.

    procspawn::enable_test_support!();
    
    #[test]
    fn test_basic() {
        let handle = procspawn::spawn((1, 2), |(a, b)| a + b);
        let value = handle.join().unwrap();
        assert_eq!(value, 3);
    }
  7. Spawn processes with serializable data using procspawn::spawn

    master

    The procspawn::spawn function allows you to run a closure in a separate process, similar to thread::spawn.

    Key requirements and behaviors:

    • Input: The data passed to the spawned function must implement serde::Serialize and serde::Deserialize.
    • Output: The return value of the closure must also be serializable so it can be sent back to the parent process.
    • Error Handling: If the spawned function panics, the panic information is serialized across the process boundary. You can retrieve the result (or the panic) by calling .join() on the returned handle.
    • Join Handle: The join() method blocks the current thread until the spawned process completes and returns the result.
    let data = vec![1, 2, 3, 4];
    let handle = procspawn::spawn(data, |data| {
        println!("Received data {:?}", &data);
        data.into_iter().sum::<i64>()
    });
    let result = handle.join().unwrap();
  8. Reset captured panic information

    master

    Call reset_panic_info() to clear the thread-local storage containing the last captured PanicInfo. This is useful if you want to ensure that subsequent panics do not accidentally retrieve stale information from a previous panic event.

    procspawn::reset_panic_info();
  9. Shutdown and clean up a Process Pool

    master

    To properly manage the lifecycle of a Pool, use the following methods:

    • join(): Blocks until all currently active and queued jobs in the pool have finished.
    • shutdown(): Performs a join() to wait for work to complete, then calls kill() to terminate all processes.
    • kill(): Immediately and forcefully kills all processes in the pool. After calling kill(), the pool cannot be used again and calling spawn() will panic.
    • Automatic Cleanup: If the Pool object is dropped, kill() is called automatically.
  10. Inspect remote panic details with PanicInfo

    master

    If a SpawnError is identified as a panic, you can use the PanicInfo struct to retrieve information about where and why the remote process failed. PanicInfo is a cross-process equivalent of std::panic::PanicInfo.

    Available methods on PanicInfo:

    • message(): Returns the panic message as a &str.
    • location(): Returns an Option<&Location> describing the file, line, and column where the panic occurred.
    • backtrace(): Returns an Option<&backtrace::Backtrace> if the backtrace feature is enabled. Note that this backtrace is typically resolved before being sent across process boundaries.
    // Accessing location details from PanicInfo
    if let Some(loc) = panic_info.location() {
        println!("Panic in file: {}", loc.file());
        println!("Line: {}", loc.line());
        println!("Column: {}", loc.column());
    }
  11. Configure and spawn processes with Builder

    master

    The Builder struct provides a fluent API to configure the environment and standard streams of a new process before spawning it. You can chain methods to set environment variables, clear the environment, or configure stdin, stdout, and stderr using std::process::Stdio types.

    To spawn a process, call .spawn(args, func), where args is the input payload (must implement Serialize and DeserializeOwned) and func is the function to execute in the child process (must be a function pointer fn(A) -> R).

    use procspawn::Builder;
    use std::process::Stdio;
    
    let handle = Builder::new()
        .env("KEY", "VALUE")
        .envs([("VAR1", "VAL1"), ("VAR2", "VAL2")])
        .env_remove("SECRET")
        .env_clear()
        .stdin(Stdio::piped())
        .stdout(Stdio::piped())
        .stderr(Stdio::null())
        .spawn(my_args, my_function);
  12. Use `with_ipc_mode()` to wrap serialization logic

    master

    The with_ipc_mode<F>() function allows you to execute a closure within a context where in_ipc_mode() returns true. This is used to signal to serializers that the data being processed is intended for IPC.

    // Example usage pattern
    with_ipc_mode(|| {
        // Any serde calls made inside this closure will 
        // report `in_ipc_mode() == true`
        let serialized = serde_json::to_vec(&my_data).unwrap();
    });
    with_ipc_mode(|| {
        // Any serde calls made inside this closure will 
        // report `in_ipc_mode() == true`
        let serialized = serde_json::to_vec(&my_data).unwrap();
    });