Pythonx

repository·main·Indexed 43 days ago

https://github.com/livebook-dev/pythonx

An Elixir library that embeds a Python interpreter into the BEAM process using NIFs. It provides seamless data conversion, code evaluation via `Pythonx.eval/3` and the `~PY` sigil, and dependency management using the `uv` package manager via `Pythonx.uv_init/2`. It includes support for remote evaluation with `Pythonx.remote_eval/4`, a `Pythonx.Encoder` protocol for custom data structures, and `Pythonx.decode/1` for converting Python objects back to Elixir terms.

Tokens
4.3K
Snippets
14
Records
19
Agent score
56%

What's inside Pythonx

  1. Understand Pythonx concurrency and the GIL

    main

    Pythonx embeds the Python interpreter in the same OS process as your Elixir application using Erlang NIFs.

    Concurrency Implications: Because of Python's Global Interpreter Lock (GIL), calling Pythonx from multiple Elixir processes does not provide true parallel execution of Python code. This can create bottlenecks.

    When to use Pythonx vs other methods:

    • Use Pythonx when: You need tight integration, easy data conversion between Elixir and Python, or when using libraries like numpy that release the GIL during heavy CPU tasks or I/O.
    • Use System.cmd/3 or Ports when: You need to run multiple independent Python programs concurrently without GIL contention.
  2. Configure Pythonx for an Elixir application

    main

    For permanent applications, add {:pythonx, "~> 0.4.0"} to your deps. Instead of manual initialization, configure the interpreter in config/config.exs using the :uv_init key. This ensures downloads happen at compile time and the interpreter initializes automatically on boot. This configuration is compatible with Elixir releases.

    # config/config.exs
    import Config
    
    config :pythonx, :uv_init,
      pyproject_toml: """
      [project]
      name = "project"
      version = "0.0.0"
      requires-python = "==3.13.*"
      dependencies = [
        "numpy==2.2.2"
      ]
      """
    
    # Optionally specify a uv version
    config :pythonx, :uv_init, 
      uv_version: "0.7.21"
  3. Install and use Pythonx in a script

    main

    To use Pythonx in a standalone Elixir script, add it to your Mix.install dependencies. You must initialize the interpreter using Pythonx.uv_init/1 with a pyproject.toml string to specify the Python version and required dependencies. Once initialized, you can evaluate Python code using Pythonx.eval/2 and decode the results using Pythonx.decode/1.

    Mix.install([
      {:pythonx, "~> 0.4.0"}
    ])
    
    Pythonx.uv_init("""
    [project]
    name = "project"
    version = "0.0.0"
    requires-python = "==3.13.*"
    dependencies = [
      "numpy==2.2.2"
    ]
    """)
    
    {result, globals} =
      Pythonx.eval(
        """
        y = 10
        x + y
        """,
        %{"x" => 1}
      )
    
    Pythonx.decode(result)
    #=> 11
  4. How Pythonx initialization and environment variables work

    main

    Pythonx can be initialized in two ways:

    1. Explicitly: Using Pythonx.uv_init/2.
    2. Via Environment Variables: If the environment variable PYTHONX_INIT_STATE is set, Pythonx will automatically initialize on boot (useful for FLAME nodes).

    Note on Environment Variables: Python's os.environ is initialized from the OS process environment. It does not reflect changes made via Elixir's System.put_env/2. To mirror Elixir environment variables to Python, you must set them on os.environ using Pythonx.eval/3.

  5. Implement the Pythonx.Encoder protocol for custom data structures

    main

    The Pythonx.Encoder protocol is used to convert Elixir data structures into Python objects. While the protocol provides built-in implementations for standard Elixir types (like Map, List, Integer, etc.), you can implement the protocol for your own structs or complex types.

    To implement the protocol, define an encode/2 function. For complex conversions, it is recommended to use Pythonx.eval/2 to execute Python code that constructs the desired object, passing necessary Elixir data as a context map.

    Important Note on Remote Objects: If you are working with Pythonx.Object instances that reside on a remote node, they cannot be encoded implicitly. You must call Pythonx.copy_remote_object/1 to retrieve a local copy before attempting to encode them.

    defmodule Complex do
      defstruct [:re, :im]
    end
    
    defimpl Pythonx.Encoder, for: Complex do
      def encode(complex, _encoder) do
        {result, %{}} =
          Pythonx.eval(
            """
            complex(re, im)
            """
            , %{"re" => complex.re, "im" => complex.im}
          )
    
        result
      end
    end
    
    # Usage:
    Pythonx.encode!(%Complex{re: 1, im: -1})
    # => #Pythonx.Object<(1-1j)>
  6. Use the Pythonx.Object struct to handle Python objects

    main

    The Pythonx.Object struct is an opaque container used to pass Python objects between Python and Elixir. It holds a reference to the underlying Python resource and metadata about its location.

    When inspecting a Pythonx.Object in an Elixir shell or via IO.inspect/2, the library performs a remote call to Python to retrieve its string representation (__repr__), allowing you to see the Python object's content directly in your Elixir console.

    #%Pythonx.Object{} is an opaque struct containing:
    # :resource - The underlying resource handle
    # :remote_info - Metadata about the remote object
  7. Install Python and dependencies using `Pythonx.uv_init/2`

    main

    Use Pythonx.uv_init/2 to install a Python interpreter and required dependencies using the uv package manager. You must provide a string containing the content of a pyproject.toml file. The configuration must include project.name and project.version fields. It is recommended to specify the Python version via project.requires-python.

    Pythonx.uv_init("""
    [project]
    name = "project"
    version = "0.0.0"
    requires-python = "==3.13.*"
    dependencies = [
      "numpy==2.2.2"
    ]
    """)
  8. Encode custom types using Pythonx.eval/2

    main

    When implementing Pythonx.Encoder for types that require specific Python library support (e.g., Explorer.DataFrame requiring polars), use Pythonx.eval/2 to run Python logic that transforms the Elixir data into a Python object.

    It is a best practice to wrap the Python import in a try/except block within the eval call and check the result in Elixir to provide a clear error message if the required Python package is missing.

    defimpl Pythonx.Encoder, for: Explorer.DataFrame do
      def encode(df, _encoder) do
        {result, %{}} =
          Pythonx.eval(
            """
            try:
              import polars
              result = polars.read_ipc(ipc)
            except ModuleNotFoundError:
              result = None
            """,
            %{"ipc" => Explorer.DataFrame.dump_ipc!(df)}
          )
    
        case Pythonx.decode(result) do
          %Pythonx.Object{} -> result
          nil ->
            raise Protocol.UndefinedError,
              protocol: @protocol,
              value: df,
              description: "cannot encode Explorer.DataFrame, because the polars Python package is not installed"
        end
      end
    end
  9. Use the ~PY sigil for dynamic evaluation

    main

    In dynamic environments like IEx or Livebook, you can use the ~PY sigil to evaluate Python code. This requires importing Pythonx. The sigil allows you to reference Elixir variables directly within the Python block, and the result of the block is returned as a #Pythonx.Object<...>.

    Note: The ~PY sigil does not work within standard Mix project code; in applications, you should manage Python globals explicitly via Pythonx.eval/2.

    import Pythonx
    
    x = 1
    
    ~PY"""
    y = 10
    result = x + y
    """
    
    result
    #=> #Pythonx.Object<11>
  10. Send data from Python to Elixir using pythonx.send_tagged_object

    main

    The Python module pythonx provides send_tagged_object(pid, tag, object) to allow Python code to send messages back to an Elixir process.

    • pid: A pythonx.PID object (passed into the evaluation via globals).
    • tag: A string that will appear as an atom in the Elixir message.
    • object: Any Python object.

    When the Elixir process receives the message, it arrives as a {tag, object} tuple, where object is a #Pythonx.Object<...>.

    Warnings:

    1. GIL Bottlenecks: If the Python evaluation holds the Global Interpreter Lock (GIL) for a long time, it can block other Elixir processes. It is recommended to run long-running evaluations in a single Elixir process.
    2. Decoding: Pythonx.decode/1 also requires the GIL. If the Python code is still holding the GIL, decoding the received object may be blocked.
    # Inside a Python evaluation
    import pythonx
    
    # pid is a pythonx.PID passed from Elixir
    pythonx.send_tagged_object(pid, "my_tag", [1, 2, 3])
  11. Use the `~PY` sigil for convenient Python evaluation

    main

    The ~PY sigil is a convenience macro for evaluating Python code. It automatically handles globals: any Elixir variables referenced in the Python code are automatically encoded and passed as globals, and any globals assigned in the Python code are defined as Elixir variables.

    Warning: This macro evaluates code at compile time. It is intended for dynamic environments like IEx or Livebook. For regular application code, use Pythonx.eval/3 to make dependency management explicit.

    import Pythonx
    
    x = 1
    ~PY"""
    y = 10
    x + y
    """
    # returns 11
    # y is now defined as #Pythonx.Object<10>
  12. Fetch Python and dependencies using Pythonx.Uv.fetch/3

    main

    Use Pythonx.Uv.fetch/3 to download and install a managed Python interpreter and its dependencies via uv. This function creates a project directory, writes the provided pyproject.toml content to it, and runs uv sync to set up the environment.

    Options:

    • force: (boolean) If true, deletes the existing project directory before fetching.
    • uv_version: (string) The version of uv to use. Defaults to default_uv_version() (currently 0.8.5).
    • native_tls: (boolean) Whether to use native TLS for downloads.
    • python: (string, optional) A specific Python version or path to use with uv.

    Environment Variables:

    • PYTHONX_CACHE_DIR: Sets the base directory for the uv cache and project storage. If not set, it defaults to the user's cache directory.
    # Example: Fetching dependencies with a specific Python version
    Pythonx.Uv.fetch("dependencies_content", false, [python: "3.11"])