LangChain NVIDIA

The OpenShellSandbox implements the LangChain BaseSandbox interface. It follows the sandbox-as-tool pattern: the agent (running on the host via ChatNVIDIA) remains outside the sandbox, while tools for shell commands, file uploads, and downloads are dispatched into a hardened, policy-enforced runtime. This prevents the agent itself from being exposed to the sandbox environment.

                  ┌──────────────────────────────┐
                  │   Host (your agent)          │
                  │                              │
                  │   ChatNVIDIA(...)            │
                  │       │                      │
                  │       ▼                      │
                  │   create_deep_agent(         │
                  │     model=...,               │
                  │     backend=OpenShellSandbox │
                  │   )                          │
                  └──────────────┬───────────────┘
                                 │  gRPC (mTLS)
                                 ▼
                  ┌──────────────────────────────┐
                  │   OpenShell Gateway          │
                  └──────────────┬───────────────┘
                                 │  ExecSandbox stream
                                 ▼
                  ┌──────────────────────────────┐
                  │   Sandbox compute driver     │
                  │   Docker/Podman/K8s/MicroVM  │
                  │   policy-enforced execution   │
                  └──────────────────────────────┘

Switchyard acts as a routing middleware for LangChain Deep Agents. It allows a Deep Agent to route individual model calls through a Switchyard libsy algorithm to select the most appropriate model target (e.g., an 'efficient' model for simple tasks or a 'capable' model for complex ones).

The Mental Model:

1. The Deep Agent maintains control over the agent loop, tools, state, and middleware composition.
2. The SwitchyardRoutingMiddleware intercepts model turns and uses a configured libsy Algorithm to make a routing decision.
3. The Algorithm selects a LangChainLlmClient target.
4. The LangChainLlmClient adapts the selected LangChain BaseChatModel to the interface required by Switchyard.
5. The Target Model processes the request and returns an AIMessage containing Switchyard decision metadata.

Because the middleware uses LangChain's dynamic-model pattern (request.override(model=...)), existing tool bindings, callbacks, and LangChain tracing continue to function normally.

Models such as nvidia/neva-22b support multimodal inputs (text and images).

Important: Local file paths are no longer accepted directly. You must explicitly read local files and pass them as base64 encoded URIs in the format data:image/...;base64,....

NVIDIA APIs also uniquely support images provided as base64 strings within HTML <img> tags.

import requests
import base64
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_core.messages import HumanMessage

# 1. Using standard image_url format (URL or Base64 URI)
image_url = "https://picsum.photos/seed/kitten/300/200"
llm = ChatNVIDIA(model="nvidia/neva-22b")
llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": image_url}},
        ])
    ])

# 2. Using Base64 encoded string
image_content = requests.get(image_url).content
b64_string = base64.b64encode(image_content).decode('utf-8')
llm.invoke(
    [
        HumanMessage(content=[
            {"type": "text", "text": "Describe this image:"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_string}"}},
        ])
    ])

# 3. Using HTML img tag (NVIDIA unique support)
base64_with_mime_type = f"data:image/png;base64,{b64_string}"
llm.invoke(
    f"What's in this image?\n<img src=\"{base64_with_mime_type}\" />"
)

Switchyard's libsy Python bindings must be built from the Switchyard source tree before installing this integration. Follow these steps:

1. Build libsy bindings:

   git clone https://github.com/NVIDIA-NeMo/Switchyard.git
   python -m pip install -e ./Switchyard

2. Install the integration: Clone the langchain-nvidia repository and install the desired extra:

   • Generic (adapter only): python -m pip install -e "./langchain-nvidia/libs/switchyard"
   • With Deep Agents: python -m pip install -e "./langchain-nvidia/libs/switchyard[deepagents]"
   • With Deep Agents & OpenRouter: python -m pip install -e "./langchain-nvidia/libs/switchyard[openrouter]"

git clone https://github.com/NVIDIA-NeMo/Switchyard.git
python -m pip install -e ./Switchyard

git clone https://github.com/langchain-ai/langchain-nvidia.git
python -m pip install -e "./langchain-nvidia/libs/switchyard[openrouter]"

You can compile your graph in three distinct modes to balance performance and safety:

1. Baseline: No optimization; executes nodes sequentially (standard LangGraph behavior).
2. Parallel: Parallelizes independent stages (fan-out/fan-in) via compile-time graph rewriting. Set enable_parallel=True.
3. Speculative: Combines parallel execution with speculative branch execution (runtime speculation). Set both enable_parallel=True and enable_speculation=True.

Warning: Speculative execution has significant limitations regarding state persistence and streaming (see Limitations record).

# Baseline — no optimization
baseline_compiled = graph.compile()

# Parallel — independent stages parallelized
optimized_parallel = graph.compile(optimization=OptimizationConfig(enable_parallel=True))

# Speculative — parallel + speculative branch execution
optimized_speculative = graph.compile(
    optimization=OptimizationConfig(enable_parallel=True, enable_speculation=True),
)