Weave

repository·master·Indexed 22 days ago

https://github.com/wandb/weave

A toolkit by Weights & Biases for building composable interactive data-driven Generative AI applications. Weave provides tools to log and debug LLM traces using the @weave.op decorator, build rigorous evaluations, and organize LLM workflows from experimentation to production. It supports remote scoring via HTTPS endpoints with OAuth or static bearer authentication and provides SDKs for Python and Node.js.

Tokens
38.2K
Snippets
119
Records
166
Agent score
75%

What's inside weave

  1. Use the Weave Display Module for console output

    master

    The weave.trace.display module provides a unified interface for console output using a pluggable viewer system. This allows you to switch between different output backends (like rich or basic print) without changing your application logic.

    While primarily intended for internal library use, you can use it to create styled console output, tables, and progress bars. You can interact with the module via a new Console instance or by using the default global display.console instance.

    from weave.trace.display import display
    
    # Use the default console
    display.console.print("Hello, World!")
    display.console.rule("Section")
  2. Integrate Weave with Agent SDKs and LLM Providers

    master

    Supported Agent SDKs

    • OpenAI Agents SDK (@openai/agents, @openai/agents-realtime)
    • Google Agent Development Kit (ADK) (@google/adk)
    • Claude Agent SDK (@anthropic-ai/claude-agent-sdk)

    Supported LLM Providers

    • OpenAI (openai)
    • Anthropic (@anthropic-ai/sdk)
    • Google Gen AI (@google/genai)
  3. Authenticate a Remote Scorer

    master

    Weave supports two primary authentication modes for remote scorers. In both cases, you should store the secret name (not the raw secret) in the W&B secret store for the entity that owns the project.

    Best for enterprise deployments. Weave fetches a bearer token from your OAuth token endpoint using a client ID and a secret stored in the Weave/entity secret store, then sends that token to the /score endpoint.

    2. Static Bearer

    Weave reads a static bearer token directly from the Weave/entity secret store and sends it to the /score endpoint.

    Note: Your service must implement bearer-token validation (e.g., JWT validation or token introspection) to verify the incoming Authorization header.

  4. Choose between Session SDK and OTEL auto-instrumentation

    master

    When adding observability to an LLM or agent codebase, you must choose a strategy based on the desired telemetry shape and the code structure.

    1. Session SDK (Agent-logging APIs)

    Use this when: You want 'agent-shaped' traces (hierarchical trees) that appear in the Agents tab of the Weave UI, or when you are using a custom/unknown framework. Mechanism: You explicitly wrap the agent's logic using Turn, LLM, Tool, and SubAgent spans.

    2. OTEL Auto-instrumentation

    Use this when: You want 'flat' call traces that appear in the Calls tab, and you are using a library that Weave already recognizes (e.g., specific versions of OpenAI, Anthropic, or LangChain). Mechanism: Simply calling weave.init() captures telemetry automatically without per-call code.

    3. Raw OTEL Export

    Use this when: The application already owns its OpenTelemetry setup or emits its own spans and you want to route them to Weave. Mechanism: Configure the environment variables to export OTEL directly to the Weave endpoint and add the Weave exporter to the app's TracerProvider.

  5. How the Session SDK models agent traces

    master

    The Session SDK is a universal mechanism that allows you to project complex logic (including DAGs) onto a tree structure. It is designed to capture agentic behavior regardless of the underlying framework.

    Supported Span Types

    • Session: The top-level container.
    • Turn (invoke_agent): Represents one input mapping to one cycle of logic.
    • LLM (chat): Represents model interactions.
    • Tool (execute_tool): Represents tool dispatches.
    • SubAgent (invoke_agent): Represents nested agent calls.

    Capabilities and Limitations

    • Streaming: You can hold an LLM span open, accumulate tokens, and then close it.
    • Concurrency:
      • TypeScript: Use runIsolated for parallel model calls in an async chain.
      • Python: Explicit wrapping is required for thread or queue boundaries.
    • Post-hoc Logging (Python only): Supports log_turn and log_session for batch logging. The TypeScript SDK does not support these batch functions.
    • Constraint: The Session SDK models a tree, not a graph. If there is no observable boundary in your code, the content cannot be captured.
  6. How Weave integrations and patching work

    master

    Weave supports two main styles of libraries: Model Vendors (e.g., OpenAI, Anthropic) which are typically patched to track single API calls, and Orchestration Frameworks (e.g., Langchain) which may require more complex integration like callbacks to handle their specific call stacks.

    There are three ways to apply integrations:

    1. Implicit Patching (Default): Uses a Python import hook to automatically patch supported libraries regardless of when they are imported relative to weave.init().
    2. Explicit Patching: Manually calling specific patch functions (e.g., weave.integrations.patch_openai()) after initialization.
    3. Manual Integration: Using provided utilities like callbacks for frameworks where patching is insufficient.
    import weave
    import openai
    
    weave.init('my-project')  # OpenAI is automatically patched via implicit patching!
  7. Implement the Weave Remote Scorer Contract

    master

    To use Weave remote scoring, you must host an HTTPS endpoint that follows a specific HTTP and JSON contract. Weave sends an HTTP POST request to your endpoint.

    Request Headers

    Weave includes the following headers in the request:

    • Authorization: Bearer <token>
    • Idempotency-Key: <stable key for this scoring attempt>
    • X-Correlation-ID: <request correlation id>
    • X-Weave-Schema-Version: 1

    Response Format

    Your endpoint must return an HTTP 200 status code with a JSON object. The response must include:

    • schema_version: An integer, must be 1.
    • result: The structured scorer output. This can be a single object or a list of objects.

    For a single score, the result object supports:

    • value: A numeric rating (0.0 to 1.0) or a tag string (max 36 characters).
    • reason: (Optional) A string explaining the score.
    • confidence: (Optional) A numeric confidence value (0.0 to 1.0).
    {
      "schema_version": 1,
      "result": {
        "value": 1.0,
        "reason": "The response is clear and concise.",
        "confidence": 0.9
      }
    }
  8. Understand trace_server_mock storage and isolation

    master

    The trace_server_mock uses in-memory storage only.

    • Persistence: Data is lost as soon as the process exits. It does not use Kafka or ClickHouse.
    • Isolation: Data is keyed by project_id. Concurrent tests running with different project_ids are isolated and will not see each other's data.
  9. Use weave.Object for structured data and querying

    master

    By subclassing weave.Object, you create structured "weave Objects" (capital 'O'). This adds metadata to the stored payload, enabling you to query for all instances of a specific class using the base_object_classes filter in objs_query or the POST objs/query API.

    Terminology: The base_object_class refers to the first subtype of Object in the hierarchy. For example, if your hierarchy is B -> A -> Object, the base_object_class filter is A.

    class ModelConfig(weave.Object):
        model_name: str
        model_version: str
    
    # Publishing a structured object
    config = ModelConfig(model_name="my_model", model_version="1.0")
    ref = weave.publish(config)
  10. How Weave references resolve files and objects

    master

    When interpreting a reference, the Weave engine follows these resolution rules:

    1. Artifact Lookup: It identifies the artifact using the path up to (but excluding) the FILE_PATH.
    2. File vs. Object Resolution:
      • If FILE_PATH matches a specific file in the artifact, the reference points to that file.
      • If FILE_PATH is not a file but FILE_PATH.type.json exists in the artifact, the reference points to a weave object. The engine uses the .type.json file to determine the object type and reconstructs the object by reading necessary peer files (e.g., obj.type.json and obj.object.json).
    3. Traversal via REF_EXTRA: If a REF_EXTRA is present, the engine traverses the resolved weave object to extract a nested property.
  11. Handle concurrency and flushing in Weave

    master

    Concurrency

    By default, the Weave context is process-wide. If you are running concurrent agents (e.g., in a web server handling multiple requests), sessions will clash. To run agents in parallel, wrap each session in await weave.runIsolated(async () => { ... }).

    Flushing

    Spans are batched and flushed automatically on process exit. However, for short-lived processes, you should explicitly call await weave.flushOTel() before exiting to ensure all traces are sent.

  12. Understand the Trace Server data flow

    master

    The Trace Server architecture manages the lifecycle of calls made from decorated functions (@op). When a user calls an @op decorated function, the execution flow typically follows this path:

    1. User Code triggers an OpExecution.
    2. OpExecution calls start_run on the GraphClient.
    3. The GraphClient communicates with a RemoteHTTPTraceServer (via call_start).
    4. The RemoteHTTPTraceServer forwards the request to the core TraceWebServer (an FlaskApp).
    5. The TraceWebServer interacts with a ClickHouseTraceServer (implementing TraceServerInterface).
    6. The ClickHouseTraceServer performs batched, asynchronous inserts into the ClickHouseDB (e.g., INSERT INTO calls_raw``).

    Note for Testing: The Web service layer can be bypassed during testing by using the trace_client pytest fixture.