Verdict: LLM Evaluation Pipelines

A Unit is a wrapper around an LLM inference call with well-defined inputs and outputs. Units are the fundamental building blocks of LLM-as-a-Judge systems in Verdict. They support arbitrary dependency structures using the >> operator and can be configured using chainable directives.

Key Features

• Model Selection: Use .via() to specify the model and parameters.
• Prompt Specification: Use .prompt() to define the template.
• Structured Extraction: Use .extract() to parse LLM responses (e.g., via regex).
• Transformation: Use .propagate() for arbitrary post-inference transformations.
• Execution: Units can be executed in parallel with built-in client-side rate-limiting.

Anatomy of a Unit

A Unit consists of three schema components:

1. InputSchema (optional): The data required from dependency units. Verdict performs automatic name-casting from previous outputs to this schema.
2. ResponseSchema (required): The raw schema representing the response produced by the LLM inference call after extraction.
3. OutputSchema (optional): The final processed data passed to the next unit. If not specified, it defaults to the ResponseSchema.

JudgeUnit(DiscreteScale((1, 5)), name="CXEmpathyJudge")
    .via('gpt-4o', retries=3, temperature=1.4)
    .prompt("""
        Score the following customer support conversation on empathy.

        <conversation>
        ...
        </conversation>
    """)
    .extract(RegexExtractor(pattern=RegexExtractor.FIRST_INT))
    .propagate(lambda output: Schema.of(score=output.score / 5))

A Scale can be used directly within a Schema to define the required response format for a judge. Depending on the scale type used, Verdict employs different extraction strategies:

• ContinuousScale: The Structured Output extractor enforces the numerical range (e.g., 0 <= score <= 1).
• DiscreteScale: The Token Probability extractor infers a probability distribution over the provided tokens (e.g., ['A', 'B', 'C', 'D', 'E', 'F']).

By default, the first value in a scale is considered the worst and the last value is the best. You can reverse this logic using the end_is_worst=True flag.

# Continuous Scale Example
class ResponseSchema(Schema):
  score: Scale = ContinuousScale(0, 1)

# Categorical Scale Example
class ResponseSchema(Schema):
  grade: Scale = DiscreteScale(['A', 'B', 'C', 'D', 'E', 'F'])

# Reversing scale direction (1 is best, 5 is worst)
class ResponseSchema(Schema):
  reward: Scale = DiscreteScale((1, 5), end_is_worst=True)

Verdict automatically attempts to cast the execution output of one Unit to the InputSchema of the next Unit in a pipeline. This allows you to connect components that use different naming conventions for the same data.

How it works:

• The operation is append-only and non-destructive. Fields are never removed from the source schema.
• If a field required by the target InputSchema is missing in the source ResponseSchema, Verdict attempts to copy a compatible field from the source to the target name.
• Compatibility is determined by matching the field type (e.g., int) and constraints (e.g., ge=1, le=5).

Example: If a JudgeUnit outputs a score: int and the next SummarizeUnit expects a rating: int, Verdict will copy the value from score to rating at runtime.

class JudgeUnit(Unit):
    class ResponseSchema(Schema):
        explanation: str
        score: int

class SummarizeUnit(Unit):
    class InputSchema(Schema):
        rating: int

# At runtime, the JudgeUnit's ResponseSchema is cast to:
# class ResponseSchema(Schema):
#     explanation: str
#     score: int
#     rating: int # has same value as `score`

A Layer is a container that groups multiple Units together for organizational purposes and shared configuration. When you apply configuration using .via() to a Layer, those settings (like model name and inference parameters) are automatically applied to all Units within that layer.

Default Behavior

By default, a Layer operates in a feedforward and fully-connected manner:

1. Within the Layer: Units are fully independent of one another (an ensemble pattern).
2. Between Layers: Every Unit in a subsequent Layer receives the output of every Unit in the previous Layer (dense connection).

Unraveling

While a Layer typically takes a single primitive (a Unit, another Layer, or a Block), you can pass a list of primitives to a Layer, and it will be "unraveled" (expanded) accordingly.

from verdict import Layer
from verdict.common.judge import JudgeUnit
from verdict.transform import MeanPoolUnit

# Creates an ensemble of 3 independent judges sharing the same config
ensemble = Layer(JudgeUnit(), 3).via('gpt-4o', temperature=0.8) \\
    >> MeanPoolUnit("score")

The pipeline manages the execution of a graph of Units across samples using a concurrency model based on ThreadPoolExecutor.

Concurrency Model

• I/O-bound tasks: Uses a ThreadPoolExecutor with a configurable max_workers parameter. Because most time is spent waiting for network I/O, max_workers can be set to a high value.
• CPU-bound tasks: Tasks like MapUnit are handled by a separate executor with its own configurable max_workers parameter.

Execution Flow for a Single Sample

1. Materialization: The graph of primitives is materialized into a graph of Units.
2. Root Execution: All root nodes are queued for execution.
3. Output Storage: Once a Unit completes its lifecycle, its output is stored.
4. Dependency Resolution: Ready dependent Units are queued for execution.
5. Iteration: This repeats until all Units in the graph are executed.
6. Collection: The outputs of all leaf nodes are gathered.

Execution Flow for a Dataset

For an entire dataset, the system builds a copy of the graph for each sample and queues all samples for execution using the same ThreadPoolExecutor. This allows execution to occur across all samples in parallel.