To ensure compatibility with the OpenEvals ecosystem, custom evaluators should follow these patterns:
1. Interface Requirements
Evaluators should accept a subset of these parameters (which can be any value, but typically a dictionary/object):
inputs: The inputs to your application.outputs: The outputs from your application.reference_outputs (Python) or referenceOutputs (TypeScript): The reference outputs to evaluate against.
2. Factory Functions
If your evaluator requires additional configuration (like a regex pattern or a specific model), use a factory function named create_<evaluator_name> (e.g., create_regex_evaluator).
3. Return Format
Evaluators must return a dictionary (or a list of dictionaries) containing:
key: A string representing the metric name.score: A boolean or number representing the score.comment: A string representing the comment/justification (optional).
4. LangSmith Integration
To ensure results are logged to LangSmith, wrap your internal logic in the _run_evaluator/_arun_evaluator (Python) or runEvaluator (TypeScript) method. This method accepts a scorer function that returns either a single score or a tuple of (score, comment).
### Custom Regex Evaluator Example (Python)
```python
import json
import re
from typing import Any
from openevals.types import EvaluatorResult, SimpleEvaluator
from openevals.utils import _run_evaluator
def create_regex_evaluator(*, regex: str) -> SimpleEvaluator:
regex = re.compile(regex)
def wrapped_evaluator(*, outputs: Any, **kwargs: Any) -> EvaluatorResult:
if not isinstance(outputs, str):
outputs = json.dumps(outputs)
def get_score():
return regex.match(outputs) is not None
return _run_evaluator(
run_name="regex_match",
scorer=get_score,
feedback_key="regex_match",
)
return wrapped_evaluator
evaluator = create_regex_evaluator(regex=r"some string")
result = evaluator(outputs="this contains some string")
