Overview of Evals System Components
mainThe architecture is composed of several specialized components:
- Next.js Web Application: The management interface for creating runs, monitoring progress via Server-Sent Events (SSE), and viewing the results dashboard.
- Controller Container: An orchestrator (based on
evals-runner) that manages an in-memory task queue usingp-queue, prepares Git workspaces, and coordinates runner containers. - Runner Containers: Isolated environments that execute single tasks. They include a fresh VS Code instance, pre-installed language runtimes (Go, Java, JavaScript, Python, Rust), and the Roo Code extension.
- Redis: Used for pub/sub messaging, runner registration tracking, and heartbeat monitoring (Note: it is not used for task queuing).
- PostgreSQL: The primary database for storing run configurations, task definitions, metrics, and results.