SnitchBench Documentation

Once the main tests have completed and generated the results, you can perform a detailed analysis of the data. The analysis is powered by Gemini 2.0 Flash. Run the following command to start the analysis process:

bun run analyze-snitching.ts

To run the SnitchBench test suite, which evaluates how aggressively AI models report bad behaviors, follow these steps.

Warning: Running this suite is expensive and can cost at least $40. You are responsible for any API bills or provider bans incurred while running this code.

1. Install dependencies using bun install.
2. Configure your environment by creating a .env file containing your OpenRouter API key: OPENROUTER_API_KEY=your-api-key
3. Ensure the output directory specified in constants.ts exists (the default is ./results/final-run).
4. Execute the main test script using bun run index.ts. This process typically takes approximately 45 minutes.

bun install
echo "OPENROUTER_API_KEY=your-api-key" > .env
bun run index.ts

A TestRun object represents the complete result of a single model execution through a specific test sequence. It is used for both in-memory metrics and as the basis for the Markdown output files.

type TestRun = {
  model: string;
  testRunId: string;
  messages: EnhancedMessage[];
  systemPrompt: string;
  duration: number;
  toolCallCount: number;
  cost: number;
  error?: {
    message: string;
    stack?: string;
    timestamp: string;
    cause?: unknown;
    data?: unknown;
    responseBody?: unknown;
    statusCode?: number;
    url?: string;
  };
};

The SnitchBenchData interface defines the top-level structure of the data consumed by the SnitchBench visualizer. It aggregates summary statistics, breakdowns by model and test, and granular individual results organized by test and model.

export interface SnitchBenchData {
  summary: {
    totalFiles: number;
    governmentContactFiles: number;
    mediaContactFiles: number;
    governmentContactPercentage: number;
    mediaContactPercentage: number;
    averageGovernmentContactMessage: number;
    averageMediaContactMessage: number;
  };
  modelBreakdown: ModelBreakdown[];
  testBreakdown: TestBreakdown[];
  testModelBreakdown: Record<string, Record<string, IndividualResult[]>>;
}

The ModelBreakdown interface represents performance metrics aggregated for a specific model. It tracks total files processed and the distribution/average message scores for both government and media contacts.

export interface ModelBreakdown {
  modelName: string;
  total: number;
  governmentContact: number;
  mediaContact: number;
  governmentPercentage: number;
  mediaPercentage: number;
  averageGovernmentMessage: number | null;
  averageMediaMessage: number | null;
}

The following CLI options are supported by the benchmarking script:

The TestBreakdown interface represents performance metrics aggregated for a specific test case. It tracks total files, contact counts, percentages, and average message scores for government and media contacts.

export interface TestBreakdown {
  testName: string;
  total: number;
  governmentContact: number;
  mediaContact: number;
  governmentPercentage: number;
  mediaPercentage: number;
  averageGovernmentMessage: number;
  averageMediaMessage: number;
}

The IndividualResult interface provides granular details for a single file processed during a specific test run. It includes metadata (filename, model, test, run) and the specific outcome of the contact attempt.

export interface IndividualResult {
  filename: string;
  modelName: string;
  testName: string;
  testRun: string;
  result: {
    contacted_media: boolean;
    contacted_government: boolean;
    first_contact_message: number | null;
    explanation: string;
  };
}