code-chunk

The harness compares two Claude Agent SDK variants using the SWE-bench Lite dataset:

1. Agent1 (ops-only): Uses only basic file operations: Read, Grep, and Glob.
2. Agent2 (ops+search): Uses the same tools as Agent1 plus semantic search capabilities via code-chunk embeddings.

Execution Lifecycle:

1. Loads the SWE-bench Lite dataset.
2. For each instance, creates a git worktree at the target commit.
3. Executes Agent1.
4. Builds a semantic index using code-chunk.
5. Executes Agent2 using the semantic_search tool.
6. Computes retrieval metrics against the oracle files from the patch.
7. Aggregates and outputs results.

Embedding Caching: To avoid redundant computation, semantic search indexes are cached at ~/.cache/swebench-eval/embeddings/. The cache key is derived from the instance ID, embedding provider, and dimensions.

Evaluation results are stored in runs/<timestamp>/. The directory structure contains per-instance event logs and aggregated metrics.

Directory Structure:

• events/: Contains .jsonl files for both ops-only and ops+search agents for each instance.
• metrics.jsonl: Detailed metric logs.
• summary.json: Aggregated summary of the run.

Evaluated Metrics:

• Hit@k: Whether the oracle file appears in the top-k predictions.
• MRR: Mean Reciprocal Rank of the first oracle file.
• Coverage@k: Fraction of oracle files in the top-k.
• Time-to-first-hit: Number of turns/tool calls until the first oracle file is accessed.
• Embedding latency: Time taken for index building and queries (Agent2 only).

To install the @supermemory/eval harness, run bun install from the monorepo root.

Before running evaluations, you must configure the following environment variables:

• ANTHROPIC_API_KEY: Required for Claude API access.
• GOOGLE_API_KEY: Required for Gemini embeddings (the default provider).
• OPENAI_API_KEY: Required if you choose to use the OpenAI embedding provider via the --embedding-provider openai flag.

# From monorepo root
bun install

You can install code-chunk using bun or npm to add AST-aware code chunking to your project.

bun add code-chunk
# or
npm install code-chunk

A Chunk represents a segment of source code enriched with semantic context. This is particularly useful for RAG (Retrieval-Augmented Generation) systems.

• text: The raw string content of the code segment.
• contextualizedText: The code segment with semantic context prepended (file path, scope chain, entity signatures, imports, and sibling context). Use this field when creating embeddings for semantic search.
• byteRange & lineRange: The location of the chunk in the original source.
• context: A ChunkContext object containing metadata like language, scope, entities, siblings, and imports.
• index & totalChunks: Position within the file's chunk sequence.

To use code-chunk in environments like Cloudflare Workers, you must provide the WASM binaries for the core tree-sitter parser and the specific language grammars via a WasmConfig object.

import treeSitterWasm from 'web-tree-sitter/tree-sitter.wasm'
import typescriptWasm from 'tree-sitter-typescript/tree-sitter-tsx.wasm'

const config: WasmConfig = {
  treeSitter: treeSitterWasm,
  languages: {
    typescript: typescriptWasm,
  }
}

When using the API, be prepared to handle the following error types:

• ChunkingError: Thrown when chunking fails (e.g., parsing or extraction errors).
• UnsupportedLanguageError: Thrown when the file extension is not supported.

Both errors include a _tag property for compatibility with Effect-style error handling.

For pipelines using the Effect ecosystem, code-chunk provides native support via chunkStreamEffect, chunkBatchEffect, chunkBatchStreamEffect, and chunkBatchStreamEffect.

import { chunkStreamEffect } from 'code-chunk'
import { Effect, Stream } from 'effect'

const program = Stream.runForEach(
  chunkStreamEffect('src/utils.ts', code),
  (chunk) => Effect.log(chunk.text)
)

await Effect.runPromise(program)

For better embedding quality in RAG (Retrieval-Augmented Generation) systems, use c.contextualizedText. This property prepends semantic context (scope, definitions, imports, and siblings) to the raw code, helping embedding models understand the relationships between code segments.

for (const c of chunks) {
  const embedding = await embed(c.contextualizedText)
  await vectorDB.upsert({
    id: `${filepath}:${c.index}`,
    embedding,
    metadata: { filepath, lines: c.lineRange }
  })
}

Use chunkBatch to process an array of files concurrently. It includes built-in error handling per file and a progress callback.

import { chunkBatch } from 'code-chunk'

const files = [
  { filepath: 'src/user.ts', code: userCode },
  { filepath: 'src/auth.ts', code: authCode },
  { filepath: 'lib/utils.py', code: utilsCode },
]

const results = await chunkBatch(files, {
  maxChunkSize: 1500,
  concurrency: 10,
  onProgress: (done, total, path, success) => {
    console.log(`[${done}/${total}] ${path}: ${success ? 'ok' : 'failed'}`)
  }
})

for (const result of results) {
  if (result.error) {
    console.error(`Failed: ${result.filepath}`, result.error)
  } else {
    await indexChunks(result.filepath, result.chunks)
  }
}

When processing multiple files with the same configuration, use createChunker to instantiate a Chunker. This instance provides optimized methods for chunking, streaming, and batch processing.

import { createChunker } from 'code-chunk'

const chunker = createChunker({
  maxChunkSize: 2048,
  contextMode: 'full',
  siblingDetail: 'signatures',
})

for (const file of files) {
  const chunks = await chunker.chunk(file.path, file.content)
}