DOCSEARCHbyAlgolia
Submit Docs

Codebuff

repository·main·Indexed 27 days ago

https://github.com/codebuffai/codebuff

Searchable repository documentation for Codebuffai Codebuff from https://github.com/codebuffai/codebuff.

Tokens
44.5K
Snippets
87
Records
254
Agent score
94%

What's inside codebuffai/codebuff

  1. Overview of BuffBench

    main
    BuffBench is an evaluation framework designed to measure the performance of AI coding agents. It works by tasking agents with reconstructing actual git commits from open-source repositories. The system uses AI-powered judging to assess implementation quality and problem-solving processes through metrics like completion score, code quality score, and overall score.

  2. Freebuff CLI Features and Commands

    main

    Freebuff includes most core features of Codebuff but with specific restrictions.

    Supported Features

    • Authentication: Login/logout flow and API key storage.
    • Chat: Message history, streaming, and agent spawning.
    • Mentions: @files for attaching files and @agents for using available free-tier agents.
    • Bash mode: Running terminal commands.
    • Attachments: Image attachments (paste or attach).
    • Knowledge: Support for knowledge.md files.
    • History: Use /history to resume conversations.
    • Feedback: Use /feedback command.
    • Customization: Light/dark themes, skills from .agents/skills, and local agents from .agents/ directory.

    Restrictions

    • The Mode Toggle (to switch between paid/free modes) is hidden.
    • Subscription/Usage commands (like /subscribe or /usage) are removed.
    • Credits/Subscription UI components are suppressed.
    • Ads are always shown in Freebuff mode.

  3. Add new E2E tests

    main

    To add a new end-to-end test to the suite:

    1. Create a new file in freebuff/e2e/tests/ using the naming convention <feature>.e2e.test.ts.
    2. Add the new test name to the .github/workflows/freebuff-e2e.yml matrix to ensure it runs in CI:
    matrix:
  test:
    - version
    - startup
    - help-command
    - agent-startup
    - your-new-test    # <-- add here

  4. Quick Start: Testing any TUI application with tmux scripts

    main

    Use the tmux-cli.sh script to automate the lifecycle of testing Terminal User Interface (TUI) applications. These scripts handle bracketed paste mode automatically to prevent character dropping during rapid input.

    Basic Workflow

    1. Start a session: Run a command (e.g., claude, python my_tui.py) in a new tmux session.
    2. Send input: Use the send command to pass text or keys to the app.
    3. Capture output: Use capture to grab the terminal state, which is automatically saved to debug/tmux-sessions/{session}/.
    4. Clean up: Stop the session when finished.
    # Start a test session with any command
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "claude")
echo "Started session: $SESSION"

# Send a command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/help"

# Wait and capture output
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"

  5. Install and use the Codebuff CLI

    main

    Install the Codebuff CLI globally via npm to use the AI programming assistant in your terminal. Once installed, navigate to your project directory and run the codebuff command to start an interactive session where you can issue natural language instructions for code modifications.

    npm install -g codebuff

cd your-project
codebuff

  6. Manage test baselines and the CI guard

    main

    The CI uses scripts/ci/test-with-guard.ts to prevent disappearing tests. The build will fail if:

    1. An error occurs outside a test body (e.g., an Unhandled error between tests during import).
    2. The test or file count falls below the baseline defined in .github/test-baselines.json.

    Updating Baselines

    • To add tests: Simply add them; the guard notes the baseline is stale but does not fail.
    • To delete tests: You must re-record the baseline using the --update flag.
    • Important: Always re-record baselines from a real CI run, not locally. Local runs may differ due to skipped tests (e.g., DB-backed suites) or missing build artifacts like sdk/dist.

    Note on sdk/dist: Some CLI tests register placeholder tests only when sdk/dist is missing. Ensure you run cd sdk && bun run build before recording a baseline to avoid inflated counts.

  8. Create new evaluations from Git commits

    main

    BuffBench provides tools to generate evaluation tasks from existing repositories:

    1. From specific commits: Provide a repository URL followed by the target SHAs to generate tasks for those specific points in history.
    2. End-to-end from a repository: Automatically clones a repository, uses AI to select high-quality commits, and generates a complete evaluation file.
    # Generate from specific commits
bun run evals/buffbench/gen-evals.ts \
  https://github.com/user/repo \
  abc123 \
  def456 \
  ghi789

# Generate from Repository (End-to-End)
bun run evals/buffbench/gen-repo-eval.ts \
  https://github.com/user/repo

  9. Handle service availability in tests

    main

    Tests requiring external services (like Postgres) should gate on reachability rather than the presence of an environment variable (e.g., !process.env.DATABASE_URL).

    • Local Dev: Tests should skip cleanly if the service is unavailable (e.g., using a docker command to fix it).
    • CI Environment: Tests must never skip in CI. If CODEBUFF_GITHUB_ACTIONS=true is set, the test should throw an error if the service is unreachable, preventing a broken service container from being reported as a passing test.

  10. Install and use Shell Shims

    main

    Shell shims allow you to run commands directly without the codebuff prefix. To set up shims for a specific agent (e.g., codebuff/base-lite@1.0.0), install the shim and evaluate the environment configuration in your current shell session.

    codebuff shims install codebuff/base-lite@1.0.0
eval "$(codebuff shims env)"
base-lite "fix this bug"

  12. Implement E2E tests for the Codebuff SDK

    main

    When writing end-to-end tests, use the CodebuffClient and ensure you handle API key availability to avoid failures in environments without credentials. Use the EventCollector utility to capture events during the agent run.

    Recommended pattern:

    1. Use skipIfNoApiKey() to gracefully skip tests if CODEBUFF_API_KEY is missing.
    2. Initialize CodebuffClient with getApiKey().
    3. Use client.run() with a handleEvent callback passed from an EventCollector instance.
    4. Check for authentication errors using isAuthError(result.output) before making assertions.
    import { describe, test, expect, beforeAll } from 'bun:test'
import { CodebuffClient } from '../../src/client'
import { EventCollector, getApiKey, skipIfNoApiKey, isAuthError, DEFAULT_AGENT, DEFAULT_TIMEOUT } from '../utils'

describe('E2E: My Test', () => {
  let client: CodebuffClient

  beforeAll(() => {
    if (skipIfNoApiKey()) return
    client = new CodebuffClient({ apiKey: getApiKey() })
  })

  test('does something', async () => {
    if (skipIfNoApiKey()) return
    const collector = new EventCollector()
    
    const result = await client.run({
      agent: DEFAULT_AGENT,
      prompt: 'Test prompt',
      handleEvent: collector.handleEvent,
    })

    if (isAuthError(result.output)) return
    
    expect(result.output.type).not.toBe('error'
  }, DEFAULT_TIMEOUT)
})