ck Semantic Code Search

repository·main·Indexed 23 days ago

https://github.com/beaconbay/ck

A semantic code search tool that finds code by meaning, patterns, or a combination of both. It provides AI-powered semantic search, traditional grep-compatible regex search, and BM25-based lexical search. The tool includes a CLI (installed via cargo install ck-search), extensions for VS Code and Cursor, and a benchmarking suite for evaluating retrieval performance against standards like SWE-bench.

Tokens
74.9K
Snippets
199
Records
413
Agent score
80%

What's inside ck

  1. Overview of CK Benchmarks

    main
    The benchmarks/ directory contains tools for evaluating CK's semantic code search performance against industry standards. These benchmarks are designed to test retrieval capabilities, specifically comparing CK's hybrid semantic + lexical search against traditional lexical baselines like BM25.
  2. Core Search Capabilities in ck

    main

    As of version 0.5.3, ck provides a robust semantic and keyword search engine with the following features:

    • Semantic Search: Uses local embedding models to find code based on meaning rather than just text.
    • Hybrid Search: Combines semantic relevance with keyword matching using Reciprocal Rank Fusion.
    • Grep-compatible CLI: Supports traditional grep-style workflows.
    • Threshold Filtering: Control search results by relevance score.
    • Full Section Extraction: Use the --full-section flag to extract complete code blocks rather than just snippets.
    • File Listing: Supports -l and -L flags for grep compatibility.
    • Visual Highlighting: Provides relevance scoring with visual indicators in the output.
  3. Compare ck interfaces for code search

    main

    ck provides four distinct interfaces depending on your workflow requirements:

    • CLI (Command-Line Interface): Best for scripting, automation, and replacing grep/ripgrep. It provides structured output (JSON/JSONL) and is highly composable with Unix tools.
    • TUI (Terminal User Interface): Best for interactive exploration and discovery. It offers live previews, keyboard-driven navigation, and visual context.
    • Editor Integration: Best for in-editor search within VSCode or Cursor. It allows for zero context switching, visual score indicators, and click-to-navigate functionality.
    • MCP (Model Context Protocol) Server: Best for AI agent integration (e.g., Claude Desktop). It provides a programmatic JSON-RPC API for AI-assisted code exploration and automated analysis.
  4. AI Agent Integration via MCP

    main

    ck includes a built-in Model Context Protocol (MCP) server, allowing AI agents (like Claude Code) to interact with your codebase semantically.

    Key MCP Features (v0.5.3):

    • Built-in Pagination: Handles large result sets for the agent.
    • Structured Output: Supports JSON and JSONL formats for automated workflows and streaming.
  5. Understand the ck workspace architecture

    main

    ck is organized as a modular Rust workspace where specialized crates handle different parts of the semantic search pipeline.

    • ck-cli: The entry point providing the command-line interface and MCP server.
    • ck-core: The foundation containing shared types (like SearchResult), configuration, and error handling.
    • ck-engine: Implements search logic, including RegexEngine (pattern matching), SemanticEngine (vector similarity), and HybridEngine (Reciprocal Rank Fusion).
    • ck-index: Manages file discovery, incremental updates via hash-based change detection, and sidecar files.
    • ck-embed: Handles embedding generation using providers like FastEmbed and supports models like BGE, Nomic, and Jina.
    • ck-chunk: Performs intelligent code segmentation using Tree-sitter parsers and semantic boundary detection.
    • ck-models: A registry for model configurations, token limits, and dimensions.
  6. MCP Server: Pagination and capabilities

    main

    The ck MCP server is read-only. It can search and read code, but it cannot write files, refactor code, or generate code.

    Pagination Requirement: Large result sets must be paginated. The default page size is 25 results. To retrieve more, AI agents must implement pagination logic using cursors. The maximum recommended page_size is 100.

  7. Understand Hybrid Search (RRF) Thresholds

    main

    Hybrid search uses Reciprocal Rank Fusion (RRF) for scoring, which operates on a significantly different scale than semantic search.

    Crucial Difference:

    • Semantic Search: Scores range from 0.0 to 1.0. A typical threshold is 0.6.
    • Hybrid Search (RRF): Scores typically range from 0.01 to 0.05.

    If you attempt to use a semantic-style threshold (e.g., --threshold 0.6) with --hybrid, you will likely receive zero results.

    Threshold Quick Reference:

    Search ModeScore RangeTypical Threshold
    Semantic0.0 - 1.00.6
    Hybrid (RRF)~0.01 - 0.050.016 - 0.025

    Tip: Use the --scores flag to see the actual RRF values returned so you can calibrate your --threshold accurately.

  8. Use preview modes in ck TUI

    main

    Cycle through preview modes using Ctrl+V to change how code matches are displayed:

    • Heatmap Mode (Default): Shows semantic similarity with color-coded highlighting:
      • Red: Lower similarity (0.6-0.7)
      • Yellow: Medium similarity (0.7-0.85)
      • Green: High similarity (0.85+)
    • Syntax Mode: Displays syntax-highlighted code using syntect. Supports 7+ languages.
    • Chunks Mode: Shows chunk boundaries and metadata (e.g., Function, Class, Method annotations). Useful for understanding how code is indexed.
  9. Implement adaptive search using near-miss hints

    main

    If a semantic search returns no results above the specified --threshold, ck may output a "near-miss" hint to stderr. This hint suggests a lower threshold that might yield results.

    An effective AI agent pattern is to monitor stderr for the string Near-miss and the suggestion Try lowering threshold to, then automatically retry the search with the suggested value.

    def adaptive_search(query: str, threshold: float = 0.6) -> list:
        """Search with adaptive threshold based on results."""
        cmd = ["ck", "--sem", query, ".", "--json", "--threshold", str(threshold), "--limit", "10"]
        result = subprocess.run(cmd, capture_output=True, text=True)
    
        # Check for near-miss hint in stderr
        if "Near-miss" in result.stderr and "Try lowering threshold to" in result.stderr:
            # Extract suggested threshold
            suggested = extract_threshold_hint(result.stderr)
            # Retry with suggested threshold
            return adaptive_search(query, suggested)
    
        return json.loads(result.stdout) if result.stdout else []
  10. Use hybrid search for refactoring known names

    main
    After you have identified a code pattern using semantic search, you can use hybrid search to find all exact references to specific function or variable names associated with that pattern. This helps bridge the gap between finding a concept and finding all usages of a specific implementation.
  11. How ck works: Indexing, Embedding, and Search

    main

    ck operates through a four-step lifecycle to enable semantic understanding:

    1. Indexing: ck automatically creates and maintains semantic indexes of your code.
    2. Embedding: It uses local AI models (such as BGE, Nomic, or Jina) to transform code into semantic embeddings.
    3. Search: It performs vector similarity searches to find code chunks semantically similar to your query.
    4. Results: It returns results in a familiar grep-style output format, which can include optional relevance scores.

    This process is designed to be privacy-first, running 100% offline with no code leaving your machine.

  12. AI Agent Workflow Patterns for ck

    main

    When using ck with AI agents, follow these effective workflow patterns:

    Iterative Refinement

    Agents should adjust search parameters based on result density. If no results are found, lower the --threshold. If too many results are returned, increase the --threshold.

    Multi-Stage Analysis

    Break complex codebase exploration into stages:

    1. Find entry points (semantic search).
    2. Discover architecture (semantic search).
    3. Locate specific logic like database or error handling (semantic search).

    Security Audit Workflow

    Automate security audits by running targeted semantic searches for patterns like authentication login password, sql query user input, or html render user content, then passing the results to an LLM for analysis.

    Code Understanding Assistant

    Use different search strategies based on the user's question:

    • Specific function names: Use hybrid search (semantic + keyword).
    • Conceptual questions: Use semantic search.
    • General exploration: Use broad semantic search.