zoekt

The Zoekt query language is used to search text within Git repositories. It allows you to combine search patterns, field filters, negations, and logical operators to create precise queries.

Core Syntax Rules:

• Conjunction (AND): Implicitly applied when expressions are separated by a space. Note that the word and is treated as a search term, not an operator.
• Disjunction (OR): Use the lowercase or operator to combine alternatives.
• Negation: Use the - symbol before an expression to exclude it.
• Grouping: Use parentheses () to control the order of operations and create complex logic.

content:test (lang:python or lang:javascript)

A full Zoekt deployment includes a service management tool that handles the lifecycle of the search engine. Its primary responsibilities are:

• Polling: Monitoring git hosting sites (e.g., GitHub, Google Source) for updates.
• Reindexing: Automatically reindexing repositories when changes are detected.
• Webserver Management: Running the webserver and automatically restarting it if it fails.
• Log Maintenance: Deleting old webserver logs to manage disk space and privacy.

Zoekt supports indexing multiple branches of a repository with minimal space overhead using bitmasks.

Each file blob in the index has a bitmask representing the branches in which that specific content is found. For example:

• branches: [master=1, staging=2, stable=4]
• A file x.java with branch mask=3 exists in both master and staging.
• A file x.java with branch mask=4 exists only in stable.

This allows the index to store different versions of the same file only once if the content is identical across branches.

Search Speed

Search latency depends on the corpus, query, result limit, hardware, and cache state.

• For specific queries (e.g., searching for a term in a specific directory), results can return in milliseconds.
• The speed for common strings is primarily dominated by the number of results requested. Producing a very large number of results (e.g., 50,000+) can take between 100ms and 1 second.

Indexing Speed

Indexing can be parallelized for speedup. As a benchmark, indexing the Linux kernel (~545MB of data) took approximately 160 seconds on a single thread using a laptop-grade CPU.

To handle special characters or spaces in your queries:

• Spaces: Wrap values in double quotes, e.g., "my text".
• Escaping: Use a backslash \ to escape the next character. To include a literal backslash in a regular expression, use two backslashes \\.
• Regex Delimiters: Do not use / delimiters for regex; slashes are matched literally.

content:"foo\"bar"

The Zoekt webserver exposes two types of APIs:

1. JSON Search API: Enabled by starting the webserver with the -rpc flag. It is available at http://localhost:6070/api/search. It supports:

   • UseBM25Scoring: Alternative BM25 scoring.
   • NumContextLines: Providing context lines around matches.

2. gRPC API: Supports structured query objects, streaming search results, and advanced search options.

The type: operator limits the kind of results returned by the query. By default, Zoekt returns file content matches (filematch).

Valid values for type::

• filematch: Returns file content matches (default).
• filename (or file): Returns only matching filenames.
• repo: Returns only repository names.

Scoping Note: type: applies to the entire expression in its current scope. For example, type:repo foo or bar is equivalent to type:repo (foo or bar). To scope it to a specific branch of an or clause, use parentheses: (type:repo foo) or bar.

type:repo content:config

To ensure optimal performance, the Zoekt search server should meet the following resource guidelines:

• Storage: Use a local SSD to store the index file. Note that the index file size is approximately 3.5x the size of the corpus.
• Memory (RAM): The server should have at least 20% more RAM than the size of the corpus.
• CPU: For large codebases, use machines with ample CPU cores, as search operations can be parallelized across shards.

Zoekt provides a single container image ghcr.io/sourcegraph/zoekt that includes the Zoekt binaries, git, and universal-ctags.

Run Webserver (Default): Runs zoekt-webserver against /data/index. Use a volume mount to provide your index data.

Run Indexserver: You can override the default command to run zoekt-indexserver. This is useful for managing cloned repositories, logs, and indexes in a persistent volume.

# Run webserver with local index volume
docker run --rm -p 6070:6070 -v "$PWD/index:/data/index" ghcr.io/sourcegraph/zoekt

# Run indexserver with custom config and data volume
docker run --rm \
  -v "$PWD/config.json:/config.json:ro" \
  -v "$PWD/token.txt:/home/zoekt/token.txt:ro" \
  -v zoekt-data:/data \
  ghcr.io/sourcegraph/zoekt \
  zoekt-indexserver -mirror_config /config.json -data_dir /data

If you need to build Universal Ctags from source to ensure compatibility with Zoekt (specifically requiring --enable-json and --enable-seccomp), use the following build process. This requires the dependencies installed via apt-get as described in the installation guide.

./autogen.sh
LDFLAGS=-static ./configure --enable-json --enable-seccomp
make -j4