DOCSEARCHbyAlgolia
Submit Docs

SCIP Code Intelligence Protocol

repository·main·Indexed 20 days ago

https://github.com/scip-code/scip

A language-agnostic protocol for indexing source code to enable high-performance navigation features such as 'Go to definition', 'Find references', and 'Find implementations'. Defined via a Protobuf schema, it provides a CLI tool, Reprolang for deterministic index testing, and language bindings for Go, Rust, TypeScript, and Haskell.

Tokens
32.4K
Snippets
112
Records
138
Agent score
71%

What's inside SCIP

  1. What is Reprolang

    main

    Reprolang ("reproducible language") is a domain-specific language designed to generate SCIP indexes in a deterministic and controlled manner. It is primarily used to test SCIP features—such as definitions, references, cross-repo navigation, relationships, and diagnostics—without requiring a real programming language indexer.

    Reprolang uses .repro files containing statements that declare symbols and their relationships. The indexer parses these files using a tree-sitter grammar to emit a complete SCIP index.

  2. What is the SCIP Code Intelligence Protocol?

    main

    SCIP (pronounced "skip") is a language-agnostic protocol designed for indexing source code. It is used to power advanced code navigation features, including:

    • Go to definition
    • Find references
    • Find implementations

    The protocol is defined via a Protobuf schema (scip.proto) and is supported by various language bindings and indexers.

  3. Ignore metadata lines in `scip test` files

    main
    The scip test command only validates occurrence-level assertions and diagnostics. Lines containing symbol metadata from scip snapshot output—such as kind, display_name, documentation, signature_documentation, relationship, and enclosing_symbol—do not carry range markers and are automatically ignored. This allows you to paste blocks of scip snapshot output directly into your test files without causing failures.

  4. Understand the SCIP Symbol Grammar

    main

    A Symbol is a unique identifier for code entities (classes, methods, variables) similar to a URI. It follows a specific string representation grammar that allows it to be used interchangeably with the Symbol object.

    Grammar Structure: <scheme> <manager> <package-name> <version> <descriptor>+ or local <local-id>

    Key Components:

    • Scheme: Any UTF-8 string (escape spaces with double space). Must not be empty or start with 'local'.
    • Package: Composed of a manager, package-name, and version.
    • Descriptor: A sequence of descriptors that form a fully qualified name (e.g., Namespace, Type, Method, Parameter).
    • Local Symbols: Used for entities accessible only within a single Document.
    <symbol>               ::= <scheme> ' ' <package> ' ' (<descriptor>)+ | 'local ' <local-id>
<package>              ::= <manager> ' ' <package-name> ' ' <version>
<scheme>               ::= any UTF-8, escape spaces with double space. Must not be empty nor start with 'local'
<manager>              ::= any UTF-8, escape spaces with double space. Use the placeholder '.' to indicate an empty value
<package-name>         ::= same as above
<version>              ::= same as above
<descriptor>           ::= <namespace> | <type> | <term> | <method> | <type-parameter> | <parameter> | <meta> | <macro>
<namespace>            ::= <name> '/'
<type>                 ::= <name> '#'
<term>                 ::= <name> '.'
<meta>                 ::= <name> ':'
<macro>                ::= <name> '!'
<method>               ::= <name> '(' (<method-disambiguator>)? ').'
<type-parameter>       ::= '[' <name> ']'
<parameter>            ::= '(' <name> ')'
<name>                 ::= <identifier>
<method-disambiguator> ::= <simple-identifier>
<identifier>           ::= <simple-identifier> | <escaped-identifier>
<simple-identifier>    ::= (<identifier-character>)+
<identifier-character> ::= '_' | '+' | '-' | '$' | ASCII letter or digit
<escaped-identifier>   ::= '`' (<escaped-character>)+ '`'
<escaped-characters>   ::= any UTF-8, escape backticks with double backtick.
<local-id>             ::= <simple-identifier>

  5. Core design principles for SCIP indexers

    main

    When building an indexer (producer) for SCIP, keep the following design principles in mind to ensure scalability and performance:

    • Avoid direct graph encoding: Do not attempt to build a complete adjacency list of all semantic entities (nodes and edges) in memory. This prevents parallelism and high memory usage. Instead, use documents and arrays to colocate relevant data, which allows for streaming and incremental indexing.
    • Use strings for IDs: Use string-based identifiers rather than integer IDs. Strings are natively supported by hash tables in most languages and avoid the "off-by-one" errors common with integer-based symbol tables that can break navigation repo-wide.
    • Enable Parallelism and Incrementality: The format is designed so that an indexer can load parts of a codebase, append index data to a file, and clear memory before moving to the next part. This supports both file-level incrementality and parallel execution.
    • Leverage Protobuf: SCIP uses Protocol Buffers (Protobuf) for its schema. This provides a compact binary format, easy code generation across many languages, and supports streaming reads/writes and merging via concatenation.

  6. Understand Reprolang symbol scoping

    main

    Reprolang supports two levels of symbol scoping:

    1. Global (default): Symbols are visible across all files within the same project. No special keyword is required for declaration or reference.
    2. Local: Symbols are scoped to a single file. Use the local keyword to declare or reference them.
    # Global scope (default)
definition global_sym
reference global_sym

# Local scope
definition local myHelper
reference local myHelper

  7. Understand the `scip test` file format

    main

    The scip test command validates a SCIP index against a human-readable test file. The syntax is inspired by Sublime Text's syntax highlighting tests. Test cases consist of a range, a type, and data.

    To create a test file, you place comments in your source code that specify where an occurrence (like a definition or reference) should be and what its properties should be.

  8. Configure Document Position Encoding

    main

    The Document.position_encoding field specifies how the character values in SingleLineRange and MultiLineRange should be interpreted. Indexers should choose an encoding that matches their implementation language for $O(1)$ performance.

    Recommended Encodings:

    • JVM / .NET / JavaScript / TypeScript: Use UTF16CodeUnitOffsetFromLineStart.
    • Python: Use UTF32CodeUnitOffsetFromLineStart.
    • Go / Rust / C++: Use UTF8ByteOffsetFromLineStart.
    1: UTF8CodeUnitOffsetFromLineStart
2: UTF16CodeUnitOffsetFromLineStart
3: UTF32CodeUnitOffsetFromLineStart

  9. Understand the purpose and role of SCIP

    main

    SCIP (SCIP Code Intelligence Protocol) is a transmission format designed for sending code intelligence data from producers (indexers) to consumers (e.g., Sourcegraph).

    Key characteristics:

    • Not a storage format: It is optimized for moving data, not for being a long-term queryable database. Consumers should ideally use a query engine to handle bidirectional lookups (like finding subclasses vs. superclasses).
    • Optimized for Producers: The design prioritizes making it easy to write indexers that are parallelizable, incremental, and language-agnostic.
    • Robustness: It is designed to limit the "blast radius" of indexer bugs. For example, an error in one entity should not cause navigation to fail across the entire repository.

  12. Publish JVM bindings to Maven Central

    main

    Java and Kotlin bindings are published to the org.scip-code namespace via the Sonatype Central Portal. This is driven by the release profile in bindings/{java,kotlin}/pom.xml and the publish-jvm-bindings job in the release workflow.

    Important Notes:

    • scip-kotlin-bindings depends on scip-java-bindings. The Java deployment includes a wait period (~10–30 min) before the Kotlin deployment begins.
    • Publications are irreversible. To fix a bad release, you must bump the version in cmd/scip/version.txt and release a new version.

    Required GitHub Actions Secrets:

    • MAVEN_USERNAME: Token from your Central Portal account.
    • MAVEN_PASSWORD: Token from your Central Portal account.
    • MAVEN_GPG_PRIVATE_KEY: The armored export of your passphrase-less primary signing key (gpg --armor --export-secret-keys $KEYID).