IndexStoreDB Documentation

IndexStoreDB is a source code indexing library designed for efficient and composable querying of source code symbols, symbol occurrences, and relations.

It works by:

1. Reading raw index data produced by compilers (like Apple Clang or Swift) using the -index-store-path option.
2. Utilizing libIndexStore for data ingestion.
3. Maintaining acceleration tables in a key-value database built with LMDB to enable high-performance queries.

The data model is derived from libIndexStore.

The IndexStore Swift library is a high-performance, type-safe wrapper around libIndexStore.dylib.

Key Characteristics:

• Iteration-focused: The library is designed for iterating through an Index Store rather than performing efficient random-access queries.
• Performance-oriented: To minimize overhead compared to the C API, the library favors speed over API convenience. This results in the use of non-Escapable types and forEach methods for collection iteration.
• Prerequisite: Users should have an understanding of the Index Store's internal structure. See Index Store.md for details on the data model.

Tibs ("Test Index Build System") is a build system designed specifically for test projects of IndexStoreDB and SourceKit-LSP. It is used to incrementally build or rebuild index data and generated module files for Swift and/or C-family language test projects.

Key capabilities:

• Builds index data and generated module files.
• Dumps compiler arguments to a clang-compatible JSON compilation database (compile_commands.json).
• Supports mixed-language targets (e.g., using bridging headers).

Important Limitation: Tibs is not designed to compile to executable code; it focuses on the module-generation and indexing parts of the build process.

An Index Store is a data dump emitted by the Swift and Apple Clang compilers during compilation. It maps source locations to unique identifiers of the symbols (such as types, functions, properties, and global variables) that are declared or referenced at those locations.

Key Characteristics:

• Not for direct querying: The raw Index Store format is not designed for efficient querying. To find occurrences of a specific symbol efficiently, you should use tools like indexstore-db.
• Symbol Identification: Every symbol is uniquely identified by a USR (Unified Symbol Resolution), such as s:4test3fibyS2iF.
  • For Swift, a USR is similar to the symbol's mangled name.
  • For C, it uses the function's base name.
  • For C++, it includes namespace information.

Unit files are identified by their output path (the path to the object file produced during compilation). The hash at the end of the unit's filename is derived from this output path.

Key Metadata in Unit files:

• module-name: The name of the module.
• main-path: The primary source file for the unit.
• out-file: The unique identifier for the build configuration (the output path). Treat this as an opaque string.
• DEPEND START/END: Lists dependencies on other units and record files.

Note on Output Paths: Using output paths instead of source paths allows the Index Store to maintain separate indices for the same source file when compiled for different targets (e.g., iOS vs. watchOS) using conditional compilation (#if).

Be aware that the same symbol (USR) may be defined multiple times in an Index Store due to:

• Conditional Compilation: A symbol might be defined differently depending on platform flags (e.g., #if os(Windows)).
• Multiple Targets: Different targets within the same project may each have their own main function, resulting in separate definitions with the same USR.

To easily reference specific lines or columns in a test project, use inline comment syntax in your source files. The name of the location is the text immediately following the comment.

Source File Example:

func /*myFuncDef*/myFunc() {
  /*myFuncCall*/myFunc()
}

Test Code Usage: You can retrieve a TestLocation by name using ws.testLoc("name"). This is commonly used to create a SymbolOccurrence at a specific spot.

let loc = ws.testLoc("myFuncDef")
let occurrence = Symbol(...).at(loc, roles: .definition)

let loc = ws.testLoc("myFuncDef")
// TestLocation(url: ..., line: 1, column: 19)

let occurrence = Symbol(...).at(ws.testLoc("myFuncDef"), roles: .definition)

Record files represent the state of an individual source file during a specific compilation.

Hashing Logic: Each record file name includes a hash. This hash is based on the interpretation of the file's contents during compilation.

• No change: Adding content to a comment at the end of a line does not change the record.
• Change: Modifying a declaration, changing how an overload resolves, or making significant modifications will result in a different hash and a new record file.

Record files can represent standard source files (.swift, .c) or SDK interfaces (.swiftinterface).

Test projects (fixtures) should be stored in the ISDBTestSupport/INPUTS directory. These projects use the Tibs build system to define sources and targets.

A project structure typically looks like this:

ISDBTestSupport/
  INPUTS/
    MyTestProj/
      a.swift
      b.swift
      c.cpp

You define the project's targets in a project.json file. For example, to include specific source files:

{ "sources": ["a.swift", "b.swift", "c.cpp"] }

Tibs supports advanced configurations including multiple Swift modules and dependencies.

When building on Linux, the C++ code requires libdispatch, which may not be found automatically. You must manually provide the search paths to the Swift toolchain's library directories using the -Xcxx flag to pass include paths to the C++ compiler.

$ swift build -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift/Block