DOCSEARCHbyAlgolia
Submit Docs

trailmark

repository·main·Indexed 19 days ago

https://github.com/trailofbits/trailmark

A tool for parsing source code into queryable graphs of functions, classes, and calls to facilitate security analysis. It leverages tree-sitter for parsing and rustworkx for high-performance graph traversal. Trailmark allows users to identify attack surfaces, find paths between nodes, detect complexity hotspots, and augment graphs with external findings from SARIF or weAudit. It provides both a CLI for analysis and structural diffing, and a Python API for programmatic graph navigation and annotation management.

Tokens
18.5K
Snippets
59
Records
78
Agent score
64%

What's inside trailmark

  1. Overview of Trailmark workflow

    main

    Trailmark parses source code into queryable graphs of functions, classes, calls, and semantic annotations for security analysis. It operates in three distinct phases:

    1. Parse: A language-specific parser walks a directory, uses tree-sitter to create an AST, and extracts Nodes (functions, classes, etc.), Edges (calls, inheritance, etc.), and Metadata (type annotations, complexity, etc.).
    2. Index: The GraphStore loads the CodeGraph into a rustworkx PyDiGraph and builds bidirectional ID/index mappings for high-performance traversal.
    3. Query: The QueryEngine provides a high-level API to traverse the graph, find paths, identify attack surfaces, and manage semantic annotations.

    The long-term goal is to integrate this graph with mutation testing and coverage-guided fuzzing to identify gaps between security assumptions and test coverage.

  2. Understand Entrypoint Detection Patterns in Trailmark

    main

    Trailmark uses entrypoint detectors to identify where external data enters a system. A detector identifies a syntactic marker (like a decorator or a specific function signature) within a specific scope (function, class, or module).

    When implementing a detector, you should:

    1. Open the file at CodeUnit.location.file_path.
    2. Inspect the span ending at start_line.
    3. Backtrack through contiguous decorator or attribute lines to find the marker.

    Each detected entrypoint is assigned metadata based on the following enums:

    • EntrypointKind: user_input, api, database, file_system, third_party
    • TrustLevel: untrusted_external, semi_trusted_external, trusted_internal
    • AssetValue: high, medium, low

  3. Implementation guidelines for entrypoint detectors

    main

    When implementing or extending entrypoint detectors in Trailmark, follow these architectural patterns:

    File IO and Caching

    Detectors should re-read source files but must cache file contents across a single detection run using a dictionary keyed by file path to optimize performance.

    Decorator and Attribute Scanning

    To detect function-like attributes (e.g., in Rust or Cairo):

    1. Start at the function's location.start_line - 1.
    2. Walk backwards through contiguous lines.
    3. Continue as long as lines match the language's decorator/attribute syntax.
    4. Stop at the first blank line, comment-only line (that isn't a decorator), or non-decorator statement.

    Visibility and Framework Disambiguation

    • Visibility: For languages like Solidity or C, inspect the signature line and any continuation lines until the opening brace { or parenthesis ).
    • Frameworks: When patterns are syntactically identical across frameworks (e.g., Flask vs. FastAPI), check the file-scope import statements to determine the correct framework defaults. If ambiguity remains, prefer the more conservative (higher-risk) classification.

    Advanced Detection

    • Confidence Levels: Detectors should always emit an EntrypointKind even if the framework is uncertain. Use an override file for manual corrections.
    • Call-graph Resolution: For frameworks that register handlers at call sites rather than via decorators (e.g., Express, Starlette, Cobra), perform a second pass. This pass should resolve entrypoints by finding functions referenced as arguments to known route-registration calls.

  4. Install Trailmark

    main

    You can install Trailmark via PyPI or by checking out the development branch. Requires Python $\ge$ 3.12.

    Latest published release

    uv pip install trailmark

    Current checkout / development branch

    uv sync --all-groups

    Offline or TLS-inspected environments

    Trailmark uses tree-sitter-language-pack for grammars. If you are in an environment that prevents direct grammar downloads, pre-populate the cache on a matching platform and copy the tree-sitter-language-pack cache directory to your target machine:

    python -c "import tree_sitter_language_pack as p; p.download_all()"

    Note: The HTTPS_PROXY environment variable is honored.

    uv pip install trailmark

  5. Haskell Parser Data Model

    main

    When using HaskellParser, the resulting CodeGraph populates several node types and edge kinds based on Haskell syntax:

    Node Kinds

    • NodeKind.MODULE: Represents the Haskell module.
    • NodeKind.STRUCT: Represents data or newtype declarations.
    • NodeKind.TRAIT: Represents type class declarations.
    • NodeKind.METHOD: Represents functions defined within a type class instance or class methods.
    • NodeKind.FUNCTION: Represents top-level function declarations.

    Edge Kinds

    • EdgeKind.CONTAINS: Connects a module to its internal components (types, classes, functions) or a class to its methods.
    • EdgeKind.IMPLEMENTS: Connects a type to a type class (e.g., an instance declaration).
    • EdgeKind.CALLS: Connects a function to the function it calls.

    Metadata Extracted

    • Docstrings: Haddock comments (-- | or --|) preceding declarations are extracted into the docstring field of the CodeUnit.
    • Complexity: Cyclomatic complexity is computed based on branch points (guards and case alternatives).
    • Parameters: Function parameters are extracted, combining pattern names from the equation with type information from the signature. If names are missing, they are assigned as _argN.

  6. Configure custom entrypoints in .trailmark/entrypoints.toml

    main

    Trailmark uses heuristics to detect entrypoints (like main() or framework-specific patterns). To override or add custom entrypoints, create a .trailmark/entrypoints.toml file at your project root.

    Supported configuration types:

    • Single-node: Define a specific node by ID or module.path:function.
    • File Glob: Match files using patterns.
    • Parameter Type: Match functions based on their argument types.
    • Regex: Match function names using regular expressions.
    • Compound Rules: Combine file_glob and name_regex (rules are joined with AND).

    Note: Later entries in the file override earlier ones for the same node. Place broad rules first and specific overrides last.

    # Single-node entry
[[entrypoint]]
node = "my_module:handle_request"
kind = "api"
trust = "untrusted_external"
asset_value = "high"
description = "HTTP POST /auth"

# Rule: every PHP script under public_html/ is a web-exposed entrypoint.
[[entrypoint]]
file_glob = "public_html/**/*.php"
kind = "user_input"
trust = "untrusted_external"
asset_value = "high"
description = "Web-exposed PHP script"

# Rule: any function that takes a PSR-7 request object.
[[entrypoint]]
param_type = "ServerRequestInterface"
kind = "api"
trust = "untrusted_external"
asset_value = "high"
description = "PSR-7 HTTP handler"

# Rule: functions named `handle_*`.
[[entrypoint]]
name_regex = "^handle_"
kind = "api"
trust = "untrusted_external"

# Rule: conditions compose with AND
[[entrypoint]]
file_glob = "public/*.py"
name_regex = "^handle_"
kind = "api"
trust = "untrusted_external"

  7. Configure cross-language links in .trailmark/links.toml

    main

    To represent relationships that are not visible in source syntax (like RPC calls, FFI, or subprocesses), use a .trailmark/links.toml file. This allows you to connect nodes across different language graphs.

    • source / target: The node ID or unique name/suffix.
    • kind: The relationship type (defaults to calls).
    • confidence: The certainty of the link (defaults to inferred).
    • target_external = true: Use this if the target is an unresolved external endpoint (e.g., a webhook or external service). This creates a proxy node.
    [[link]]
source = "backend:submit"
target = "contract:Verifier.verify"
kind = "calls"
confidence = "certain"
description = "JSON-RPC eth_call"

[[link]]
source = "backend:notify"
target = "payments-webhook"
target_external = true

  8. How diagram emitters handle node labels and styles

    main

    Trailmark diagram emitters use specific logic to transform graph data into Mermaid syntax:

    • Node Labels: Labels are constructed using the node's name, kind, and cyclomatic complexity (e.g., my_func, function, CC=12).
    • Edge Styles: Edge confidence levels determine the Mermaid arrow type:
      • certain: -->
      • inferred: -.->
      • uncertain: -..->
    • Complexity Coloring: In complexity diagrams, nodes are assigned CSS classes based on their cyclomatic complexity:
      • low (< 5): Green
      • medium (5-10): Yellow
      • high (> 10): Red
    • Sanitization: Node IDs are sanitized using sanitize_id to ensure they are Mermaid-safe (replacing non-alphanumeric characters with _ and prefixing numeric IDs with n_).

  9. Understand the Cairo CodeGraph output

    main

    When parsing Cairo code, the resulting CodeGraph contains several types of CodeUnit nodes and CodeEdge relationships:

    Supported Node Kinds

    • MODULE: A standard module.
    • CONTRACT: A module identified by the starknet::contract attribute.
    • TRAIT: A trait definition.
    • STRUCT: A struct definition.
    • ENUM: An enum definition.
    • FUNCTION: A top-level function.
    • METHOD: A function defined within a container (like a module or trait).

    Supported Edge Kinds

    • CONTAINS: Represents ownership (e.g., a module contains a struct, or a trait contains a method).
    • IMPLEMENTS: Represents a trait implementation (e.g., Type IMPLEMENTS Trait).
    • CALLS: Represents a function call from one unit to another.

    Function Metadata

    Functions and methods include:

    • parameters: A tuple of Parameter objects (name and type).
    • return_type: The TypeRef of the returned value.
    • cyclomatic_complexity: An integer representing code complexity.
    • docstring: The extracted /// documentation comments.

  10. Understand the JavaScriptParser output (CodeGraph)

    main

    When using JavaScriptParser, the resulting CodeGraph contains structured information about the JavaScript code, including:

    • Nodes: Representing CodeUnits such as FUNCTION, CLASS, and METHOD.
    • Edges: Representing relationships like CALLS (function calls), INHERITS (class inheritance), and CONTAINS (module/class containing a function/method).
    • Metadata: Each CodeUnit includes its name, location, parameters, cyclomatic complexity, branches, and extracted JSDoc docstrings.
    • Dependencies: A list of top-level module dependencies extracted from import statements.

  11. How CParser extracts code information

    main

    When parsing C code, CParser populates a CodeGraph with several types of CodeUnit nodes and CodeEdge connections:

    Extracted Nodes (CodeUnit)

    • Functions: Includes name, parameters, return type, cyclomatic complexity, branches, and docstrings (extracted from /** ... */ or /// comments).
    • Structs: Extracted from struct_specifier or type_definition nodes.
    • Enums: Extracted from enum_specifier or type_definition nodes.
    • Modules: Represented by the file/directory structure.

    Extracted Edges (CodeEdge)

    • CALLS: Connects a function to the target it calls.
      • If the call is a direct identifier (e.g., func()), the confidence is EdgeConfidence.CERTAIN.
      • If the call involves member access (e.g., obj.field or ptr->field), the confidence is EdgeConfidence.INFERRED.
    • CONTAINS: Connects a module to the functions, structs, or enums defined within it.

    Dependencies

    • Includes: #include directives are extracted and added to the graph.dependencies list as strings (stripping quotes or angle brackets).

  12. Generate Mermaid diagrams from Trailmark graphs

    main

    The trailmark.diagram module provides utilities to generate Mermaid-compatible diagram text from Trailmark code graphs. This can be used via the CLI or by calling the emitter functions programmatically with a QueryEngine instance.

    Supported Diagram Types

    TypeDescription
    call-graphA flowchart of function/method call relationships.
    class-hierarchyA classDiagram showing inheritance and implementation relationships.
    module-depsA flowchart of module import relationships.
    containmentA classDiagram showing class members (functions/methods) within containers.
    complexityA flowchart where nodes are color-coded based on their cyclomatic complexity.
    data-flowA flowchart showing paths from entrypoints (attack surface) to complexity hotspots.

    CLI Usage

    You can run the diagram generator from the command line. Use --focus to scope large graphs to a specific node and --depth to limit the BFS traversal distance.

    # Example: Generate a call graph for a specific function
trailmark diagram --target ./src --language python --type call-graph --focus my_function --depth 3

# Example: Generate a complexity diagram
trailmark diagram --target ./src --language cpp --type complexity --threshold 15
    trailmark diagram --target ./src --language python --type call-graph --focus my_function --depth 3