DOCSEARCHbyAlgolia
Submit Docs

sig Documentation

repository·main·Indexed 19 days ago

https://github.com/syndica/sig

Sig is a high-performance Solana validator client implemented in Zig, featuring a multi-process architecture with sandboxed processes communicating via shared memory. The documentation covers the sig validator, the parseout tool (v0.1.0) for comparing conformance test outputs, and guides for running conformance tests using the Zig test runner, solana-conformance, and Nix.

Tokens
50.5K
Snippets
154
Records
245
Agent score
55%

What's inside sig

  2. Understand the status of Sig v1

    main

    Sig v1 is the original implementation of Sig. It implements the full Solana protocol, excluding block production.

    Note: Sig v1 is currently in maintenance mode. Only critical bug fixes are being addressed. For new development and features, use the v2/ directory.

  4. Understand the motivation behind Sig

    main

    Sig is designed to address specific scaling and reliability challenges in the Solana ecosystem by focusing on three core pillars:

    1. RPS Performance: Optimizing Reads-Per-Second (RPS) through low-latency and high-throughput RPC methods to prevent validator slot lag.
    2. Client Diversity: Reducing systemic risk and improving network fault tolerance by providing an alternative client to minimize reliance on a single implementation.
    3. Protocol Accessibility: Prioritizing readability and simplicity to encourage a modular, contributor-friendly ecosystem.

    By addressing these areas, Sig aims to provide a more robust and scalable foundation for on-chain data interaction.

  5. How fork-aware queries work in AccountsDB

    main

    AccountsDB v2 supports Solana's fork-based execution model. When performing a query, the ancestors set defines which slots are visible. The lookup follows this priority:

    1. Unrooted Storage: Searched first for the most recent version of the account within the ancestor set.
    2. Rooted Storage: If not found in unrooted storage, the system checks the persistent SQLite storage.
    3. Deleted Accounts: Zero-lamport accounts are treated as deleted and return null.

    Example Scenario: If a query is made for slot 6 with ancestors {1, 3, 6}:

    • It returns the slot 6 version if modified there.
    • If not, it returns the slot 3 version if modified there.
    • If not, it returns the slot 1 version from rooted storage.
    • It will not see modifications from slots 2, 4, or 5, as they are not in the ancestor set.

  6. How parseout architecture and layers work

    main

    The parseout architecture is split into two composable layers that can be imported into Python scripts for custom interpretation.

    Parsing Layers

    1. Parse Layer 1 (parseout.parser): A generic parser that converts the protobuf-text-like format into OrderedDict[str, dict]. It handles headers, key-value pairs, nested { } blocks, and record separators (lines of 20 hyphens).
    2. Parse Layer 2 (parseout.transaction.parser): A transaction-specific parser that converts generic dicts into a typed dataclass hierarchy including Record, SanitizationError, ExecutedSuccess, ExecutedError, FeeDetails, and AccountEntry.

    Differ Layers

    1. Generic Differ (parseout.differ): Operates on Layer 1 output. It recursively walks fields and generates dynamic categories based on paths.
    2. Transaction Differ (parseout.transaction.differ): Operates on Layer 2 output. It compares records by shared test IDs and uses a fixed set of semantic categories.

  7. How Topology Internals work

    main

    A topology defines a set of services that run together and how they interact. Services run as isolated processes or as threads (when .sandboxing_mode = .threaded is used, which is required for Tracy).

    The orchestration process follows these steps:

    1. Define a Topology struct listing the services to run.
    2. Create shared-memory regions using Region(T) from init/topology.zig for inter-process communication.
    3. Initialize region states.
    4. Initialize the Topology struct, which maps regions to services as ReadOnly or ReadWrite access.
    5. Spawn children via children.spawn(mode, topology), which forks services, maps regions, and installs seccomp filters in sandboxed mode.
    6. Monitor health via children.wait(...). The parent process uses Activity and Connection primitives from lib/runner.zig to track service health.

  8. Handle Write Concurrency and Errors

    main

    Webzockets enforces a strict concurrency model for data writes:

    • Single Write Rule: Only one data write can be active at a time. If you attempt a second sendText or sendBinary before the first has finished, the call will return error.WriteBusy. You should queue subsequent writes and trigger them from the onWriteComplete callback.
    • Control Frames: sendPing and sendPong use a separate internal queue (256 bytes). If this queue overflows, the call returns error.QueueFull.

  9. Research findings on using LMDB for Accounts DB

    main

    An experiment was conducted to evaluate LMDB as a storage engine for the Accounts DB. While LMDB offers excellent read performance and zero-copy querying, it was ultimately rejected for this specific use case due to performance degradation at scale.

    Key Findings

    • Read Performance: LMDB provides linear scaling for parallel reads and zero-copy querying within transactions. However, for the Accounts DB, most queries involve unordered Pubkeys, forcing $O(\log n)$ lookups. As the number of keys approaches 10 million, performance degrades significantly.
    • Write Performance: Initial write speeds were slow but could be improved by approximately 35x using the MDB_NOSYNC flag.
    • Scalability Issues: The unordered nature of Pubkeys prevents leveraging LMDB's fast paths for sorted data. Sharding was considered as a solution but rejected due to lost consistency (transactions cannot span databases) and increased complexity.

  10. Understand Sig v2 architecture and code categories

    main

    Sig v2 uses a multi-process architecture where services communicate through typed shared-memory regions. The codebase is organized into strict categories to enforce boundaries:

    • Main files (main.zig, tests/**/main.zig): The entry points that instantiate a topology and spawn services. Only these can import topology.
    • Service implementations (services/*.zig): Lightweight glue code that runs in its own process. They orchestrate components and hook up shared memory regions. A service must declare its components using a pub const components declaration.
    • Components (components/{name}/): High-level domain logic (e.g., accounts_db, gossip). Each component consists of a component.zig (the implementation) and an api.zig (the public interface). Components do not invoke each other directly.
    • Lib (lib/): General-purpose primitives (IPC, crypto, Solana types) that can be imported by any module.
    • Init (init/): Plumbing used to turn topology descriptions into running processes.
    • Tools (tools/): Standalone developer utilities.

  11. Understand snapshot types and management

    main

    AccountsDB supports two types of snapshots for state recovery:

    • Full snapshots: Contain all accounts at a specific slot.
    • Incremental snapshots: Contain only accounts that changed since a full snapshot.

    Snapshot Workflow:

    1. Download: Handled by src/accountsdb/snapshot/download.zig.
    2. Decompress: Snapshots are .tar.zstd files. Use parallelUnpackZstdTarBall for high-performance, parallelized decompression.
    3. Load: The snapshot module unpacks the archive, loads account files, and builds the account index.

  12. How the sig update lifecycle works

    main

    The sig service uses a systemd-orchestrated lifecycle to ensure it stays up-to-date with the latest code. The process follows this call graph:

    sig-update.timer $\rightarrow$ sig-update.service $\rightarrow$ sig-update $\rightarrow$ sig.service $\rightarrow$ sig

    Lifecycle Stages

    1. Scheduling: sig-update.timer periodically triggers sig-update.service.
    2. Update Check: The sig-update binary runs as root, then de-escalates to the sig user to check the configured BRANCH for new commits.
    3. Rebuild: If new commits are found, sig-update builds a new sig binary.
    4. Restart: As root, the script restarts sig.service and restarts metrics via docker-compose.
    5. Execution: sig.service runs the newly built sig binary (located in the sig user's home folder) using the CLI_ARGS defined in /etc/sig.conf.