Sonic Search Backend

The Sonic Channel protocol is used for searching, ingesting index data, and administration via TCP (default port 1491).

Integration Requirements:

1. Command Termination: Every command sent to the server must be terminated with a newline character (\n) to commit the command.
2. Buffer Management: Upon a successful START command, the server returns a STARTED response containing a buffer(N) parameter. This value (in bytes) indicates the maximum size for a single command. Libraries should use this value to truncate or split large data payloads into multiple sub-commands to prevent buffer overflows.

telnet ::1 1491

Sonic communicates with clients using the Sonic Channel protocol over a raw TCP socket. It does not expose an HTTP API to minimize network and processing overhead.

Key characteristics of the protocol:

• Raw TCP: Clients interact directly with a TCP socket.
• Synchronous by default: Most commands are synchronous. On a single connection, you must wait for a command to return before issuing the next one. To achieve parallelism, open multiple Sonic Channel connections.
• Asynchronous support: Certain commands (like search queries) can be handled asynchronously using a special eventing protocol format, allowing them to be processed by a dedicated thread pool.

Example command/response pattern:

• Request: QUERY collection bucket "search query"
• Response: EVENT QUERY result_id result_1 result_2

To minimize disk space, Sonic uses two types of identifiers:

1. OID (Object Identifier): The original, user-provided identifier (e.g., a UUID or string like session_77f2e05e...). These are used for user-facing input and output.
2. IID (Internal Identifier): A compact 32-bit integer representation of the OID. Sonic uses IIDs for internal storage and mapping to indexed words to significantly reduce the index size.

Sonic internally maps OID <-> IID using a RocksDB-powered key-value store.

The tasker module performs periodic background maintenance tasks to optimize performance and resource usage:

1. Janitor: Closes cached collection and bucket stores that haven't been used recently to free up RAM.
2. Consolidate: Writes in-memory FST changes to the on-disk FST data structure.

Note on Latency: Because the tasker performs heavy-duty work on KV or FST stores, you may experience higher-than-expected latency during these periods due to locking. The system is optimized to minimize this contention.

Sonic organizes indexed data into a two-layer hierarchy: Collections contain Buckets, and Buckets contain Indexed Objects.

• Collections: Top-level grouping for your data.
• Buckets: A second layer of organization within a collection. If your use case doesn't require buckets, you can use a generic value like default.
• Objects: The actual search results.

Important Design Choice: Sonic does not store full documents. It only stores identifiers (OIDs) that refer to primary keys in your external database. This keeps the index compact and lightweight. You should use the identifiers returned by Sonic to fetch the actual documents from your own database.

Sonic does not provide an HTTP endpoint. All interactions, including performing searches and managing objects (data ingestion), must be handled via the Sonic Channel protocol (a TCP-based protocol).

To interact with the database, you must use a Sonic Channel client library. If you are building a client for an unsupported language, you can refer to the detailed protocol documentation or the node-sonic-channel implementation.

Certain sensitive or environment-specific configuration keys support interpolation using the ${env.VAR_NAME} syntax. This allows you to keep secrets out of your static TOML files.

Supported keys for interpolation:

• server.log_level
• channel.inet
• channel.auth_password
• store.kv.path
• store.fst.path

[channel]
auth_password = "${env.SONIC_PASSWORD}"

[server]
log_level = "${env.SONIC_LOG_LEVEL}"

Before text is indexed, it passes through a lexer (tokenizer) that performs the following:

1. Normalization: e.g., converting text to lower-case.
2. Cleaning: Removing stopwords (common words like 'the' that add little search value).
3. Tokenization: Splitting text into individual words using an iterator pattern.

Language Detection: Since stopwords are language-specific, Sonic detects the text language using a hybrid approach:

• Fast Method: Counting stopwords in the text (reliable for long texts).
• Slow Method: Performing an n-gram pass (more accurate for small texts).

Sonic uses ISO 639-3 codes for language references.

Sonic provides typo tolerance and autocomplete using a Finite-State Transducer (FST).

• Mechanism: The FST acts as a graph of characters where nodes are characters and edges connect them to form words. This allows Sonic to find matches even if the user inputs englich instead of english, or provides an incomplete word like eng.
• Storage: Sonic stores one FST file per bucket. These files are memory-mapped and read directly from the disk.
• Immutability & Updates: FSTs are immutable once built. To handle new words without rebuilding the entire index constantly, Sonic uses an FST consolidation tasker that buffers changes in memory and periodically commits them to disk.