DOCSEARCHbyAlgolia
Submit Docs

libSQL

repository·main·Indexed 12 days ago

https://github.com/tursodatabase/libsql

An open-source fork of SQLite maintained by Turso that extends SQLite's capabilities with embedded replicas, remote access via a server (sqld), and Bottomless replication for S3-compatible storage. It maintains high compatibility with the standard SQLite file format and API, providing client libraries for TypeScript, JavaScript, Rust, Go, and Python.

Tokens
91.6K
Snippets
296
Records
425
Agent score
96%

What's inside libSQL

  1. Overview of the testrunner.tcl script

    main

    The testrunner.tcl script is a Tcl-based utility used to execute multiple SQLite tests in parallel using multiple jobs. It supports running Tcl test scripts and tests triggered via make commands.

    Output and Monitoring:

    • testrunner.log: A log file created in the current working directory containing all test output. Use grep "^!" testrunner.log or grep failed testrunner.log to find errors.
    • testrunner.db: A SQLite database containing the state of all tests. You can query it to find failures: SELECT * FROM script WHERE state='failed'.
    • Status Reporting: Run ./testfixture $(TESTDIR)/testrunner.tcl status to get a succinct report of the current test state. Use watch ./testfixture $(TESTDIR)/testrunner.tcl status to monitor long-running tests in real-time.
    grep "^!" testrunner.log
SELECT * FROM script WHERE state='failed'
./testfixture $(TESTDIR)/testrunner.tcl status

  2. Overview of SQLite3 Multiple Ciphers

    main
    SQLite3 Multiple Ciphers is an encryption extension for SQLite that supports multiple cipher schemes. Unlike older encryption extensions that were tightly coupled with SQLite internals, this implementation uses SQLite's Virtual File System (VFS) feature. This approach allows it to support SQLite version 3.32.0 and later, though it makes accessing SQLite's internal data structures more complex. The project is primarily developed for Windows but is also tested on Linux.

  3. What is Jaccwabyt?

    main

    Jaccwabyt is a JavaScript API designed for two-way communication between JavaScript and C structs via WASM byte arrays. It creates JavaScript proxies for C structs, allowing changes made to struct members in JavaScript to be visible in C/WASM, and vice versa, by marshaling reads and writes through a shared flat byte array allocated from the WASM heap.

    Browser Compatibility:

    • Requires a recent browser (released mid-2018 or later).
    • Late 2021 releases are required for some optional features (e.g., BigInt64Array in Safari).
    • Relies on TextEncoder and TextDecoder.

  4. Overview of repair extensions and utilities

    main
    The libsql-sqlite3/ext/repair directory contains experimental extensions and utility programs designed to analyze, detect, and potentially fix problems in live database files. These tools are specifically intended for very large databases (in the terabyte range) where hardware malfunctions or cosmic rays may cause corruption. The primary goal is to provide mechanisms for detecting and fixing errors while the databases remain in active use, minimizing application downtime.

  5. Overview of sqld (SQL daemon)

    main

    The sqld project provides a server mode for libSQL, allowing you to consume your database as a service rather than an embedded engine. This is particularly useful for applications running on serverless infrastructure where fitting a full database engine locally might be difficult or inconvenient.

    Key features include:

    • HTTP-based SQLite dialect: Layered on top of HTTP for remote access.
    • Drop-in replacement: Use LD_PRELOAD in your application to switch from a local database to a remote sqld instance.
    • Read replica support: Enables scaling read operations.
    • High Availability: Integration with mvSQLite for fault tolerance.

  6. Understand the capabilities and limitations of sqlite3-parser

    main

    The sqlite3-parser is a Rust port of the SQLite lexer and parser, generated using a modified version of the LEMON parser generator. It is designed to generate an Abstract Syntax Tree (AST) and supports streamable and resumable parsing (e.g., stopping at the end of a statement and restarting).

    Key Features:

    • Tracks position (line and column).
    • Streamable: Can stop at the end of a statement.
    • Resumable: Can restart after the end of a statement.

    Known Limitations & Unsupported Features:

    • Grammar Syntax: Does not support %token_destructor, %default_destructor, or %destructor.
    • Directives: The #line directive is unsupported.
    • Rust-specific issues: %extra_argument is not supported, and terminal symbols generated by lemon are not yet dumped to a specified file.
    • Parsing Behavior: There is a known issue where keywords in double quotes might be incorrectly understood as string literals instead of identifiers in certain contexts.
    • API Changes: The ParseAlloc and ParseFree methods have been removed.

  8. Explore libSQL features and extensions

    main

    libSQL provides several features designed to evolve SQLite for modern use cases:

    Core Features

    • Embedded Replicas: Allows you to have replicated databases inside your application.
    • libSQL server: Provides remote SQLite access, similar to how PostgreSQL or MySQL operates.
    • Multi-language support: Drivers available for Rust, JavaScript, Python, Go, and more.

    SQLite Extensions

    • ALTER TABLE extension: Allows modifying column types and constraints.
    • Randomized ROWID: An improvement to the default row ID behavior.
    • WebAssembly User Defined Functions: Support for WASM-based UDFs.
    • Virtual write-ahead log interface: A virtual WAL interface.
    • Virtual table SQL string passing: Allows passing down SQL strings to virtual table implementations.

  10. Use SQLite no_std bindings for embedded or WASM environments

    main

    The sqlite-rs-embedded bindings provide a lightweight, no_std interface to the SQLite C-API. They are designed for environments where the Rust standard library is unavailable, such as embedded systems or WASM-based browser extensions.

    Key characteristics:

    • no_std support: Does not require the Rust standard library.
    • Zero-copy: Uses techniques to pass Rust strings directly to SQLite without converting to CString or performing extra copies.
    • Memory management: If no allocator is present, the bindings use the SQLite memory subsystem.
    • WASM compatible: Can be used to write SQLite extensions that compile to WASM for browser execution.

    ⚠️ Safety Warning: These bindings aim for minimal Rust<->C overhead by remaining faithful to the base SQLite C-API. Consequently, they are not entirely safe. For example, a SQLite statement object may clear returned values if you step or finalize the statement while those references still exist in your Rust program.

  11. Use the FTS3 full-text search extension

    main

    This extension provides full-text search capabilities for SQLite. While it maintains API compatibility with the original FTS1 extension, it uses a different internal storage schema.

    Important Migration Note: Because the storage schema is substantially different from fts1, you cannot simply swap the extensions; any existing FTS1 tables must be rebuilt to use FTS3.

  12. What is the Hrana protocol?

    main

    Hrana is a protocol designed for connecting to a SQLite database over a WebSocket. It is optimized for edge runtimes (like Cloudflare Workers, Deno Deploy, or Lagon) where low latency and small overhead are critical.

    Key benefits include:

    • Edge Compatibility: Uses WebSockets, which are widely supported in edge environments where general TCP sockets might not be.
    • Fast Cold Starts: Requires only a single roundtrip (via the WebSocket handshake) before queries can be sent, compared to multiple roundtrips in the Postgres wire protocol.
    • Multiplexing: A single WebSocket connection can host multiple concurrent SQL streams, acting as a connection pool.
    • Simplicity: The protocol is lightweight, making it suitable for environments with strict code size limits.