OP-SQLite

repository·main·Indexed 21 days ago

https://github.com/op-engineering/op-sqlite

A high-performance SQLite library for cross-platform applications (iOS, Android, macOS, and Web) and Node.js. It supports vanilla SQLite, Turso, Libsql, and SQLCipher, featuring reactive queries, prepared statements, and plugins like FTS5, Rtree, cr-sqlite, and sqlite-vec. The @op-engineering/op-sqlite-node package provides a Node.js adapter using better-sqlite3 to share database logic between mobile and backend services.

Tokens
25.4K
Snippets
88
Records
113
Agent score
76%

What's inside op-sqlite

  1. Overview of OP-SQLite supported features

    main

    OP-SQLite is a high-performance SQLite wrapper supporting multiple platforms and advanced SQLite extensions. Key capabilities include:

    • Platform Support: iOS, Android, macOS, and Web.
    • SQLite Engines: Vanilla SQLite, Turso, Libsql, and SQLCipher (as compilation targets).
    • Plugins: FTS5, Rtree, cr-sqlite, and sqlite-vec.
    • Advanced Features:
      • Reactive queries
      • Custom tokenizers
      • Loading runtime extensions
      • JSONB support
      • Native query interruption via db.interrupt()
    • Built-in Key-Value Store: Includes a simple Key-Value storage implementation to avoid extra dependencies.
  2. Use @op-engineering/op-sqlite-node for Node.js

    main

    The @op-engineering/op-sqlite-node package is a Node.js adapter that provides the same TypeScript API as the React Native version of @op-engineering/op-sqlite. It uses better-sqlite3 under the hood, allowing developers to share database logic between React Native mobile applications and Node.js backend services.

    import { open } from '@op-engineering/op-sqlite-node';
  3. Use the NodeJS façade for testing

    main

    While the core JSI module cannot run in Node.js, op-sqlite provides a NodeJS-compatible façade with the same API as the React Native version.

    Purpose: This is intended for writing Jest tests to verify your SQL queries. It is not intended for production usage and is not a guarantee of the library's correctness, but rather a convenience for testing logic.

  4. How reactive queries work in op-sqlite

    main

    Reactive queries allow you to subscribe to database changes. When a change is detected in a specified table or row ID, the query is automatically re-executed.

    Key Mechanics:

    • Implementation: Reactivity is achieved via SQLite's native update hook and filtering is implemented in C++ for high performance.
    • Optimization: Queries are internally stored as prepared statements to optimize callbacks.
    • Row ID Requirement: Reactivity relies on SQLite's internal rowid column. If your table uses a different primary key, you must explicitly retrieve the rowid to subscribe to specific rows.
    • Trigger Mechanism: Reactive queries are only triggered on transactions. The query re-runs at the end of a transaction that mutates the observed data.
  5. Integrate op-sqlite with native Android and iOS code

    main

    To use the native C++ functions of op-sqlite without going through the React Native/JavaScript layer:

    • Android: Implement your logic via JNI/NDK code.
    • iOS: You can use Objective-C++ (.mm) files to import and use the headers directly.

    Note: Swift integration may require setting the minimum Swift version to 5.9.

  6. Understanding HostObjects limitations in executeWithHostObjects

    main

    The executeWithHostObjects API returns C++ objects exposed to JavaScript. While extremely fast, they have specific limitations:

    1. Scalar assignment works: You can assign single properties with scalar values (strings, numbers, etc.).
    2. Object assignment fails: You cannot assign a JavaScript object to a property of a HostObject because the C++ side requires properties to be explicitly stored and cast to C++ types.

    Workaround: To add complex objects to a result row, spread the HostObject into a new pure JavaScript object. If your transpiler converts spread syntax to Object.assign, use the ...{} pattern to ensure a new object is created.

    // Scalar assignment (Works)
    let results = await db.executeWithHostObjects('SELECT * FROM USER;');
    results._array[0].newProp = 'myNewProp';
    
    // Object assignment (Fails)
    // results._array[0].newProp = { foo: 'bar' }; 
    
    // Workaround: Create a new pure JS object
    let newUser = { ...{}, ...results._array[0], newProp: { foo: 'bar' } };
  7. Handling large integers with BigInt

    main

    Because JavaScript numbers are 64-bit floats (double), they can only safely represent integers up to Number.MAX_SAFE_INTEGER ($2^{53} - 1$). SQLite supports 64-bit integers (long long), but values exceeding the JS safe limit will be truncated when returned to JavaScript.

    To handle larger numbers, you must store them as TEXT in SQLite and manually serialize/deserialize them using JavaScript BigInt objects.

    // 1. Create table with TEXT type and STRICT typing
    db.executeSync(
      'CREATE TABLE IF NOT EXISTS NumbersTable (myBigInt TEXT NOT NULL) STRICT'
    );
    
    // 2. Insert by converting BigInt to string
    db.executeSync('INSERT INTO NumbersTable VALUES (?)', [
      BigInt('12345678901234567890').toString(),
    ]);
    
    // 3. Retrieve by converting string back to BigInt
    let res = db.executeSync('SELECT * FROM NumbersTable');
    let myBigint = BigInt(res.rows[0].myBigInt);
  8. Install @op-engineering/op-sqlite

    main

    To install the library in a standard React Native project, use npm:

    npm i -s @op-engineering/op-sqlite

    In recent React Native versions, pod installation should be automatic. If not, run pod install manually.

    Note for Expo users: You cannot use this library in the expo-go app. You must use a development build (pre-build). No specific Expo plugin is required, but ensure pods are properly set up.

    npx expo install @op-engineering/op-sqlite
    npx expo prebuild --clean
  9. Implement Custom Tokenizers in op-sqlite

    main

    Custom tokenizers are C++ functions that allow you to define how a stream of characters is broken into "tokens" (e.g., breaking on whitespace or special characters) to improve full-text search (FTS5) accuracy.

    To implement a custom tokenizer, follow these steps:

    1. Declare the tokenizer in your package.json under the op-sqlite key. You must also ensure fts5 is set to true.
    2. Initialize the code generation by running pod install. This creates a c_sources folder at your project root and a tokenizers.h file.
    3. Implement the logic in a new file named c_sources/tokenizers.cpp. You must wrap your implementation in the namespace opsqlite and maintain specific function signatures provided in the generated tokenizers.h.
    4. Register the tokenizer by running pod install again. This compiles your C++ code directly into the op-sqlite binary.

    Important Notes:

    • Do not touch c_sources/tokenizers.h; it is auto-generated and will be overwritten.
    • The code generation step is currently implemented for Cocoapods (iOS). For Android, the header generation step is supported, but you must still follow the pattern.
    • On CI environments, you must run pod install even if pods are cached to ensure the generated sources are copied correctly.
    "op-sqlite": {
    	"fts5": true,
    	"tokenizers": ["word_tokenizer"]
    }
  10. Use JSONB support in SQLite

    main

    op-sqlite includes built-in JSONB support. Because it is a direct binding to SQLite, you must handle data serialization yourself. You can insert JSON as a string or as a blob (ArrayBuffer).

    When using blobs, you must manually convert your JavaScript object to an ArrayBuffer using a TextEncoder.

    // Inserting as a string
    await db.execute('INSERT INTO states VALUES (?)', [JSON.stringify(states)]);
    
    // Inserting as a blob (ArrayBuffer)
    function objectToArrayBuffer(obj) {
      const jsonString = JSON.stringify(obj);
      const encoder = new TextEncoder();
      const uint8Array = encoder.encode(jsonString);
      return uint8Array.buffer;
    }
    
    await db.execute('INSERT INTO states VALUES (?)', [
      objectToArrayBuffer(states),
    ]);
    
    // Querying JSONB data
    let res = await db.execute(
      `SELECT data->>'country' FROM states WHERE data->>'capital'=='Amsterdam';`
    );
  11. Retrieve the database file path

    main

    To inspect data directly or export the database for support, you can retrieve the absolute file path of the current database using the getDbPath() method on the database instance. This is useful for locating the .sqlite file on a simulator or for attaching the file to support tickets in on-device scenarios.

    const db = open({ name: 'dbPath.sqlite' });
    await db.execute('CREATE TABLE test (id INTEGER PRIMARY KEY AUTOINCREMENT)');
    const path = db.getDbPath();
    console.warn(path);
    // You can then use the path to copy to clipboard or upload