DOCSEARCHbyAlgolia
Submit Docs

LokiDB

repository·master·Indexed 41 days ago

https://github.com/lokijs-forge/lokidb

A high-performance, document-oriented NoSQL in-memory database written in TypeScript and the official successor to LokiJS. Version 2.1.0 features a modular plugin system for persistence (Node.js fs, IndexedDB, Local Storage), full-text search, and customizable comparators and operator packages for optimized data retrieval and filtering.

Tokens
30.5K
Snippets
128
Records
150
Agent score
49%

What's inside LokiDB

  1. Learn about LokiDB core topics

    master

    LokiDB covers several advanced concepts and features that you can explore to build more complex database logic:

    • Query Examples: Practical patterns for querying data.
    • Persistence: How to save and load database state.
    • CollectionTransforms: Mechanisms for transforming collections.
    • Understanding Comparators: How data comparison logic works within the engine.
    • LokiOperatorPackage: Using operator packages for advanced querying.
    • Loki Ranged Indexes: Utilizing ranged indexes for optimized data retrieval.

  2. What is a LokiOperatorPackage?

    master

    A LokiOperatorPackage is the set of unindexed or "fallback" implementations for all supported query operators (e.g., $eq, $gt, $lt) used in collection.find() or query chains.

    While Ranged Indexes provide high performance for a specific subset of operators ($eq, $lt, $lte, $gt, $gte, $between), any other operators used in a query—or any operators used when a ranged index is not applicable—fall back to the LokiOperatorPackage registered as the collection's default.

  3. Understand LokiDB Comparators

    master

    Comparators in LokiDB are functions used for sorting, indexing, and filtering. They accept two values and must return:

    • 0: if the values are equal
    • -1: if the first value is less than the second
    • 1: if the first value is greater than the second

    LokiDB uses a ComparatorMap (a collection of named comparator functions) to decouple these functions from specific components. Comparators are utilized for:

    • Unindexed sorting
    • RangedIndexes (like the avl index) for high-performance filtering
    • Unindexed find() operations (e.g., $eq, $gt, $lt) when using the ComparatorOperatorPackage.

  4. How RangedIndexes work in LokiDB

    master

    RangedIndexes is an interface that allows LokiDB to use high-performance indexing algorithms for property range queries.

    Currently, LokiDB uses an AVL self-balancing binary tree (which supports duplicates) as its primary implementation of the IRangedIndex interface. This implementation is optimized for scalability, maintaining high performance for operations like $eq, $lt, $lte, $gt, $gte, $between, and $in even as collection sizes grow.

    Because IRangedIndex is an abstraction, the system is designed to allow for future alternative implementations (such as Red-Black trees) or user-provided custom implementations.

  5. Implement and register a custom IRangedIndex

    master

    You can extend LokiDB by implementing the IRangedIndex<T> interface and registering your own implementation via a factory function.

    To use a custom index:

    1. Implement the IRangedIndex<T> interface (including methods like insert, update, remove, rangeRequest, etc.).
    2. Create a factory function that returns an instance of your implementation.
    3. Register the factory function in the Loki constructor using the rangedIndexFactoryMap option.
    4. Reference your custom index by its name in the rangedIndexes configuration when calling addCollection.

    Note: The indexTypeName in addCollection must match the key used in rangedIndexFactoryMap.

    // 1. define index implementation
class customRangedIndex<T> implements IRangedIndex<T> {
  public name: string;
  public comparator: ILokiComparer<T>;

  constructor(name: string, comparator: ILokiComparer<T>) {
    this.name = name;
    this.comparator = comparator;
  }

  insert(id: number, val: T) {
    if (!id || !val) throw new Error("");
    return;
  }
  update(id: number, val: T) {
    if (!id || val === null) throw new Error("");
    return;
  }
  remove(id: number) {
    if (!id) throw new Error("");
    return;
  }
  restore(tree: any) {
    if (!tree) throw new Error("");
    return;
  }
  backup() {
    return this;
  }
  rangeRequest(range?: IRangedIndexRequest<T>) {
    if (range === null) {
      // return everything
      return <number[]> [];
    }
    return <number[]> [];
  }
  validateIndex() {
    return true;
  }
}

// 2. ranged index implementations need factory function
let myCustomIndexFactory = (name: string, cmp: ILokiComparer<any>) => { 
  return new customRangedIndex<any>(name, cmp); 
};

// 3. register ranged index factory function with loki constructor
let db = new Loki("test.db", {
  rangedIndexFactoryMap: {
    "MyCustomRangedIndex": myCustomIndexFactory
  }
});

// 4. utilize your registered ranged index within a collection
let items = db.addCollection<TestUserType>("users", {
  rangedIndexes: {
    "name": { indexTypeName: "MyCustomRangedIndex", comparatorName: "js" }
  }
});

  6. Define and register custom comparators

    master

    If built-in comparators do not meet your requirements, you can define custom comparator functions and add them to the ComparatorMap during database instantiation via the Loki constructor.

    Note: Registration must be performed every time you re-instance the database. Only the names of the comparators are persisted in serialized databases.

    Warning: If a custom comparator is used with a ranged index, the comparison logic must not change over time. Ranged indexes rely on consistent comparison results to maintain their ordered representation during inserts, updates, and removals. If the comparator is not used by a ranged index, you can safely update it.

    // define comparator function
let myComparator = (a, b) => {
   if (a === b) return 0;
   if (a > b) return 1;
   return -1;
}

// pass in any number of comparators to add to or override those already registered
let db = new Loki("test.db", {
   comparatorMap: {
      "MyComparator": myComparator
   }
});

  7. Set up the LokiDB development environment

    master

    To build and test LokiDB locally, ensure your development machine has the following software installed:

    • Git: For version control.
    • Node.js (version >=7.0.0): Used to run tests and generate distributable files.
    • npm (version >=4.0.0): Node's package manager (included with Node.js).
    • Elasticsearch: Required specifically for unit testing LokiDB's full-text search capabilities.

  8. Create and register a custom LokiOperatorPackage

    master

    To create a custom operator package, extend the LokiOperatorPackage class and override specific operator methods. To make it available to your database, register it in the lokiOperatorPackageMap option during the Loki constructor initialization.

    Note: Registering a package with an existing name will overwrite the native implementation.

    // extend and override with some opposite logic
class MyOperatorPackage extends LokiOperatorPackage {
  $gt(a, b): boolean {
    if (a < b) return true;
    return false;
  }

  $lt(a, b): boolean {
    if (a > b) return true;
    return false;
  }
}

// inject our new operator into global LokiOperatorPackageMap.
let db = new Loki("test.db", {
  lokiOperatorPackageMap : {
    "MyOperatorPackage": new MyOperatorPackage()
  }
});

  9. Run LokiDB tests

    master

    LokiDB provides several npm scripts to execute tests across different environments. Note that all tests must pass on the Continuous Integration (CI) infrastructure before a Pull Request can be merged.

    Use the following commands to run tests:

    • npm run test: Runs all LokiDB tests.
    • npm run test:web: Runs tests specifically in a web browser environment.
    • npm run test:node: Runs tests specifically in a Node.js environment.
    $ npm run test            # Run all LokiDB tests
$ npm run test:web        # Run tests only in web browser
$ npm run test:node       # Run tests only in node

  11. Create an operator package based on comparators

    master

    If you want to use a custom comparator to drive the logic for $eq, $lt, $lte, $gt, $gte, and $between operations, instantiate a ComparatorOperatorPackage with your comparator function and register it in the lokiOperatorPackageMap during database initialization.

    // create a custom case-insensitive string comparator
let customComparator: ILokiComparer = (a, b) => {
  if (typeof a === "string" && typeof b === "string") {
    a = a.toLocaleLowerCase();
    b = b.toLocaleLowerCase();
  }

  if (a === b) return 0;
  if (a > b) return 1;
  return -1;
};

// instantiate an operator package based on that comparator
let myComparatorOperatorPackage = new ComparatorOperatorPackage(customComparator);

// inject as named operator package within the Loki constructor
let db = new Loki("test.db", {
  lokiOperatorPackageMap: {
    "MyComparatorOperatorPackage": myComparatorOperatorPackage
  }
});

  12. Apply an AVL ranged index to a collection property

    master

    To improve performance for range-based queries (like $lt, $gt, $between), you can apply an avl ranged index to a specific property in a collection. The AVL index is a self-balancing binary tree that is highly scalable and performs significantly better than standard binary indices for large collections (over a few thousand documents) due to its logarithmic maintenance costs.

    Use the rangedIndexes option in addCollection to specify the indexTypeName as "avl".

    const db = new Loki("idxtest");

// create a "users" collection, applying an avl index to the "name" property of its documents
const items = db.addCollection("users", {
  rangedIndexes: {
    name: { indexTypeName: "avl", comparatorName: "js" }
  }
});