WatermelonDB

repository·master·Indexed 11 days ago

https://github.com/nozbe/watermelondb

A reactive, lazy-loaded database framework optimized for high-performance React and React Native applications. It is designed to scale from hundreds to tens of thousands of records while maintaining responsiveness by performing queries on a SQLite foundation on a separate native thread. Key features include offline-first synchronization, multiplatform support (iOS, Android, Windows, web, and Node.js), and an observable pattern for automatic UI re-rendering.

Tokens
64.8K
Snippets
201
Records
266
Agent score
93%

What's inside WatermelonDB

  1. Overview of the {fmt} library

    master
    The {fmt} library is a high-performance C++ formatting library designed to address the limitations of existing methods like printf, iostreams, Boost Format, and FastFormat. It provides a safe, fast, and feature-rich alternative for string formatting, supporting user-defined types and positional arguments (useful for internationalization).
  2. What is WatermelonDB and when to use it

    master

    WatermelonDB is a reactive database framework designed for React and React Native applications that need to scale from hundreds to tens of thousands of records while maintaining high performance.

    Unlike traditional approaches (like Redux or MobX with persistence) that load the entire database into JavaScript memory—which can cause slow app launches—WatermelonDB is lazy loaded. It only loads data when it is explicitly requested. Most querying is performed directly on a SQLite foundation on a separate native thread, ensuring the main thread remains responsive.

    Key Features:

    • Scalability: Handles large datasets efficiently.
    • Lazy Loading: Minimizes initial load time and memory usage.
    • Offline-first: Supports synchronization with your own backend.
    • Multiplatform: Works on iOS, Android, Windows, web, and Node.js.
    • Reactive: Uses an observable pattern (optionally via RxJS) so UI components automatically re-render when underlying data changes.
  3. Introduction to {fmt}

    master

    {fmt} is an open-source C++ formatting library that provides a fast and safe alternative to C stdio and C++ iostreams. It implements the C++20 std::format syntax and uses a Python-like format string syntax.

    Key features include:

    • Type Safety: Errors in format strings can be reported at compile time (in C++20).
    • Performance: Faster than printf, iostreams, and to_string.
    • Extensibility: Supports user-defined types.
    • Portability: Consistent output across platforms and support for older compilers.
    • Small Footprint: Minimal configuration requires only core.h, format.h, and format-inl.h.
  4. Understand null behavior in WatermelonDB queries

    master

    WatermelonDB query operators (like Q.gt, Q.lt, Q.oneOf, Q.notIn, Q.like) follow SQLite semantics, which differ from JavaScript regarding null values.

    Key Rules:

    • No null comparisons: Standard operators do not match null. For example, Q.where('likes', Q.lt(10)) will include records where likes is 0, but not where likes is null.
    • Column comparisons: Q.where('likes', Q.gt(Q.column('dislikes'))) will only return records where both columns are non-null.
    • oneOf / notIn: To include null values in a oneOf query, you must explicitly use Q.or with a null check. Similarly, notIn will not match records where the column is null.

    The Solution: Q.weakGt

    If you need to treat null as being less than any number (matching JavaScript behavior), use the Q.weakGt operator. For weakGt, any number is considered greater than null.

    // To include nulls in a oneOf query:
    postsCollection.query(
      Q.or(
        Q.where('status', Q.oneOf(['published', 'draft'])),
        Q.where('status', null)
      )
    )
    
    // Using weakGt to allow null comparisons:
    // This WILL match comments where likes is 5 and dislikes is null
    commentsCollection.query(
      Q.where('likes', Q.weakGt(Q.column('dislikes')))
    )
  5. Optimize Queries with Indexing

    master

    You can speed up queries by adding isIndexed: true to a column in tableSchema.

    Best Practices:

    • When to index: Most _id fields should be indexed. Indexing boolean fields is possible but provides low-quality indexing.
    • When NOT to index: Avoid indexing date (_at) columns or string columns (especially long-form text).
    • Performance Cost: Indexing increases database size and slows down create/update operations. Do not index all columns to try and make WatermelonDB faster.
  6. Use Turbo Login for high-performance initial sync

    master

    Turbo Login is an advanced optimization for the initial (login) sync that can be up to 5.3x faster and more memory-efficient. It is intended for very large datasets.

    Constraints & Requirements:

    • Use Case: Only for the first sync when the database is empty. Using it on an existing database is a serious error.
    • Data Format: pullChanges must return the raw JSON text via the syncJson key, rather than a parsed object.
    • Environment: Only works with SQLiteAdapter with JSI enabled. It does NOT work on the web or when Chrome Remote Debugging is enabled.
    • API Status: Marked as unsafe (the API may change).

    Implementation Pattern: Set unsafeTurbo: true in the synchronize options. In pullChanges, if useTurbo is true, return { syncJson: rawJsonText } instead of the standard { changes, timestamp } object.

    Handling Extra Data: Since you cannot process JSON in pullChanges during a Turbo sync, use the onDidPullChanges callback to process additional metadata or messages sent from the server.

    const isFirstSync = ...
    const useTurbo = isFirstSync
    
    await synchronize({
      database,
      pullChanges: async ({ lastPulledAt, schemaVersion, migration }) => {
        const response = await fetch(`https://my.backend/sync?${...}`)
        if (!response.ok) {
          throw new Error(await response.text())
        }
    
        if (useTurbo) {
          // NOTE: DO NOT parse JSON, we want raw text
          const json = await response.text()
          return { syncJson: json }
        } else {
          const { changes, timestamp } = await response.json()
          return { changes, timestamp }
        }
      },
      unsafeTurbo: useTurbo,
      onDidPullChanges: async ({ messages }) => {
        if (messages) {
          messages.forEach((message) => {
            alert(message)
          })
        }
      },
    })
  7. Handle database resets and migration failures

    master

    Understanding how WatermelonDB handles schema mismatches and failures is critical for data safety:

    • No migrations used: If you change the schema version without providing migrations, the database will be cleared (reset) at launch.
    • Migration failure: If a migration fails (e.g., trying to create a table that already exists), the database will fail to initialize and roll back to the previous version rather than resetting. This prevents accidental data loss.
    • Newer database version: If the database on the device has a version newer than the version defined in your code, the database will be reset.
    • Missing migration path: If a user's database version is higher than the latest available migration (e.g., user is on v4, but your migrations only go up to v3), the database will be reset.

    Rolling back changes in development

    If you make a mistake during development, follow this order to reset the database to its previous state:

    1. Comment out changes in schema.js.
    2. Comment out changes in migrations.js.
    3. Decrement the schema version number in schema.js to its original value.
    4. Refresh the app.
  8. Modify the database using Writers

    master

    All database modifications (create, update, delete) must be performed within a Writer. If you attempt to modify data outside of a writer, the operation will fail.

    There are two ways to implement a writer:

    1. Inline via database.write(): Wrap your logic in an async callback passed to database.write().
    2. Model Decorator via @writer: Define a method on your Model class and decorate it with @writer from @nozbe/watermelondb/decorators. This is the preferred way to encapsulate model-specific logic.

    Note: Refer to the Writers documentation for more details.

    // Option 1: Using database.write()
    await database.write(async () => {
      const someComment = await database.get('comments').find(commentId)
      await someComment.update((comment) => {
        comment.isSpam = true
      })
    })
    
    // Option 2: Using @writer decorator on a Model
    import { writer } from '@nozbe/watermelondb/decorators'
    
    class Comment extends Model {
      @writer async markAsSpam() {
        await this.update(comment => {
          comment.isSpam = true
        })
      }
    }
  9. Observe records and queries

    master

    You can reactively observe changes to data using RxJS Observables. This is useful for manually updating UI or logic outside of React components.

    • Model.observe(): Returns an Observable that emits the record immediately upon subscription and every time the record is updated. If the record is deleted, the Observable completes.
    • Query.observe() and Relation.observe(): Analogous to Model.observe(), but for sets of records or relationships.
    • Query.observeWithColumns(): Specifically used for maintaining sorted lists.
    • Collection.findAndObserve(id): A convenience method that combines .find(id) and .observe().
  10. Handle permissions and descendants in Sync

    master

    If your application uses permissions, granting or revoking access must be treated as a change to the data to ensure the client stays in sync.

    • Granting Access: When permission to access records is granted, the pull endpoint must include those records in the created object.
    • Revoking Access: When permission is revoked, the pull endpoint must include those records in the deleted object.
    • Hierarchy: When a permission change affects a record, you must also return all of its descendants in the sync response.
  11. How Models and Collections interact

    master

    WatermelonDB follows a delegation pattern between Model and Collection to maintain consistency:

    • Model manages the state of a specific instance (e.g., a single Task). It provides instance-level APIs like update(), markAsDeleted(), and destroyPermanently().
    • Collection manages the entire set of records.

    Example Workflow: When you call model.markAsDeleted(), the model changes its own local state and then delegates the operation to its parent Collection. The collection then notifies all collection-level observers and performs the actual database removal.