OceanBase seekdb

repository·develop·Indexed 24 days ago

https://github.com/oceanbase/seekdb

A MySQL-compatible, ACID-compliant state store designed for AI agents. It supports hybrid search (vector, full-text, and scalar) and features Copy-on-Write (COW) sandboxing for safe agent experimentation. The system includes an async index pipeline with incremental HNSW for real-time vector searchability and provides a Python SDK via pyseekdb.

Tokens
41.4K
Snippets
108
Records
230
Agent score
74%

What's inside seekdb

  1. Overview of seekdb features

    develop

    seekdb is a state store designed for AI agents, offering:

    • MySQL Compatibility: Built on the OceanBase SQL engine, supporting the MySQL protocol and ACID compliance.
    • Hybrid Search: Combines vector, full-text, and scalar filtering in one SQL statement.
    • High Performance: Uses an asynchronous index pipeline (Change Stream) and 2-level HNSW (incremental + snapshot) to provide high QPS with stable P99 latency during concurrent streaming writes and searches.
    • Sandboxing: Kernel-level Copy-on-Write (COW) via FORK DATABASE for safe agent exploration.
    • Deployment Modes: Supports embedded (in-process), single-node server, and OceanBase distributed cluster modes.
  2. SeekDB Ecosystem and Integrations

    develop

    seekdb is integrated with several AI and LLM orchestration frameworks, including:

    • LangChain
    • LlamaIndex
    • Dify
    • LangGraph
    • Coze
    • HuggingFace

    Additional integrations include Camel-AI, DB-GPT, FastGPT, Firecrawl, Spring-AI-Alibaba, Cloudflare Workers AI, Jina AI, Ragas, Instructor, and Baseten. For a complete list, consult the User Guide.

  3. Choose a Map implementation in seekdb

    develop

    seekdb provides several Map implementations depending on your performance and ordering requirements.

    • Balanced Search Trees: Use ObRbTree if you need ordered keys (Red-Black Tree implementation).
    • Hash Maps: Use these for high-efficiency insertion and query when order is not required. Available implementations include:
      • ObHashMap
      • ObLinkHashMap
      • ObLinearHashMap

    It is recommended to use the standard implementations unless you have a specific requirement for a specialized hash map.

  4. Access seekdb official documentation

    develop
    The official documentation for seekdb, including product overviews, developer guides, tutorials, and integration references, is hosted in a dedicated repository and a documentation site. Use these resources to learn about hybrid search (relational, vector, text, JSON, and GIS), end-to-end development workflows using SQL or the seekdb SDK, and deployment/configuration guides.
  5. Quickstart with pyseekdb

    develop

    You can try seekdb in 30 seconds using the pyseekdb Python library. It works in embedded mode (no server required, schema-less) and uses an asynchronous indexing pipeline. If you need to query immediately after writing, call refresh_index() to ensure the index is prepared.

    import pyseekdb
    
    client = pyseekdb.Client(path="./agent_state.db")
    memory = client.get_or_create_collection(name="episodic")
    
    # Write observations
    memory.upsert(
        ids=["1", "2", "3"],
        documents=[
            "user prefers dark mode",
            "user speaks English and Chinese",
            "user timezone is UTC+8",
        ],
    )
    memory.refresh_index()
    
    # Query
    results = memory.query(query_texts="ui preferences?", n_results=1)
    print(results["documents"])
    # -> [['user prefers dark mode']]
    
    # Write new observation and refresh
    memory.upsert(ids=["4"], documents=["user saw pricing page 3 times today"])
    memory.refresh_index()
    
    results = memory.query(query_texts="purchase intent signals", n_results=1)
    print(results["documents"])
    # -> [['user saw pricing page 3 times today']]
  6. Quickstart with pyseekdb (Embedded Mode)

    develop

    You can run seekdb in-process without a separate server. Use pyseekdb.Client to initialize a local database file, create a collection, and perform upserts and queries. Note that you must call refresh_index() to make new writes immediately queryable.

    import pyseekdb
    
    client = pyseekdb.Client(path="./agent_state.db")
    memory = client.get_or_create_collection(name="episodic")
    
    # Write Agent observations
    memory.upsert(
        ids=["1", "2", "3"],
        documents=[
            "user prefers dark mode",
            "user speaks English and Chinese",
            "user timezone is UTC+8",
        ],
    )
    memory.refresh_index()
    
    # Query the data
    results = memory.query(query_texts="ui preferences?", n_results=1)
    print(results["documents"])
    # -> [['user prefers dark mode']]
    
    # Write new observation and refresh
    memory.upsert(ids=["4"], documents=["user saw pricing page 3 times today"])
    memory.refresh_index()
    
    results = memory.query(query_texts="purchase intent signals", n_results=1)
    print(results["documents"])
    # -> [['user saw pricing page 3 times today']]
  7. Declare and manage local variables

    develop

    Follow these rules for local variable management to ensure readability and performance:

    Best Practices:

    • Placement: Declare variables at the beginning of a statement block. If the declaration and use are far apart, refactor the block.
    • Initialization: Simple variables should be initialized upon declaration.
    • Loop Efficiency: Avoid declaring complex variables (e.g., class instances) inside a loop body, as this causes repeated construction/destruction. Extract them outside the loop.
      • Note: Declaring references inside a loop is allowed for readability.

    Constraints:

    • Stack Limit: The function stack should not exceed 32K.
    • Variable Size: A single local variable should not exceed 8K.
    • Complex Variables in Loops: Prohibited unless approved by a team leader with detailed comments.
    // Inefficient implementation
    for (int i = 0; i < 100000; ++i) {
      ObFoo f;  // Constructor/destructor called every iteration
      f.do_something();
    }
    
    // Efficient implementation
    ObFoo f;
    for (int i = 0; i < 100000; ++i) {
      f.do_something();
    }
    
    // References are allowed inside loops for readability
    for(int i = 0; i < N; ++i) {
       const T &t = very_long_variable_name.at(i);
       t.f1();
       t.f2();
    }
  8. Apply typography and indentation styles

    develop

    Standardize code formatting using these rules:

    • Indentation: Use two spaces for indentation. Do not use the Tab key.
    • Line Length: Maximum 100 characters per line. Exception: Up to 120 characters for long URLs, commands in comments, or long paths.
    • Empty Lines: Minimize blank lines. Only use them to separate distinct logical parts of a function. Do not place blank lines at the very beginning or end of a function body or code block.
  9. Set the seekdb log level

    develop

    You can adjust the system log level using one of the following three methods:

    1. At Startup: Set the syslog_level configuration item in the configuration file or via command-line parameters.
    2. After Startup (SQL): Connect via a MySQL client and execute:
      alter system set syslog_level='DEBUG';
    3. Per Request (SQL Hint): Use a SQL hint to change the level for the current request only:
      select /*+ log_level("ERROR") */ * from foo;
  10. Build seekdb for Android

    develop

    Follow these steps to initialize dependencies and build the seekdb binaries for Android.

    1. Initialize dependencies

    Run the build script with the --android and --init flags to download and extract pre-built NDK dependency tarballs into deps/3rd/.

    2. Configure and build

    Navigate to the build directory and use make to build the seekdb observer binary.

    3. Build unit tests (optional)

    To build a combined all_tests binary containing all unit tests, run make all_tests.

  11. Handle global variables and functions

    develop

    The use of global variables and functions is strictly limited.

    Guidelines:

    • New Globals: Do not add new global variables or functions without prior discussion and approval.
    • Singletons: If a variable must be shared globally, place it in a server singleton (e.g., ObUpdateServerMain in UpdateServer).
    • Constants: Global constants should be defined in ob_define.h. Global functions should be in common/ob_define.h or utility headers like common/utility.h or common/ob_print_utils.h.
    • Header Restriction: It is prohibited to define global const variables in header files. Doing so creates internal linkage, resulting in multiple copies of the variable in the binary. Use extern const in headers and define the value in a .cpp file instead.
  12. Configure and control Debug Sync via SQL

    develop

    Debug Sync is disabled by default. Use the following SQL commands to manage its lifecycle:

    1. Enable the mechanism

    Set debug_sync_timeout to a value greater than 0 (unit is microseconds).

    alter system set debug_sync_timeout='100000s';

    2. Activate a specific sync point

    Use set ob_global_debug_sync to tell a thread to wait at a specific point. Format: 'POINT_NAME wait_for signal_name execute MAX_EXECUTIONS'

    set ob_global_debug_sync = 'BEFORE_UNIT_MANAGER_LOAD wait_for my_signal execute 10000';

    3. Signal the thread to continue

    Wake up the hanging thread using signal (single thread) or broadcast (all threads waiting on that signal).

    set ob_global_debug_sync = 'now signal my_signal';
    -- or
    set ob_global_debug_sync = 'now broadcast my_signal';

    4. Clear or Disable

    To remove a specific point or disable the mechanism entirely:

    -- Clear a specific point
    set ob_global_debug_sync = 'BEFORE_UNIT_MANAGER_LOAD clear';
    
    -- Disable debug sync globally
    alter system set debug_sync_timeout=0;