DOCSEARCHbyAlgolia
Submit Docs

Apache Kvrocks Documentation

repository·unstable·Indexed 26 days ago

https://github.com/apache/kvrocks

Apache Kvrocks is a distributed key-value NoSQL database that uses RocksDB as its storage engine. Designed to be Redis-compatible, it allows the use of existing Redis clients while providing lower memory costs and higher capacity. The documentation covers installation, building from source, Docker deployment, namespace management, the kvrocks2redis migration utility, and detailed command references for bitwise operations and Bloom filters.

Tokens
22.7K
Snippets
16
Records
242
Agent score
88%

What's inside Apache Kvrocks

  1. Supported query languages for Kvrocks Search (KQIR)

    unstable

    Kvrocks Search (powered by the KQIR engine) supports two primary frontend query languages:

    1. SQL: An extended subset of MySQL syntax processed by the SQL Parser.
    2. RediSearch Query Syntax: Supports RediSearch query syntax (specifically targeting DIALECT 2 or greater) processed by the Redis Query Parser.

  2. Security properties and adversary model

    unstable

    Security Properties Provided

    • Namespace Isolation: Strict per-namespace keyspace confinement (except for pub/sub).
    • Privilege Separation: Only the admin (requirepass) token can run sensitive commands like config, slaveof, bgsave, and cluster operations.
    • Lua Sandboxing: Scripts run in a constrained LuaJIT environment with no host/file access and are confined to the caller's namespace.
    • Memory Safety: Designed to handle well-formed and malformed RESP input without memory corruption.

    Adversary Model

    The primary adversary is an untrusted TCP client (unauthenticated or with a limited namespace token) attempting to:

    • Access data in other namespaces.
    • Run admin commands without the admin token.
    • Break the Lua sandbox.
    • Exhaust CPU, memory, or disk resources.
    • Intercept tokens/data on unencrypted wires.

  3. Understand Kvrocks Authentication and Access Control

    unstable

    Kvrocks uses a tiered authentication model based on tokens. Access levels are determined by the token presented via the AUTH command:

    1. Unauthenticated Client: A TCP peer that has not presented a valid token. Access is restricted if requirepass or namespace tokens are configured.
    2. Namespace Client: Authenticated with a specific namespace token. This client is confined to its assigned namespace's keyspace and is denied access to administrative, namespace management, or cluster commands.
    3. Admin Client: Authenticated with the requirepass admin token. This client has full control over the instance, including running sensitive commands like config, slaveof, bgsave, namespace, cluster, and debug.

    Note on Security: If requirepass is unset in kvrocks.conf, no admin token is required, and all clients effectively have admin privileges. It is the operator's responsibility to restrict network access in this configuration.

  4. Security Properties of Lua Scripting in Kvrocks

    unstable

    Kvrocks supports Lua scripting via EVAL and FUNCTION commands using an embedded LuaJIT sandbox.

    • Isolation: Scripts are confined to the namespace of the caller.
    • Sandbox Guarantees: The sandbox is intended to deny arbitrary host access and prevent cross-namespace or host resource access.
    • Risk: While sandboxed, the operator should be aware that Lua scripting adds a code-execution surface. Scripts run by non-admin callers are restricted to their namespace.

  6. Migrate data from Kvrocks to Redis using kvrocks2redis

    unstable

    The kvrocks2redis utility migrates data from a Kvrocks instance to Redis. It follows a two-stage logic:

    1. Incremental Synchronization: It first attempts to connect to the running Kvrocks instance to perform incremental synchronization.
    2. Full Data Parsing: If the connection or incremental synchronization fails, the tool parses the full Kvrocks data directly from the configured data directory to an AOF (Append Only File).

    Synchronization Process:

    • If incremental synchronization is active, kvrocks2redis parses incremental data into an AOF file.
    • A dedicated thread named redis-writer continuously reads the AOF file and sends its contents to the Redis instance.

  7. Run Kvrocks locally or via Docker

    unstable

    Run from local build

    Execute the compiled binary with a configuration file:

    ./build/kvrocks -c kvrocks.conf

    Run using Docker

    Use the official image or the nightly image. Port 6666 is mapped by default.

    docker run -it -p 6666:6666 apache/kvrocks --bind 0.0.0.0
# or get the nightly image:
docker run -it -p 6666:6666 apache/kvrocks:nightly
    $ ./build/kvrocks -c kvrocks.conf

  8. Secure Kvrocks deployment configuration

    unstable

    To prevent unauthenticated access and data exposure, operators must follow these security configuration requirements:

    1. Authentication: Set requirepass in kvrocks.conf before binding to any non-localhost interface. If requirepass is left unset, you must restrict network access to trusted personnel only.
    2. Network Binding: By default, Kvrocks binds to 127.0.0.1. Do not change this to a routable interface without setting an authentication password.
    3. Transport Encryption: Enable TLS by configuring tls-port, providing certificates, and setting tls-auth-clients if operating on untrusted networks. Without TLS, all data and authentication tokens are sent in plaintext.
    4. Privilege Management: Treat the requirepass token as a root-equivalent admin credential. Distribute only namespace tokens to tenants/applications. Rotate namespace tokens manually using namespace set if they are exposed.
    5. Resource Limits: Configure maxclients and size limits for values/protocols to mitigate Denial of Service (DoS) risks, as Kvrocks does not provide intrinsic DoS guarantees beyond these configured limits.

  9. Install Prerequisites for Kvrocks

    unstable

    Before building Kvrocks, install the necessary dependencies based on your operating system.

    Ubuntu / Debian

    sudo apt update
sudo apt install -y git build-essential cmake libtool python3 libssl-dev

    CentOS / RedHat

    sudo yum install -y centos-release-scl-rh
sudo yum install -y git devtoolset-11 autoconf automake libtool libstdc++-static python3 openssl-devel
# download and install cmake via https://cmake.org/download
wget https://github.com/Kitware/CMake/releases/download/v3.26.4/cmake-3.26.4-linux-x86_64.sh -O cmake.sh
sudo bash cmake.sh --skip-license --prefix=/usr
# enable gcc and make in devtoolset-11
source /opt/rh/devtoolset-11/enable

    openSUSE / SUSE Linux Enterprise

    sudo zypper install -y gcc11 gcc11-c++ make wget git autoconf automake python3 curl cmake

    Arch Linux

    sudo pacman -Sy --noconfirm autoconf automake python3 git wget which cmake make gcc

    macOS

    brew install git cmake autoconf automake libtool openssl
# please link openssl by force if it still cannot be found after installing
brew link --force openssl

  10. Build Kvrocks from source

    unstable

    Build Kvrocks using the ./x.py script.

    Standard Build

    git clone https://github.com/apache/kvrocks.git
cd kvrocks
./x.py build

    Build with TLS support

    Requires OpenSSL development libraries (e.g., libssl-dev on Debian/Ubuntu).

    ./x.py build -DENABLE_OPENSSL=ON

    Build with Lua (instead of LuaJIT)

    ./x.py build -DENABLE_LUAJIT=OFF

    Build in Debug mode

    Sets the build type to Debug (optimization level -O0).

    ./x.py build -DCMAKE_BUILD_TYPE=Debug
    $ ./x.py build

  11. Configure Kvrocks Network and Authentication via kvrocks.conf

    unstable

    Use the kvrocks.conf file to define the security posture of your Kvrocks instance. Key configuration knobs include:

    KnobDefaultDescription
    requirepassunsetSets the admin token. If unset, no authentication is required.
    bind127.0.0.1Limits the server to localhost connections. Change this to expose the service to a network.
    tls-port / tls-*offEnables TLS transport encryption. When off, tokens and data are transmitted in plaintext.
    namespace tokensnoneConfigures multi-tenant isolation via namespace-specific tokens.

    Security Warning: When TLS is disabled, the confidentiality and integrity of the wire (including auth tokens) are the responsibility of the deployment (e.g., using firewalls or private networks).