SugarDB Documentation

repository·main·Indexed 19 days ago

https://github.com/echovault/sugardb

A highly configurable, distributed, in-memory data store and cache written in Go. SugarDB supports embedded usage as a library and standalone client-server deployment with RAFT-based replication. It features RESP protocol compatibility for Redis clients, supporting a wide range of commands for Strings, Hashes, Lists, Sets, Sorted Sets, and Pub/Sub. The system includes support for Access Control Lists (ACL), TLS/mTLS security, AOF persistence, and Docker-based clustering.

Tokens
60.1K
Snippets
336
Records
380
Agent score
68%

What's inside SugarDB

  1. What is SugarDB?

    main

    SugarDB is a highly configurable, distributed, in-memory data store and cache implemented in Go. It is designed to provide a rich set of data structures for in-memory data manipulation and can be used in two primary ways:

    1. As a Go library: Import it directly into your Go projects to enhance them with in-memory data structures.
    2. As an independent service: Run it as a standalone server or a distributed cluster.

    Core capabilities include support for various data structures (Lists, Sets, Sorted Sets, Hashes, etc.), a persistence layer for recovery, and replication via the RAFT algorithm for fault tolerance.

  2. Understand SugarDB deployment modes

    main

    SugarDB supports different operational modes depending on your consistency and scaling requirements:

    • Standalone mode: A single instance running in isolation. Best for simple, single-node use cases.
    • Replication cluster: A strongly consistent cluster using the RAFT consensus algorithm. Use this mode when you require high availability and strong data consistency across multiple nodes.
    • Sharding: (Planned/To be implemented) A mechanism for horizontal scaling across multiple nodes.
  3. Understand passive vs active eviction in SugarDB

    main

    SugarDB employs two mechanisms for handling expired keys:

    1. Passive Eviction: Expired keys are not deleted immediately upon expiry. Instead, they remain in the store until a user attempts to access them. The deletion is triggered at the moment of access.
    2. Active Eviction: A background process periodically samples keys to find and delete expired ones. This prevents expired data from sitting in memory indefinitely if it is never accessed again. This process is controlled by --eviction-sample and --eviction-interval.
  4. Understand SugarDB persistence strategies

    main

    SugarDB operates primarily in-memory for performance but provides mechanisms to persist data to disk to ensure data recovery after an instance restart.

    There are two primary persistence strategies:

    1. Append-Only Files: Records changes sequentially to a file.
    2. Snapshots: Periodically saves the entire state of the database to disk.

    Important Behavior Note: When running in standalone mode, if both the Append-Only and Snapshot strategies are configured simultaneously, SugarDB will prioritize and use the append-only strategy.

  5. Key Features of SugarDB

    main

    SugarDB provides several enterprise-grade features for distributed data management:

    • Security: Supports TLS and mTLS with multiple server and client RootCAs, alongside an ACL (Access Control List) layer for Authentication and Authorization.
    • High Availability: Replication cluster support using the RAFT algorithm for fault tolerance.
    • Data Structures: Built-in support for Sets, Sorted Sets, Hashes, and Lists.
    • Messaging: Distributed Pub/Sub functionality with support for consumer groups.
    • Reliability: A persistence layer utilizing both Snapshots and Append-Only files (AOF).
    • Memory Management: Configurable Key Eviction Policies.
  6. Define a Command with Subcommands

    main

    You can create hierarchical commands by providing a slice of SubCommandOptions to the SubCommand property of CommandOptions.

    Behavioral Rules:

    • Handler Priority: If a top-level command has subcommands, the top-level HandlerFunc is ignored. SugarDB will attempt to match the second element of the command string against the registered subcommands.
    • Subcommand Matching: If a subcommand is not found, an error is returned.
    • Independence: While subcommands can share handlers, it is best practice for each subcommand to provide its own unique KeyExtractionFunc and HandlerFunc.
    // Define the subcommands
    subCommands := []db.SubCommandOptions{
      {
        Command:           "SUB1",
        Module:            "mymodule",
        Categories:        []string{"subcategory1"},
        Description:       "This is subcommand 1",
        Sync:              false,
        KeyExtractionFunc: mySubCommandKeyExtractionFunc,
        HandlerFunc:       mySubCommandHandler,
      },
      {
        Command:           "SUB2",
        Module:            "mymodule",
        Categories:        []string{"subcategory2"},
        Description:       "This is subcommand 2",
        Sync:              true,
        KeyExtractionFunc: mySubCommandKeyExtractionFunc,
        HandlerFunc:       mySubCommandHandler,
      },
    }
    
    // Define the main command options
    command := db.CommandOptions{
      Command:     "MYCOMMAND",
      Module:      "mymodule",
      Categories:  []string{"category1"},
      Description: "This is a sample command with subcommands",
      Sync:       true,
      SubCommand:  subCommands,
    }
    
    err := server.AddCommand(command)
  7. Configure Memory Management and Eviction Policies

    main

    To prevent the server from exceeding memory limits, use --max-memory and --eviction-policy.

    Available Eviction Policies:

    1. noeviction: (Default) Reject all new write operations when limit is reached.
    2. allkeys-lfu: Evict least frequently used keys.
    3. allkeys-lru: Evict least recently used keys.
    4. volatile-lfu: Evict least frequently used keys that have an expiration.
    5. volatile-lru: Evict least recently used keys that have an expiration.
    6. allkeys-random: Evict random keys.
    7. volatile-random: Evict random keys that have an expiration.

    Tuning Eviction:

    • Use --eviction-sample <integer> to set the number of keys sampled (default is 20).
    • Use --eviction-interval <string> to set the sampling frequency (e.g., 10s, 100ms).
    # Example: Limit memory to 2GB and use LRU eviction
    sugardb --max-memory 2gb --eviction-policy allkeys-lru
  8. Configure Raft Replication Cluster

    main

    When running SugarDB in a replication cluster, use the following flags to manage node identity and discovery:

    • --server-id <string>: Assign a unique ID to the node.
    • --bootstrap-cluster <boolean>: Set to true on the first node to initialize the cluster.
    • --join-addr <string>: Use this on subsequent nodes to join the cluster. Format: <target-server-id>/<target-ip>:<target-port>.
    • --discovery-port <integer>: Port for memberlist communication (default 7946).
    • --forward-commands <boolean>: If true, nodes can accept write commands and forward them to the leader.
    • --in-memory <boolean>: Use only for testing to keep Raft logs/snapshots in memory.
    # Bootstrap the first node
    sugardb --server-id node1 --bootstrap-cluster
    
    # Join a second node to the cluster
    sugardb --server-id node2 --join-addr node1/127.0.0.1:7480
  9. How the Append-Only File (AOF) works in SugarDB

    main

    SugarDB uses an append-only log file to track every write command. This provides a mechanism for data durability and recovery.

    Lifecycle:

    1. Logging: Every write command is appended to the log file.
    2. Compaction: When a configured threshold of write commands is reached, SugarDB performs a compaction. It creates a snapshot of the current data and starts a fresh log file.
    3. Restoration: Upon startup, SugarDB loads the latest data snapshot and then replays all write commands from the current log file to reach the most recent state. If no snapshot exists, it replays the entire log file.

    Note: This AOF mechanism applies only to standalone nodes. In a replication cluster, logging and compaction are managed via the hashicorp/raft package (currently backed by boltdb).

  10. Configure TLS and mTLS Security

    main

    SugarDB supports TLS and mutual TLS (mTLS) for secure client connections.

    TLS Configuration:

    • --tls: Enables TLS.
    • --cert-key-pair <string>: Provides the server certificate and key. Format: <path-to-cert>,<path-to-key>. Can be specified multiple times.

    mTLS Configuration:

    • --mtls: Enables mTLS. This takes higher priority than --tls if both are provided.
    • --client-ca <string>: Path to the RootCA used to verify client certificates. Can be specified multiple times to support several RootCAs.
    # Example: Enable mTLS with specific certs and CA
    sugardb --mtls --cert-key-pair /etc/sugardb/server.crt,/etc/sugardb/server.key --client-ca /etc/sugardb/client-ca.crt
  11. Ways to extend SugarDB functionality

    main

    SugarDB provides several mechanisms to add new commands and extend its core functionality as the built-in command set evolves. Currently, you can extend SugarDB using:

    1. Embedded API: Integrate directly with the SugarDB engine within your application.
    2. Shared Object Plugins: Load external shared object files to add functionality.
    3. Lua Modules: (Planned/Coming soon) Use Lua scripts to extend capabilities.

    Note that if you are migrating from Redis, some Redis commands may not be natively implemented in SugarDB and will require one of these extension methods.