DOCSEARCHbyAlgolia
Submit Docs

otter

repository·main·Indexed 25 days ago

https://github.com/maypok86/otter

A high-performance in-memory caching library for Go, designed with principles from Caffeine. It features adaptive W-TinyLFU for high hit rates, low memory overhead, and excellent throughput under contention. Otter v2 supports size-based eviction, time-based expiration, asynchronous refresh, and automatic loading of entries. It requires Go version 1.24 or above.

Tokens
17.2K
Snippets
34
Records
97
Agent score
83%

What's inside otter

  1. Overview of Otter In-memory Caching Library

    main
    Otter is a high-performance, in-memory caching library for Go. It is designed to provide high hit rates using adaptive W-TinyLFU, excellent throughput under high contention, and low memory overhead. It features self-tuning data structures that automatically configure themselves based on workload patterns and parallelism.

  2. Understand Otter's concurrency and consistency model

    main

    Otter uses an eventually consistent model for its page replacement algorithms. Updates to the underlying xsync.Map and the recording of reads may not be immediately reflected in the eviction policy's data structures.

    To minimize lock contention, operations are applied in batches. This means the amortized cost of an operation is slightly higher than a simple xsync.Map operation, but it prevents performance degradation under high concurrency.

    Key State Transitions: Entries in the cache transition through three states:

    • Alive: Present in both the hash table and the page replacement policy.
    • Retired: Deleted from the hash table, but pending deletion from the policy.
    • Dead: Absent from both structures.

  3. Optimize memory with Otter's Fastpath and Code Generation

    main

    Otter includes optimizations to reduce memory overhead and improve performance during specific lifecycle stages:

    Fastpath (Low Capacity Mode)

    When the cache is below 50% of its maximum capacity, Otter enters a 'Fastpath' mode:

    • The frequency sketch is not initialized to save memory.
    • Accesses are not recorded (unless required by other features) to avoid read buffer contention.
    • The eviction policy is not yet fully enabled.

    Code Generation for Memory Efficiency

    To minimize per-entry overhead, Otter uses code generation to create optimal entry implementations. Instead of including all possible configuration fields in every entry (which would be wasteful), Otter generates specific implementations based on the enabled features. This reduces runtime memory consumption at the cost of a slightly larger binary size.

  4. Compare Otter v1 vs Otter v2 features

    main

    When choosing between versions of Otter, consider your workload and feature requirements:

    Otter v1

    Best for high-throughput scenarios where you do not need loading or refreshing features and your workload is not frequency-skewed.

    • Pros: High throughput (via xsync.Map), high hit rate for most workloads, low memory overhead.
    • Cons: Lacks cache stampede protection (loading) and asynchronous refreshing; hit rate may be lower than TinyLFU/W-TinyLFU on frequency-skewed workloads.

    Otter v2

    Designed to be a comprehensive, high-performance caching library with an extensible API.

    • Key Features: Loading, refreshing, entry pinning, and Compute* methods.
    • Eviction Policy: Uses adaptive W-TinyLFU for high hit rates across all workloads.
    • Security: Includes HashDoS protection.
    • Improvements: Rethought API (e.g., TTL resets on every access), reworked task scheduling, and auto-configurable lossy read buffers.

  5. Evaluate Sturdyc for caching needs

    main

    Sturdyc is a Go caching library focused on advanced features, though it may have performance trade-offs.

    • Advantages: Provides cache stampede protection, supports refreshing, and supports bulk loading/refreshing.
    • Disadvantages:
      • Lacks a strong eviction policy (O(n) complexity, often worse hit rate than LRU).
      • Lacks a strong expiration policy.
      • Performance lags behind Ristretto, Theine, and Otter.
      • Only supports string keys (no generics).
      • Cannot deduplicate calls when some keys are batched while others are not.
      • May permit dirty data during concurrent invalidation and loading.

  6. Understand Otter throughput benchmarks

    main

    Otter's throughput benchmarks are a Go port of the Caffeine benchmarks. They use a pre-populated cache with a zipf distribution for access requests to create hot spots, mimicking real-world cache contention.

    Key characteristics of these benchmarks:

    • Isolates cache costs: Measures performance in concurrent workloads by minimizing overhead (thread-local index increments and array lookups).
    • Avoids uniform distribution: Uses skewed distributions to ensure locks suffer higher contention, testing the implementation's ability to handle hardware cache coherence and branch prediction.
    • Pre-populated cache: The cache is initially full to ensure the benchmark measures cache hits/updates rather than the cost of cache misses.

    Otter maintains high throughput across most workloads, including various read/write ratios, though performance may decrease in extreme write-heavy (update-heavy) scenarios.

  7. Understand Otter's hit rate performance characteristics

    main

    Otter uses a W-TinyLFU admission policy which provides high hit rates across diverse workloads. Key characteristics include:

    • General Purpose Suitability: Provides substantial improvement over LRU across various workloads.
    • Efficiency: Achieves high hit rates without requiring non-resident entries and maintains a low memory footprint.
    • Stability: Maintains stable performance in looping access patterns (e.g., the Glimpse trace) where other implementations like Theine may exhibit fluctuations.
    • Workload Versatility: Performs well across traditional traces including Zipf, S3 (search engine disk reads), DS1 (ERP database), P8 (Windows NT disk operations), Glimpse (looping patterns), OLTP (CODASYL database), Scarab (frequency-negative workloads), and Mixed (shifting between recency-skewed and frequency-skewed patterns).

  8. Understand Otter memory consumption characteristics

    main

    Otter is designed to maintain low memory overhead across various cache capacities, even when using expiration policies and its internal read/write buffer implementation.

    When planning your cache size, note that Otter's memory footprint scales with the number of entries. Benchmarks are typically conducted using fixed-size 32-byte key-value pairs with expiration enabled to represent common real-world usage patterns.

  10. Configure time-based eviction in Otter v2

    main

    Otter v2 supports several time-based expiration strategies:

    • Expiration after creation: Entries expire after a fixed duration from when they were first inserted.
    • Expiration after last write: Entries expire after a fixed duration since they were last updated.
    • Expiration after last access: Entries expire after a fixed duration since they were last read or written (sliding window).
    • Custom ExpiryCalculator: Implement a custom logic for determining when an entry should expire.

  11. Configure key refreshing in Otter v2

    main

    In Otter v2, you can configure automatic background refreshing by providing a RefreshCalculator in the otter.Options.

    Unlike eviction, refreshing is asynchronous: when a key is refreshed, the old value is returned immediately while the new value is loaded in the background. A refresh is only triggered when the entry is queried after the duration specified by the RefreshCalculator has passed. If an entry is not queried after becoming eligible for refresh, it will eventually expire based on the ExpiryCalculator.

    To implement smart refresh logic, override the Loader.Reload method. This allows you to use the existing (old) value to compute the new value. Note that the Loader must return ErrNotFound if the entry does not exist in the data source. If a refresh operation fails, the old value is retained, the error is logged via the configured Logger, and the error is swallowed.

    cache := otter.Must(&otter.Options[string, string]{
	ExpiryCalculator: otter.ExpiryWriting[string, string](time.Hour),
	RefreshCalculator: otter.RefreshWriting[string, string](30*time.Minute),
})

  12. Refresh entries in Otter v2

    main

    Refresh allows you to update the value of an entry in the cache. This can be done via:

    • Get with refresh: Automatically refreshing the value during a Get operation.
    • Manual refresh: Explicitly calling a refresh operation.

    Note: The same logic applies to both single-key and BulkGet/BulkRefresh operations.