django-cacheops

Cacheops has several design compromises and technical limitations regarding query invalidation granularity and supported query types. Understanding these helps avoid unexpected cache behavior:

• Invalidation Granularity: Conditions other than __exact, __in, and __isnull=True do not make invalidation more granular. Conditions on TextFields, FileFields, and BinaryFields also do not improve granularity.
• Relationships: Updating a select_related object does not invalidate the cache for the parent queryset. Use .prefetch_related() instead to ensure proper invalidation.
• Mass Updates: Mass updates do not trigger invalidation by default. Use .invalidated_update() to ensure the cache is cleared.
• Query Types: Cacheops does not work correctly with .raw() SQL queries, subqueries (conditions on subqueries don't affect invalidation), or multi-table inheritance.
• Slicing: Sliced queries are invalidated as if they were non-sliced queries.

If you encounter these limitations, you can often bypass them using the @cached_as() decorator.

Follow these best practices to maximize performance:

1. Serialization: Use django-pickling to speed up model instance pickling/unpickling.
2. Querysets: Use .inplace() for micro-optimizations in hot code paths to avoid queryset cloning.
3. Template Caching: Use template fragment caching where possible. Caching a string is significantly faster than pickling/unpickling a list of model instances.
4. Redis Configuration: Run a separate Redis instance for the cache and disable persistence. Manually trigger SAVE or BGSAVE to maintain data if necessary.
5. Filter Complexity: Avoid caching querysets with many complex filters. High filter complexity can lead to frequent cache misses and slow down invalidation processes. Consider disabling cache for requests that use many primary fields in filters.
6. Granularity: Split queries into smaller parts to allow for more granular invalidation.

The CACHEOPS dictionary defines which operations (ops) are cached for specific models and for how long (timeout).

Key options:

• ops: A set of operations to cache. Supported values: get, fetch, count, aggregate, exists. Use 'all' as an alias for all five. An empty set () or None disables caching.
• timeout: Cache duration in seconds.
• local_get: If True, caches simple gets in process local memory (very fast, but not invalidated until process restart).
• cache_on_save: If True, writes an instance to cache upon save (caches by primary key). If a field name is provided, it caches by that field.
• CACHEOPS_DEFAULTS: A dictionary to set default ops and timeout for all models to avoid repetition.

Warning: Using '*.*' with non-empty ops is not recommended as it may cache unintended tables like migrations. Prefer 'app_name.*'.

CACHEOPS_DEFAULTS = {
    'timeout': 60*60
}

CACHEOPS = {
    # Cache User.get() calls for 15 minutes
    'auth.user': {'ops': 'get', 'timeout': 60*15},

    # Cache all gets and fetches for auth models for an hour
    'auth.*': {'ops': {'fetch', 'get'}, 'timeout': 60*60},

    # Cache all queries (get, fetch, count, aggregate, exists) for Permission
    'auth.permission': {'ops': 'all', 'timeout': 60*60},

    # Enable manual caching on all other models with 1 hour timeout
    '*.*': {'timeout': 60*60},

    # Explicitly forbid caching for a specific app
    'some_app.*': None,
}

To achieve more precise cache invalidation, split complex queries into smaller, simpler ones. This allows individual parts of the data to be invalidated independently rather than flushing larger chunks of the cache.

Example: Inefficient (Broad Invalidation)

Post.objects.filter(category__slug="foo")
# A single query that invalidates on ANY Post change OR any Category with slug='foo' change.

Example: Efficient (Granular Invalidation)

Post.objects.filter(category=Category.objects.get(slug="foo"))
# Two queries: one for the category and one for the post. 
# This invalidates only on specific category changes or specific post changes.

Post.objects.filter(category__slug="foo")

# A single database query, but will be invalidated not only on
# any Category with .slug == "foo" change, but also for any Post change

Post.objects.filter(category=Category.objects.get(slug="foo"))
# Two queries, each invalidates only on a granular event:
# either category.slug == "foo" or Post with .category_id == <whatever is there>