DiskCache Documentation

DiskCache provides several core data types for different caching and data storage needs:

• Cache: Manages a SQLite database and filesystem directory to store key-value pairs.
• FanoutCache: Provides a sharding layer to utilize multiple Cache objects.
• DjangoCache: Integrates DiskCache with the Django web framework.
• Deque: A cross-process, persistent replacement for Python's collections.deque.
• Index: A cross-process, persistent replacement for Python's dict.

Additionally, DiskCache provides recipes for cross-process synchronization, including memoize_stampede, Lock, and throttle.

Transactions in Cache, Deque, and Index ensure that a group of operations occurs atomically. This is critical for maintaining consistency (e.g., updating a total and a count together).

Best Practices:

• Keep transactions as short as possible to avoid blocking other writers.
• Transactions can be nested to improve performance.
• Grouping multiple writes into a single transaction can improve performance by 2x to 5x.

Note: FanoutCache and DjangoCache do not support transactions directly due to sharding. To use transactions with them, you must access a specific shard using .cache(name), .deque(name), or .index(name).

If you need to install from a cloned repository or a downloaded source archive, you can use the setup.py script to install it into your site-packages.

$ git clone https://github.com/grantjenks/python-diskcache.git
$ python setup.py install

DiskCache supports testing across multiple Python versions using tox.

Using tox (recommended):

$ tox

Using setup.py (minimal infrastructure): If you do not want to install all development requirements, you can run the tests using setup.py. This will download a minimal testing infrastructure automatically.

$ python setup.py test

Running coverage tests: Use nosetests with coverage flags to check code coverage:

$ nosetests --cover-erase --with-coverage --cover-package diskcache

To prevent multiple concurrent workers from all attempting to regenerate the same expired cache item simultaneously (a cache stampede), you can use a combination of two @cache.memoize decorators and the @dc.barrier decorator.

This pattern uses double-checked locking: the outer decorator performs an optimistic lookup with expire=0 (which checks the cache but skips the set operation), and the @dc.barrier synchronizes workers so only one proceeds to the inner @cache.memoize decorator to perform the actual computation and update.

import diskcache as dc

cache = dc.Cache()

@cache.memoize(expire=0)
@dc.barrier(cache, dc.Lock)
@cache.memoize(expire=1)
def generate_landing_page():
    time.sleep(0.2)

The simplest way to install DiskCache is using pip. It is recommended to upgrade to the latest version to ensure you have the most recent features and bug fixes.

$ pip install --upgrade diskcache

Benchmarks are performed in two steps: running the benchmark script and then plotting the results. Benchmark scripts are located in the tests directory and are prefixed with benchmark_.

Step 1: Run the benchmark

$ python tests/benchmark_core.py --help

Benchmark arguments:

• -h, --help: Show help message
• -p PROCESSES, --processes PROCESSES: Number of processes to start (default: 8)
• -n OPERATIONS, --operations OPERATIONS: Number of operations to perform (default: 100000)
• -r RANGE, --range RANGE: Range of keys (default: 100)
• -w WARMUP, --warmup WARMUP: Number of warmup operations before timings (default: 1000)

Output is stored in text files prefixed with timings_ in the tests directory.

Step 2: Plot the results Pass the generated timings_ file as an argument to plot.py.

To develop for DiskCache, clone the repository, install the required dependencies using pip, and use tox for testing.

Clone the repository:

$ git clone https://github.com/grantjenks/python-diskcache.git

Install dependencies:

$ pip install -r requirements.txt

Note: For running benchmarks, additional packages like pylibmc and redis (and their respective servers) are required.

When using DjangoCache.set with read=True, values are guaranteed to be stored in files. You can access the file path via the name attribute on the file handle returned by cache.read(path) to serve files efficiently using X-Sendfile or X-Accel-Redirect headers.

from django.core.cache import cache
from django.http import HttpResponse

def media(request, path):
    try:
        with cache.read(path) as reader:
            response = HttpResponse()
            response['X-Accel-Redirect'] = reader.name
            return response
    except KeyError:
        # Handle cache miss.
        pass

Because diskcache.Deque and diskcache.Index are backed by the file system, they can be shared across multiple processes. This allows you to scale a single task (like a web crawl) by running multiple Python processes that all operate on the same persistent queue and result index simultaneously.

from multiprocessing import Process
from diskcache import Deque, Index

# Assuming crawl() uses Deque and Index as described in the case study
def run_parallel_crawl():
    results = Index('data/results')
    results.clear()
    
    # Start 4 parallel processes targeting the same crawl logic
    processes = [Process(target=crawl) for _ in range(4)]
    for process in processes:
        process.start()
    for process in processes:
        process.join()
        
    print(f'Total results: {len(results)}')

Under heavy concurrent load, standard diskcache.Cache objects can suffer from high maximum latency because cache writers block each other. To mitigate this, use diskcache.FanoutCache to distribute writes across multiple shards.

Increasing the number of shards reduces maximum latency. For example, using shards=4 can reduce maximum latency by a factor of ten compared to a single-shard Cache. For even better performance in high-concurrency environments, consider allocating one shard per worker and setting a low timeout to keep maximum latency within reasonable bounds.