python-xxhash Documentation

repository·master·Indexed 17 days ago

https://github.com/ifduyue/python-xxhash

A high-performance Python binding for the xxHash library. It provides hashlib-compliant streaming APIs and oneshot functions for various algorithms, including xxh32, xxh64, and XXH3 (xxh3_64 and xxh3_128).

Tokens
1.3K
Snippets
7
Records
8
Agent score
17%

What's inside python-xxhash

  1. Understand xxhash digest endianness

    master

    As of python-xxhash 0.3.0, the digest() method returns bytes representing the big-endian representation of the integer digest.

    This means h.digest() is equivalent to h.intdigest().to_bytes(size, 'big').

    Example:

    import xxhash
    h = xxhash.xxh64()
    # The bytes returned by digest() match the big-endian integer conversion
    print(h.digest() == h.intdigest().to_bytes(8, 'big')) # True
    import xxhash
    >>> h = xxhash.xxh64()
    >>> h.digest()
    b'\xefF\xdb7Q\xd8\xe9\x99'
    >>> h.intdigest().to_bytes(8, 'big')
    b'\xefF\xdb7Q\xd8\xe9\x99'
  2. Configure seeds for xxh32 and xxh64

    master

    You can provide an optional seed (defaulting to 0) to alter the hash result predictably.

    Important Seed Constraints:

    • xxh32 expects an unsigned 32-bit integer seed.
    • xxh64 expects an unsigned 64-bit integer seed.

    While Python handles large integers, providing a seed that exceeds these bit-widths will result in the same hash as the seed modulo the bit-width (e.g., seed=2**32 is treated as 0 for xxh32).

    Example:

    import xxhash
    # Using a seed with xxh64
    print(xxhash.xxh64(b'xxhash', seed=20141025).hexdigest())
    # 'b559b98d844e0635'
    import xxhash
    >>> xxhash.xxh64(b'xxhash', seed=20141025).hexdigest()
    'b559b98d844e0635'
  3. Install xxhash from source

    master

    To install from source without using pre-built binaries, use the --no-binary flag.

    Prerequisites:

    • Debian/Ubuntu: apt-get install python-dev gcc
    • CentOS/Fedora: yum install python-devel gcc redhat-rpm-config

    Installation command:

    $ pip install --no-binary xxhash xxhash

    Linking to an external libxxhash.so: By default, python-xxhash uses a bundled version of xxHash. To link against a system-installed libxxhash.so instead, set the XXHASH_LINK_SO environment variable during installation:

    $ XXHASH_LINK_SO=1 pip install --no-binary xxhash xxhash
    $ pip install --no-binary xxhash xxhash
  4. Check xxhash module and backend versions

    master

    You can retrieve the version of the Python module and the version of the underlying xxHash library using the VERSION and XXHASH_VERSION properties.

    import xxhash
    print(xxhash.VERSION)      # e.g., '2.0.0'
    print(xxhash.XXHASH_VERSION) # e.g., '0.8.0'
    import xxhash
    >>> xxhash.VERSION
    '2.0.0'
    >>> xxhash.XXHASH_VERSION
    '0.8.0'
  5. Use XXH3 hash algorithms (xxh3_64 and xxh3_128)

    master

    Since v2.0.0, XXH3 hashes are supported. They are available as both streaming classes and oneshot functions.

    Streaming Classes:

    • xxh3_64
    • xxh3_128 (also aliased as xxh128)

    Oneshot Functions:

    • xxh3_64_digest(bytes, seed=0)
    • xxh3_64_intdigest(bytes, seed=0)
    • xxh3_64_hexdigest(bytes, seed=0)
    • xxh3_128_digest(bytes, seed=0)
    • xxh3_128_intdigest(bytes, seed=0)
    • xxh3_128_hexdigest(bytes, seed=0)
    • xxh128_digest(bytes, seed=0) (alias)
    • xxh128_intdigest(bytes, seed=0) (alias)
    • xxh128_hexdigest(bytes, seed=0) (alias)
  6. Use xxhash oneshot functions for performance

    master

    To avoid the overhead of allocating state on the heap, use the oneshot functions. These are faster for single-pass hashing of a complete byte string.

    Available functions:

    • xxh32_digest(bytes, seed=0)
    • xxh32_intdigest(bytes, seed=0)
    • xxh32_hexdigest(bytes, seed=0)
    • xxh64_digest(bytes, seed=0)
    • xxh64_intdigest(bytes, seed=0)
    • xxh64_hexdigest(bytes, seed=0)

    Example:

    import xxhash
    # Oneshot is faster than creating an object and calling update()
    result = xxhash.xxh64_hexdigest(b'xxhash', seed=20141025)
    print(result) # 'b559b98d844e0635'
    import xxhash
    >>> xxhash.xxh64_hexdigest(b'xxhash', seed=20141025)
    'b559b98d844e0635'
  7. Use xxhash streaming API (hashlib-compliant)

    master

    The xxhash module is hashlib-compliant, meaning you can use it with the same method signatures as hashlib.md5. This is useful for streaming data in chunks.

    Available methods:

    • update(data): Update the current digest with additional bytes.
    • digest(): Return the current digest as bytes (big-endian representation).
    • hexdigest(): Return the current digest as a hexadecimal string.
    • intdigest(): Return the current digest as an integer.
    • copy(): Return a copy of the current object.
    • reset(): Reset the state.

    Example using xxh32:

    import xxhash
    
    x = xxhash.xxh32()
    x.update(b'Nobody inspects')
    x.update(b' the spammish repetition')
    print(x.digest())      # b'\xe2);/'
    print(x.digest_size)   # 4
    print(x.block_size)    # 16
    import xxhash
    >>> x = xxhash.xxh32()
    >>> x.update(b'Nobody inspects')
    >>> x.update(b' the spammish repetition')
    >>> x.digest()
    b'\xe2);/'
    >>> x.digest_size
    4
    >>> x.block_size
    16