DOCSEARCHbyAlgolia
Submit Docs

deepdiff

repository·master·Indexed 25 days ago

https://github.com/qlustered/deepdiff

A Python library for performing deep differences between objects, searching within nested structures, and managing object deltas. It provides tools for content-based hashing via DeepHash, object searching with DeepSearch and grep, and the ability to recreate objects by applying deltas. The package includes a CLI for comparing, patching, and searching files in formats such as JSON, YAML, TOML, CSV, and TSV, as well as a ColoredView for visualizing diff results.

Tokens
39.7K
Snippets
76
Records
206
Agent score
79%

What's inside deepdiff

  1. Overview of DeepDiff modules

    master

    DeepDiff is composed of several specialized modules:

    ModuleDescription
    DeepDiffComputes the deep difference between two Python objects or calculates the distance between them.
    DeepSearchAllows searching for specific objects within larger, nested structures.
    DeepHashProvides content-based hashing for any object, including those that are not natively hashable in Python.
    DeltaGenerates a 'delta' (similar to a git commit) of objects that can be applied to other objects to transform them.
    ExtractUsed to extract a specific path from an object.
    CommandlineProvides a CLI interface for most of the functionality listed above.

  2. What is Deep Distance?

    master

    Deep Distance is a floating-point number between 0 and 1 that represents the distance between two objects. It is inspired by Levenshtein Edit Distance but is calculated based on the number of operations needed to convert one object to another, divided by the sum of the sizes of the two objects (capped at 1).

    Key characteristics:

    • 0 means there is no difference between the objects.
    • 1 means the objects are very different.
    • Granularity matters: The value is highly dependent on the granularity of the diff results, which is controlled by the parameters passed to DeepDiff.
    • Usage: This value should only be used to compare the similarity of two objects.

  3. What is DeepDiff Delta?

    master

    A Delta is a directed delta object that represents the difference between two structured data objects (t1 and t2). When applied to t1, it yields t2. Think of Delta objects as git commits for your data: you can generate them from a DeepDiff result, store them, and later apply them to other objects to recreate a specific state.

    Important Note: If you are generating Delta objects from a DeepDiff result and using ignore_order=True, you must also set report_repetition=True in your DeepDiff call.

    from deepdiff import DeepDiff, Delta

t1 = [1, 2, 3]
t2 = ['a', 2, 3, 4]
diff = DeepDiff(t1, t2)
delta = Delta(diff)

# Applying the delta to t1 yields t2
assert t1 + delta == t2

  4. How DeepHash works and how to get an object's hash

    master

    DeepHash calculates a deterministic hash of an object based on its contents. This is particularly useful for hashing unhashable objects like dictionaries or sets.

    Important Mental Model: DeepHash does not return a single hash value. Instead, it returns a dictionary where keys are the objects (or their IDs) and values are their corresponding hashes. This is because DeepHash recursively calculates hashes for all nested objects within the input.

    To get the hash of the specific object you passed in, you must access it via the returned dictionary using the object itself as the key.

  5. How custom operators work in DeepDiff

    master

    Custom operators allow you to intercept the diffing process at specific nodes in the object tree.

    1. Matching: As DeepDiff traverses the tree, it checks if any provided custom_operators match the current level (using the match method). Matching can be based on the object's path (regex) or the object's type.
    2. Comparison: Once matched, the operator's give_up_diffing method is called.
    3. Decision:
      • If give_up_diffing returns True, DeepDiff considers the two objects at that level to be 'equal' for the purpose of this diff and stops traversing deeper into that branch.
      • If it returns False, DeepDiff continues its standard diffing process.
    4. Custom Reporting: Within give_up_diffing, you can call diff_instance.custom_report_result(name, level, details) to inject custom metadata into the resulting diff object if certain criteria (like a distance threshold) are met, even if you ultimately decide the objects are 'equal' (by returning True).

  6. Understand datetime handling and UTC conversion

    master

    DeepDiff converts all datetime objects into UTC. If a datetime object is timezone-naive, DeepDiff assumes it is in UTC. This differs from standard Python behavior, where naive datetimes are often treated as local time.

    To avoid unexpected results when comparing naive datetimes with aware datetimes, you can configure a default_timezone in DeepDiff settings.

  7. Use group_by_sort_key to handle duplicate keys in lists of dictionaries

    master

    When using group_by to compare lists of dictionaries, if multiple dictionaries share the same key (e.g., the same 'id'), group_by alone might cause one dictionary to overwrite another in the internal grouping logic.

    To prevent this and ensure all dictionaries are preserved and correctly compared, use the group_by_sort_key parameter. This parameter defines how the dictionaries within a group are sorted. When provided, group_by converts the lists into a dictionary where keys map to lists of dictionaries, and group_by_sort_key is used to sort those lists. This allows DeepDiff to track changes to specific items within a group even when they share a common identifier.

    >>> t1 = [
...     {'id': 'AA', 'name': 'Joe', 'last_name': 'Nobody', 'int_id': 2},
...     {'id': 'BB', 'name': 'James', 'last_name': 'Blue', 'int_id': 20},
...     {'id': 'BB', 'name': 'Jimmy', 'last_name': 'Red', 'int_id': 3},
...     {'id': 'CC', 'name': 'Mike', 'last_name': 'Apple', 'int_id': 4},
... ]
>>> t2 = [
...     {'id': 'AA', 'name': 'Joe', 'last_name': 'Nobody', 'int_id': 2},
...     {'id': 'BB', 'name': 'James', 'last_name': 'Brown', 'int_id': 20},
...     {'id': 'CC', 'name': 'Mike', 'last_name': 'Apple', 'int_id': 4},
... ]
>>> diff = DeepDiff(t1, t2, group_by='id', group_by_sort_key='name')
>>> pprint(diff)
{'iterable_item_removed': {"root['BB'][1]": {'int_id': 3,
                                                 'last_name': 'Red',
                                                 'name': 'Jimmy'}},
 'values_changed': {"root['BB'][0]['last_name']": {'new_value': 'Brown',
                                                       'old_value': 'Blue'}}}

  8. Choose between cPython and PyPy for diffing

    master

    The choice of Python runtime affects performance depending on your data type:

    • Diffing mostly numbers: Use cPython with numpy installed for the best performance.
    • Diffing large blobs of mixed strings and numbers: PyPy may provide better CPU performance, though it typically uses more memory than cPython.

  9. Use Custom Operators to modify diffing behavior

    master

    Custom operators allow you to change how DeepDiff determines if two objects are different. This can be used to implement specialized logic like calculating L2 distance, ignoring specific paths, or stopping at the first difference found.

    You can use built-in operators or define your own and pass them via the custom_operators list in the DeepDiff constructor.

  10. Compare Text view vs. Tree view vs. pretty() method

    master

    DeepDiff provides different ways to format and consume comparison results depending on your use case:

    • Text view (default): The standard dictionary output. Best when you only need to know what changed without needing the full traversal history of the objects.
    • Tree view: Accessed via view='tree'. This view provides a more detailed structure that allows you to traverse the comparison tree and see exactly which objects were compared to which others. The values in the resulting dictionary are objects that allow access to properties like .t1 and .t2.
    • pretty() method: Unlike the views (which return dictionaries), pretty() returns a formatted string designed for human consumption.

  11. Determinism guarantees in DeepDiff multiprocessing

    master

    When using the multiprocessing mode in DeepDiff, the library adheres to a strict determinism contract to ensure that parallel execution does not change the outcome of the diff.

    Key invariants include:

    • Identical Results: A supported multiprocessing run produces the exact same public DeepDiff result as an equivalent serial run.
    • Order Independence: Pair selection (in ignore_order=True) and result merging are based on serial traversal order, not the order in which workers complete their tasks.
    • Hash Consistency: Hash aggregation follows existing semantics (e.g., sorting hash components for dictionaries/unordered iterables and preserving index order for ordered iterables).
    • Error Handling: Any exception occurring within a worker process is surfaced as a normal DeepDiff exception rather than being swallowed or resulting in partial output.
    • Safety Fallback: The system includes a reliable serial fallback for inputs that are unsupported or unsafe for multiprocessing.