DiffSync

repository·develop·Indexed 41 days ago

https://github.com/networktocode/diffsync

A utility library for comparing and synchronizing different datasets, such as inventory systems. DiffSync uses a model-adapter architecture to translate between disparate data sources and perform bidirectional synchronization, featuring methods like diff_to, diff_from, and sync_to to align data sources.

Tokens
16.2K
Snippets
55
Records
71
Agent score
51%

What's inside DiffSync

  1. Overview of DiffSync

    develop

    DiffSync is a tool designed to synchronize state between two different systems. It allows you to identify differences between two representations of a system and provides mechanisms to reconcile those differences in either direction.

    Primary Use Cases

    • Difference Detection: Identifying what would change if you applied changes from System B to System A.
    • Unidirectional Synchronization (A $\rightarrow$ B): Updating System B to align with the current status of System A.
    • Unidirectional Synchronization (B $\rightarrow$ A): Updating System A to align with the current status of System B.
  2. Work with a remote system via REST API

    develop

    DiffSync can be used to compare and synchronize data between a local data source (like a JSON file) and a remote system (like Nautobot) via a REST API. This pattern is useful for ensuring that a remote system's state aligns with a local source of truth.

    In this workflow:

    1. Compare: Use DiffSync to identify differences between the local data and the remote system.
    2. Synchronize: Use DiffSync to apply changes to the remote system to match the local state.

    This example demonstrates advanced features such as:

    • Using Global Flags to ignore objects that do not match.
    • Providing a custom Diff class to modify the ordering of object groups.
  3. Customize element processing order with custom Diff classes

    develop

    By default, objects of the same type are stored in a dictionary, meaning the processing order during diff or sync operations is not guaranteed. To enforce a specific order (e.g., for top-level objects or children of a specific object), you must implement ordering methods within a subclass of Diff.

    Ordering Logic

    When the core engine iterates over a list of objects, it looks for specific class methods to determine order:

    1. Type-specific ordering: It looks for a method named order_children_<type>, where <type> is the name of the object type (e.g., order_children_device for device objects).
    2. Fallback ordering: If no type-specific method is found, it calls order_children_default.

    Requirements:

    • Both order_children_default and any type-specific methods you implement must be @classmethod.
    • These methods must accept children (a dictionary of names to DiffElement objects) and must yield an Iterator of DiffElement objects.
    from collections import defaultdict
    from diff import Diff
    
    class MixedOrderingDiff(Diff):
        """Alternate diff class to list children in alphabetical order, 
        except devices to be ordered by CRUD action."""
    
        @classmethod
        def order_children_default(cls, children):
            """Simple diff to return all children in alphabetical order."""
            for child_name, child in sorted(children.items()):
                yield children[child_name]
    
        @classmethod
        def order_children_device(cls, children):
            """Return a list of device sorted by CRUD action and alphabetically."""
            children_by_type = defaultdict(list)
    
            # Organize the children's name by action create, update or delete
            for child_name, child in children.items():
                action = child.action or "skip"
                children_by_type[action].append(child_name)
    
            # Create a global list, organized per action
            sorted_children = sorted(children_by_type["create"])
            sorted_children += sorted(children_by_type["update"])
            sorted_children += sorted(children_by_type["delete"])
            sorted_children += sorted(children_by_type["skip"])
    
            for name in sorted_children:
                yield children[name]
  4. Diff and Sync operations with Adapters

    develop

    After loading data into adapters using the .load() method, you can use the following methods to compare or synchronize datasets:

    Diffing (Comparing)

    To see the differences between two systems without changing them:

    • adapter_a.diff_from(adapter_b): Produces a structured representation of what would change in adapter_a to match adapter_b.
    • adapter_a.diff_to(adapter_b): (Note: The README implies diff_to and diff_from are used to generate the difference representation).

    Syncing (Updating)

    To update one dataset to match another:

    • adapter_a.sync_from(adapter_b): Updates adapter_a to align with the current status of adapter_b.
    • adapter_a.sync_to(adapter_b): Updates adapter_b to align with the current status of adapter_a.
    A = DiffSyncSystemA()
    B = DiffSyncSystemB()
    
    A.load()
    B.load()
    
    # Show the difference (what would change in A to match B)
    diff_a_b = A.diff_from(B)
    print(diff_a_b.str())
    
    # Update A to match B
    A.sync_from(B)
    
    # Update B to match A
    A.sync_to(B)
  5. Understand DiffSync Store Backends

    develop

    DiffSync uses a store to hold models loaded from adapters, making them available for diff calculation and synchronization.

    By default, DiffSync uses an in-memory storage. This is suitable for single-process execution. For distributed environments where tasks run in different processes or workers, you should use a distributed backend like RedisStore.

    While store is a class attribute of the Adapter class, the following operations are abstracted through the store interface:

    • get_all_model_names
    • get
    • get_by_uids
    • add
    • update
    • remove
    • get_or_instantiate
    • update_or_instantiate
    • count
  6. Understand Model Processing Ordering (Preorder Tree Traversal)

    develop

    DiffSync processes models using a Preorder Tree Traversal. This means a parent node is always processed before its child nodes.

    The traversal logic follows this order:

    1. Start with the first model in top_level.
    2. If that model has _children, process each child in order.
    3. For each child, recursively process its own children until the end of the lineage is reached.
    4. Once a branch is exhausted, proceed to the next sibling or the next top_level model.

    Example Traversal: If top_level = ["site", "device"] and Site has children vlan and prefix, the order is:

    • site
      • vlan
      • prefix
    • device
      • ...
  7. How DiffSync works: Adapters and Models

    develop

    DiffSync acts as an intermediate translation layer between different datasets. To use the library, you must implement two core components:

    1. Data Models: Define your data structure by creating classes that subclass DiffSyncModel.
    2. Adapters: Create translation logic by creating classes that subclass Adapter. Adapters are responsible for loading data from a source into the defined DiffSyncModel records.

    Once data is loaded into these models via adapters, you can perform diffing and synchronization operations between them.

  8. Manage dependencies with Poetry

    develop

    DiffSync uses Poetry for dependency management and virtual environments.

    Key Commands

    • poetry shell: Creates and enables a virtual environment. All subsequent commands run within this environment.
    • poetry install: Installs both project dependencies (tool.poetry.dependencies) and development dependencies (tool.poetry.group.dev.dependencies) into the virtual environment.

    Dependency Structure

    Dependencies are defined in pyproject.toml:

    • tool.poetry.dependencies: Main project dependencies.
    • tool.poetry.group.dev.dependencies: Development-only dependencies used for linting, testing, and documentation.
  9. Create changelog fragments for pull requests

    develop

    All pull requests targeting the next or develop branches must include a changelog fragment file located in the ./changes directory.

    Filename Format: {issue_number}.{fragment_type} (e.g., 2362.added).

    Valid Fragment Types:

    • added
    • changed
    • deprecated
    • fixed
    • removed
    • security

    Content Rules:

    • Use plain text.
    • Summaries must be complete sentences.
    • Start with a capital letter and end with a period.
    • Use the past tense.
    • Each line in the file generates a single entry in the release notes. To generate multiple entries in the same category, use multiple lines in one file. If entries belong in different categories, create separate files.
    Fixed critical bug in documentation.
    Fixed release notes generation.
    Changed release notes generation.
  10. Manage flags using bitwise operations

    develop

    DiffSync flags are stored in binary format, allowing multiple flags to be stored in a single variable. You can manage them using standard Python bitwise operators.

    Enable a flag (Bitwise OR)

    Use the |= operator to enable a flag without affecting existing flags.

    Check a flag (Bitwise AND)

    Use the & operator to check if a flag is set. Wrapping the result in bool() converts it to a boolean value.

    Disable a flag (Bitwise NOT)

    Use the &= ~ operator to unset a specific flag while leaving others intact.

    from diffsync.enum import DiffSyncFlags
    
    # 1. Enable a flag
    flags = DiffSyncFlags.CONTINUE_ON_FAILURE
    flags |= DiffSyncFlags.SKIP_UNMATCHED_DST
    
    # 2. Check a flag
    is_enabled = bool(flags & DiffSyncFlags.SKIP_UNMATCHED_DST)
    
    # 3. Disable a flag
    flags &= ~DiffSyncFlags.SKIP_UNMATCHED_DST