DOCSEARCHbyAlgolia
Submit Docs

Ivy

repository·main·Indexed 11 days ago

https://github.com/unifyai/ivy

A tool for converting machine learning code, models, and libraries between frameworks such as PyTorch, TensorFlow, JAX, and NumPy. It features a transpiler using ivy.transpile() for source-to-source translation, a Graph Tracer to transform code into optimized Directed Acyclic Graphs (DAGs) via trace_graph, and Autograph_Ivy for converting Python functions with structured control flow into functional form.

Tokens
62.5K
Snippets
144
Records
255
Agent score
95%

What's inside Ivy

  1. Overview of supported array computation frameworks

    main

    Ivy interacts with and abstracts various prominent frameworks used for array computation. These frameworks serve as the underlying engines that Ivy's wrapper frameworks target.

    Key frameworks mentioned include:

    • MATLAB: Proprietary numeric computing environment.
    • SciPy: Python framework for scientific/technical computing (optimization, linear algebra, etc.).
    • Torch: Lua-based machine learning library.
    • NumPy: The standard Python framework for multi-dimensional arrays and mathematical functions.
    • SciKit Learn: Python framework for machine learning algorithms (classification, regression, clustering).
    • Theano / Aesara: Python framework for evaluating mathematical expressions with an optimizing compiler.
    • Pandas: Python framework for data manipulation and analysis (DataFrames, time series).
    • Julia: High-level, dynamic programming language designed for numerical analysis and computational science.

  2. What is Ivy and why use it?

    main

    Ivy is a source-to-source transpiler designed to address the fragmentation in the machine learning ecosystem. It allows for the simple and painless conversion of machine learning models and code between different cutting-edge frameworks such as PyTorch, TensorFlow, JAX, NumPy, and MLX.

    Developers use Ivy to:

    • Switch between frameworks (e.g., moving a model from PyTorch to TensorFlow) without rewriting the entire codebase.
    • Avoid being locked into a specific framework's syntax, APIs, or paradigms.
    • Enable more efficient research and deployment by targeting the most suitable framework for a given environment (e.g., using JAX for performance efficiency or TensorFlow for large-scale production).

  3. Compare Machine Learning Frameworks supported by Ivy

    main

    Ivy provides context on various machine learning frameworks that it can interact with or convert between. This documentation segment provides a high-level overview of the characteristics, strengths, and historical context of several major frameworks, which helps users understand the landscape of frameworks Ivy supports or relates to.

    Key frameworks discussed include:

    • PyTorch: The preferred choice for researchers and hobbyists due to its Pythonic nature, asynchronous scheduling, and vast ecosystem.
    • JAX: Targeted at deeply technical researchers needing high flexibility for higher-order gradients (Jacobians, Hessians) and custom optimization schemes via XLA bindings.
    • TensorFlow 2: A mature framework with strong focus on edge and mobile deployment via TensorFlow Lite.
    • Flux (Julia): A high-performance library for production pipelines that leverages Julia's extensibility and high-performance AD tools like Zygote.jl.
    • MXNet: Supports a mix of symbolic and imperative programming with a dynamic dependency scheduler.
    • Apache Spark MLlib: A distributed machine learning framework built on Spark for large-scale data processing.

  4. Module structure of Autograph_Ivy

    main

    The Autograph_Ivy implementation is organized into three main components:

    1. Core: The entry point. Contains to_functional_form() in api.py to convert functions from SCF to FCF.
    2. Converters: Contains the transformation passes applied to the AST to perform the conversion.
    3. Pyct: An independent module used for core source code transformations. It provides utilities for parsing, annotating, and modifying ASTs without containing TensorFlow-specific code.

  5. What is ivy-lint and how does it work?

    main

    ivy-lint is a specialized suite of custom code formatters designed specifically for the Ivy codebase to handle formatting requirements that standard Python formatters do not address.

    A key component is the FunctionOrderingFormatter, which standardizes the order of declarations in Python files (specifically targeting frontends and tests) using the following logic:

    1. Header Management: Removes existing headers based on specific patterns.
    2. Comments Handling: Retains leading comments by extracting them alongside their respective code components.
    3. Dependency Handling: Uses dependency graphs to maintain relationships between classes and assignments during reordering.
    4. Sorting Logic: Follows a specific hierarchy:
      • Module-level docstrings (preserved at the top)
      • Imports
      • Assignments (based on dependencies)
      • Classes
      • Functions (organized into helper and primary sections)
    5. File Processing: Rearranges content in files that match specific patterns.

  6. What is Ivy's transpiler and why use it?

    main

    Ivy's transpiler is a core feature designed to convert machine learning models, functions, and libraries between different frameworks. It addresses the fragmentation in the ML landscape by providing:

    • Interoperability: Combine the best tools and libraries from different frameworks into a single application.
    • Flexibility: Develop in your preferred framework (e.g., PyTorch) without being constrained by deployment requirements or team preferences.
    • Collaboration: Share models and tools across the community and organizations regardless of the original framework used.
    • Efficiency and Optimization: Leverage specific framework strengths at different stages of the lifecycle. For example, prototype in PyTorch for ease of use, then transpile to TensorFlow for production serving, or to JAX for high-performance accelerator-oriented computing.
    • Legacy Integration: Convert older codebases from deprecated frameworks to state-of-the-art frameworks, reducing migration effort.

  7. What is the Ivy Graph Tracer?

    main
    The Graph Tracer is a tool that transforms arbitrary code—whether written in Ivy or native frameworks like TensorFlow, PyTorch, or JAX—into an efficient Directed Acyclic Graph (DAG). This process optimizes execution by removing redundant operations, stripping unused functions, and preserving only the essential path from inputs to outputs. This allows for significant performance improvements and enables native compiler optimizations like tf.function or jax.jit by removing Python overhead.

  8. Understand TensorFlow 2 and TensorFlow Lite for deployment

    main

    TensorFlow 2 supports eager execution of computation graphs, making it easier to debug than TensorFlow 1. While it was not designed as an eager-first framework, it has matured significantly.

    A primary use case for TensorFlow 2 is industrial enterprise deployment, specifically targeting edge and mobile devices through TensorFlow Lite.

  9. Support for Mixed Functions via handle_partial_mixed_function

    main

    The @handle_partial_mixed_function decorator enables switching between a 'compositional' implementation and a 'primary' implementation of a Mixed Function based on a condition.

    To use this, you must:

    1. Define a lambda function that evaluates to True when the primary implementation should be used.
    2. Add a partial_mixed_handler attribute to the backend implementation containing this lambda.

    This allows the function to choose the most efficient implementation path based on the provided arguments.

  10. How Compositional Functions work

    main

    Compositional functions are implemented using other Ivy functions rather than backend-specific code. They exist only in the ivy/functional/ivy/ directory. This allows complex operations to be built by combining simpler primary or mixed functions.

    Example of a compositional function implementation (e.g., cross_entropy):

    def cross_entropy(
    true: Union[ivy.Array, ivy.NativeArray],
    pred: Union[ivy.Array, ivy.NativeArray],
    /,
    *,
    axis: int = -1,
    epsilon: float = 1e-7,
    reduction: str = "mean",
    out: Optional[ivy.Array] = None
) -> ivy.Array:
    ivy.utils.assertions.check_elem_in_list(reduction, ["none", "sum", "mean"])
    pred = ivy.clip(pred, epsilon, 1 - epsilon)
    log_pred = ivy.log(pred)
    return _reduce_loss(reduction, log_pred * true, axis, out)
    def cross_entropy(
    true: Union[ivy.Array, ivy.NativeArray],
    pred: Union[ivy.Array, ivy.NativeArray],
    /,
    *,
    axis: int = -1,
    epsilon: float = 1e-7,
    reduction: str = "mean",
    out: Optional[ivy.Array] = None
) -> ivy.Array:
    ivy.utils.assertions.check_elem_in_list(reduction, ["none", "sum", "mean"])
    pred = ivy.clip(pred, epsilon, 1 - epsilon)
    log_pred = ivy.log(pred)
    return _reduce_loss(reduction, log_pred * true, axis, out)

  11. Debug backend implementation errors vs numerical instability

    main

    When troubleshooting AssertionError failures where backend results do not match the ground truth, use the following logic to determine the cause:

    1. Numerical Instability: If the values are very close (e.g., 0.2583 vs 0.2585), increase rtol and atol in the test helper.
    2. Backend Bug: If the results are significantly different (e.g., one contains inf where the other has a large finite number, or the matrix shapes/values are fundamentally different) and the test passes on all other backends, the issue is likely a bug in that specific backend's implementation (e.g., a bug in permute_dims for the torch backend).

  12. How Ivy's Backend Functional APIs work

    main

    Ivy does not implement its own C++ or CUDA kernels. Instead, it wraps the functional APIs of existing frameworks (JAX, TensorFlow, PyTorch, and NumPy) to bring them into syntactic and semantic alignment.

    For simple operations, Ivy provides a wrapper in a backend-specific module that calls the underlying framework's function (e.g., jnp.stack, tf.experimental.numpy.stack, etc.). For operations that are missing in a specific framework (like logspace in TensorFlow), Ivy constructs the function using a composition of existing operations from that framework to ensure a unified API across all backends.

    # Example of how Ivy constructs a missing op (logspace) for TensorFlow using existing ops
def logspace(
    start: Union[tf.Tensor, tf.Variable, int],
    stop: Union[tf.Tensor, tf.Variable, int],
    num: int,
    base: float = 10.0,
    axis: Optional[int] = None,
    *,
    dtype: tf.DType,
    device: str,
) -> Union[tf.Tensor, tf.Variable]:
    power_seq = ivy.linspace(start, stop, num, axis, dtype=dtype, device=device)
    return base**power_seq