Pykan: Kolmogorov-Arnold Networks Implementation

repository·master·Indexed 29 days ago

https://github.com/kindxiaoming/pykan

Official implementation of Kolmogorov-Arnold Networks (KANs), an alternative to Multi-Layer Perceptrons (MLPs) using learnable activation functions on edges. Designed for high accuracy and interpretability in scientific computing, pykan supports symbolic regression, model pruning, sparsification, and CUDA acceleration.

Tokens
59.5K
Snippets
85
Records
248
Agent score
90%

What's inside pykan

  1. Overview of the kan package modules

    master

    The kan package provides the core components for Kolmogorov-Arnold Networks. The primary modules include:

    • kan.MultKAN: Implements MultKAN architectures, which support explicit multiplication nodes and variable arity.
    • kan.KANLayer: Defines the fundamental KAN layer structure.
    • kan.Symbolic_KANLayer: Provides layers specialized for symbolic mathematics and symbolic-spline hybrid modes.
    • kan.LBFGS: Contains implementations for the L-BFGS optimization algorithm, often used for fine-tuning KAN models.
    • kan.spline: Contains the underlying spline parametrization logic used for learnable activation functions.
    • kan.compiler: Provides tools for compiling symbolic formulas into KAN architectures.
    • kan.utils: Contains utility functions for dataset manipulation and model management.
    • kan.hypothesis: Provides tools for hypothesis testing or scientific discovery workflows.
  2. Perform symbolic discovery with MultKAN

    master

    To convert a trained MultKAN into a readable symbolic formula, follow this workflow:

    1. Train: Initialize a KAN with nested width parameters and use model.fit() to train on your dataset.
    2. Prune: Use model.prune() to remove unnecessary nodes and edges, simplifying the network.
    3. Auto-symbolic: Call model.auto_symbolic() to automatically identify and fix symbolic activation functions.
    4. Retrain: Run model.fit() again to refine the model with the fixed symbolic functions.
    5. Extract: Use model.symbolic_formula() to retrieve the final mathematical expression.
  3. Identify symbolic and numeric-symbolic modes in KAN plots

    master

    When visualizing a KAN model, the color of the activation functions indicates their mode:

    • Red: The function is set to be purely symbolic (using fix_symbolic).
    • Purple: The function is in numeric-symbolic mode (mode='ns'), meaning its output is the sum of a symbolic function and a spline.
    • Default: Standard spline-based activation functions.
  4. Implement Physics-Informed KAN (PINN) for Navier-Stokes

    master

    You can use KAN to solve fluid dynamics problems like the Navier-Stokes equations by treating the neural network as a Physics-Informed Neural Network (PINN). This involves defining a loss function that combines the residuals of the partial differential equations (PDEs) with boundary condition losses.

    Key Steps:

    1. Define the Model: Initialize a KAN model where the output dimensions correspond to the physical variables (e.g., u, v, and p for velocity components and pressure).
    2. Compute Derivatives: Use torch.autograd.functional.jacobian and custom Hessian implementations to compute the spatial derivatives required by the Navier-Stokes equations from the model's predictions.
    3. Formulate Loss:
      • PDE Residuals: Calculate the continuity equation and momentum equations (x and y) using the computed derivatives.
      • Boundary Conditions (BC): Add losses for no-slip conditions, inlet velocity, and outlet pressure.
      • Total Loss: total_loss = torch.mean(pde_residuals) + bc_loss.
    4. Optimization: Use the LBFGS optimizer, which is often effective for PINN training tasks.

    Note: This implementation is a community contribution and has not been officially verified by the KAN authors.

    import torch
    from kan import KAN, LBFGS
    
    # 1. Initialize KAN model
    # width=[2,3,3,3] means 2 inputs (x, y) and 3 outputs (u, v, p)
    model = KAN(width=[2,3,3, 3], grid=5, k=10, grid_eps=1.0, noise_scale_base=0.25)
    
    # 2. Define the loss function (simplified logic)
    def navier_stokes_residuals(coords):
        y_pred = model(coords)
        # ... compute gradients/hessians using autograd ...
        # ... compute continuity, x_momentum, y_momentum residuals ...
        # ... compute boundary condition losses ...
        return total_loss
    
    # 3. Train using LBFGS
    optimizer = LBFGS(model.parameters(), lr=1, history_size=10, line_search_fn="strong_wolfe")
    
    def closure():
        optimizer.zero_grad()
        loss = navier_stokes_residuals(coordinates)
        loss.backward()
        return loss
    
    optimizer.step(closure)
  5. Use CUDA for KAN training

    master

    By default, Pykan uses the CPU. To utilize a GPU (CUDA), you must explicitly pass the device argument to both the KAN model and the create_dataset function. This ensures that both the model parameters and the training data are located on the same hardware device, preventing runtime errors.

    To implement this, use torch.device to detect availability and then apply .to(device) to your model instance.

  6. Tune KAN training hyperparameters for sparsity and interpretability

    master

    Regularization in KANs helps improve interpretability by making the models sparser. You can tune several hyperparameters within the model.fit() method to control this behavior:

    • lamb ($\lambda$): The overall penalty strength for regularization. Setting lamb=0.0 removes regularization, while higher values increase the penalty.
    • lamb_entropy ($\lambda_{\rm ent}$): The relative penalty strength of entropy. The absolute magnitude of the entropy penalty is $\lambda \cdot \lambda_{\rm ent}$. Increasing this value affects the sparsity and distribution of the activation functions.
    • seed: Controls the randomness of the model initialization. Varying the seed can help in observing different convergence behaviors.

    To use these, pass them as arguments to model.fit().

    from kan import *
    import torch
    
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    
    # Example setup
    f = lambda x: torch.exp(torch.sin(torch.pi*x[:,[0]]) + x[:,[1]]**2)
    dataset = create_dataset(f, n_var=2, device=device)
    
    # Initialize model
    model = KAN(width=[2,5,1], grid=5, k=3, seed=1, device=device)
    
    # Train with specific hyperparameters
    # lamb: overall penalty strength
    # lamb_entropy: relative entropy penalty strength
    model.fit(dataset, opt="LBFGS", steps=20, lamb=0.01, lamb_entropy=10.0)
    
    model.plot()
  7. Manage KAN model checkpoints and versioning

    master

    KAN models automatically save checkpoints whenever they are altered (e.g., via .fit() or .prune()). Checkpoints are stored in a directory named model.ckpt by default (or the name specified during initialization).

    Versioning System

    Versions follow a a.b format:

    • a (Round Number): Starts at 0. Increments by 1 whenever model.rewind() is called.
    • b (Version Number): Increments within each round whenever the model is modified.

    Example lifecycle:

    1. Initialization: Version 0.0.
    2. After .fit(): Version 0.1.
    3. After .prune(): Version 0.2.
    4. After .rewind('0.1'): The model reverts to version 0.1, but because a new round has started, it is renamed to 1.1. Subsequent changes will result in 1.2, 1.3, etc.
  8. Discover constitutive laws using kanpiler and KAN with priors

    master

    This workflow demonstrates how to use kanpiler to initialize a KAN model with a known symbolic prior (e.g., a linear elastic law) and then use it to discover a more complex law (e.g., a Neo-Hookean law) through training and architecture expansion.

    Workflow Steps:

    1. Initialize with Prior: Use kanpiler(input_vars, symbolic_expression, base_fun='identity') to create a model based on a known formula.
    2. Expand Architecture: Increase model capacity using model.expand_depth() and model.expand_width(..., mult_arity=2) to allow the network to capture non-linearities and multiplication terms.
    3. Transition to Hybrid/Numeric: Use model.perturb(mode='all') to move from a purely symbolic model to a trainable hybrid model.
    4. Train on Target Data: Use model.fit(dataset, steps=N) to optimize the model against the target physical law.
    5. Prune and Discover: Use model.prune() to remove unnecessary edges/nodes, followed by model.auto_symbolic() to automatically identify the learned symbolic functions.
  9. Prune a KAN model to increase sparsity

    master

    Pruning helps reduce the complexity of a KAN model by removing unnecessary connections. There are two ways to use model.prune():

    1. Keep original shape: Call model.prune() to prune edges/nodes internally. Use model.plot(mask=True) to visualize the pruned structure while maintaining the original network dimensions.
    2. Reduce model size: Assign the result of the pruning back to the model (model = model.prune()). This physically removes the pruned components, resulting in a smaller architecture.

    After pruning, you can continue training the model to refine the remaining connections.

    # Option 1: Prune but keep original shape
    model.prune()
    model.plot(mask=True)
    
    # Option 2: Prune and get a smaller shape
    model = model.prune()
    model(dataset['train_input'])
    model.plot()
    
    # Continue training after pruning
    model.train(dataset, opt="LBFGS", steps=50)
  10. Initialize and train a basic KAN model

    master

    To use a Kolmogorov-Arnold Network (KAN), initialize the KAN class with a width list defining the number of neurons in each layer. You can specify the spline degree using k and the number of grid intervals using grid.

    After initializing, you can create a dataset using create_dataset and train the model using model.train(). For better interpretability and sparsity, use lamb (L1 regularization) and lamb_entropy (entropy regularization) during training.

    Training modes include specifying an optimizer like LBFGS via the opt parameter.

  11. Install pykan

    master

    You can install pykan via PyPI, directly from GitHub, or using Conda.

    Prerequisites:

    • Python 3.9.7 or higher
    • pip

    Via PyPI:

    pip install pykan

    Via GitHub:

    pip install git+https://github.com/KindXiaoming/pykan.git

    Via Conda:

    conda create --name pykan-env python=3.9.7
    conda activate pykan-env
    pip install pykan

    For Developers (Editable Install):

    git clone https://github.com/KindXiaoming/pykan.git
    cd pykan
    pip install -e .
    pip install pykan
  12. Encourage linearity in KAN models

    master

    When using a model that is larger than necessary (wider or deeper than the minimal required architecture), you can encourage the model to find 'shortcuts' and simplify its activation functions using two primary strategies:

    1. Set base_fun='identity': This sets the base function of the KAN to be linear. This helps the model default to linear behavior if the spline component is not needed.
    2. Penalize spline coefficients: During training, use the lamb and lamb_coef parameters in the .fit() method to penalize the spline coefficients. When spline coefficients are driven toward zero, the activation function becomes linear.

    These techniques help in pruning a large model down to its functional essence, effectively making it behave like a smaller, more efficient model.