DOCSEARCHbyAlgolia
Submit Docs

Ludwig: Declarative Deep Learning Framework

repository·main·Indexed 11 days ago

https://github.com/ludwig-ai/ludwig

A declarative deep learning framework for training, fine-tuning, and deploying AI models (LLMs, multimodal, and tabular) using YAML configurations. It supports end-to-end machine learning pipelines, LLM alignment via DPO, KTO, ORPO, and GRPO, anomaly detection using Deep SVDD, Deep SAD, and DROCC, and temperature scaling for model calibration.

Tokens
81.8K
Snippets
268
Records
353
Agent score
95%

What's inside Ludwig

  1. Advanced PEFT Adapters in Ludwig

    main

    Ludwig supports an extended range of Parameter-Efficient Fine-Tuning (PEFT) adapters beyond standard LoRA. These adapters allow for fine-tuning large models with significantly reduced memory and parameter requirements. Supported advanced methods include:

    • Advanced LoRA Initializers: PiSSA, EVA, CorDA, and LoftQ.
    • Scaling & Stability: rsLoRA (rank-stabilized LoRA).
    • Extreme Low-Rank: TinyLoRA (LoRA-XS variant).
    • Contextual/Compositional: C3A adapters.
    • Orthogonal Methods: OFT (Orthogonal Fine-Tuning) and HRA (Householder reflections).
    • Domain/Layer Specific: WaveFT (wavelet-domain), LN-Tuning (Layer Normalization only), and VBLoRA (vector bank LoRA).

  2. Ensure data accessibility for Ray clusters

    main

    When running Ludwig on a remote Ray cluster, the --dataset path must be a location that the cluster nodes can reach. Local file paths from your machine will not work.

    Supported storage types and examples:

    • S3: s3://bucket/data.csv (Cluster requires AWS credentials)
    • GCS: gs://bucket/data.csv (Cluster requires GCP credentials)
    • NFS: /shared/data/train.csv (Must be mounted on all cluster nodes)
    • HDFS: hdfs://namenode/data.csv (Requires a Hadoop cluster)

    If your data is currently local, upload it to cloud storage first:

    aws s3 cp my_data.csv s3://my-bucket/data/my_data.csv

  3. Configure Anomaly Detection output features

    main

    To perform anomaly detection in Ludwig, use the anomaly output feature type. The model learns a hypersphere representation of 'normal' data. At inference time, each sample receives an anomaly_score representing its squared distance from the learned hypersphere centre; higher scores indicate higher anomaly probability.

    There are three primary loss variants available for the anomaly type:

    1. Deep SVDD (Unsupervised): Minimises the mean squared distance of normal training representations to a centre c. Use this when you only have normal samples for training.
    2. Deep SAD (Semi-supervised): Extends Deep SVDD by using a small set of labeled anomalies (label 1) to push them away from the centre, while pulling normal/unlabeled samples (label 0 or -1) toward it.
    3. DROCC (Robust Unsupervised): Uses an adversarial perturbation regulariser to prevent hypersphere collapse. This is recommended when using expressive encoders (like Transformers) that might otherwise collapse all representations to a single point.
    output_features:
  - name: anomaly
    type: anomaly
    loss:
      type: deep_svdd
      nu: 0.1

  4. Align LLMs using DPO, KTO, ORPO, or GRPO

    main

    Ludwig provides built-in preference learning trainers to align Large Language Models (LLMs) with human values or programmatic rewards. This process is typically performed after an initial Supervised Fine-Tuning (SFT) stage.

    Trainer Comparison

    TrainerData FormatUse CaseCompute Requirement
    dpoprompt, chosen, rejectedHuman-ranked response pairs (most widely studied).Medium (requires policy and reference model)
    ktoprompt, response, label (bool)Single-label feedback (e.g., thumbs up/down).Low (simpler loss than DPO)
    orpoprompt, chosen, rejectedSingle-stage SFT + alignment (skips separate SFT).Low (no reference model)
    grpoprompt, custom reward functionRL-style training with group-normalized rewards (e.g., DeepSeek-R1).High (multiple rollouts per prompt)

    When to choose which trainer:

    • DPO: Use when you have paired human preferences.
    • KTO: Use when collecting binary feedback is easier than pairwise comparisons.
    • ORPO: Use to combine SFT and alignment into a single step.
    • GRPO: Use when you have a programmatic reward function (like math verification or code execution).

  5. Handle imbalanced datasets with class balancing

    main
    Ludwig provides a class balancing feature designed to handle imbalanced datasets. You can use this feature to over-sample minority classes during the training process to ensure the model learns effectively from all classes.

  6. Prepare a dataset for VLM fine-tuning

    main

    For VLM fine-tuning (e.g., Visual Question Answering), provide a CSV file containing the following three columns:

    • image_path: The file path to the image (JPEG or PNG).
    • question: The natural-language question related to the image.
    • answer: The expected target answer for fine-tuning.

  7. Conceptual overview of adding a new feature type

    main

    When adding a new feature type to Ludwig, you must implement two parallel layers that are wired together via the feature registry:

    1. Schema Layer (ludwig/schema/features/<type>_feature.py): Uses Pydantic-backed config classes to declare hyperparameters, defaults, and validation rules. These are used for config validation and serialization.
    2. Feature Module Layer (ludwig/features/<type>_feature.py): Contains PyTorch modules that implement the actual logic for preprocessing, encoding, decoding, and postprocessing. These are instantiated at model-build time using the schema configs.

    Neither layer knows about the other at import time; the feature registry handles the connection.

  8. Multi-Task Learning with Loss Balancing in Ludwig

    main

    Multi-task learning allows you to train a single model to predict multiple outputs simultaneously using a shared representation. This is useful when tasks are related and can benefit from shared features.

    The Loss Balancing Problem: When training multiple tasks (e.g., a regression task and a classification task), the loss scales often differ. Without balancing, the task with the larger loss magnitude will dominate the gradient updates, causing the other tasks to under-train. Ludwig provides several loss balancing methods to assign adaptive weights to each task's loss so that all tasks contribute proportionately to the total gradient.

  9. Use the native Optuna executor for single-machine HPO

    main

    Ludwig 0.15+ provides a native Optuna executor for Hyperparameter Optimization (HPO) that runs trials directly without the overhead of a Ray cluster. This is the recommended choice for single-machine HPO. It supports various samplers, pruning algorithms, and SQLite-backed persistence for resumable studies.

    If you require distributed trials across multiple GPUs or nodes, use the ray executor instead (which wraps OptunaSearch).

    hyperopt:
  executor:
    type: optuna
    num_samples: 50
    sampler: auto
    pruner: null
    study_name: my_study
    storage: null
    time_budget_s: 1800

  parameters:
    trainer.learning_rate:
      space: loguniform
      lower: 1e-5
      upper: 1e-1
    # ... other parameters

  output_feature: quality
  metric: root_mean_squared_error
  goal: minimize
  split: validation

  10. Quantify model uncertainty with Temperature Scaling and MC Dropout

    main

    Ludwig supports two primary techniques for managing model uncertainty:

    1. Temperature Scaling Calibration: A post-hoc calibration method used to adjust overconfident predicted probabilities so they better match empirical frequencies. This is enabled via a configuration change.
    2. MC Dropout (Monte Carlo Dropout): A method that runs multiple stochastic forward passes at inference time to produce per-sample uncertainty estimates. This allows you to interpret an uncertainty output alongside standard predictions.

    These techniques are useful for datasets with class imbalance where models tend to be overconfident.

  11. Fine-tune LLMs and VLMs with Ludwig

    main

    Ludwig provides comprehensive support for Large Language Model (LLM) and Vision-Language Model (VLM) fine-tuning via declarative YAML configurations.

    Key Capabilities:

    • Supervised Fine-Tuning (SFT): Train on instruction/response pairs.
    • Alignment Training: Support for DPO, KTO, ORPO, and GRPO (reward-model-free RLHF).
    • PEFT (Parameter-Efficient Fine-Tuning): Includes LoRA, DoRA, VeRA, LoRA+, TinyLoRA, OFT, HRA, WaveFT, LN-Tuning, VBLoRA, and C3A.
    • Advanced LoRA Initializers: PiSSA, EVA, CorDA, and LoftQ.
    • Multi-adapter PEFT: Use multiple named adapters on one base model, switchable at runtime, with merging options like TIES, DARE, SVD, and magnitude pruning.
    • Quantization: 4-bit/8-bit QLoRA (via bitsandbytes) and torchao int4/int8/float8 with QAT.
    • VLM Fine-tuning: Support for LLaVA, Qwen2-VL, and InternVL by setting is_multimodal: true.
    • Efficiency: Sequence packing for variable-length inputs and paged/8-bit optimizers for memory efficiency.

  12. Compare Semantic Segmentation decoders

    main

    When choosing a decoder for semantic segmentation in Ludwig, consider the following trade-offs:

    DecoderArchitectureRecommended EncoderBest For
    unetSymmetric encoder-decoder with skip connectionsBuilt-in unet encoderGeneral purpose baseline; no pretrained backbone required.
    segformerLightweight all-MLP head fusing multi-scale ViT featuresdinov2 (pretrained)Highest accuracy; transformer features transfer well to dense prediction.
    fpnFeature Pyramid Network top-down pathwayefficientnet (pretrained)Fast inference; handles objects at multiple scales efficiently.