DOCSEARCHbyAlgolia
Submit Docs

LightGBM

repository·main·Indexed 12 days ago

https://github.com/lightgbm-org/lightgbm

A high-performance, distributed gradient boosting framework based on tree-learning algorithms, optimized for speed, memory efficiency, and accuracy. Version 4.7.0.99 supports CPU and GPU modes with APIs for Python and R, as well as a Command Line Interface (CLI). It provides various Docker configurations for interactive development (including Jupyter), minimal CLI footprints, and RStudio integration.

Tokens
52.1K
Snippets
137
Records
251
Agent score
98%

What's inside LightGBM

  1. Overview of LightGBM

    main

    LightGBM (Light Gradient Boosting Machine) is a distributed and efficient gradient boosting framework that uses tree-based learning algorithms. It is designed for high performance with the following key advantages:

    • Efficiency: Faster training speed and lower memory usage.
    • Accuracy: High predictive accuracy.
    • Scalability: Support for parallel, distributed, and GPU learning, capable of handling large-scale datasets.

    LightGBM is widely used in machine learning competitions due to its ability to outperform other boosting frameworks in both efficiency and accuracy.

  2. Overview of LightGBM features

    main

    LightGBM is a gradient boosting framework that utilizes tree-based learning algorithms. It is optimized for distributed and efficient machine learning with the following key characteristics:

    • High Efficiency: Faster training speeds and lower memory usage compared to many other frameworks.
    • Accuracy: Designed for high-performance predictive accuracy.
    • Scalability: Capable of handling large-scale datasets.
    • Hardware Support: Supports parallel, distributed, and GPU-accelerated learning.

  3. Available LightGBM GPU Dockerfile variants

    main

    LightGBM provides different Dockerfile configurations depending on your deployment needs (CLI-only vs. Python-enabled) and desired image size:

    1. dockerfile.gpu (Python Version): Includes LightGBM (CPU/GPU) and a full Python stack (Conda, scikit-learn, pandas, matplotlib, Jupyter). Best for interactive development.
    2. dockerfile-cli-only.gpu (Small CLI Version): A multi-stage build using nvidia/opencl:devel and nvidia/opencl:runtime. Resulting image size is ~100 MB. Supports CLI-only usage in GPU and CPU modes.
    3. dockerfile-cli-only-distroless.gpu (Tiny CLI Version): A multi-stage build using nvidia/opencl:devel-ubuntu18.04 and distroless/cc-debian10. Resulting image size is ~15 MB. Optimized for minimal footprint in GPU/CPU CLI modes.

  4. Use LightGBM Callbacks during training

    main

    Callbacks allow you to inject custom logic into the training loop. Common built-in callbacks include:

    • early_stopping: Stops training when the validation metric stops improving.
    • log_evaluation: Periodically logs training progress to the console.
    • record_evaluation: Records evaluation results for later analysis.
    • reset_parameter: Resets specific parameters during training.

  5. Understand LightGBM parameter merging order

    main

    LightGBM merges parameters from multiple sources using a specific precedence order. If a parameter is defined in multiple places, the later item in this list will overwrite the earlier ones:

    1. LightGBM's default values
    2. Special files for weight, init_score, query, and positions
    3. (CLI only) Configuration in a file passed via config=train.conf
    4. (CLI only) Configuration passed directly via the command line
    5. (Python, R) Special keyword arguments to specific functions (e.g., num_boost_round in train())
    6. (Python, R) The params function argument (including **kwargs in Python and ... in R)
    7. (C API) The parameters or params function argument

  6. Comparison of LightGBM with XGBoost settings

    main

    LightGBM's performance is often compared against XGBoost. Because LightGBM uses a leaf-wise algorithm (controlled by num_leaves) and XGBoost typically uses a depth-wise algorithm (controlled by max_depth), a direct parameter-for-parameter comparison is not possible.

    To achieve a comparable model complexity, a common tradeoff is to set XGBoost's max_depth=8 (which results in a maximum of 255 leaves) to compare against LightGBM with num_leaves=255.

  7. How GPU acceleration works in LightGBM

    main
    LightGBM accelerates the most computationally expensive part of training—building feature histograms—using an efficient GPU algorithm. The implementation is modular and supports all learning tasks (classification, ranking, regression, etc.) and distributed learning settings. The GPU implementation is based on OpenCL, allowing it to work across a wide range of hardware.

  8. Use the LightGBM Data Structure API

    main

    The Data Structure API provides the core objects required for managing data and models in LightGBM. Key components include:

    • Dataset: The primary container for training data, optimized for memory efficiency and speed.
    • Booster: The core model object used for making predictions and managing the boosting process.
    • CVBooster: Used for cross-validation tasks.
    • EvalResult: Contains evaluation results from training.
    • Sequence: A utility for handling sequences of data.

  10. Visualize LightGBM models with Plotting utilities

    main

    LightGBM provides several functions to visualize model behavior and structure:

    • plot_importance: Shows feature importance.
    • plot_split_value_histogram: Visualizes the distribution of split values.
    • plot_metric: Plots training/validation metrics over iterations.
    • plot_tree: Visualizes a specific tree from the model.
    • create_tree_digraph: Generates a tree structure as a digraph.

  11. Load data into a LightGBM Dataset

    main

    LightGBM uses a Dataset object to store data. It supports various input formats including LibSVM/TSV/CSV text files, NumPy arrays, SciPy sparse matrices, pandas/polars DataFrames, pyarrow Tables, and LightGBM binary files.

    Key features for Dataset construction:

    • Categorical Features: You can pass categorical_feature names directly. LightGBM handles them without one-hot encoding, providing a significant speed-up. Note: You must convert categorical features to int type before constructing the Dataset.
    • Weights: Use the weight parameter or Dataset.set_weight() to assign weights to samples.
    • Feature Names: Use the feature_name parameter to specify names.
    • Memory Efficiency: To save memory, set free_raw_data=True (default) or explicitly set raw_data=None after construction.
    import numpy as np
import lightgbm as lgb

# From NumPy arrays
rng = np.random.default_rng()
data = rng.uniform(size=(500, 10))
label = rng.integers(low=0, high=2, size=(500, ))
train_data = lgb.Dataset(data, label=label, feature_name=['c1', 'c2', 'c3'], categorical_feature=['c3'])

# From LibSVM/Binary files
train_data = lgb.Dataset('train.svm.bin')

# From SciPy sparse matrix
import scipy
csr = scipy.sparse.csr_matrix((dat, (row, col)))
train_data = lgb.Dataset(csr)

  12. Use the LightGBM Dask API

    main

    For distributed computing with large datasets, LightGBM supports Dask. This API was added in version 3.2.0.

    Available classes include:

    • DaskLGBMClassifier: Distributed classification.
    • DaskLGBMRegressor: Distributed regression.
    • DaskLGBMRanker: Distributed ranking.