DOCSEARCHbyAlgolia
Submit Docs

MLJ (Machine Learning in Julia)

repository·dev·Indexed 23 days ago

https://github.com/juliaai/mlj.jl

A comprehensive machine learning toolbox for Julia providing a unified interface for over 200 models. MLJ enables standardized workflows for model selection, tuning, evaluation, and composition. Key features include the @iload macro for dynamic model loading, a system for building model pipelines with the pipe operator, and integrated tools for hyperparameter tuning via TunedModel and performance assessment using the evaluate() function.

Tokens
36.5K
Snippets
85
Records
189
Agent score
83%

What's inside MLJ.jl

  1. Overview of MLJ (Machine Learning in Julia)

    dev

    MLJ is a machine learning toolbox for Julia that provides a unified interface and meta-algorithms for the entire machine learning workflow. It allows users to select, tune, evaluate, compose, and compare over 200 machine learning models. These models can be written natively in Julia or in other languages, provided they adhere to the MLJ interface.

    MLJ acts as an umbrella package that integrates various components distributed across the MLJ ecosystem.

  2. Explore community and workshop resources for MLJ

    dev

    Beyond official documentation, you can use the following resources to deepen your understanding of MLJ and Julia-based data science:

    • MLJTutorial.jl: Material designed for a 4-hour MLJ workshop.
    • MLCourse: Teaching material for an introductory machine learning course at EPFL.
    • Julia Data Science: A broader community resource for Julia-based data science workflows.

  3. Identify third-party packages integrated with MLJ

    dev

    MLJ integrates with various third-party packages to extend its capabilities. These integrations fall into three main categories:

    1. Models in the MLJ Model Registry: Packages that provide models that can be discovered and used via the standard MLJ model registry interface.
    2. Unregistered Models: Packages that provide models which are not yet part of the official MLJ model registry but are compatible with MLJ workflows.
    3. Extended Functionality: Packages that provide specialized tools such as hyper-parameter optimization strategies, feature interpretation, fairness metrics, outlier detection, or uncertainty quantification.

    For a complete list of models available in the registry, refer to the [List of Supported Models](@ref model_list).

  4. Compare MLJ with ScikitLearn.jl

    dev

    While ScikitLearn.jl provides access to the mature Python scikit-learn library via a Julia wrapper, MLJ is designed as a native Julia ecosystem. Key advantages of MLJ include:

    • Native Julia Implementation: Algorithms implementing the MLJ interface are 100% Julia, allowing for superior interoperability with libraries like Flux.jl (for automatic differentiation/gradient-descent tuning) and CuArrays.jl (for GPU acceleration) without major refactoring.
    • Model Registry: Unlike ScikitLearn.jl where metadata must be found in documentation, MLJ uses a structured, searchable model registry that provides metadata (e.g., handling of categorical inputs, probabilistic prediction capabilities) without requiring models to be loaded.
    • Flexible Model Composition: MLJ is built around a "learning network" API, allowing models to be connected in arbitrary ways (like Wolpert model stacks). These networks support "smart" training, where only necessary components are retrained after parameter changes.
    • Standardized Probabilistic API: MLJ provides a universal standard for probabilistic predictions, improving support for Bayesian statistics and probabilistic graphical models.
    • Robust Categorical Data Handling: MLJ uses dedicated categorical data types that track the full pool of possible classes. This prevents common errors where test sets contain categories not seen during training; MLJ models preserve these class pools so that probabilistic predictions can correctly account for missing classes with zero probability.

  5. What is a Machine in MLJ?

    dev

    A Machine is an object that binds a model (an algorithm plus its hyperparameters) to specific data. It serves as the primary container for storing learned parameters after training.

    Key behaviors:

    • Training: Calling fit! on a machine triggers the training process.
    • Warm Restarts: If a model supports it, calling fit! after increasing an iteration parameter (like epochs) can avoid redundant calculations by performing a 'warm restart' instead of full retraining.
    • Retraining: Changing hyperparameters typically triggers full retraining on subsequent fit! calls, whereas increasing an iteration parameter might only add to the existing state.
    using MLJ; color_off() # hide
tree = (@load DecisionTreeClassifier pkg=DecisionTree verbosity=0)()
forest = EnsembleModel(model=tree, n=10);
X, y = @load_iris;
mach = machine(forest, X, y)
fit!(mach, verbosity=2);

  6. Overview of Model Tuning in MLJ

    dev

    In MLJ, tuning is implemented as a model wrapper called TunedModel. This wrapper transforms specific hyperparameters into learned parameters.

    Workflow:

    1. Wrap a model in a TunedModel using a tuning strategy (e.g., Grid, RandomSearch).
    2. Define a range for the hyperparameters to be optimized.
    3. Bind the TunedModel to data in a machine (called mach).
    4. Call fit!(mach) to execute the search for optimal hyperparameters and train the best model on the full dataset.
    5. Call predict(mach, Xnew) to use the optimized model for predictions.

    Note on Evaluation: Evaluating a TunedModel instance using evaluate! implies nested resampling, which is a best practice to avoid overfitting during the tuning process.

  7. Understand scientific types for discrete data

    dev

    MLJ uses scientific types to define data requirements. Discrete data falls into three categories:

    1. Count (<: Infinite): Used for frequency or unbounded data (e.g., number of phone calls, population). Note: You cannot use raw integers to represent categorical data in MLJ; integers are reserved for Count.
    2. OrderedFactor (<: Finite): Used for ordered categorical data (e.g., number of rooms in a house, or an exam result like 'Pass' vs 'Fail').
    3. Multiclass (<: Finite): Used for unordered categorical data (e.g., animal species, colors).

    Binary Data: There is no specific Binary type. Use OrderedFactor{2} if the order matters (the second class is treated as the 'positive' class for metrics like true_positive) or Multiclass{2} if it does not. Note: Bool data is treated as Count and should generally be coerced to Multiclass or OrderedFactor.

  8. Build flexible model compositions with Learning Networks

    dev
    For complex compositions that go beyond simple chains or ensembles, use Learning Networks. These act as 'blueprints' that allow you to combine models in flexible ways. You can transform existing workflows and 'export' them to define new, stand-alone model types.

  9. Compose machine learning pipelines

    dev

    MLJ allows for advanced model composition, including preprocessing steps and iterative model wrapping. You can use the pipe operator (|>) to chain components together.

    Common composition patterns include:

    1. Iterative Models: Wrapping a model with IteratedModel (from MLJIteration) to automatically learn the number of iterations based on a criterion (e.g., NumberSinceBest).
    2. Preprocessing: Chaining a transformer (e.g., ContinuousEncoder()) with a model using the |> operator.
    3. Self-Tuning Models: Wrapping a pipeline in a TunedModel to optimize hyper-parameters using strategies like RandomSearch() over defined range objects.
    using MLJ
using MLJIteration

# 1. Load and instantiate a model
Booster = @load EvoTreeRegressor
booster = Booster(max_depth=2)

# 2. Wrap to make it self-iterating
iterated_booster = IteratedModel(model=booster,
                                 resampling=Holdout(fraction_train=0.8),
                                 controls=[Step(2), NumberSinceBest(3), NumberLimit(300)],
                                 measure=l1,
                                 retrain=true)

# 3. Preprocess features via pipeline
pipe = ContinuousEncoder() |> iterated_booster

# 4. Wrap in TunedModel for hyper-parameter optimization
max_depth_range = range(pipe, :(deterministic_iterated_model.model.max_depth), lower=1, upper=10)

self_tuning_pipe = TunedModel(model=pipe,
                              tuning=RandomSearch(),
                              ranges=max_depth_range,
                              resampling=CV(nfolds=3, rng=456),
                              measure=l1,
                              acceleration=CPUThreads(),
                              n=50)

  10. Use training losses instead of out-of-sample loss in IteratedModel

    dev

    Some iterative models report training losses during fit!. You can use these to supplement out-of-sample estimates or as a substitute when you want to train on all supplied data without resampling.

    To force an IteratedModel to use training losses and bind all data to the training machine (instead of using resampling/cross-validation), set resampling=nothing.

    To check if a specific model supports training losses, use supports_training_losses(ModelType) or inspect the model with info(ModelType).

  11. Re-use learned parameters across different nodes

    dev

    In a learning network, multiple nodes can point to the same machine. This is particularly useful for applying an inverse transformation to predictions using parameters learned from the target variable during training.

    function MLJBase.prefit(composite::CompositeD, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    mach1 = machine(:preprocessor, Xs)
    W = transform(mach1, Xs)

    # mach2 learns the target transformation
    mach2 = machine(:target_transformer, ys)
    z = transform(mach2, ys)

    mach3 = machine(:regressor, W, z)
    zhat = predict(mach3, W)

    # Re-use mach2 to perform the inverse transform on the prediction
    yhat = inverse_transform(mach2, zhat)

    return (predict = yhat,)
end

  12. Access nested hyperparameters in TransformedTargetModel

    dev
    When using TransformedTargetModel, the hyperparameters of both the underlying model and the transformer are accessible as nested hyperparameters. This allows you to tune the model and the transformation process simultaneously during training or evaluation.