SynapseML Documentation

repository·master·Indexed 26 days ago

SynapseML (formerly MMLSpark) is an open-source machine learning library for Apache Spark that simplifies the creation of scalable ML pipelines. It provides a unified ecosystem for deep learning, image analysis, and recommendation systems, featuring specialized modules such as LightGBM on Spark, Vowpal Wabbit on Spark, ONNX on Spark, and LangchainTransformer for integrating LangChain with Spark DataFrames. It supports Python, R, Scala, Java, and .NET.

Tokens: 122K
Snippets: 352
Records: 467
Agent score: 87%

What's inside SynapseML

Overview of Synapse Machine Learning (PySpark)
master
Synapse Machine Learning is a PySpark library that provides Spark estimators, transformers, and utility functions designed for performing machine learning tasks on Spark. It is the Python interface for the SynapseML ecosystem.
https://github.com/microsoft/synapseml/blob/master/core/src/main/python/synapse/README.txt
Overview of SynapseML Distributed Deep Learning
master
SynapseML provides a simple deep learning toolkit designed to run modern deep learning methods on Apache Spark clusters without requiring extensive domain expertise in distributed systems.
Key capabilities include:
- Visual Classification: Distributed transfer learning for image classification using pretrained models.
- Text Classification: Implementation of NLP tasks like sentiment analysis and language modeling.
The toolkit leverages Horovod for efficient scaling across multiple GPUs and nodes using ring-allreduce algorithms, and PyTorch Lightning to provide a clean, modular, and scalable code structure for training.
https://github.com/microsoft/synapseml/blob/master/docs/Explore%20Algorithms/Deep%20Learning/Distributed%20Training.md
Overview of Synapse Machine Learning
master
SynapseML (formerly MMLSpark) is an open-source library designed to simplify the creation of massively scalable machine learning (ML) pipelines. It is built on top of the Apache Spark distributed computing framework and shares the same API as SparkML/MLLib, allowing for seamless integration into existing Spark workflows.
Key Capabilities:
- Scalability: Train and evaluate models on single-node, multi-node, or elastically resizable clusters.
- Multi-language Support: Usable across Python, R, Scala, Java, and .NET.
- Data Abstraction: Provides APIs that abstract over various databases, file systems, and cloud data stores.
- Task Variety: Supports text analytics, computer vision, anomaly detection, deep learning, and more.
https://github.com/microsoft/synapseml/blob/master/README.md
Overview of SynapseML
master
SynapseML (formerly MMLSpark) is an open-source library designed to simplify the creation of massively scalable machine learning (ML) pipelines. It provides composable, distributed APIs for tasks including text analytics, computer vision, anomaly detection, and deep learning.
Key characteristics:
- Built on Apache Spark: It shares the same API as SparkML/MLLib, allowing seamless integration into existing Apache Spark workflows.
- Scalability: Supports training and evaluation on single-node, multi-node, and elastically resizable clusters.
- Multi-language Support: Usable across Python, R, Scala, Java, and .NET.
- Data Abstraction: The API abstracts over various databases, file systems, and cloud data stores.
https://github.com/microsoft/synapseml/blob/master/docs/Overview.md
Quickstart: Install and run the SynapseML Docker image
master
To get started with SynapseML using Docker, install Docker for your OS, then run the following command to pull the image and start a Jupyter notebook server:
```
docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release
```
Once running, navigate to http://localhost:8888/ in your browser. You will be prompted to accept the EULA. To bypass the EULA prompt automatically, add the -e ACCEPT_EULA=y environment variable to your command.
```
docker run -it -p 8888:8888 -e ACCEPT_EULA=y mcr.microsoft.com/mmlspark/release
```
https://github.com/microsoft/synapseml/blob/master/docs/Reference/Docker%20Setup.md
Prepare input data for the Smart Adaptive Recommendations (SAR) algorithm
master
The SAR algorithm requires two types of input data to generate personalized recommendations:
1. Transaction (Usage) Data: Contains information on interactions between users and items. This data must be provided in a comma-separated format with the following schema: <User Id>,<Item Id>,<Time>
  Each row represents a single interaction (transaction). SAR uses implicit events (like clicks or purchases) rather than explicit ratings.
2. Catalog Data: Contains item descriptions used to handle 'cold' items (items with low or no transaction history) via feature-based similarity.
Example Transaction Data (CSV format):
```
User 1,Item 1,2015/06/20T10:00:00
User 1,Item 1,2015/06/28T11:00:00
User 1,Item 2,2015/08/28T11:01:00
User 1,Item 2,2015/08/28T12:00:01
```
https://github.com/microsoft/synapseml/blob/master/docs/Explore%20Algorithms/Other%20Algorithms/Smart%20Adaptive%20Recommendations.md
Run a specific version of the SynapseML Docker image
master
By default, mcr.microsoft.com/mmlspark/release uses the latest tag, which points to the most recent stable version. To run a specific version, append the version tag to the image name using the format image:tag.
Example for version 1.1.3: mcr.microsoft.com/mmlspark/release:1.1.3
https://github.com/microsoft/synapseml/blob/master/docs/Reference/Docker%20Setup.md
Install SynapseML in Azure Synapse Analytics
master
SynapseML is pre-installed in Synapse Analytics notebooks. To change the version, use the %%configure -f magic in the first cell of your notebook based on your Spark pool version.
- Spark 3.5: Use version 1.1.3.
- Spark 3.4: Use version 1.0.15.
- Spark 3.3: Use version 0.11.4-spark3.3.
In Azure Synapse, it is recommended to set spark.yarn.user.classpath.first to true to override existing packages.
```
# For Spark 3.5 pools
%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}
```
https://github.com/microsoft/synapseml/blob/master/docs/Get%20Started/Install%20SynapseML.md
Generate and interpret User Recommendations with SAR
master
SAR produces User Recommendations by multiplying the Item-to-Item similarity matrix with a user's affinity vector.
Key Advantages:
- Real-time updates: When a user performs a new action (e.g., adding an item to a cart), the affinity vector is updated. New recommendation scores can be calculated immediately without retraining the entire similarity model.
- Interpretability: Every recommendation score is explainable. The score for an item is the sum of similarities to all items the user has interacted with, weighted by affinity.
Example Interpretation: If an item has a high recommendation score, it is often because it has a high similarity to an item the user has a high affinity for. This allows you to generate natural language explanations like: "The algorithm recommends Item X because it is similar to Item Y, which you have interacted with frequently."
https://github.com/microsoft/synapseml/blob/master/docs/Explore%20Algorithms/Other%20Algorithms/Smart%20Adaptive%20Recommendations.md

Tune hyperparameters with TuneHyperparameters

master

Use TuneHyperparameters to automate the search for optimal hyperparameters across multiple models.

Workflow:

Define Models: Create your Spark ML estimators (e.g., LogisticRegression, RandomForestClassifier).
Wrap Models: Wrap them in TrainClassifier instances.
Build Search Space: Use HyperparamBuilder to define which parameters to tune. You can use RangeHyperParam for continuous ranges or DiscreteHyperParam for specific values.
Define Search Strategy: Use RandomSpace to convert the search space into a format suitable for the tuner.
Execute Tuning: Initialize TuneHyperparameters with your models, the paramSpace, evaluationMetric, numFolds, and numRuns.

from synapse.ml.automl import *
from synapse.ml.train import *
from pyspark.ml.classification import LogisticRegression, RandomForestClassifier, GBTClassifier

# ... (dataframe df definition) ...

logReg = LogisticRegression()
randForest = RandomForestClassifier()
gbt = GBTClassifier()
smlmodels = [logReg, randForest, gbt]
mmlmodels = [TrainClassifier(model=model, labelCol="Label") for model in smlmodels]

# Define the search space
paramBuilder = (HyperparamBuilder()
    .addHyperparam(logReg, logReg.regParam, RangeHyperparam(0.1, 0.3))
    .addHyperparam(randForest, randForest.numTrees, DiscreteHyperparam([5,10]))
    .addHyperparam(randForest, randForest.maxDepth, DiscreteHyperparam([3,5]))
    .addHyperparam(gbt, gbt.maxBins, RangeHyperparam(8,16))
    .addHyperparam(gbt, gbt.maxDepth, DiscreteHyperparam([3,5])))

searchSpace = paramBuilder.build()
randomSpace = RandomSpace(searchSpace)

# Run the tuning
bestModel = TuneHyperparameters(
              evaluationMetric="accuracy", 
              models=mmlmodels, 
              numFolds=2,
              numRuns=len(mmlmodels) * 2, 
              parallelism=2,
              paramSpace=randomSpace.space(), 
              seed=0).fit(df)

Build SynapseML R bindings from source
master
If you are developing SynapseML, you can build the R bindings using sbt. After running the build, the generated R files can be installed locally using devtools::install_local().
https://github.com/microsoft/synapseml/blob/master/docs/Reference/R%20Setup.md
Install SynapseML via Spark Submit, Spark Shell, or PySpark
master
You can install SynapseML on existing Spark clusters using the --packages option.
```
spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3
pyspark --packages com.microsoft.azure:synapseml_2.12:1.1.3
spark-submit --packages com.microsoft.azure:synapseml_2.12:1.1.3 MyApp.jar
```
https://github.com/microsoft/synapseml/blob/master/README.md