Apache PredictionIO

Apache PredictionIO is an open-source machine learning framework designed for developers and data scientists. It implements a Lambda Architecture and is built upon scalable open-source services including Hadoop, HBase (and other databases), Elasticsearch, and Spark.

Key capabilities include:

• Event Collection: Gathering data for machine learning.
• Algorithm Deployment: Deploying ML models into production.
• Evaluation: Assessing model performance.
• Querying: Accessing predictive results via REST APIs.

You can follow and interact with the Apache PredictionIO community through the following channels:

• GitHub: View the source code, track issues, and contribute at https://github.com/apache/predictionio.
• Twitter: Follow @predictionio for updates.
• Facebook: Like the official page at https://www.facebook.com/predictionio.

PredictionIO is a machine learning stack composed of three primary components:

1. PredictionIO platform: The core open-source stack used for building, evaluating, and deploying machine learning engines.
2. Event Server: An analytics layer that unifies data from multiple platforms. It collects data from your application (in real-time or batch) to serve two purposes:
   • Providing data to Engines for model training and evaluation.
   • Offering a unified view for data analysis.
3. Template Gallery: A collection of pre-built engine templates for various machine learning applications (e.g., recommendation engines, classification, etc.).

In a typical workflow, the Event Server collects data, an Engine uses that data to build predictive models, and the Engine is then deployed as a web service to respond to real-time queries.

PredictionIO provides several SDKs to interact with the PredictionIO engine from different programming environments. These are categorized into officially supported SDKs and community-powered SDKs.

Officially Supported SDKs

Use these SDKs for production environments where official support is required:

• Java & Android SDK: For JVM-based applications and Android mobile development.
• PHP SDK: For integrating PredictionIO with PHP web applications.
• Python SDK: For data science workflows and Python-based backend services.
• Ruby SDK: For integrating with Ruby on Rails or other Ruby applications.

Community Powered SDKs

For other languages or specialized integrations, check the community projects list to find unofficial SDKs maintained by the community.

PredictionIO provides several specialized guides for the evaluation process:

• Hyperparameter Tuning: An end-to-end workflow for using the evaluation module to select and deploy the optimal engine parameters.
• Evaluation Dashboard: A centralized dashboard for viewing detailed breakdowns of all previous evaluation runs.
• Choosing Evaluation Metrics: Guidance on selecting appropriate machine learning metrics for your use case.
• Building Evaluation Metrics: Instructions on how to implement custom metrics (often requiring only a few lines of code).

PredictionIO supports exporting data for use with the following analytics tools:

1. IPython Notebook: For interactive computing and data visualization.
2. Tableau: For business intelligence and advanced data visualization.
3. Zeppelin: For interactive data analytics and notebook-based workflows.

Data should be exported to Apache Parquet format via pio export before being used with these tools.

An Engine represents a specific type of prediction task (for example, a product recommendation engine). It is a composite entity consisting of four primary components:

1. Data Source and Data Preparator: Handles the ingestion and transformation of data.
2. Algorithm: The core logic that performs the predictive modeling.
3. Serving: The mechanism that provides predictions to users or applications.
4. Evaluation Metrics: The framework used to measure the performance of the engine.

A Metric is used by the hyperparameter tuning module to determine the quality of an engine variant. It evaluates an engine by comparing the engine's output (predicted result) against the original label (actual result).

In the context of hyperparameter tuning, the goal is to find the engine parameters that yield the highest Metric score. In machine learning literature, this is sometimes referred to as a loss function, though in the context of a loss function, the goal is typically to minimize the value rather than maximize it.

A Training Model is an abstraction implemented as a set of Scala classes that take feature observations as input and produce a predictive model.

In the provided text classification engine template, the Training Model is implemented using the NBModel class. This specific implementation produces a Multinomial Naive Bayes classifier based on t.f.-i.d.f. vectorized text. When building your own engine, you can implement this abstraction to output any predictive model that the Algorithm component can then leverage for real-time queries.

In the PredictionIO DASE architecture, the Serving component is the final stage where post-processing occurs. After the Algorithm component makes predictions, the Serving component receives the results and can modify them before they are returned to the user.

Common use cases for the Serving component include:

• Filtering out items that are out of stock.
• Applying real-time blacklists.
• Re-ranking results based on business logic.

In a standard engine template, the Serving component is implemented in src/main/scala/Serving.scala by extending LServing[Query, PredictedResult] and overriding the serve method.