SciKit-Learn Laboratory (SKLL)

The SKLL package is organized into several functional modules:

• skll.config: Parses experiment configuration files.
• skll.experiments: Handles creating and running experiments, collecting metrics, and writing results to disk.
• skll.learner: Contains the Learner and VotingLearner classes. Learner is used for standard learners, while VotingLearner is used specifically for VotingClassifier and VotingRegressor.
• skll.metrics: Contains custom metrics (e.g., kappa, kendall_tau, spearman) and powers user-defined custom metrics.
• skll.data: Manages data via FeatureSet metadata, readers.py (reading files into FeatureSet), writers.py (writing FeatureSet to disk), and dict_vectorizer.py (an enhanced sklearn.feature_extraction.DictVectorizer).
• skll.utils.commandline: A collection of utility scripts for common tasks like generating predictions, filtering features, and summarizing results.

SKLL provides a pipeline attribute on Learner objects to allow seamless interoperability with scikit-learn. When a Learner is initialized with pipeline=True, the pipeline attribute contains a (deep) copy of the fitted pipeline components. This allows you to use standard scikit-learn methods like .predict() directly on the pipeline, avoiding the need to manually call transformations (like scaling or vectorization) on your input data.

Key Behaviors:

• Deep Copies: The pipeline components are copies, so modifying them in scikit-learn space will not affect the original SKLL model.
• Automatic Densification: SKLL automatically inserts a skll.learner.Densifier stage in the pipeline if:
  • You use FeatureHasher along with feature scaling (with_mean or both).
  • You use SkewedChi2Sampler for feature sampling (as it requires dense input).
• DictVectorizer Behavior: When using DictVectorizer with feature_scaling set to with_mean or both, the sparse attribute of the vectorizer stage is automatically set to False to accommodate centering.

from sklearn.preprocessing import LabelEncoder
from skll.data import Reader
from skll.learner import Learner

# 1. Train a learner with pipeline=True
fs1 = Reader.for_path('examples/iris/train/example_iris_features.jsonlines').read()
learner1 = Learner('LogisticRegression', pipeline=True)
_ = learner1.train(fs1, grid_search=True, grid_objective='f1_score_macro')

# 2. Access the pipeline to predict on raw dictionary data
D1 = {"f0": 6.1, "f1": 2.8, "f2": 4.7, "f3": 1.2}
pipeline1 = learner1.pipeline

# 3. Use standard sklearn-style prediction
# (No need to manually call scaler.transform or vectorizer.transform)
prediction = pipeline1.predict(D1)

A valid custom metric function must accept two required positional arguments: y_true (true labels/scores) and y_pred (predicted labels/scores).

To control how the metric interacts with estimators, you can provide two optional keyword arguments (matching sklearn.metrics.make_scorer):

1. greater_is_better (bool): Indicates if a higher value is better. Defaults to True.
2. response_method (str or tuple): Specifies how to obtain predictions from an estimator. Possible values:
   • "predict": Uses estimator.predict() (default if None).
   • "predict_proba": Uses estimator.predict_proba().
   • "decision_function": Uses estimator.decision_function().
   • A tuple or list of these strings (e.g., ("decision_function", "predict_proba")) tells the scorer to use the first method implemented by the estimator.

Note on Deprecations: Do not use needs_proba or needs_threshold. Use response_method="predict_proba" for probability requirements and response_method=("decision_function", "predict_proba") for threshold requirements.

from sklearn.metrics import fbeta_score

def f075(y_true, y_pred):
    return fbeta_score(y_true, y_pred, beta=0.75)

The run_experiment function uses a Python configuration file (INI format) to define the type of experiment to execute. The task field in the [General] section determines the workflow:

• cross_validate: Performs cross-validation on training feature files. It uses StratifiedKFold. You can optionally provide specific folds using the folds_file setting. For classifiers, SKLL automatically adjusts the number of folds to match the minimum class count in the training data.
• evaluate: Trains a model and evaluates it on a separate test set. Requires a training location, a test location, and a results directory.
• predict: Trains a model and generates predictions on a test set. Requires a training location and a test location.
• train: Simply trains a model. Requires a training location.
• learning_curve: Generates learning curves using the SKLL feature pre-processing pipeline. Requires a training location.
  • Note: For reliable results, a minimum of 500 training examples is expected.
  • Note: If probability is set to True, probabilities are converted to the most likely label via argmax before computing the curve.

[General]
experiment_name = my_experiment
task = cross_validate

[Input]
# Example for cross_validate
train_file = path/to/data.csv
learners = RandomForestClassifier

SKLL supports several feature file formats:

• arff: Weka-compatible format. Supports simple numeric, string, and nominal values.
  • Requires an attribute for instance IDs (defaults to id) and labels (defaults to y).
  • The label attribute must be the final attribute in the file.
• csv/tsv: Comma or tab-delimited formats. Uses pandas for fast loading.
  • Requires a column for labels (defaults to y) and instance IDs (defaults to id).
  • Warning: SKLL will error on blank values. You must drop or replace them using filter_features or the Reader API.
• jsonlines/ndj (Recommended): A sparse format where each line is a JSON dictionary or a comment (//). Each dictionary must contain:
  • y: The class label.
  • x: A dictionary of feature values.
  • id: An optional instance ID.
• libsvm: Supports LibSVM, LibLinear, and SVMLight formats. You can include metadata in comments at the end of each line to provide names for IDs, labels, and features using the format: ID | 1=ClassX | 1=FeatureA 2=FeatureB. Note that |, #, and = are reserved characters and cannot be used in names.