LogBERT: Log Anomaly Detection using BERT

During execution, LogBERT creates the following directories:

• ~/.dataset: Stores the original datasets after they are downloaded.
• project/output: Stores intermediate files and final results generated during the execution process.

Alternatively, you can use Conda to set up the environment using the provided environment.yml file.

conda create -f ./environment/environment.yml
conda activate logbert

To run the LogBERT anomaly detection pipeline on the HDFS dataset, follow these steps within the HDFS directory:

1. Initialize: Run sh init.sh to download and prepare the raw data.
2. Process Data: Run python data_process.py to parse logs into structured data and create sequences.
3. LogBERT Pipeline:
   • Generate vocabulary: python logbert.py vocab
   • Train the model: python logbert.py train
   • Perform prediction: python logbert.py predict

cd HDFS
sh init.sh
python data_process.py

# run logbert
python logbert.py vocab
python logbert.py train
python logbert.py predict

On macOS and Linux, you can install LogBERT by creating a Python virtual environment and installing the dependencies from the provided requirements file.

Follow these steps:

1. Install virtualenv using pip.
2. Create a new virtual environment named env.
3. Activate the environment.
4. Install the required packages from ./environment/requirements.txt.
5. Deactivate the environment when finished.

python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate
pip install -r ./environment/requirements.txt
deactivate

To ensure compatibility, the following environment configuration is recommended:

• OS: Ubuntu 20.04
• NVIDIA driver: 460.73.01
• CUDA: 11.2
• Python: 3.8
• PyTorch: 1.9.0

To prepare log data for anomaly detection, use loglizer.dataloader.load_data to load the dataset and loglizer.preprocessing.FeatureExtractor to convert raw logs into feature vectors.

When loading data, ensure is_mapping=True is used if you are working with mapped log templates. After loading, you must fit_transform the training data and transform the test data using the same feature extractor instance to ensure consistent feature dimensions.

from loglizer import dataloader, preprocessing

# Load data
(x_train, y_train), (x_test, y_test) = dataloader.load_data(ouput_dir, middle_dir, log_file, is_mapping=True)

# Feature extraction
feature_extractor = preprocessing.FeatureExtractor()
x_train = feature_extractor.fit_transform(x_train)
x_test = feature_extractor.transform(x_test)

To use LogBERT baselines, you must first load your log data and transform it into a feature matrix. Use loglizer.dataloader.load_data to load the raw logs and loglizer.preprocessing.FeatureExtractor to convert the logs into numerical features suitable for machine learning models.

Note: When loading data, is_mapping=True is used to handle log mapping.

from loglizer import dataloader, preprocessing

ouput_dir = "../output/tbird/"
middle_dir = ""
log_file = "Thunderbird_20M.log"

# Load data
(x_train, y_train), (x_test, y_test) = dataloader.load_data(ouput_dir, middle_dir, log_file, is_mapping=True)

# Extract features
feature_extractor = preprocessing.FeatureExtractor()
x_train = feature_extractor.fit_transform(x_train)
x_test = feature_extractor.transform(x_test)

To prepare data for anomaly detection, use loglizer.dataloader.load_data to load datasets from a directory and loglizer.preprocessing.FeatureExtractor to transform the raw data into features. The load_data function returns training and testing sets as tuples of (features, labels).

from loglizer import dataloader, preprocessing

output_dir = "../output/hdfs/"
(x_train, y_train), (x_test, y_test) = dataloader.load_data(data_dir=output_dir)

feature_extractor = preprocessing.FeatureExtractor()
x_train = feature_extractor.fit_transform(x_train)
x_test = feature_extractor.transform(x_test)

The OneClassSVM model implements a One-Class Support Vector Machine, typically used for novelty or anomaly detection.

from loglizer.models import OneClassSVM

model = OneClassSVM(kernel='rbf')
model.fit(x_train, y_train)
precision, recall, f1 = model.evaluate(x_test, y_test)

The LogClustering model uses clustering to identify anomalies. It requires two threshold parameters:

• max_dist: The threshold to stop the clustering process.
• anomaly_threshold: The threshold used for anomaly detection.

Note: When training LogClustering, it is common practice to fit the model using only normal samples (where y_train == 0).

from loglizer.models import LogClustering

max_dist = 0.3
anomaly_threshold = 0.3
model = LogClustering(max_dist=max_dist, anomaly_threshold=anomaly_threshold)

# Fit using only normal samples
model.fit(x_train[y_train == 0, :])
precision, recall, f1 = model.evaluate(x_test, y_test)

The OneClassSVM model in loglizer.models can be used to identify outliers in a dataset.

Key parameters:

• kernel: The kernel type to be used in the algorithm (e.g., 'rbf').

from loglizer.models import OneClassSVM

model = OneClassSVM(kernel='rbf')
model.fit(x_train, y_train)
precision, recall, f1 = model.evaluate(x_test, y_test)