DOCSEARCHbyAlgolia
Submit Docs

DeepLog

repository·master·Indexed 43 days ago

https://github.com/thijsvanede/deeplog

A PyTorch implementation of the DeepLog model (CCS'17) for detecting and diagnosing anomalies in system logs using LSTM networks. It features anomaly detection, online adaptation, and root cause analysis, providing both a scikit-learn style API and a command-line interface for training and prediction.

Tokens
3.9K
Snippets
14
Records
22
Agent score
53%

What's inside DeepLog

  1. Overview of DeepLog

    master

    DeepLog is a PyTorch implementation of the DeepLog model (CCS'17), designed for anomaly detection and diagnosis in system logs. It uses Long Short-Term Memory (LSTM) networks to model system logs as natural language sequences.

    Key capabilities include:

    • Anomaly Detection: Automatically learns log patterns from normal execution and detects deviations.
    • Online Adaptation: Supports incremental model updates to adapt to new log patterns over time.
    • Root Cause Analysis: Constructs workflows from system logs to help users diagnose detected anomalies effectively.

  2. Use the DeepLog class for log anomaly detection

    master

    The deeplog.DeepLog class implements the neural network architecture described in the paper "Deeplog: Anomaly detection and diagnosis from system logs through deep learning". It is built on top of the torch-train library and is used to perform anomaly detection and diagnosis from system logs.

    Key capabilities include:

    • Training: Uses the fit method inherited from torch-train.
    • Inference: Uses a customized predict method that returns the k most likely outputs instead of a standard probability distribution.

  4. Install DeepLog dependencies

    master

    DeepLog requires several Python packages. If you are installing from source or need to manage dependencies manually, you can install them using the provided requirements.txt file or by installing the specific packages directly.

    Required Packages:

    • argformat
    • numpy
    • scikit-learn
    • torch (PyTorch)
    # Install via requirements.txt
pip install -r requirements.txt

# Or install packages manually
pip install -U argformat numpy scikit-learn torch

  5. Access DeepLog Installation and Reference Documentation

    master

    Detailed installation instructions, usage guides, and technical references are hosted on Read the Docs. For setup and API details, visit the official documentation site.

    https://deeplog.readthedocs.io/en/latest

  6. Cite DeepLog and DeepCASE

    master

    If you use DeepLog or DeepCASE in your research, please cite the following publications:

    DeepCASE: van Ede, T., Aghakhani, H., Spahn, N., Bortolameotti, R., Cova, M., Continella, A., van Steen, M., Peter, A., Kruegel, C. & Vigna, G. (2022, May). DeepCASE: Semi-Supervised Contextual Analysis of Security Events. In 2022 Proceedings of the IEEE Symposium on Security and Privacy (S&P). IEEE.

    DeepLog: Du, M., Li, F., Zheng, G., & Srikumar, V. (2017). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS) (pp. 1285-1298).

  8. Train and predict with DeepLog using the scikit-learn API

    master

    DeepLog provides high-level methods for training and inference on event sequences, following a scikit-learn style API. You can use the following methods to work with event sequences:

    • fit(data): Trains the model on the provided data.
    • predict(data): Predicts events given previous event sequences.
    • fit_predict(data): Performs both training and prediction in a single step.

  10. Use the Preprocessor to extract event sequences

    master

    The Preprocessor class is used to automatically extract event sequences from common data formats. To begin, instantiate a Preprocessor object and then call the appropriate method based on your file format.

    from deeplog import preprocessor

# Create the preprocessor object
prep = preprocessor.Preprocessor()

# Extract sequences (example using CSV)
sequences = prep.csv('path/to/your/file.csv')

  11. Use the DeepLog command line tool

    master

    DeepLog can be executed from the command line via the deeplog module. It provides a quick interface for training models and predicting sequences from log files in .csv or .txt formats. The tool operates in two primary modes: train and predict.

    python3 -m deeplog train --csv <data.csv> --save deeplog.save
python3 -m deeplog predict --csv <data.csv> --load deeplog.save

  12. Train and use DeepLog for prediction

    master

    The DeepLog object handles the neural network logic for anomaly detection.

    Initialization

    • input_size: The number of different events expected in the input.
    • hidden_size: The hidden dimension of the network (64 is suggested).
    • output_size: The number of different events expected in the output.

    Workflow

    1. Device Placement: You can move the model and data to a GPU using .to("cuda").
    2. Training: Call .fit() with your sequences X, targets y, number of epochs, and batch_size.
    3. Prediction: Call .predict() to get predicted targets y_pred and their associated confidence scores. Use the k parameter to specify the number of top predictions to return.
    from deeplog import DeepLog

deep_log = DeepLog(
    input_size  = 300,
    hidden_size = 64,
    output_size = 300,
)

# Move to GPU if available
deep_log = deep_log.to("cuda")
X, y = X.to("cuda"), y.to("cuda")

# Train
deep_log.fit(X=X, y=y, epochs=10, batch_size=128)

# Predict
y_pred, confidence = deep_log.predict(X=X, y=y, k=3)
    from deeplog import DeepLog

deep_log = DeepLog(
    input_size  = 300,
    hidden_size = 64,
    output_size = 300,
)

deep_log = deep_log.to("cuda")
X, y = X.to("cuda"), y.to("cuda")

deep_log.fit(
    X          = X,
    y          = y,
    epochs     = 10,
    batch_size = 128,
)

y_pred, confidence = deep_log.predict(
    X = X,
    y = y,
    k = 3,
)