Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics

BGL is an open dataset of logs collected from a BlueGene/L supercomputer system (131,072 processors, 32,768GB memory). The dataset contains both alert and non-alert messages, making it suitable for research in log parsing, anomaly detection, and failure prediction.

Log Format Details:

• The first column contains alert category tags.
• A '-' in the first column indicates a non-alert message.
• Any other value in the first column indicates an alert message.

Thunderbird is an open dataset of logs collected from a Thunderbird supercomputer system at Sandia National Labs (SNL). The dataset features 9,024 processors and 27,072GB of memory.

Key Characteristics:

• Content: Contains both alert and non-alert messages identified by alert category tags.
• Labeling Format: In the first column of the log, a hyphen (-) indicates a non-alert message, while any other value indicates an alert message.
• Use Case: The label information is designed for alert detection and prediction research.

For more detailed information, visit the project page.

The Hadoop dataset contains logs generated from a Hadoop cluster consisting of five machines (46 cores total, Intel(R) Core(TM) i7-3770 CPU, 16GB RAM per machine). It includes logs from two specific testing applications:

• WordCount: A MapReduce example application that counts word occurrences.
• PageRank: A program used for ranking web pages.

The logs capture both normal operations and simulated service failures, including:

• Machine down: Simulates machine failure by turning off a server.
• Network disconnection: Simulates network failure by disconnecting a server.
• Disk full: Simulates disk failure by manually filling a server's hard disk.

Labels for abnormal vs. normal job IDs are provided in abnormal_label.txt.

The OpenStack dataset contains both normal logs and abnormal cases with failure injection, suitable for anomaly detection research. The logs were generated on CloudLab.

Raw logs can be downloaded from the main Loghub repository: https://github.com/logpai/loghub

Loghub provides a collection of system logs for AI-driven log analytics research. Datasets are categorized into Distributed systems, Super computers, Operating systems, Mobile systems, Server applications, and Standalone software.

Logs are provided as raw files (often in .zip or .tar.gz formats) and are generally not sanitized or anonymized. You can download them directly via the links provided in the dataset table.

HealthApp is a dataset consisting of mobile application logs collected from an Android smartphone after more than 10 days of continuous use. The raw logs can be downloaded from the Loghub repository.

https://github.com/logpai/loghub

MacOS logs collected from /var/log/system.log after 7 days of use on a Macbook are available for download at the Loghub repository.

https://github.com/logpai/loghub

The HPC dataset contains logs collected from System 20 of a high performance computing cluster at Los Alamos National Laboratories. Since the original source link is out of service, you can download the raw logs directly from the Loghub repository.

https://github.com/logpai/loghub

Android_v1 logs consist of sample log files printed by the Android Application Framework, collected from smartphones with heavily instrumented modules. You can download the raw log files from the main Loghub repository.

https://github.com/logpai/loghub

The Spark log dataset consists of aggregated logs from 32 machines in a lab environment. The logs are provided in their raw format (over 2GB) without modification or labeling, containing both normal and abnormal application runs. Note that some logs from three machines may be missing due to repairs.

You can download the raw logs from the Loghub repository.

https://github.com/logpai/loghub

If you use Loghub datasets in your research, please cite the following two papers:

1. Loghub: Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023.
2. Loghub-2.0: Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, Michael R. Lyu. A Large-scale Evaluation for Log Parsing Techniques: How Far are We?. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024.