DOCSEARCHbyAlgolia
Submit Docs

logparser

repository·main·Indexed 24 days ago

https://github.com/logpai/logparser

A tool for parsing system logs to transform unstructured log messages into structured formats. The project includes access to various benchmark datasets from Loghub, covering distributed systems (HDFS, Hadoop, Spark, Zookeeper, OpenStack), supercomputers (BGL, HPC, Thunderbird), operating systems (Windows, Linux, Mac), mobile systems (Android, HealthApp), and server applications (Apache, OpenSSH).

Tokens
14K
Snippets
60
Records
131
Agent score
84%

What's inside logparser

  1. Overview of the Thunderbird dataset

    main

    Thunderbird is an open dataset of logs collected from a Thunderbird supercomputer system at Sandia National Labs (SNL). The dataset features 9,024 processors and 27,072GB of memory.

    Key characteristics for log analysis:

    • Alert Categorization: The logs contain both alert and non-alert messages identified by alert category tags.
    • Labeling Format: In the first column of the log, a hyphen (-) indicates a non-alert message, while any other value indicates an alert message.
    • Research Use Case: The label information is specifically designed for research in alert detection and prediction.

  2. Overview of available log datasets

    main

    The logparser project provides access to several benchmark datasets for evaluating log parsing performance. These datasets vary in scale and annotation quality:

    • Loghub_2k: Sampled datasets containing 2,000 lines per log. Templates are extracted via regular expressions and manually validated. Used for benchmarking in ICSE 2019.
    • Loghub_2k_corrected: A refined version of Loghub_2k that fixes incorrect ground-truth event templates, based on ICSE 2022 guidelines.
    • Loghub: Large-scale collections of raw system logs available for AI-driven log analytics research.
    • LogPub: Large-scale raw logs from Loghub that include manual annotations for event templates, designed for more rigorous and practical evaluation.

  3. What is LogSig

    main
    LogSig is a message signature-based algorithm designed to generate system events from raw textual log messages. It works by identifying the most representative message signatures to categorize log messages into a set of event templates. It is capable of handling various log data types and allows for the incorporation of human domain knowledge to improve performance.

  4. Overview of Logparser functionality

    main
    Logparser is a suite of open-source tools designed for production environments to automate the transformation of unstructured logs into structured data. The core workflow involves automatically learning event templates from raw log messages and converting those messages into a sequence of structured events.

  5. Overview of the BGL dataset

    main

    BGL is a dataset of logs from a BlueGene/L supercomputer system. It contains both alert and non-alert messages, making it suitable for research in log parsing, anomaly detection, and failure prediction.

    Data Structure:

    • The log includes alert category tags.
    • In the first column, a - indicates a non-alert message.
    • Other values in the first column indicate alert messages.

  6. Overview of LogCluster

    main
    LogCluster is a tool designed for log file clustering and mining line patterns from log files. It is originally a Perl-based tool, but this repository provides a Python wrapper around the original Perl source code to provide a common interface and facilitate benchmarking. It is compatible with both Linux and Windows systems (on Windows, Strawberry Perl is recommended to run the underlying Perl program).

  7. What is Logram?

    main
    Logram is an automated log parsing technique that utilizes n-gram dictionaries to achieve efficient log parsing. It is based on the research presented in the paper: Logram: Efficient Log Parsing Using n-Gram Dictionaries (IEEE Transactions on Software Engineering, 2020).

  8. Available log parsers in logparser

    main

    The repository implements a wide variety of log parsing algorithms from various research publications. Common parsers include:

    • Drain (ICWS'17): Online log parsing with a fixed-depth tree.
    • Spell (ICDM'16): Streaming parsing of system event logs.
    • LogMine (CIKM'16): Fast pattern recognition.
    • NuLog (ECML-PKDD'20): Self-supervised log parsing.
    • DivLog (ICSE'24): Prompt enhanced in-context learning.
    • Brain (TSC'23): Bidirectional parallel tree.

    Each parser is located in its own subdirectory under logparser/ and includes its own demo.py and benchmark.py.

  9. What is Logmatch and how does it work?

    main
    Logmatch is a regular expression-based method designed to match log messages against a predefined set of message templates. It is intended for streaming log parsing scenarios where templates have already been extracted (for example, by training a learning-based parser like Drain).

  10. What is ULP (Universal Log Parsing)?

    main

    ULP (Universal Log Parsing) is a tool designed to extract templates from unstructured log data. It uses a combination of pattern matching and frequency analysis to recognize log events.

    How it works:

    1. Grouping: Log events are first organized into groups using a text processing method.
    2. Frequency Analysis: Local frequency analysis is applied to instances within the same group to distinguish between static content (templates) and dynamic content (parameters).

    ULP is optimized for both accuracy and speed, capable of parsing up to four million log events in less than 3 minutes.

  11. What is NuLog?

    main
    NuLog is a log parsing technique designed to transform semi-structured records with free-form text log messages into structured templates. It uses a self-supervised learning model that formulates parsing as a masked language modeling (MLM) task. During parsing, the model extracts vector embeddings as summarizations, which allows the MLM pre-training to be coupled with downstream anomaly detection tasks.

  12. What is LFA (Log File Abstraction)?

    main
    LFA (Log File Abstraction) is a log parsing approach that extends the SLCT method. While SLCT may struggle to abstract event templates for all log messages, LFA addresses this by comparing token frequencies within each individual log message rather than across the entire dataset. This allows the algorithm to identify parameters by distinguishing token frequencies specifically within a single log message.