logmerger

Log files are merged by interleaving lines based on their timestamps. logmerger automatically attempts to detect matching timestamp formats for each input file.

Multi-line logs: Lines that do not contain a timestamp (such as Python tracebacks or indented log details) are automatically combined with the preceding timestamped line.

The --start and --end flags allow you to clip the merged output to a specific time window.

Timestamp Formats:

• Use YYYY-MM-DD HH:MM:SS.SSS (milliseconds are optional).
• You can use T between date and time (e.g., YYYY-MM-DDTHH:MM:SS) to avoid quoting spaces on the command line.
• A comma , can be used as a decimal point for milliseconds.

Relative Times:

• You can use relative time strings like 15m (15 minutes ago).
• Valid units: s (seconds), m (minutes), h (hours), d (days).

Install the base package from PyPI to use logmerger as a shell command:

pip install logmerger

To enable experimental support for merging packet capture files (.pcap), install with the pcap extra:

pip install logmerger[pcap]

The primary use case for logmerger is to view a merged display of multiple log files interleaved by their timestamps. By default, it opens an interactive TUI (Terminal User Interface) browser using the textual library.

To merge files and enter the interactive mode:

logmerger log1.txt log2.txt

Inside the interactive TUI, press h to view a help menu of all available key commands.

You can merge multiple log files that cover the same time span even if they use different timestamp formats. logmerger supports compressed files like .gz and will unpack them internally.

If you are using a custom or non-standard timestamp format (such as the one found in log7.txt), you must specify it using the --timestamp_format command-line option with a regex pattern.

# Example for custom timestamp formats like those in log7.txt
logmerger --timestamp_format '(.*m\[)((...)\|)' log7.txt

logmerger can combine plaintext log files with network packet capture data. When you pass a .pcap file as an argument, logmerger unpacks its contents and merges the captured packets into a single timeline alongside the plaintext log messages.

# Example merging server logs, client logs, and a Wireshark PCAP
logmerger server_log.txt client_log.txt mirror_server_8881.pcap

The JsonlFileReader handles .jsonl (JSON Lines) files. It automatically detects the timestamp column by attempting to parse values using TimestampedLineTransformer.

Once the timestamp key is identified, it formats each line as: <timestamp_value> <key1>: <value1> <key2>: <value2> ... (excluding the timestamp key itself).

The CsvFileReader handles .csv files by:

1. Reading the first row as headers.
2. Treating the first column as the timestamp.
3. Formatting subsequent rows as: <timestamp> <header1>=<value1> <header2>=<value2> ....

It includes a reader_guard to catch csv.Error exceptions (e.g., due to malformed lines) and yields a descriptive error string instead of crashing.

The TimestampedLineTransformer is an abstraction designed to normalize log lines by separating their temporal component from their content.

The Lifecycle:

1. Detection: A subclass (either built-in or custom) defines a timestamp_pattern (regex) and a strptime_format (or a callable formatter).
2. Matching: The transformer uses a combined pattern to identify the timestamp within a line.
3. Transformation: Upon calling the transformer, it extracts the timestamp string, converts it to a datetime object, and uses a substitution pattern (sub_repl) to 'clip' the timestamp out of the original string.
4. Normalization: It handles edge cases like missing dates (e.g., in syslog or strace) by using the file's creation time to fill in missing year/month/day components, and it strips terminal escape sequences.

The PcapFileReader uses pyshark to parse .pcap files. It filters for packets containing 'IP' and converts them into a standardized string format: <timestamp> <protocol> <source> <direction> <destination> <details>.

Supported protocols and details include:

• NFS: Reports procedure, filename, and status (e.g., OK or error codes).
• HTTP: Reports protocol, source/destination, and the request/response content.
• TCP: Reports source/destination, sequence/ack numbers, TCP flags (e.g., SYN, ACK, FIN), and payload snippets.

The jump action allows you to move the cursor by a specific quantity of lines or time. The input follows a specific syntax: [quantity][unit].

Units:

• l: Lines
• us: Microseconds
• ms: Milliseconds
• s: Seconds
• m: Minutes
• h: Hours
• d: Days

Examples:

• 10l: Jump forward 10 lines.
• 5s: Jump forward 5 seconds.
• -1m: Jump backward 1 minute.