fmtlog

repository·main·Indexed 21 days ago

https://github.com/mengrao/fmtlog

A high-performance, asynchronous C++ logging library leveraging the fmtlib formatting engine. Designed for low runtime latency and high throughput in multi-threaded applications, it features per-thread queues, compile-time and runtime filtering, and flexible polling mechanisms. It supports log frequency limiting via FMTLOG_LIMIT, infrequent logging with FMTLOG_ONCE, and customizable log header patterns.

Tokens
3.1K
Snippets
9
Records
12
Agent score
27%

What's inside fmtlog

  1. How asynchronous polling works in fmtlog

    main

    Because fmtlog is asynchronous, log statements only push data into a queue. To ensure logs are actually written to their destination, you have two options:

    1. Manual Polling: Call fmtlog::poll() periodically in your application. This gives you full control over when flushing occurs. Note that only one thread should call fmtlog::poll().
    2. Background Polling Thread: Call fmtlog::startPollingThread(interval) to let fmtlog manage a background thread that polls at the specified interval. Warning: If a background polling thread is running, you cannot call fmtlog::poll() manually.

    You can also force an immediate flush by calling fmtlog::poll(true).

    // Manual polling approach
    logi("Message 1\n");
    fmtlog::poll();
    
    logi("Message 2\n");
    fmtlog::poll();
    
    // Background thread approach
    fmtlog::startPollingThread(1000); // Poll every 1000ms
    // Do NOT call fmtlog::poll() manually while this thread is running
  2. Handle full log queues via blocking or callbacks

    main

    When the per-thread log queue is full, fmtlog's default behavior is to drop additional log messages and return immediately to minimize latency. You can change this behavior using the following methods:

    1. Blocking: To prevent log loss by blocking the front-end logging thread when the queue is full, define the macro FMTLOG_BLOCK=1.
    2. Callbacks: To be notified when the consumer (polling thread) cannot keep up, register a callback function using fmtlog::setLogQFullCB(cb, userData). This allows you to implement custom logic when the queue reaches capacity.
  3. Install fmtlog

    main

    fmtlog requires C++17 and the fmtlib library. You must install fmtlib before installing fmtlog.

    Header-only version

    1. Copy fmtlog.h and fmtlog-inl.h to your project.
    2. Either define the macro FMTLOG_HEADER_ONLY before including fmtlog.h, or include fmtlog-inl.h in one of your source files.

    Static/Shared library version (via CMake)

    Clone the repository, initialize submodules, and run the build script:

    $ git clone https://github.com/MengRao/fmtlog.git
    $ cd fmtlog
    $ git submodule init
    $ git submodule update
    $ ./build.sh

    After building, copy fmtlog.h and the generated libfmtlog-static.a or libfmtlog-shared.so from the .build directory to your project.

  4. Optimize logging latency with fmt::preallocate()

    main

    fmtlog uses a single producer single consumer queue for each logging thread to avoid contention. These queues are automatically created upon the first log message from a thread, which can cause a slight latency spike for that first call. To ensure even the first log message has low latency, call fmt::preallocate() immediately after a thread is created.

    By default, the thread queue size is 1 MB. You can change this by defining the macro FMTLOG_QUEUE_SIZE.

    // Recommended pattern for new threads
    void my_thread_func() {
        fmt::preallocate(); 
        // ... rest of thread logic
    }
  5. Basic Usage and Logging Macros

    main

    You can log messages using the FMTLOG macro or convenient shortcut macros.

    Shortcut Macros:

    • logd: Debug level (DBG)
    • logi: Info level (INF)
    • logw: Warning level (WRN)
    • loge: Error level (ERR)

    Note on Asynchronicity: fmtlog is asynchronous. Log messages are pushed into a queue and are not written to the file/console immediately. You must call fmtlog::poll() to process the queue and write the logs. You can also use fmtlog::startPollingThread(interval) to create a background thread that handles polling automatically, but if you do this, you must not call fmtlog::poll() manually.

    #include "fmtlog/fmtlog.h"
    
    int main() 
    {
      // Using the main macro
      FMTLOG(fmtlog::INF, "The answer is {}.\n", 42);
    
      // Using shortcut macros
      logi("A info msg\n");
      logd("This msg will not be logged as the default log level is INF\n");
      
      fmtlog::setLogLevel(fmtlog::DBG);
      logd("Now debug msg is shown\n");
    
      // IMPORTANT: Must poll to see output
      fmtlog::poll();
      return 0;
    }
  6. Configure log file output and flushing

    main

    By default, fmtlog outputs to stdout. To write to a file, use fmtlog::setLogFile(filename, truncate).

    Flushing Behavior: fmtlog buffers data for performance. The buffer is flushed to the file when:

    1. The buffer size exceeds 8 KB (configurable via fmtlog::setFlushBufSize(bytes)).
    2. The oldest data in the buffer exceeds a specific duration (default 3s, configurable via fmtlog::setFlushDelay(ns)).
    3. A log message reaches a specific level (configurable via fmtlog::flushOn(logLevel)).
    4. The user calls fmtlog::poll(true).

    Advanced File Management:

    • You can pass an existing FILE* to fmtlog::setLogFile(fp, manageFp). If manageFp is false, fmtlog will not buffer (e.g., fmtlog::setLogFile(stderr, false) for unbuffered stderr logging).
    • Use fmtlog::closeLogFile() to stop logging to the file.
    // Set log file with truncation
    fmtlog::setLogFile("app.log", true);
    
    // Set custom flush buffer size (e.g., 16KB)
    fmtlog::setFlushBufSize(16 * 1024);
    
    // Set flush delay to 1 second
    fmtlog::setFlushDelay(1000000000);
    
    // Flush immediately on ERROR logs
    fmtlog::flushOn(fmtlog::ERR);
  7. Filter logs at compile time or runtime

    main

    You can control which logs are included in your binary using the following macros:

    Compile-time Filtering

    Use FMTLOG_ACTIVE_LEVEL to discard logs below a certain level at compile time. The default value is FMTLOG_LEVEL_INF (meaning debug logs are discarded). Note: This only applies to the shortcut macros (e.g., logi), not the generic FMTLOG macro.

    Runtime Filtering

    By default, fmtlog checks log levels at runtime. You can disable this check to increase performance and reduce generated code size by defining the macro FMTLOG_NO_CHECK_LEVEL.

  8. Register a custom log callback

    main

    You can register a callback function to handle log messages in real-time (e.g., for alerting) using fmtlog::setLogCB(cb, minCBLogLevel). Unlike the file output, callbacks are not buffered and can be triggered even if the log file is closed.

    Callback Signature:

    typedef void (*LogCBFn)(int64_t ns, LogLevel level, fmt::string_view location, size_t basePos, 
                              fmt::string_view threadName, fmt::string_view msg, size_t bodyPos, size_t logFilePos);

    Parameters:

    • ns: Nanosecond timestamp.
    • level: The log level.
    • location: Full file path with line number.
    • basePos: File base index in the location.
    • threadName: Thread ID or name set via setThreadName().
    • msg: Full log message including the header.
    • bodyPos: Index where the log body starts in msg.
    • logFilePos: Log file position of this message.
    void my_callback(int64_t ns, fmtlog::LogLevel level, fmt::string_view location, size_t basePos,
                     fmt::string_view threadName, fmt::string_view msg, size_t bodyPos, size_t logFilePos) {
        if (level >= fmtlog::WRN) {
            // Handle warning or error (e.g., send to an alerting system)
        }
    }
    
    // Register the callback
    fmtlog::setLogCB(my_callback, fmtlog::WRN);
  9. Use FMTLOG_ONCE for infrequent or latency-insensitive logs

    main

    fmtlog's primary optimization stores static log information (format string, level, location) in a table and pushes only an index and dynamic arguments to the queue. While extremely fast, this adds ~50 bytes of program size (via decoding functions) and ~50 bytes of runtime memory per log statement.

    For infrequent logs where this overhead is undesirable (e.g., program initialization), use the FMTLOG_ONCE macros. These push the static info and the formatted message body directly to the queue without creating table entries or decoding functions.

    Note: Passing arguments by pointer is not supported when using FMTLOG_ONCE.

    Available macros:

    • FMTLOG_ONCE
    • logdo (debug once)
    • logio (info once)
    • logwo (warning once)
    • logeo (error once)
  10. Limit log frequency with FMTLOG_LIMIT macros

    main

    To prevent high-frequency log statements (e.g., error spamming) from overwhelming the system, fmtlog provides macros that limit how often a log is recorded. You must pass the minimum interval in nanoseconds as the first argument.

    Available macros:

    • FMTLOG_LIMIT
    • logdl (debug limit)
    • logil (info limit)
    • logwl (warning limit)
    • logel (error limit)

    Example usage (limiting to once per second):

    logil(1e9, "this log will be displayed at most once per second");
  11. Format log messages using fmtlib syntax

    main

    fmtlog uses the fmt library for formatting. It supports most fmtlib features, including named arguments, ranges, tuples, and user-defined types (though color support is excluded).

    Pointer Support for Performance: To avoid copy overhead in an asynchronous environment, you can pass pointers (including std::shared_ptr and std::unique_ptr) to arguments. This is useful if the lifetime of the object is guaranteed to last until the next poll(). For example, passing a std::string* will only copy the pointer, whereas passing a std::string copies the entire content.

    #include "fmtlog/fmtlog.h"
    #include "fmt/ranges.h"
    
    void example() {
      // Standard fmtlib formatting
      logi("Hello, {}!\n", "World");
      
      // Using ranges
      std::vector<int> v = {1, 2, 3};
      logi("vector: {}\n", v);
    
      // Passing pointers to avoid copies (ensure lifetime until poll())
      std::string str = "important data";
      logi("Pointer log: {}\n", &str);
      fmtlog::poll();
    }
  12. Configure the log header pattern

    main

    You can customize the prefix of every log line using fmtlog::setHeaderPattern(format_string). The pattern uses fmtlib named arguments.

    Default Pattern: {HMSf} {s:<16} {l}[{t:<6}]

    Available Named Arguments:

    NameMeaningExample
    lLog levelINF
    sFile base name and line numlog_test.cc:48
    gFile path and line num/path/to/log_test.cc:48
    tThread id (or name set via setThreadName)main
    aWeekdayMon
    bMonth nameMay
    YYear2021
    CShort year21
    mMonth05
    dDay03
    HHour16
    MMinute08
    SSecond09
    eMillisecond796
    fMicrosecond796341
    FNanosecond796341126
    YmdYear-Month-Day2021-05-03
    HMSHour:Minute:Second16:08:09
    HMSfHour:Minute:Second.Microsecond16:08:09.796341
    YmdHMSYear-Month-Day Hour:Minute:Second2021-05-03 16:08:09

    Tip: Using concatenated arguments like {YmdHMS} is more efficient than {Y}-{m}-{d} {H}:{M}:{S}.

    // Example: Customizing header to show date and time
    fmtlog::setHeaderPattern("{Ymd HMSf} [{l}] {s}\n");