BqLog Documentation

BqLog (BianQue Log) is a lightweight, high-performance, industrial-grade logging system designed to balance the 'impossible triangle' of logging: easy problem traceability (comprehensive logging), high performance (minimal overhead), and low storage consumption (compressed formats). It is widely used in large-scale projects like Honor of Kings and supports a wide range of platforms and languages.

Key Features:

• High Performance: Significantly faster than common open-source libraries like spdlog, glog, and Log4j2, especially in compressed modes.
• Low Memory Footprint: Uses minimal memory (approx. 1-3 MB on mobile, 2-4 MB in high-concurrency benchmarks).
• Compressed & Encrypted Logs: Provides high-compression real-time log formats and optional high-strength asymmetric hybrid encryption with near-zero performance overhead.
• Cross-Language Support: A unified solution for C++, Java, C#, Kotlin, TypeScript/JavaScript, and Python.
• Game Engine Ready: Works seamlessly with Unity and Unreal Engine (including Blueprint support for Unreal).

BqLog is designed to break the 'impossible triangle' of logging: providing high performance (minimal log writing overhead), comprehensive traceability (writing all necessary logs), and efficient storage usage.

In benchmarks comparing the time taken to write 2 million logs (each with 4 formatted parameters, log level, timestamp, and thread info) across multiple threads, BqLog significantly outperforms industry standards like Log4j2:

• C++ BqLog (Compressed): Approximately 9x faster than Log4j2.
• Java BqLog (Compressed): Approximately 7x faster than Log4j2.

Key performance optimization techniques used in BqLog include:

• Cache line isolation to avoid False Sharing.
• Efficient use of Memory Order (optimized via CPU profiling tools like AMD uProf).
• IO operation merging.
• Avoiding runtime formatting.
• Increasing CPU cache hit rates.
• A custom high-concurrency ring buffer that avoids traditional CAS (Compare And Swap) operations.

The BqLog documentation is organized into several specialized guides to help you integrate and use the library:

• Integration Guide (docs/INTEGRATION_GUIDE_CHS.md): Complete integration steps for all platforms and language-specific demos.
• Game Engine Integration (docs/ENGINE_INTEGRATION_CHS.md): Instructions for using Unity, Union Engine, and Unreal Engine plugins (including Blueprint support).
• API Reference (docs/API_REFERENCE_CHS.md): Details on core APIs, synchronous/asynchronous logging, Appenders, builders, and tools.
• Configuration Reference (docs/CONFIGURATION_CHS.md): Full reference for appenders, log, and snapshot settings.
• Advanced Usage (docs/ADVANCED_USAGE_CHS.md): Deep dives into zero Heap Allocation, Categories, crash recovery, custom types, and encryption.
• Benchmarks (docs/BENCHMARK_CHS.md): Benchmark code (C++, Java, Log4j) and performance results.

Because fetch_add can over-increment the in_ pointer beyond the available buffer capacity (since it doesn't check for success like CAS does), BqLog implements a rollback mechanism. If a thread claims memory that exceeds the buffer boundary (from + len > this->out_ + this->size_), it attempts to roll back the in_ pointer using CAS. This ensures that the pointer is only moved back if it hasn't been modified by another thread in a way that would cause data overlap. If space becomes available during the rollback process, the thread stops rolling back.

// Pseudocode of the BqLog rollback allocation logic
void* bq::miso_ring_buffer::alloc(size_t len)
{
    // 1. Check available space
    size_t free_space = this->size_ - (this->in_ - this->out_);
    if (len > free_space) {
        return nullptr;
    }

    // 2. Claim memory using fetch_add
    size_t from = __sync_fetch_add(this->in_, len);

    // 3. Validate if the claimed range is within bounds
    while(from + len > this->out_ + this->size_)
    {
        // 4. Rollback using CAS to prevent data overlap
        size_t expected_in = from + len;
        if(__sync_bool_compare_and_swap(expected_in, this->in_, from))
        {
            return nullptr; // Insufficient space
        }
        yield(); // Wait for consumer to free space
    }
    
    return to_addr(from);
}

When a Data Item's type bit is 0, it is a Log Template. Templates are used to store repetitive information once, which is then referenced by index to save space.

1. Format Template (Template Type = 0)

Stores the invariant parts of a log message. It includes:

• LogLevel: A single byte representing the severity (Verbose, Debug, Info, Warning, Error, Fatal).
• Extra Data: The log category/context (e.g., Shop.Order). Stored as Extra Data Size (VLQ) + Extra Data Content.
• Format String: The actual template string containing placeholders (e.g., New order, order ID:{}, price:{}).

Referencing: Log Entries reference these via a zero-based incrementing index based on their appearance in the file.

2. Thread Info Template (Template Type = 1)

Stores thread-specific metadata to avoid repeating thread names and IDs.

• Thread Info Index: A zero-based incrementing index (resets to 0 upon process restart).
• Thread ID: The unique ID of the thread.
• Thread Name: The string name of the thread.

Referencing: Log Entries reference these via the Thread Info Index.

BqLog supports two logging modes controlled by the log.thread_mode configuration key. Choosing the right mode depends on your performance requirements and whether you can tolerate slight delays in log output.

• Synchronous (sync): The log is processed immediately before the function returns. This provides lower performance as the caller blocks until output is complete, but ensures logs are written immediately.
• Asynchronous (async or independent): Logs are written to an internal high-concurrency ring buffer and processed later by a worker thread. This offers much higher performance as the caller returns immediately.

Thread Safety Rule: BqLog copies all parameters to the internal ring buffer during the call. To ensure thread safety, do not modify a parameter in another thread while the log call is currently in progress. Once the log function returns, the data is safely stored and the worker thread will not access the caller's stack.

// Example of an UNSAFE scenario:
static std::string global_str = "hello";  // modified by other threads concurrently

void thread_a() {
    // This is undefined behavior if global_str is modified by another thread 
    // exactly while this call is executing.
    log_obj.info("param: {}", global_str);
}

BqLog uses a real-time compressed logging algorithm that avoids the performance overhead of post-processing traditional compression (like Gzip). Instead of storing full text strings, it decomposes logs into two types of Data Items: Log Templates and Log Entries.

Core Abstractions

1. Log Templates: These store unchanging parts of the logs to avoid repetition.

   • Format Template: Stores the LogLevel, ExtraData, and the Format String (the template with {} placeholders). Each template is assigned a unique index.
   • Thread Info Template: Stores thread-specific information like Thread ID and Thread Name. These are indexed and reset upon process restarts.

2. Log Entries: These represent the actual log event. Instead of a full string, an entry contains:

   • A Timestamp (stored as a VLQ-encoded offset from the previous log entry).
   • An index referencing a Format Template.
   • An index referencing a Thread Info Template.
   • Parameters: The actual values that fill the {} placeholders in the template (e.g., IDs, names, prices).

Benefits

• Performance: Reduces CPU usage by avoiding heavy string concatenation and formatting during the logging call. Write performance is typically 3-4x faster than text formats.
• Storage: Significantly reduces footprint. In highly repetitive scenarios (server/network logs), file size can be ~10% of the original text size; for game client logs, it is typically ~20%.