logly

repository·main·Indexed 18 days ago

https://github.com/muhammad-fiaz/logly

A high-performance, Rust-powered logging library for Python designed for structured logging, telemetry, and observability. It features a Rust-native engine, 10 built-in log levels, and support for multiple sinks including console, file, and network (HTTP, TCP, UDP, Syslog). Logly provides advanced file management with size-based and time-based rotation, various compression codecs (gzip, zip, bz2, xz, zstd), and over 40 integrations with frameworks like FastAPI, Django, and Flask, as well as cloud providers and data stores.

Tokens
150.3K
Snippets
641
Records
737
Agent score
63%

What's inside logly

  1. Overview of Logly features

    main

    Logly is a high-performance, Rust-powered logging library for Python. Key capabilities include:

    • High Performance: Rust-native engine with zero unsafe Rust.
    • Rich Log Levels: 10 built-in levels (TRACE, DEBUG, INFO, NOTICE, SUCCESS, WARNING, ERROR, FAIL, CRITICAL, FATAL) plus support for custom levels.
    • Flexible Sinks: Simultaneous output to Console, file, callable, and network (HTTP, TCP, UDP, Syslog).
    • Advanced File Management: Size-based, time-based, clock-based, and weekday rotation with support for gzip, zip, bz2, xz, and zstd compression.
    • Structured Logging: JSON output support and context binding (attaching persistent key-value pairs).
    • Concurrency & Reliability: Thread-safe logging, non-blocking background workers (enqueue=True), and exception catching via decorators/context managers.
    • Integrations: Over 40 integrations including FastAPI, Django, Flask, Redis, and Kafka.
  2. Compare Logly with other Python logging libraries

    main
    Logly is a high-performance logging library featuring a Rust-native engine (via PyO3). Compared to structlog and the Python standard library (stdlib logging), Logly provides built-in support for 10 logging levels, native file rotation, retention policies, compression, and background workers. It also offers first-class support for structured JSON output, scoped context, and integrations with frameworks like FastAPI, Django, and Flask, as well as observability tools like OpenTelemetry and Prometheus.
  3. Handle Sink errors and their subtypes

    main

    A SinkError is raised when an operation to write logs to a destination (sink) fails. This includes issues like permission denied, network timeouts, or full disks.

    Subtypes of SinkError include:

    • RotationError: Raised when log rotation fails (e.g., invalid rotation policy or permission issues during rename).
    • CompressionError: Raised when log compression fails (e.g., unknown codec or missing compression libraries like zstd).
    from logly.exceptions import SinkError, RotationError, CompressionError
    
    # Handling general SinkError
    try:
        logger.info("Test message")
    except SinkError as e:
        print(f"Sink failed: {e}")
    
    # Handling specific subtypes
    try:
        logger.add("app.log", rotation="invalid_policy")
    except RotationError as e:
        print(f"Rotation failed: {e}")
    
    try:
        logger.add("app.log", compression="unknown_codec")
    except CompressionError as e:
        print(f"Compression failed: {e}")
  4. Compare compression codec performance

    main

    Choose a codec based on your application's performance requirements:

    CodecSpeedRatioCPU UsageBest For
    gzipFastGoodLowGeneral purpose
    zipFastGoodLowArchive compatibility
    bz2SlowHighHighMaximum compression
    xzSlowestHighestHighestLong-term archival
    zstdFastestGoodLowHigh-throughput

    Recommendations:

    • gzip: Recommended default for most use cases.
    • zstd: Best speed-to-ratio tradeoff for high-volume logging.
    • xz: Best for long-term archival where disk space is critical.
    • bz2: Higher compression than gzip at the cost of speed.
    | Codec | Speed | Ratio | CPU Usage | Best For |
    |-------|-------|-------|-----------|----------|
    | gzip | Fast | Good | Low | General purpose |
    | zip | Fast | Good | Low | Archive compatibility |
    | bz2 | Slow | High | High | Maximum compression |
    | xz | Slowest | Highest | Highest | Long-term archival |
    | zstd | Fastest | Good | Low | High-throughput |
  5. Best practices for PostgreSQL logging

    main

    When using PostgresHandler in production:

    • Schema Management: Set create_table=False if you prefer to manage the schema manually or if you need to add custom columns to the log table.
    • Performance: Use connection pooling in production environments to avoid the overhead of opening a new connection for every log message.
  6. Choose between Redis List and Stream modes

    main

    When configuring RedisHandler, choose a mode based on your requirements:

    • List Mode (mode="list"): Uses LPUSH. Best for simple log queuing.
    • Stream Mode (mode="stream"): Uses XADD. Best if you require Redis consumer group support.

    Tip: Use the max_stream_len argument to cap the length of the list or stream to prevent high-volume logs from consuming excessive memory.

    from logly import logger
    from logly.integrations.redis import RedisHandler
    
    # List mode for simple queuing
    handler = RedisHandler(
        "redis://localhost:6379/0",
        key="app:logs",
        mode="list",
        max_stream_len=5000,
    )
    logger.add(handler, level="WARNING")
    
    # Stream mode for consumer groups
    stream_handler = RedisHandler(
        "redis://localhost:6379/0",
        key="app:logs:stream",
        mode="stream",
        max_stream_len=10000,
    )
    logger.add(stream_handler, level="ERROR")
  7. Manage application lifecycle with logger.start() and logger.stop()

    main

    Logly uses logger.start() and logger.stop() to manage background processing and application lifecycle hooks.

    • logger.start(): Initializes background workers and starts processing. It accepts optional lifecycle hooks for compatibility with service startup code.
    • logger.stop(): Flushes all sinks (including queued messages when enqueue=True) and stops background workers. This should be called during application shutdown to ensure no logs are lost.

    These methods are thread-safe. When stop() is called, it drains the queues and waits for all messages to be processed, allowing multiple threads to continue logging safely until the drain is complete.

    from logly import logger
    
    # Configure logger
    logger.add("app.log", level="INFO")
    
    # Start background processing
    logger.start()
    
    # Application runs...
    
    # Flush and stop on shutdown
    logger.stop()
  8. Combine exception handling with context and binding

    main

    Exception handling works seamlessly with Logly's contextual features like bind() and contextualize().

    • With bind(): Use a bound logger (e.g., containing service or endpoint info) inside a logger.catch() block to ensure all logs in that scope carry the same metadata.
    • With contextualize(): Use with logger.contextualize(...) to add request-scoped metadata (like request_id) around a try/except block where logger.exception() is called.
    from logly import logger
    
    # Combined with bind
    api_logger = logger.bind(service="api", endpoint="/users")
    
    with logger.catch(level="ERROR"):
        response = api_client.get("/users")
    
    # Combined with contextualize
    with logger.contextualize(request_id="req-123"):
        try:
            risky_operation()
        except Exception:
            logger.exception("Request failed")
  9. Combine foreground colors with text styles

    main

    You can create compound styles by combining a color with a text style. Logly supports two syntaxes for this:

    1. Underscore-Separated: style_color (e.g., bold_blue, dim_cyan).
    2. Space-Separated: style color (e.g., bold blue, underline yellow).

    Available text styles: dim, bold, italic, underline, blink, reverse, strike.

    from logly import logger
    
    logger.remove()
    logger.add("stderr", level="TRACE", colorize=True)
    
    # Underscore syntax
    logger.level("DEBUG", no=10, color="bold_blue")
    
    # Space syntax
    logger.level("WARNING", no=40, color="underline yellow")
    
    logger.debug("bold blue debug")
    logger.warning("underline yellow warning")
  10. Enable Backtrace and Diagnose modes in Logly

    main

    You can enhance exception output by enabling backtrace and diagnose modes when adding a logger handler.

    • Backtrace Mode (backtrace=True): Includes the function call history (stack trace) in the exception output.
    • Diagnose Mode (diagnose=True): Includes the values of local variables at the time of the exception in the output.

    You can also override these settings for a single log call using .opt().

    from logly import logger
    
    # Enable both modes globally for a handler
    logger.add(
        "app.log",
        backtrace=True,
        diagnose=True,
        format="{time} | {level} | {message}",
    )
    
    # Or override for a specific call
    logger.opt(backtrace=True).error("This has backtrace")