DOCSEARCHbyAlgolia
Submit Docs

sniffly

repository·main·Indexed 22 days ago

https://github.com/chiphuyen/sniffly

An analytics dashboard for analyzing Claude Code logs. Sniffly helps developers understand usage patterns, identify error types, and review message histories. It includes features for sharing project statistics and instruction histories, as well as detailed documentation on Claude log structures, entry types, and processing strategies.

Tokens
15K
Snippets
37
Records
83
Agent score
77%

What's inside sniffly

  1. Understand the Sniffly repository structure

    main

    The Sniffly repository is organized into several key areas:

    • sniffly/: The core Python package containing the CLI, FastAPI server, log processing logic (core/), API endpoints (api/), and utility modules (utils/).
    • sniffly-site/: A static site used for sharing dashboards, including Cloudflare Pages Functions and assets for rendering shared views.
    • docs/: Comprehensive documentation covering CLI references, log structures, pricing models, and technical specifications.
    • tests/: The test suite covering CLI, deduplication, performance, and core modules.
    • scripts/ & Root Analysis Scripts: Utility scripts for starting local servers and performing specific data analysis (e.g., analyze_response_times.py).
    • assets/: Branding and logos for Sniffly and Lemongrass.

  2. How Sniffly's two-tier caching architecture works

    main

    Sniffly uses a two-tier caching system to balance speed and persistence, similar to CPU cache levels. This reduces dashboard load times for large projects by avoiding expensive log reprocessing.

    1. L1 Cache (Memory Cache)

    • Location: sniffly/utils/memory_cache.py
    • Storage: Python OrderedDict in RAM.
    • Speed: <1ms retrieval.
    • Capacity: Defaults to 5 projects, with a 500MB limit per project.
    • Persistence: Lost on server restart.
    • Eviction: Uses LRU (Least Recently Used) with a 5-minute protection window. Projects accessed within the last 5 minutes cannot be evicted by background processes.

    2. L2 Cache (File Cache)

    • Location: sniffly/utils/local_cache.py
    • Storage: JSON files in ~/.sniffly/cache/.
    • Speed: ~200-300ms (disk read + JSON parse).
    • Persistence: Survives server restarts.
    • Validation: Uses timestamp and checksum checking to detect changes.

    Data Flow (Read Path)

    1. Check L1: If hit, return instantly (<1ms).
    2. Check L2: If hit, promote data to L1 and return (~200ms).
    3. Process Source: If both miss, parse raw logs, save to L2 and L1, then return (~1000ms+).

    Data Flow (Write Path)

    When data is processed, it is stored in both L1 and L2 caches simultaneously to ensure immediate reuse and long-term persistence.

    # Simplified cache lookup
def get_project_data(project_path):
    # L1 - Memory Cache
    if data := memory_cache.get(project_path):
        return data  # <1ms
    
    # L2 - File Cache
    if data := local_cache.get(project_path):
        memory_cache.put(project_path, data)  # Promote to L1
        return data  # ~200ms
    
    # Process from source
    data = process_logs(project_path)  # ~1000ms
    memory_cache.put(project_path, data)
    local_cache.save(project_path, data)
    return data

  3. Configure Sniffly via Environment Variables

    main

    You can override any configuration key using environment variables. The mapping follows the pattern of converting the key to uppercase with underscores.

    Priority Order:

    1. Command-line arguments
    2. Environment variables
    3. Config file (~/.sniffly/config.json)
    4. Built-in defaults
    # Example: Setting variables for the current session
export PORT=9000
export AUTO_BROWSER=false
export CACHE_MAX_PROJECTS=10

# Example: Setting variables inline for a single command
PORT=9000 sniffly init

  4. Understand the Sniffly Configuration Priority System

    main

    Sniffly uses a layered priority system for configuration managed by the Config class in sniffly/config.py. When retrieving a configuration value, the system checks sources in the following order (highest priority first):

    1. Environment Variables: Includes variables loaded from .env or .env.sniffly.dev files.
    2. Config File: Values stored in ~/.sniffly/config.json.
    3. Default Values: Hardcoded values defined in the DEFAULTS dictionary within the code.

    Example: If you set PORT=9000 in your .env file, config.get("port") will return 9000, even if the default is 8081 or if ~/.sniffly/config.json specifies a different port.

  5. How Sniffly handles cache misses and large projects

    main

    Large Projects (Exceeding CACHE_MAX_MB_PER_PROJECT)

    If a project's processed size exceeds the configured CACHE_MAX_MB_PER_PROJECT limit:

    1. The project is not stored in the L1 (Memory) cache.
    2. The server logs a size rejection: [Cache] Skipping {project} - too large ({size}MB > {limit}MB limit).
    3. Impact: The project remains fully functional, but you lose the <1ms memory cache speed boost. Every load will require a ~1s disk-based processing step (L2 cache hit or full reprocess).

    Smart Refresh Behavior

    When clicking the refresh button, Sniffly performs intelligent change detection:

    • No changes detected: Only the memory cache is invalidated. This is extremely fast (~5ms).
    • Changes detected: A full reprocessing of the logs is triggered, updating both L1 and L2 caches (~1600ms).

  6. Handle Task tool logging limitations

    main

    When analyzing logs, be aware that the Task tool (which launches sub-agents) has significant logging limitations:

    • Only the initial Task invocation and the final result are logged.
    • Internal tool operations performed by sub-agents are not visible in the logs.
    • Token usage by sub-agents is not tracked.

    This results in apparent 'missing' tool counts in analytics compared to the actual work performed by the sub-agents.

  7. Sniffly Performance and Caching

    main

    Sniffly is optimized for large log files using a two-tier caching system and server-side aggregation.

    Caching Tiers

    • L1 (Memory Cache): Fast in-memory storage with LRU eviction. Provides ~74,000x speedup for retrieval.
    • L2 (File Cache): Persistent JSON storage on disk. Uses file metadata (size + mtime) for change detection, providing ~2.4x speedup over reprocessing.

    Performance Characteristics

    • Processing Speed: ~27,000 messages/second.
    • Memory Usage: ~36KB per message.
    • Cache Hit Latency: <5ms.
    • Full Refresh: ~1.6s for a 124MB project.

  8. How Sniffly's architecture works

    main

    Sniffly is composed of three distinct components that separate local analytics from public sharing:

    1. Local Analytics Tool (sniffly/): A Python package that runs on the user's machine. It analyzes Claude Desktop logs locally via a FastAPI server (server.py) with multi-layered caching. Data processing is entirely local and private.
    2. Static Sharing Site (sniffly-site/): A Cloudflare Pages website (hosted at sniffly.dev) used to view shared dashboards. It includes a public gallery (index.html), a share viewer (share.html), and an admin dashboard (admin.html).
    3. Cloud Storage & Functions: The backend for the sharing feature, utilizing Cloudflare R2 to store shared dashboard data as JSON and Pages Functions for dynamic routing (e.g., /share/[id]).

    Key Relationship: The analytics tool and sharing site are separate systems. Shared dashboards are static snapshots of data, not live streams. Users can use the local tool without ever enabling the sharing feature.

  9. How the Sniffly Site architecture works

    main

    The Sniffly Site is a static site architecture designed to run on Cloudflare Pages, utilizing R2 for storage and Cloudflare Functions for dynamic routing.

    1. Build Process

    When building, build.py bundles assets from the main sniffly package (CSS from sniffly/static/css/dashboard.css and JS modules from sniffly/static/js/) into a single, self-contained share.html. This allows shared dashboards to function independently of the main Sniffly server.

    2. Share Viewing Lifecycle

    When a user accesses a URL like sniffly.dev/share/abc123:

    1. The Cloudflare Pages Function (functions/share/[[id]].js) intercepts the request.
    2. It fetches the corresponding JSON data from R2 storage.
    3. It injects that data into the pre-built share.html template.
    4. It returns the complete, interactive dashboard to the user.

    3. Public Gallery

    The homepage (index.html) acts as a discovery layer. It fetches data from /gallery-index.json (served from R2) to display featured projects, all public shared dashboards, and project statistics (commands, tokens, duration, cost, etc.).

    4. Admin Dashboard

    The admin.html interface provides management capabilities for authorized users (authenticated via Google OAuth). Admins can:

    • View all shared projects.
    • Feature or unfeature projects.
    • Remove inappropriate content.
    • View share statistics.

  10. Explore the core Sniffly Python package

    main

    The sniffly/ directory contains the main application logic:

    • Entry Points: cli.py for command-line interaction and server.py for the FastAPI backend.
    • core/: Contains the engine for log processing (processor.py), statistics generation (stats.py), and cross-project aggregation (global_aggregator.py).
    • api/: Handles data loading (data_loader.py), message endpoints (messages.py), and formatting (data.py).
    • utils/: Provides essential helpers including log_finder.py (detects Claude logs), memory_cache.py (L1 LRU cache), and local_cache.py (L2 file-based cache).
    • services/: Contains business logic like pricing_service.py for dynamic token pricing.
    • templates/ & static/: HTML templates and frontend assets (CSS/JS) for the dashboard UI.

  11. Handle streaming responses and conversation compaction

    main

    Claude logs certain complex states using specific patterns:

    Streaming Responses

    Claude logs streaming responses as multiple entries sharing the same message.id. For example, one entry might contain the text response, while a subsequent entry with the same ID contains the tool_use instruction.

    Conversation Compaction

    When context limits are approached, Claude creates summaries. These appear as user messages with the isCompactSummary: true flag:

    {
  "type": "user",
  "isCompactSummary": true,
  "message": {
    "role": "user",
    "content": [{
      "type": "text",
      "text": "This session is being continued from a previous conversation..."
    }]
  }
}

  12. Understand Sniffly's interaction-based processing strategy

    main

    To solve issues like duplicate messages (caused by crashes/restarts) and inconsistent tool counts, Sniffly uses Interaction-Based Processing instead of processing individual messages in isolation.

    The Interaction Model

    Sniffly groups messages into complete Interaction objects. An interaction typically consists of:

    • A user_message
    • A list of assistant_messages
    • A list of tools_used
    • Metadata like session_id and a unique interaction_id (a hash of user content + timestamp).

    Deduplication and Merging

    1. Session Continuation Detection: Maps sessions to predecessors by looking for isCompactSummary or 'continue' commands.
    2. Interaction Grouping: Groups user messages with their subsequent assistant messages, even if they are split across different log files.
    3. Intelligent Merging: When duplicates are found, Sniffly scores them and prefers the most 'complete' interaction (e.g., the one with more tools, more output tokens, or a later session timestamp).
    4. Tool Count Reconciliation: Reconciles counts by verifying assistant messages against tool results and handling the Task tool as a single unit.