Cordon: Semantic Anomaly Detection for System Logs

The Cordon benchmark on the HDFS v1 dataset (from Loghub) evaluates the system's ability to detect anomalous log patterns, specifically focusing on template recall and rare pattern detection.

Key Performance Drivers:

• Sample Size: This is the most critical factor. Template recall improves significantly as sample size increases (e.g., from 58% at 50k lines to 97% at 5M lines). Stability (measured by Coefficient of Variation) also improves as sample size grows.
• Model Size: Embedding model size (e.g., BGE-large vs. MiniLM) has no significant impact on performance for HDFS logs.
• Parameter Tuning: Adjusting window size (w) and k-neighbors (k) provides only marginal improvements (~2%) and does not solve instability issues caused by small sample sizes.
• Thresholds: Varying the anomaly percentile threshold (p) affects traditional precision/recall but has minimal impact on template-level metrics.

Recommendation: Cordon is most effective and stable when used with large log files (≥500k lines) where comprehensive sampling is possible.

Cordon supports three primary backends for embedding generation:

1. sentence-transformers (Default)

Best for native installations with local GPU access.

• Use --device cuda or --device cpu to specify the hardware.

2. llama.cpp Backend

Best for container deployments using GPU acceleration via Vulkan.

• Use --backend llama-cpp.
• Specify layers with --n-gpu-layers <int>.
• Specify a custom model with --model-path <path>.

3. Remote API Backend

Uses LiteLLM to connect to providers like OpenAI, Gemini, or Cohere.

• Use --backend remote.
• Specify the model with --model-name <provider>/<model>.
• Provide an API key via --api-key <key> or environment variables (e.g., OPENAI_API_KEY, GEMINI_API_KEY, COHERE_API_KEY).
• Specify a custom endpoint with --endpoint <url> for OpenAI-compatible APIs.

cordon --backend remote --model-name openai/text-embedding-3-small --api-key $OPENAI_API_KEY system.log

Cordon prioritizes the diversity of anomaly types over the total count of anomalous lines. Therefore, users should evaluate performance using Template-Level Metrics rather than traditional line-level metrics.

Primary Metrics

• Template Recall: The fraction of unique anomaly template types detected. High recall indicates comprehensive coverage of different anomaly patterns.
• Rare Template Recall: The fraction of templates with < 100 occurrences that are detected. This measures the ability to find infrequent but critical patterns.
• Frequency-Weighted Recall: An inverse-frequency weighted score where rare patterns are weighted higher. A higher score indicates the system is successfully biasing toward unique pattern detection.

Reference Metrics

• Line-Level F1, Precision, Recall: Standard metrics that count every instance. These are expected to be low (e.g., 0.05-0.08) because Cordon ignores repetitive patterns to focus on type diversity.

When running with --generate-plots, two main visualizations are produced:

1. UMAP Projection (umap_*.png)

Shows a 2D projection of the high-dimensional embedding space.

• Left Panel (k-NN score): Uses a plasma scale (dark purple = low score $\rightarrow$ orange/yellow = high score). Isolated orange/yellow points indicate detected anomalies.
• Right Panel (Ground Truth): Colored by dataset labels (Red = Anomaly sessions, Green = Normal sessions).
• Interpretation: High scores (orange/yellow) on the left that align with red points on the right indicate correct detection. Dense dark purple regions indicate normal, repetitive patterns.

2. Template Coverage Chart (template_coverage_*.png)

A horizontal bar chart showing detection success rates for each unique anomaly template.

• Y-axis: Template IDs (sorted by frequency).
• X-axis: Occurrence count (log scale).
• Green Bars: Successfully detected templates.
• Red Bars: Missed templates.
• Goal: Look for green bars in the shorter (rarer) bars to verify the system's ability to detect rare patterns.

Cordon uses a template-level evaluation methodology instead of traditional line-level metrics (Precision, Recall, F1). Traditional metrics reward systems for flagging repetitive, common errors. Cordon's methodology prioritizes semantic uniqueness, measuring the diversity of anomaly types (templates) detected rather than the quantity of individual log lines flagged.

Key Metrics

• Template Recall: The fraction of unique anomaly templates detected. Template Recall = |Detected Templates ∩ Anomaly Templates| / |Anomaly Templates|
• Rare Template Recall: The fraction of rare templates (defined as templates with < 100 occurrences) detected.
• Frequency-Weighted Recall: Uses inverse-frequency weighting to give higher value to finding rare templates. Weight(template) = 1 / occurrence_count FW Recall = Σ weights(detected) / Σ weights(all anomalies)

Line-Level Template Attribution

To ensure credit is only given for templates the system actually identifies, the evaluation follows these steps for each flagged log line:

1. Parse the line.
2. Match it to a template using regex patterns.
3. Credit the template only if it appears in the flagged line itself (not just within the same session).

When interpreting Cordon's performance or applying it to new datasets, consider these technical limitations:

Data & Model Constraints

• Template Dependency: Evaluation requires pre-parsed templates. If your dataset lacks template ground truth, you must perform template extraction (e.g., using Drain) first.
• Embedding Dependency: Results are highly dependent on the quality of the semantic embeddings used.
• Anomaly Diversity: The current benchmark is based on HDFS v1, which has limited anomaly types. Performance may vary on datasets with hundreds of anomaly types.

Algorithmic Constraints

• No Temporal Metrics: Cordon does not measure detection latency, temporal ordering, or streaming performance.
• No Severity Weighting: All anomalies are treated equally; the tool does not account for business impact or severity.
• k-NN Density Assumptions: The approach assumes semantic isolation indicates an anomaly. This may fail for normal but rare operations (like startup sequences) or clustered anomalies.
• Percentile Thresholding: Using anomaly_percentile is adaptive but might flag near-identical content as anomalous in extremely uniform logs.

Cordon uses density scoring in semantic embedding space to reduce log volume while preserving signal. The process follows these steps:

1. Chunking: Logs are divided into non-overlapping windows (e.g., 10 lines per window).
2. Embedding: Each window is converted into a 384-dimensional vector using a transformer model.
3. k-NN Distance Calculation: For every embedding, the algorithm finds its $k$ nearest neighbors.
4. Scoring: A score is calculated as the average distance to these $k$ nearest neighbors.
5. Anomaly Identification: Higher scores indicate an embedding is farther from its neighbors, marking it as more anomalous.
6. Filtering: The system keeps the top $X$% highest-scoring windows.
7. Merging: Adjacent windows are merged into contiguous blocks to maintain context.

1. Chunk logs into non-overlapping windows (e.g., 10 lines per window)
2. Embed each window using a transformer model → 384-dimensional vectors
3. For each embedding, compute k-NN distance to find "neighbors"
4. Score = average distance to k nearest neighbors
5. Higher score = farther from neighbors = more anomalous
6. Keep top X% highest-scoring windows
7. Merge adjacent windows into contiguous blocks