DOCSEARCHbyAlgolia
Submit Docs

YARA Performance Guidelines

repository·master·Indexed 41 days ago

https://github.com/neo23x0/yara-performance-guidelines

A collection of best practices and technical guidelines for writing high-performance YARA rules. Covers optimizing string selection, Aho-Corasick atom generation, condition short-circuiting, and efficient regular expression usage to avoid 'too many matches' errors and scanning degradation.

Tokens
2.3K
Snippets
3
Records
10
Agent score
2%

What's inside YARA Performance Guidelines

  1. Understand YARA's scanning process

    master

    YARA scanning is a four-step process. Understanding this helps you optimize rules by targeting the right stage:

    1. Compiling the rules: YARA extracts atoms (up to 4-byte substrings) from your defined strings.
    2. Aho-Corasick search: YARA scans the file for these atoms using a prefix tree. This is the first line of defense.
    3. Bytecode engine: If atoms match, the engine verifies the full string (e.g., checking if a regex matches the surrounding context).
    4. Condition evaluation: YARA checks the condition block. It uses short-circuiting to skip expensive checks if earlier conditions fail.

    Key takeaway: You cannot fix a poorly chosen string with a well-formed condition. Efficiency starts with the strings.

  2. Manage RAM usage with Metadata

    master
    All data defined in the metadata section of a YARA rule is loaded into RAM during a scan. If you are operating in a memory-constrained environment, large amounts of metadata (such as thousands of hashes) can significantly increase the memory footprint of the YARA process. Consider stripping unnecessary metadata from rules immediately prior to scanning if RAM is limited.

  3. Optimize Conditions using Short-Circuit Evaluation

    master

    YARA evaluates conditions from left to right. To improve performance, place the cheapest (fastest) conditions first and the most likely to be False conditions first. This allows the engine to 'short-circuit' and skip expensive evaluations if a preceding condition fails.

    Key Rules:

    • Order Matters for Mixed Costs: If one statement is significantly faster than another, place the fast one first. Reordering statements with similar costs provides no benefit.
    • Protect 'for' Loops: Use short-circuiting to prevent expensive for loops from running on files that don't meet basic criteria (e.g., check filesize or a magic number before the loop).
    • Regex Exception: Regular expressions do not support short-circuiting in the condition block. They are fed into the string matching engine immediately, meaning an expensive regex will slow down the scan regardless of its position in the condition section.
    // SLOW: Expensive check first
math.entropy(0, filesize) > 7.0 and uint16(0) == 0x5A4D

// FAST: Cheap check first
uint16(0) == 0x5A4D and math.entropy(0, filesize) > 7.0

// OPTIMIZED FOR LOOP
$mz at 0 and filesize < 100KB and for all i in (1..filesize) : ( whatever )

// REGEX WARNING: This will be slow regardless of order
strings:
  $expensive_regex = /\$[a-z0-9_]+\(/ nocase
conditions:
  filesize < 200 and $expensive_regex

  4. Use modules with caution

    master

    Modules like pe, elf, or magic can significantly increase scan time because they often require parsing the entire file before the condition can be evaluated.

    Alternatives

    • Manual Header Checks: Instead of using pe.is_pe, use direct byte checks like uint16(0) == 0x5A4D to identify PE files.
    • Avoid magic module: The magic module is not available on Windows and is generally slower than manual header checks.

    Example: Manual GIF header check vs Magic module

    // Faster: Manual check
rule gif_1 {
  condition:
    (uint32be(0) == 0x47494638 and uint16be(4) == 0x3961) or
    (uint32be(0) == 0x47494638 and uint16be(4) == 0x3761)
}

// Slower: Using magic module
import "magic"
rule gif_2 {
  condition:
    magic.mime_type() == "image/gif"
}

  5. Optimize conditions and short-circuiting

    master

    YARA evaluates conditions sequentially and stops at the first failure (short-circuiting).

    Best Practices

    • Order by cost: Put quick, inexpensive checks (e.g., filesize < 20KB) at the beginning of your condition before expensive checks (e.g., math.entropy).
    • Avoid large loops: Do not use loops that iterate over large data, such as for all i in (1..filesize). This is highly inefficient.
    • Avoid loops on common strings: If a string is very common, #string_name will be high, making loops like for all i in (1..#a) very slow.
    • Use direct offsets: Use the @ operator for sequence checks instead of regex where possible.

    Note: Regex conditions do not short-circuit and are always evaluated last.

  6. Optimize Regular Expressions for YARA

    master

    Regular expressions are slower and more memory-intensive than plain string matching. To maintain performance:

    1. Prefer Hex Strings: Use hex strings with jumps and wildcards instead of regex whenever possible.
    2. Avoid Greedy Quantifiers: Do not use .*, .+, or reluctant quantifiers .*?.
    3. Use Exact Bounds: Always specify an upper bound for quantifiers (e.g., use .{1,30} instead of .{2,}).
    4. Use Anchors and Offsets: Instead of using regex to find the relationship between two strings, use the @ symbol to check offsets.
    5. Include Long Anchors: Include long, specific sequences of strings to act as anchors to speed up the matching process.
    6. Avoid Multiple Matches: When matching patterns like email addresses, avoid using quantifiers on the character set (e.g., [set]*) as this can cause YARA to match a single instance multiple times, leading to errors.
    // SLOW: Regex approach
$ = /exec.*\/bin\/sh/

// FAST: Offset approach
strings:
  $exec = "exec" 
  $sh   = "/bin/sh"
conditions:
  $exec and $sh and @exec < @sh

  7. Optimize string selection for performance

    master

    String selection is the most critical factor for YARA performance because YARA searches for strings before evaluating conditions.

    Best Practices

    • Avoid short strings: Strings < 4 bytes create excessive false positives and slow down scanning.
    • Use unique 4-byte atoms: YARA relies on these for fast Aho-Corasick scanning. Avoid strings that don't contain any fixed substrings (e.g., /[0-9]+\n/ or /\w.*\d/), as they force YARA to evaluate every offset.
    • Minimize wildcards: In hex strings, keep at least one long concrete segment. Avoid excessive wildcards that prevent the creation of good atoms.
    • Use regex sparingly: If you must use regex, include a fixed 4-byte anchor to improve efficiency.
    • Avoid uniform/repetitive patterns: Strings like \x00\x00\x00\x00 or long sequences of \x00\x20 appear too frequently and cause performance degradation.
    • Use nocase carefully: This attribute generates exponentially more search variations (atoms), increasing memory and CPU usage. If you only need specific casing, use a regex like /[Pp]assword/ instead.

  8. Troubleshoot 'too many matches' errors

    master

    If you encounter errors stating a string caused too many matches, it means the string is too common or lacks unique atoms.

    Fixes

    • Check regex quantifiers: Avoid unanchored or unbounded quantifiers like .*, .+, or {x,}.
    • Reduce wildcards: Tighten hex strings to ensure they contain more concrete bytes.
    • Split alternations: Instead of using a single string with many alternations (e.g., /(a|b|c)de/), write them as separate strings to provide better atoms for the Aho-Corasick engine.
    • Avoid common patterns: Ensure your strings don't match common null-byte sequences or wide-formatted spaces.

  9. Troubleshoot 'Too Many Matches' and 'Slowing Down Scanning' errors

    master

    These errors are typically caused by overly general strings or inefficient matching patterns.

    Causes:

    • Too Many Matches: Caused by strings that appear too frequently in the input or patterns that match a single instance multiple times.
    • Slowing Down Scanning: Caused by strings that generate very short 'atoms' (or none at all), forcing YARA to use a slower, naïve pattern matching algorithm.

    Resolution Checklist:

    1. Replace .*, .+, or .*? with bounded quantifiers.
    2. Ensure quantifiers have upper bounds (avoid x{14,}).
    3. Avoid excessively large ranges (e.g., x{1,300000}).
    4. Reduce big jumps in hexadecimal strings.
    5. Specify wildcards more precisely or split strings to omit wildcards.
    6. Split large alternations into multiple individual strings.
    7. Use word matching specifications like fullword or \b.

  10. Reference: String modifier combinations

    master

    YARA string modifiers affect how many atoms are generated.

    • LOW impact: Only one atom is generated.
    • HIGH impact: nocase generates all combinations of upper and lowercase for the chosen 4 bytes, significantly increasing search variations.
    $s1 = "cmd.exe"             // (ascii only)
$s2 = "cmd.exe" ascii       // (ascii only, same as $s1)
$s3 = "cmd.exe" wide        // (UTF-16 only)
$s4 = "cmd.exe" ascii wide  // (both ascii and UTF-16) two atoms will be generated
$s5 = { 63 6d 64 2e 65 78 65 } // ascii char code in hex
$s6 = "cmd.exe" nocase      // (all different cases, e.g. "Cmd.", "cMd.", "cmD.")