repository·master·Indexed 22 days ago
https://github.com/cloudflare/wirefilterA high-performance filter parser and execution engine for Wireshark-like filters. It allows users to define a Scheme of fields and types, parse filter strings into an AST, and execute them against dynamic runtime data. The project provides WebAssembly (WASM) bindings for use in web environments and Node.js, and supports hardware acceleration via AVX2 and SIMD128.
A simple demo of the WebAssembly bindings is available in the wasm directory. You can view it by opening index.html directly from your filesystem or by serving it via a local HTTP server.
# Example using a local HTTP server (e.g., python)
python3 -m http.serverTo run fuzz tests for the bytes component, follow these steps:
Install AFL:
cargo install afl --forceBuild the fuzz target (from the fuzz/bytes directory):
cd fuzz/bytes
cargo afl buildRun the fuzz test (from the fuzz/bytes directory):
cargo afl fuzz -i in -o out ../../target/debug/fuzz-bytesTroubleshooting: If you encounter the error Looks like the target binary is not instrumented!, delete the existing compiled binary and re-run cargo afl build.
cargo install afl --forceTo build the experimental WebAssembly bindings for the filter parser, you must have wasm-pack installed. Running the build command generates a Node.js package in the pkg directory, which can be used directly or published.
Prerequisites:
Build Command: Execute the following in the root directory of the project:
wasm-pack build -t no-modulesWirefilter is an execution engine for Wireshark-like filters. The workflow consists of three main steps:
Scheme: Create a map of possible filter fields and their types (e.g., Bytes, Int) using the Scheme! macro.Scheme to parse a filter string into an Abstract Syntax Tree (AST), then call .compile() on the AST to produce an executable filter.ExecutionContext from the Scheme, populate it with runtime field values using .set_field_value(), and run the filter using .execute(&ctx).This allows you to test filter expressions against dynamic runtime data.
use wirefilter::{ExecutionContext, Scheme};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// 1. Define the schema
let scheme = Scheme! {
http.method: Bytes,
http.ua: Bytes,
port: Int,
}
.build();
// 2. Parse and Compile
let ast = scheme.parse(
r"#""
http.method != "POST" &&
not http.ua matches "(googlebot|facebook)" &&
port in {80 443}
""#,
)?;
let filter = ast.compile();
// 3. Execute against context
let mut ctx = ExecutionContext::new(&scheme);
ctx.set_field_value(scheme.get_field("http.method").unwrap(), "GET")?;
ctx.set_field_value(
scheme.get_field("http.ua").unwrap(),
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0",
)?;
ctx.set_field_value(scheme.get_field("port").unwrap(), 443)?;
println!("Filter matches: {:?}", filter.execute(&ctx)?); // true
Ok(())
}The in operator allows you to check if a value exists within a predefined list (e.g., $even, $odd, or a custom list).
Syntax: field in $list_name
This is commonly used for checking ports, IP addresses, or other discrete values against a set of allowed or disallowed items.
// Check if a port is in the 'even' list
tcp.port in $even
// Check if any port in an array is in the 'even' list
any(tcp.ports[*] in $even)Wirefilter supports various comparison operators for different data types:
ssl (Implicit IsTrue check)Int, Ip, and Bytes)==, !=, <, <=, >, >=Bytes can be compared using hex notation (e.g., 10:20:30...) or string literals (e.g., "example.org").& (Bitwise AND) for integer fields like tcp.port.in { ... }: Checks if a value is within a set of values, ranges, or CIDR blocks.tcp.port in { 80 443 2082..2083 }ip.addr in { 127.0.0.0/8 ::1 10.0.0.0..10.0.255.255 }contains: Checks if a Bytes field contains a specific substring or byte sequence.http.host contains "abc"http.host contains 6F:72:67You can access specific elements within complex fields using index notation:
[index] to access an element by its position.http.cookies[0]["key"] to access a value by its key.http.headers["host"]In Wirefilter, Filter and FilterValue are the primary IR (Intermediate Representation) structures used to execute compiled expressions. They act as a facade for a tree of compiled expressions.
Filter<U>: Represents a compiled expression that evaluates to a single boolean value. It is used for filtering logic (e.g., deciding if a packet or record matches a rule).FilterValue<U>: Represents a compiled expression that evaluates to an LhsValue (a specific data value). It is used when the expression is intended to extract or compute a value rather than just a boolean result.Both types are bound to a specific Scheme. When executing, they verify that the provided ExecutionContext matches the expected Scheme. If the schemes do not match, they return a SchemeMismatchError.
When writing filter expressions for string fields, you can use different quoting and escaping styles. Wirefilter supports:
r (e.g., r"foo") or r# (e.g., r#"foo"#) to handle backslashes and quotes more easily."foo") where backslashes must be escaped (e.g., \\).\xHH (e.g., "\xaa\x22") to represent specific byte values.Case Sensitivity:
"a" matches "A")."a" does not match "A").Note that the ? character is treated as a literal character and is not a special wildcard in string expressions.
// Example of different string formats in expressions
// Raw string with wildcard
http.host r"foo?*\*"
// Quoted string with hex escape
http.host "\xaa\x22"
// Case sensitivity examples
http.host "a" // matches "A" in normal mode
http.host strict "a" // does not match "A"The not operator can be applied to a single logical expression to negate its result. Supported tokens are:
not!When passing a MapEach index to a function, the function is applied to every element of the collection.
If the function call includes additional arguments that are not part of the mapped collection, those arguments are evaluated once and then reused (memoized) for every element in the iteration. This is particularly efficient for expensive operations like nested function calls.
Example: concat(http.cookies[*], lowercase(http.host)) will evaluate lowercase(http.host) once and append the result to every cookie in the array.
// The result of lowercase(http.host) is computed once and applied to all cookies
concat(http.cookies[*], lowercase(http.host))A ComparisonExpr consists of a left-hand side (LHS) index expression and an operator with a right-hand side (RHS) value.
Common patterns include:
field_name), which is treated as IsTrue.field >= 100 or ip_field < 192.168.1.1.field & 0xFF.field contains "abc" or field ~ "regex_pattern".field in {1, 2, 3} or field in $my_list.