Spicy Parser Generator

Spicy is a parser generator used to create robust C++ parsers for network protocols and file formats. It uses an attributed grammar language to describe both the syntax and semantics of an input format in a single unified language.

Key Characteristics:

• Toolchain: Converts Spicy grammars into efficient C++ parsing code.
• Runtime: Supports fully incremental and highly concurrent parsing on input streams of arbitrary size.
• Compilation Modes:
  • Just-in-time (JIT): Compilation occurs at startup via a C++ compiler.
  • Ahead-of-time (AOT): Creates pre-compiled shared libraries or generates C++ code that can be linked into an application.
• Zeek Integration: Allows adding new protocol and file analyzers to Zeek without writing C++ code by defining grammars and specifying Zeek events.

# Example of a Spicy grammar output for an HTTP request
# echo "GET /index.html HTTP/1.0" | spicy-driver http-request.spicy
[$method=b"GET", $uri=b"/index.html", $version=[$number=b"1.0"]]

The switch construct allows branching into different parsing alternatives. There are two primary modes:

1. Branch by expression

Evaluates an expression at the time of the switch. If a value matches, parsing continues with the corresponding field.

switch ( EXPR ) {
    VALUE_1 -> FIELD_1;
    VALUE_2 -> FIELD_2;
    *      -> DEFAULT_FIELD; # Use '*' for a default case
};

Features:

• Multiple values: Use commas to match multiple values: b"A", b"B" -> x: int8;.
• Multiple fields: Enclose fields in braces: VALUE -> { FIELD_1; FIELD_2; }.
• Anonymous branches: Use void if you only need to branch without storing data: b"A" -> : void { DoSomething(); }.
• Field reuse: If alternatives use the same field name and type, the unit will contain only one instance of that member.

2. Branch by look-ahead

If no expression is provided, Spicy attempts to identify the correct branch by looking ahead at the upcoming input.

switch {
    -> a: b"A";
    -> b: b"B";
};

3. Switching with common size

You can apply a size constraint to all branches within a switch using the &size=EXPR attribute on the switch itself. This limits the number of raw bytes available to the switch logic.

module Test;

public type Foo = unit {
    x: bytes &size=1;
    switch ( self.x ) {
        b"A" -> a8: int8;
        b"B" -> a16: int16;
        b"C" -> a32: int32;
    };

    on %done { print self; }
};

In Spicy, data structures are defined using the unit type. A unit is an ordered set of named fields, each with a specific type. Units can be instantiated, and their fields can be assigned or retrieved values.

Key behaviors:

• Unset Fields: Fields are initially unset. Accessing an unset field triggers a runtime error.
• Default Values: Use the &default=VALUE attribute to provide a fallback value. If a field is accessed but has no explicitly assigned value, the default is returned. Note that the field remains technically 'unset' even when a default is used.
• Development Tip: When using spicy-driver, use the --debug (or -d) option. This prevents the optimizer from eliding fields it deems unused, which is helpful during development.

type Foo = unit {
    version: uint32 &default=42;
};

module Test;

global f: Foo;
print f.version; // Prints 42 if unset

You can move the current input position backwards to any location between the start of the current unit and the current position using built-in unit methods.

Note: Using random access requires Spicy to buffer the unit's data, which may incur a performance penalty.

Key Methods

• unit::input(): Returns a stream iterator pointing to the current input position.
• unit::set_input(iterator): Sets the current input position to the location of the specified iterator. The new position must be between the start of the unit and the current position.
• unit::offset(): Returns the numerical offset of the current position relative to the first byte of the unit.
• unit::position(): Returns an iterator to the current input position in the stream.

module Test;

public type Foo = unit {
    on %init() { self.start = self.input(); }

    a: A { self.set_input(self.start); } // Reverts to the start of the unit
    b: B;

    on %done() { print self; }

    var start: iterator<stream>;
};

type A = unit { x: uint32; };
type B = unit { y: bytes &size=4; };

Unit types can be parameterized upon instantiation. This is useful for passing configuration or context down to subunits.

Parameter Passing Rules

• Read-only by default: Like functions, parameters are read-only unless specified otherwise.
• Passing Units (inout): If a parameter is a unit type, you can declare it as inout to allow the subunit to modify the parent unit's state.
• Passing Other Types (References): To allow a subunit to modify non-unit types (like a string), you must pass the parameter as a reference using the & suffix (e.g., s: string&).

Common Pattern: Passing self

A common idiom is passing the parent unit's self into a subunit so the subunit can access or modify the parent's state.

type Bar = unit(foo: Foo) {
    # Access to 'foo' state
}

type Foo = unit {
    b: Bar(self);
}

type Bar = unit(msg: string, mult: int8) {
    x: int8 &convert=($$ * mult);
    on %done { print "%s: %d" % (msg, self.x); }
};

public type Foo = unit {
    y: Bar("My multiplied integer", 5);
};