Wingfoil Stream Processing Engine

Wingfoil is a high-performance, scalable stream processing framework optimized for latency-critical applications like electronic trading and real-time AI. It uses a Directed Acyclic Graph (DAG) model to execute transformations over data streams.

Key characteristics:

• Engine: The core execution is handled by a high-performance Rust engine.
• Python Interface: The wingfoil package provides Python bindings to the Rust engine, exposing the graph model, combinator surface, I/O adapters, and latency-tracing tools.
• Extensibility: You can author custom operations (ops), sub-graphs, and adapters in Rust and then compose them within your Python workflows.
• Execution Modes: Supports both live data processing and deterministic historical replay.

import wingfoil as wf

g = wf.Graph()
out = g.counter(period_nanos=100).map(lambda n: n * 2)
g.run(cycles=3)          # deterministic historical replay from t=0
assert out.value() == 6  # 3rd tick -> 3 * 2

Wingfoil is composed of six specialized crates divided into three functional groups: the core engine and its Python bindings, procedural macros for boilerplate generation, and the web/browser communication layer.

Core Engine & Python

• wingfoil: The dual-mode stream-processing engine. Includes the Op trait, interpreter, fluent wiring layer, op catalog, I/O adapters, and the shared runtime core.
• wingfoil-derive: Provides procedural macros nitro! and #[op] to automate engine wiring and Op implementation boilerplate.
• wingfoil-python: PyO3 bindings for Python, importable via import wingfoil.
• wingfoil-python-derive: Provides the #[pyop] macro to derive Python-callable functions from Op implementations without manual glue code.

Web & Browser

• wingfoil-wire-types: Defines the shared wire-format types used by both the web adapter and the browser client to ensure synchronization.
• wingfoil-wasm: The browser-side WASM codec (targets wasm32-unknown-unknown). It is the engine behind the @wingfoil/client TypeScript package.

Wingfoil is a Reactive/DAG compute engine designed for latency-critical compute graphs where you can write logic once and replay it over history or run it live.

Key differentiators from other frameworks:

• Vs. Reactive/DAG engines (e.g., csp): Unlike csp, which is Python-only for graph building, Wingfoil supports Rust as a first-class user language, with Python and TypeScript support. Wingfoil also offers a compiled tier via 'Nitro' for performance, whereas csp relies on an in-process interpreter.
• Vs. Trading engines (e.g., NautilusTrader): While NautilusTrader provides a closed domain model (e.g., enum Data), Wingfoil uses a generic Stream<T> approach resolved at wiring time. Wingfoil also uses Burst to group same-instant events by construction, whereas NautilusTrader consumers may need to sample mid-instant.
• Vs. Distributed processors (e.g., Arroyo, Flink): Wingfoil is a single-process low-latency engine, whereas distributed processors are cluster-shaped and optimized for horizontal scaling and SQL-first workloads.

Wingfoil is currently suited for mid-tier latency trading (tens-of-microseconds budgets and up), such as crypto, market-making, and signals operating off commodity co-location.

Key differentiators include:

• Deterministic backtest/live parity: The engine ensures the same graph used for backtesting runs live with deterministic behavior. This is enforced via types (e.g., HistoricalFrom, timestamped channel replay) rather than manual discipline.
• High-performance hot path: The engine is designed with no locks on the cycle path and no per-cycle allocation. A Nitro-compiled tier can run a 37-node graph in ~19 ns/cycle.
• Built-in components: Includes a serious FIX session engine (adapters/fix), a venue-neutral market-data layer (adapters/market), and per-stage latency stamping (latency, Traced<T, L>).

Wingfoil is not currently intended for competitive single-digit-µs software HFT (which requires kernel-bypass adapters) or sub-microsecond wire-to-wire racing (which requires FPGAs).

Wingfoil is designed for high-performance graph execution by visiting every node once per tick in topological order. This allows the engine to compile the entire graph into a single function, significantly reducing dispatch overhead compared to reactive or per-path models.

Key performance metrics (measured on shared 4-core cloud VMs):

• Compiled Tier: Scales flatly with graph depth. At depth 10, it achieves ~23.9 ns per cycle.
• Interpreted Tier: Scales linearly with depth (approx. 68 ns + 22.0 ns × depth).
• Engine Overhead: Approximately 20 ns per node cycle.
• Clock Reading: Approximately 24 ns (optimized to only pay this cost in cycles that actually stamp latency).
• Channel Ingress: 0.87 µs per message with zero payload allocations at steady state.

Note on Scaling: Unlike per-path libraries (like tokio async streams or rxrust) which see exponential complexity ($2^N$ paths) as graph depth increases, Wingfoil's complexity remains flat because the scheduler visits each node exactly once per tick regardless of the number of upstream paths.

Wingfoil examples are categorized into three main groups based on their purpose:

Core

Focuses on engine concepts such as wiring, run modes, execution tiers, threading, and dynamism. These require no special setup.

• Key topics: run_mode, topological_sort, feedback, statistics, tracing, latency, threading, async, and dynamism.

Adapters

Focuses on I/O adapters for connecting to external systems. These typically require specific feature flags and some may require a running server.

• No server needed: lines, csv, augurs, zmq.
• Brokers: kafka, fluvio, redis, etcd.
• Low latency: iceoryx2, aeron.
• Stores: kdb, postgres.
• Protocols/Web: fix, web.
• Market Data: market.
• Telemetry: prometheus, otlp, telemetry.

Showcase

Demonstrates multi-process end-to-end latency. These should be run with the --release flag and require several services to be running.

• Key topics: latency (per-hop stamping over iceoryx2) and trading_e2e (nine-stage browser-to-venue flow).

The Wingfoil engine enforces structural causality through its feedback mechanism.

Builder::feedback_send pushes values onto the paired source's queue at time + 1, meaning the source emits the value on the next cycle. This ensures that an order emitted at time t cannot produce a fill that the strategy observes at the same instant t.

Key Properties:

• One-nanosecond floor: The engine imposes a structural 1ns floor for feedback. This is a minimum latency guarantee, not a full latency model. Realistic latency (ack/fill delays) should be modeled as a larger, configurable delay scheduled on this same mechanism.
• Burst Shape: Because orders travel as Burst<Order>, multiple identical orders in a single cycle are preserved within a single burst value. This prevents the TimeQueue deduplication logic from accidentally collapsing distinct orders into one.

// Conceptual usage of the feedback mechanism
// Values arrive on the source one cycle later (time + 1)
Builder::feedback_send(value);