Morgan Stanley Hobbes

repository·main·Indexed 22 days ago

https://github.com/morganstanley/hobbes

A system for embedding dynamic expressions and evaluation within C++ processes, featuring strong type integration and binding capabilities. It provides tools for compiling expressions into C++ callables via hobbes::cc, binding C++ functions and class instances, and pushing application data to storage using the HSTORE macro and the hog utility for data consumption and fault recovery.

Tokens
26.1K
Snippets
79
Records
153
Agent score
78%

What's inside hobbes

  1. Overview of hog for consuming Hobbes logs

    main
    When Hobbes producer code runs, it initializes a shared memory ringbuffer for logging. To prevent this ringbuffer from filling up and causing writes to block or fail, a performant consumer must service the queue. hog is a pre-written consumer designed to record structured data either locally to disk or to a remote process.
  2. Overview of the Hobbes programming language

    main

    Hobbes is a domain-targeting programming language and execution environment designed for ultra-low latency, high-performance integration with C/C++ applications. It is primarily used for managing the runtime of low-latency processes (like equities trading engines) that require dynamic, in-process rewriting of processing rules and persistence of structured logs without requiring restarts during working hours.

    Key capabilities include:

    • Dynamic Rule Rewriting: In-process rewriting of processing rules for domain objects such as trades, orders, and executions.
    • Structured Logging: Persistence and out-of-band processing of logs for order managers and surrounding processes.
    • Low Latency: Optimized for rock-solid, ultra-low latency execution.
    • Functional Paradigm: A variant of the pure-functional language Haskell.

    Security and Safety Warning: Hobbes is designed for performance over sandboxing. It does not have a sandboxed runtime or runtime safety features. It provides direct access to memory and does not perform array bounds checks. It also supports remote compilation and execution of native code over a network (RPC), which should only be used within trusted internal networks due to the security implications of these design choices.

    nil :: () -> (^x.(()+(a*x)))
    nil _ = roll(|0=()|)
    
    cons :: (a, ^x.(()+(a*x))) -> (^x.(()+(a*x)))
    cons x xs = roll(|1=(x,xs)|)
  3. Understand the Hobbes domain and use cases

    main

    Hobbes is a specialized language designed for DevOps staff to manage the in-process configuration of extremely low-latency Order Managers (OM).

    An Order Manager is responsible for maintaining the state of trade orders and executing logic based on market conditions (e.g., executing a limit order when a stock price hits a specific threshold).

    Key requirements addressed by Hobbes:

    • Low Latency: Logic must run in the 'hot path' of trading systems where execution speed is critical. Hobbes produces compiled, in-process logic to maximize processor efficiency (mechanical sympathy).
    • Dynamic Runtime Changes: The ability to change both the static portion (the operations/logic) and the dynamic portion (the operands/data like stock prices) at runtime without restarting the process.
    • Complex Decision Trees: Managing complex trading strategies (algo trading), venue splitting, and credit constraints that would be too slow if implemented with standard type hierarchies or long if-else chains in a high-frequency environment.
  4. Understand the core components of Hobbes

    main

    Hobbes consists of two primary components designed to balance performance with flexibility:

    1. A Programming Language: A Haskell-like language with a rich type system. It is designed to be embedded within C++ applications. This allows developers to write high-performance, structured code in C++ while implementing dynamic business logic in Hobbes. Hobbes code is compiled into efficient x86 instructions.
    2. A Persistence Format: A typesafe and space-efficient format used for real-time data storage and retrieval. It supports inter-process communication (IPC) over TCP, querying/filtering of application logs, and post-hoc analysis of application behavior using Hobbes' internal decision tree structure.
  5. Embed Hobbes in a C++ application

    main

    Hobbes is designed to be hosted within C++ programs. You can achieve a hybrid architecture where:

    • C++ binds to a Hobbes environment: You can call into Hobbes functions from C++ to execute dynamic logic.
    • Hobbes calls C++ functions: You can make C++ functions available to the embedded Hobbes code.
    • Data Marshalling: Data can be marshalled between the C++ host and the Hobbes environment.

    This approach allows you to keep highly structured application parts in C++ while updating business logic in Hobbes without changing the core C++ codebase. For implementation details, refer to the Embedding Hobbes guide.

  6. Understand Type Classes and Polymorphism in Hobbes

    main

    Hobbes implements polymorphism through Type Classes, which allow you to externally declare behaviors that a type supports. This enables writing generic functions that work across any data type implementing a specific capability (e.g., addition, multiplication, or printing).

    Anonymous Function Syntax

    You can define polymorphic functions using lambda syntax. The backslash \ starts the function, and the period . separates the argument list from the function body. Hobbes uses type inference to determine the necessary Type Class restrictions based on the operations used inside the function.

    Type Inference and Notation

    You can inspect the inferred type of an expression using the :t command. The resulting type notation follows the pattern: Restrictions => (Input Types) -> Return Type

    • Restrictions: The part before => specifies the Type Classes required (e.g., Add a b c =>).
    • Function Type: The part after => describes the mapping from input to output (e.g., (a * b) -> c).

    Common Type Classes

    • Add: Required for types supporting the + operator.
    • Equiv: For types supporting equivalence/equality.
    • Multiply: For types supporting the * operator.
    • Print: For types whose values can be printed.
  7. Pattern matching in Hobbes

    main

    Hobbes supports pattern matching for classifying and destructuring data using match expressions. This generalizes C++ switch statements by allowing matching on multiple values simultaneously, binding variables to parts of the matched value, and using guard expressions for conditional branching.

    Key Features

    • Multi-value matching: Match against multiple expressions at once (e.g., match x y with).
    • Variable Binding: Bind payloads from variant types or sub-elements of structures to variables (e.g., |paymentReceived=x|).
    • Regex Matching: When matching character arrays, use regular expressions with named capture groups for binding.
    • Guard Expressions: Use the where keyword to add a condition to a match row. The row is only selected if both the pattern matches and the guard evaluates to true.

    Requirements for Validity

    • Exhaustiveness: Every possible input must have a matching row. It is common practice to include a catch-all row using the _ pattern.
    • Reachability: No row should be redundant (i.e., no prior row should fully subsume it). Reachability checks are enabled by default in the compiler.
    match x y with
    | 0 0 -> "foo"
    | 0 1 -> "foobar"
    | 1 0 -> "bar"
    | 1 1 -> "barbar"
    | 2 0 -> "chicken"
    | 2 1 -> "chicken bar!"
    | _ _ -> "beats me"
  8. Understand the Hobbes logfile format

    main

    Hobbes uses a space-efficient binary format for persisted data. Files follow a header/body structure:

    1. Header: Describes the names, types, and sizes of each column.
    2. Body: Contains the actual persisted data.

    Because the header defines the schema, tools can perform efficient equality searches by calculating offsets based on the known sizes of the struct members. While the files are not human-readable, the hi REPL provides type-safe access to the data using the schema extracted from the header.

  9. Define LALR(1) parsers using context-free grammars

    main

    For complex parsing tasks that regular expressions cannot handle (like expression languages), hobbes provides a syntax to define LALR(1) parsers based on context-free grammars. You use the parse { RULES } syntax to construct a parser. Rules define both the syntax and "actions" (arbitrary hobbes code) that produce semantic values from the matched rules.

    Key features include:

    • Rule definitions: Rules can be composed of other rules or lexical constants.
    • Value binding: You can bind values matched by sub-rules to variables (e.g., v:V) to use them in the rule's action.
    • Left recursion: The parser supports left recursion for handling indefinite-length sequences.
    • Ambiguity detection: Hobbes rejects parser definitions that introduce ambiguous choices, ensuring $O(n)$ parse time.
    calc = parse {
      E := x:E "+" y:T { x + y }
        |  x:E "-" y:T { x - y }
        |  x:T         { x }
    
      T := x:T "*" y:F { x * y }
        |  x:T "/" y:F { x / y }
        |  x:F         { x }
    
      F := "(" x:E ")" { x }
        |  x:V         { x }
    
      V := v:V d:D { v*10 + d }
        |  d:D     { d }
    
      D := "0" {0} | "1" {1} | "2" {2} | "3" {3} | "4" {4}
        |  "5" {5} | "6" {6} | "7" {7} | "8" {8} | "9" {9}
    }
  10. Use pattern matching with match expressions

    main

    Match expressions allow you to perform actions based on the value or type of an expression. They work top-down: the first valid case encountered is executed.

    Key Rules:

    • Wildcards: The underscore _ acts as a wildcard (default case) or an instruction not to bind a name to the matched element.
    • Exhaustiveness: The compiler requires that all potential options are covered. If a match is not exhaustive, you must add a default case (e.g., _).
    • Reachability: Every case must be reachable; a case that matches everything must not appear before more specific cases.
    • Type Consistency: Since match is an expression, all branches must return the same type. Failing to do so results in a type unification error.
    • Terse Syntax: Match expressions can be written on a single line for idiomatic, concise code.

    Matching Multiple Values

    When matching against multiple values (like a tuple), you must provide the correct number of underscores/patterns to match the expected column count.

    Example of a simple match:

    match 3 with 
    | 1 -> show("hello")
    | 2 -> show("hobbes")
    | _ -> show("oops!")

    Example of a match expression assigned to a variable:

    hostport = match env with | "prod" -> "lnprd" | "qa" -> "euqa" | _ -> "ln123dev"
    match 3 with 
    | 1 -> show("hello")
    | 2 -> show("hobbes")
    | _ -> show("oops!")
  11. Perform polymorphic arithmetic using Type Classes

    main

    Hobbes handles arithmetic through Type Classes like Add, Subtract, Multiply, and Divide. These are available in the default namespace, allowing you to use operators like + on basic types (e.g., int, long) implicitly.

    Note that operators are not hard-coded into the language; they are resolved by the compiler via these Type Class instances. You can extend arithmetic support to your own custom types by implementing the corresponding Type Class instance.

    type counter = { count: int}
    
    counterAdd = (\x y. { count = iadd(x.count,y.count)})
    
    instance Add counter counter counter where
      (+) = counterAdd