krabsetw

repository·master·Indexed 21 days ago

https://github.com/microsoft/krabsetw

A C++ library that simplifies interaction with Event Tracing for Windows (ETW), providing mechanisms for enabling traces, registering for event notifications, and parsing event data into strongly typed structures. It includes a .NET wrapper called 'Lobsters' (Microsoft.O365.Security.Native.ETW) and a specialized version for .NET Core (Microsoft.O365.Security.Native.ETW.NetCore). The library supports x64 and ARM64 architectures on Windows 7/Windows 2008R2 and above.

Tokens
4.4K
Snippets
10
Records
21
Agent score
74%

What's inside krabsetw

  1. Overview of krabsetw and Lobsters

    master

    krabsetw

    A C++ library designed to simplify interacting with Event Tracing for Windows (ETW). It enables managing multiple traces and providers simultaneously and allows client code to register for event notifications. It also includes utilities for parsing generic event data into strongly typed data structures.

    Microsoft.O365.Security.Native.ETW (Lobsters)

    A C++ CLI (.NET) wrapper around krabsetw. It exposes the same functionality to .NET applications and is used in production by the Office 365 Security team.

  2. How traces and providers work in ETWLib

    master

    ETWLib uses two primary abstractions to manage event streams:

    Traces

    A Trace represents a stream of events. ETWLib distinguishes between Kernel traces and User traces.

    • UserTrace: Represents events from ETW-aware applications. You can create an unnamed trace or provide a specific name.
    • Kernel traces: (Note: Kernel trace implementation is currently not supported in ETWLib).

    Providers

    A Provider represents a specific source of ETW events, identified by a unique GUID. Providers use bitflags for event filtering:

    • Any: If an event meets any of the flags in this property, registered callbacks are triggered.
    • All: If an event meets all the bits in this property, registered callbacks are triggered.

    Note: The exact semantics of Any and All depend on the specific ETW provider being used; some providers may ignore the All flag if Any is not set.

    To receive events, you must:

    1. Create a UserTrace.
    2. Create a Provider with its GUID.
    3. Configure the provider's Any or All bitflags.
    4. Register a callback to the provider's OnEvent event.
    5. Enable the provider on the trace using trace.Enable(provider).
    6. Start the trace using trace.Start().
    // 1. Create a trace
    UserTrace namedTrace = new UserTrace("Muffins McGoo");
    
    // 2. Define a provider with a GUID
    Provider powershellProvider = new Provider(Guid.Parse("{A0C1853B-5C40-4B15-8766-3CF1C58F985A}"));
    
    // 3. Set filtering flags and register a callback
    powershellProvider.Any = 0x10;
    powershellProvider.OnEvent += MyCallbackFunction;
    
    // 4. Enable the provider on the trace
    namedTrace.Enable(powershellProvider);
    
    // 5. Start the trace (blocking call)
    // Use Task.Run to prevent blocking the main thread
    var t = Task.Run(() => namedTrace.Start());
    
    void MyCallbackFunction(EventRecord record) 
    {
        // Handle the event
    }
  3. The pattern for processing individual ETW events

    master

    Once an event is received, the processing logic follows these steps:

    1. Receive the event.
    2. Retrieve the schema associated with that specific event.
    3. Parse the data field of the event using the retrieved schema to extract the required information.
  4. Important usage notes and limitations

    master

    When using krabsetw or Microsoft.O365.Security.Native.ETW, be aware of the following:

    • Architecture Support: Only supports x64 and ARM64. x86 is not supported.
    • OS Requirements: Supported on Windows 7 or Windows 2008R2 and above.
    • Exception Handling: Throwing exceptions within an event handler callback (or within the library itself) will cause the trace to stop processing events.
    • Threading: The call to start on a trace object is blocking. You may need to manage threads manually to avoid blocking your main application execution.
    • Native Compilation: When building native code binaries using the krabsetw package, consult krabs/README.md regarding the TYPEASSERT and NDEBUG compilation flags.
  5. Understand ETW core concepts

    master

    Event Tracing for Windows (ETW) is a logging infrastructure used for diagnostics and performance analysis. To use krabsetw effectively, you must understand these key abstractions:

    • trace provider: A source of events (e.g., .NET, COM, WinINet).
    • trace session: An instance of an ETW trace that registers one or more providers. It handles events from all enabled providers in semi-chronological order on a single thread. There is a system limit of 64 traces (2 reserved).
    • kernel trace: A special session for providers exposed by kernel components. There is a limit of 8 sessions (2 reserved). Note: Kernel traces require Administrator privileges to start.
    • user trace: The standard type of ETW trace; most providers are user traces and can be registered by any process.
    • manifest: Defines the event schema for a specific trace provider.
    • schema: The data structure used to unpack and interpret the raw data from an ETW event buffer.
  6. How UserTrace and Provider work together

    master

    In O365.Security.Native.ETW, event listening is structured around two main abstractions: UserTrace and Provider.

    • UserTrace: Represents a stream of events. You can create an unnamed trace or a named trace using a string.
    • Provider: Represents a specific source of ETW events, identified by a GUID.

    To listen to events, you must associate a Provider with a UserTrace using the .Enable(provider) method, and then call .Start() on the trace. Providers use bitflags (Any and All) for event filtering. Note that the exact semantics of these flags depend on the specific ETW provider being used.

    // 1. Create a trace
    var trace = new UserTrace("MyTraceName");
    
    // 2. Define a provider with a GUID and filtering flags
    var provider = new Provider(Guid.Parse("{A0C1853B-5C40-4B15-8766-3CF1C58F985A}"));
    provider.Any = 0x01;
    provider.All = 0x10;
    
    // 3. Attach an event handler
    provider.OnEvent += (record) => { /* handle IEventRecord */ };
    
    // 4. Enable the provider on the trace and start
    trace.Enable(provider);
    trace.Start();
  7. Best practices for ETW event processing performance

    master

    Because starting an ETW trace session requires 'donating' a thread to the ETW subsystem, performance is critical to avoid dropping events:

    • Minimize work on the processing thread: The thread responsible for the trace session must dequeue events from the buffer as quickly as possible.
    • Offload heavy processing: When your event callback is invoked, you should parse the event and immediately push the actual data processing to a different thread.
    • Consolidate sessions: A single thread can process a massive volume of events from multiple providers. It is generally better to use one trace session with multiple providers rather than starting multiple trace sessions, unless you encounter event loss.
  8. Create a user trace

    master

    In Krabs, a user_trace represents a stream of events from ETW-aware applications. You can create an unnamed trace or a named trace by passing a wide-string literal to the constructor.

    Note that Krabs distinguishes between user_trace and kernel traces because their APIs differ.

    user_trace trace(); // unnamed trace
    user_trace namedTrace(L"Muffins McGoo");
  9. Enable providers and start a trace

    master

    To begin receiving events, you must follow these steps in order:

    1. Enable Providers: Call trace.enable(provider) for every provider you wish to listen to on that specific trace.
    2. Start the Trace: Call trace.start().

    Important: user_trace::start() is a blocking call. If your application needs to perform other tasks while listening for events, you must call start() on a separate thread. 3. Stop the Trace: Call trace.stop() to cease event collection.

    // 1. Enable the provider
    namedTrace.enable(powershellProvider);
    
    // 2. Start listening (on a separate thread to avoid blocking the main thread)
    void startListening()
    {
        namedTrace.start();
    }
    
    std::thread t(startListening);
    
    // ... do other work ...
    
    // 3. Stop the trace
    namedTrace.stop();
    t.join();