PerfView Documentation

NetTrace is a binary file format (typically using the *.nettrace extension) designed for efficient streaming of profiling or logging traces. It consists of a small header followed by a sequence of serialized blocks. The format is optimized for high throughput (millions of events per second) and low file size.

Data is organized into five primary tables:

• Event Table: Represents individual events.
• Metadata Table: Stores event types (names, IDs, and field schemas).
• Stack Table: Stores stack traces (sequences of instruction pointers).
• Thread Table: Stores thread information (OS thread/process IDs, names).
• Label List Table: Stores arbitrary key/value labels (e.g., correlation IDs).

To resolve a single event, a reader must fetch the event row and then resolve its references to the metadata, stack, thread, and label list rows. These referenced rows must have appeared in blocks earlier in the file.

The TraceEvent library is a .NET NuGet package designed to control and consume strongly typed Event Tracing for Windows (ETW) events. It works alongside System.Diagnostics.Tracing.EventSource to create an end-to-end semantic logging system.

The architecture consists of three primary components:

1. Event Session (Microsoft.Diagnostics.Tracing.TraceEventSession): Controls the logging process. It can start/stop providers, control verbosity, and route data to files or directly to the session for real-time processing.
2. Event Provider (Microsoft.Diagnostics.Tracing.EventSource): The component integrated into the application being monitored. It calls logging APIs to emit events.
3. Event Consumer (Microsoft.Diagnostics.Tracing.TraceEventSource): Processes data from files or sessions to generate statistics or alerts.

Key Characteristics:

• Strongly Typed (Semantic): Events follow a schema (manifest) defining names, types, and metadata, making processing more efficient and less fragile than string-based logging.
• Asynchronous: Logging is a 'fire and forget' operation. Events are written to a buffer, allowing the application to continue immediately. This ensures high scalability and minimal impact on application performance, though it introduces the possibility of lost events if the consumer cannot keep up with the provider volume.

The TraceEvent library follows a specific pipeline for processing ETW data:

1. TraceEventSession: Used to start new ETW sessions, enable providers, and direct output.
2. ETW Providers: Sources of data, such as the Windows OS Kernel, .NET/JScript Runtimes, or custom System.Diagnostics.Tracing.EventSource providers.
3. ETWTraceEventSource: The component that hooks into the event stream (either a live session or an ETL file).
4. TraceEventParser: Converts unparsed events from the source into strongly typed objects.
   • Static Parsers: Efficient, compile-time parsers (e.g., built-in parsers or those generated via TraceParserGen).
   • Dynamic Parsers: Runtime parsers (e.g., DynamicTraceEventParser, RegisteredTraceEventParser, WPPTraceEventParser) used when schemas are unknown at compile time.
5. Subscription: You subscribe to C# events provided by the parser.
6. ETWTraceEventSource.Process(): The method that must be called to begin the processing loop and trigger callbacks.

Errors in EventSource (like duplicate event IDs) are often swallowed by the runtime, making them hard to detect. Use these techniques to diagnose them:

1. Use the EventSource NuGet package: The Microsoft.Diagnostics.Tracing.EventSource package provides compile-time warnings for common errors.
2. Run with the provider enabled constantly: Use PerfView to start a circular logging session for your provider so you can catch exceptions immediately:

   PerfView /CircularMB=10 /OnlyProviders:*Microsoft-Demos-MySource start

   Then, in your debugger, enable stopping on any thrown CLR exception (Debug $\rightarrow$ Exceptions).
3. Check ConstructionException: In your DEBUG code, explicitly check and throw the ConstructionException property:

   if (MySource.ConstructionException != null)
     throw MySource.ConstructionException;

4. Inspect Exceptions in PerfView: Even if swallowed by the runtime, the exceptions often appear as Exceptions events within PerfView.

When updating the EventPipe file format, follow these patterns to ensure compatibility:

Backward Compatibility (New readers, old writers)

To allow new readers to process old files:

1. Increment the Version number for the Trace Type.
2. Set the MinimumReaderVersion to the new version number.
3. Implement logic in the reader to check the object's Version. If the version is less than the new version, use the legacy reading logic; otherwise, use the new format logic.

Forward Compatibility (Old readers, new writers)

To allow old readers to process new files without crashing:

• Use Tagged values: Add new fields to an object using tagged data (e.g., tags for bool, byte, short, int, long, string, blob). The FastSerializable library can skip unknown tags until it finds the endObject tag.
• Add new object types: Insert entirely new object types into the stream. Ensure these new objects use tagged fields so old readers can skip them.
• Extend Headers: Add new data to EventBlock or MetadataBlock headers. These headers include a header size field, allowing old readers to skip the additional bytes they do not recognize.