Understanding Event Tracing for Windows in the .NET Runtime
Event Tracing for Windows is the operating system’s built-in, high-speed logging facility. For a .NET developer, it’s a way to instrument both your own code and the runtime itself without dragging in file I/O or string allocations on the hot path. When I first got pulled into diagnosing hangs on production ASP.NET services, ETW became the thing I reached for long before I ever attached a debugger. The reason is straightforward: ETW records what the runtime, the JIT, the garbage collector, and your framework libraries are actually doing—not what your code assumes they’re doing.
At its core you’ve got providers, controllers, consumers, and sessions. A provider is anything that fires events. The CLR is a generous provider, but you can write your own. A controller starts and stops trace sessions, flipping on specific providers and keywords. A consumer swallows the resulting .etl file and makes sense of it. Sessions can live in memory or on disk, and the whole setup leans on kernel-level ring buffers to keep the traced process running smoothly. In practice, turning on a detailed trace against a live production process often costs less than 1% CPU overhead. No reflection-based profiler gets close to that.
Provider Registration and Manifest-Free Events
Years ago, ETW providers demanded a manifest—an XML file tucked away as a resource—to spell out the event schema. The newer way for .NET is the EventSource class, which arrived in .NET Framework 4.5 and came along for the .NET Core ride. EventSource knows how to emit self-describing events, so the schema gets baked right into the payload. No manifest deployment headaches. You derive from EventSource, call WriteEvent, and the runtime records the event name, field names, and types. No offline compilation step at all.
One mistake I keep seeing in code reviews: folks grab EventSource methods that accept a format string. The WriteEvent overloads that take a string message force the runtime to pinvoke FormatMessage at trace time and allocate a temporary string on the tracing thread. The better path is to define a method with typed parameters that match the event payload and call the overload that takes an event ID plus the parameters directly. That way the event gets written as a structured blob, and any consumer—PerfView, WPA, you name it—can chew on the fields separately.
Keywords, Levels, and the Economics of Verbosity
Every event carries a level (Informational, Verbose, Warning, Error, Critical) and a keyword bitmask. Controllers use these to filter right at the session level, before the events ever touch a consumer. This isn’t some after-the-fact filter: the kernel drops events that don’t match the enabled keywords inside the provider’s write path. You pay zero for events you aren’t collecting. Designing a sensible keyword taxonomy for your custom EventSource is, honestly, the single most impactful choice you’ll make for production diagnostics.
I usually carve up keywords by subsystem: one bucket for database calls, one for outbound HTTP, one for caching operations, and so forth. The EventKeywords enum gives you a 64-bit flags type, so there’s room to burn. High bits I reserve for debug-only events that would never fly in production—per-iteration loop counters, object allocation timestamps. Low bits map to the business transactions your service actually cares about.
Collecting Traces with PerfView and dotnet-trace
PerfView is the canonical ETW analysis tool on Windows. Vance Morrison, one of the CLR architects, wrote it. It acts as both controller and consumer: you launch a collection, pick your CLR providers (Microsoft-Windows-DotNETRuntime, and usually the kernel provider for context switches), then crack open the .etl file when you’re done. PerfView’s stack viewer resolves managed call stacks by walking the JIT-compiled code, which only works if the Rundown keyword was enabled during the trace. Skip rundown, and you’ll stare at hex addresses instead of method names.
On Linux and macOS, the cross-platform player is dotnet-trace, part of the .NET diagnostics CLI. It talks to the runtime’s EventPipe layer rather than the kernel’s ETW subsystem, but the provider names and event payloads stay identical. A command like dotnet-trace collect --process-id 1234 --providers Microsoft-Windows-DotNETRuntime:0x1F000000018:5 captures GC, JIT, and ThreadPool events at the Verbose level. The resulting .nettrace file can be converted to speedscope format for a flame-graph view or opened directly in PerfView on Windows.
Choosing the Right Providers for Common Scenarios
When I’m chasing a memory leak, I turn on the GC provider with the GCHeapAndTypeNames keyword. That surfaces the object allocation graph and the finalization queue events. For a CPU spike, I combine the runtime provider’s Default keyword with the kernel’s Profile provider to grab sampled call stacks at 1 ms intervals. Investigating a networking hang? The HttpHandler provider in .NET Core spits out events at every stage of an HTTP request lifecycle—DNS resolution, connection pooling, TLS negotiation, the works.
One gotcha that bites newcomers: some providers demand admin privileges. The kernel provider is one; the CLR’s private provider (used for internal debugging) is another. In production you’ll typically run your trace collection tool under the same account as the process or with the SeSystemProfilePrivilege enabled. On Kubernetes that often translates to adding a sidecar container with the right capabilities.
Building a Custom EventSource for Application Telemetry
The EventSource class in .NET is built to be inherited. The usual pattern: create a sealed internal class, slap on the [EventSource(Name = "MyCompany-MyService")] attribute, and define a static singleton instance. Every event method should be non-inlined, return void, and hand off to one of the WriteEvent overloads. The event ID has to be unique within the source; I start at 1 and bump the number for each new event, never reusing a retired ID.
Here’s a minimal but correct example. Notice the use of the EventSource generator in .NET 6 and later—it gets rid of the manual WriteEvent calls. The source generator inspects the method signatures and emits the right call at compile time, so you won’t trip over mismatched parameter counts.
[EventSource(Name = "Contoso-Inventory")]
public sealed class InventoryEventSource : EventSource
{
public static readonly InventoryEventSource Log = new();
private const int OrderPlacedEventId = 1;
[Event(OrderPlacedEventId, Level = EventLevel.Informational, Keywords = Keywords.Orders)]
public void OrderPlaced(string orderId, int itemCount)
{
WriteEvent(OrderPlacedEventId, orderId, itemCount);
}
public static class Keywords
{
public const EventKeywords Orders = (EventKeywords)0x1;
public const EventKeywords Caching = (EventKeywords)0x2;
public const EventKeywords Diagnostics = (EventKeywords)0x1000;
}
}Activity IDs and End-to-End Correlation
ETW has a built-in way to track logical operations that cross thread boundaries and async continuations. An Activity ID is a GUID that propagates through Task and async state machines when you either call EventSource.SetCurrentThreadActivityId or let the runtime’s implicit propagation kick in via the System.Threading.Tasks.TplEventSource. Enable the TPL provider with the TaskTransfer keyword, and the runtime emits events that link a continuation’s activity back to the original task. That gives you a causal chain from the incoming HTTP request all the way to the final database query.
What I do in practice: set the activity ID at the entry point of each service operation—usually inside an ASP.NET middleware or a message handler—then use PerfView’s “Any Stacks” view grouped by activity ID. That reconstructs the whole request timeline, async gaps included. Without activity IDs, you’re looking at disjointed call stacks with no sense of temporal order.
Advanced Analysis Techniques with WPA and TraceProcessor
PerfView’s built-in views cover most investigations. But when I need to compute custom numbers—say, the 99th percentile latency of a particular SQL query over an hour-long trace—I reach for the TraceProcessor NuGet package. It hands you a programmatic API over .etl and .nettrace files. You crack open a trace, enumerate events by provider, and throw LINQ filters at them. TraceProcessor rides on the same native parsing engine as PerfView, so it handles multi-gigabyte traces without stuffing them into memory.
On Windows, the Windows Performance Analyzer (WPA) gives you a GUI for slicing traces by process, thread, and event type. WPA groks ETW’s extended data items—things like stack walks attached to individual events. Load the CLR’s symbol tables through WPA’s symbol resolution service, and you get fully resolved managed call stacks for any event that carries a stack key, like GC/AllocationTick. That’s pure gold when you’re hunting down the exact allocation site that’s driving gen-2 GC pressure.
Sampling vs. Instrumentation: When ETW Replaces a Profiler
Commercial profilers work by instrumenting bytecode or injecting hooks into JIT-compiled methods. They give you exact invocation counts and timing, but the overhead can turn a production server into molasses. ETW’s strength is that it can answer most of the same questions with a statistical approach. The kernel’s sampling profiler fires at a configurable interval (1 ms by default) and records the instruction pointer of each logical processor. Pair that with the JIT events that map instruction pointers to method names, and you get a statistical call-tree that honestly reflects where CPU time is spent—without touching a single instruction of your code.
For latency analysis, I combine the Microsoft-Windows-DotNETRuntime provider’s ThreadPool and Task keywords with custom EventSource events at service boundaries. That gives you a full picture of request duration and the contributing wait intervals. I once used this technique to root out a 300 ms delay caused by a misconfigured DNS suffix search list. No CPU profiler would have ever surfaced that.
Common Pitfalls and How to Avoid Them
Buffer loss is the headache I see most often. When the event rate outruns the consumer’s ability to flush the ring buffer, events get dropped. The trace log records a LostEvents event with the count, but plenty of developers just ignore it. The fix: bump the buffer size (the -buffersize flag in PerfView) or dial down the verbosity. A 256 MB buffer is often what you need for production GC traces when allocations are hammering the system.
Another trap: enabling the StackWalk keyword everywhere without thinking. Stack walking in ETW is the kernel’s job—it suspends the target thread and walks the call stack using the debugging API. On a hot path that can introduce noticeable pauses. I turn on stack walks only for specific events where I actually need the call site, like GCAllocationTick, and keep them off for high-frequency informational events.
One more gotcha: the interaction between EventSource and dynamic assembly loading. If your EventSource class lives in an assembly loaded via Assembly.Load and later unloaded, the provider stays registered in the kernel session until the process exits. That can cause the trace session to hang onto a reference to the unloaded assembly’s memory, creating a leak that’s a nightmare to diagnose. Always define EventSource instances in long-lived assemblies—ideally the application’s main .exe or a foundational library that never gets unloaded.
FAQ
Can I use ETW on Linux in production?
Yes, but through the EventPipe layer rather than the kernel’s ETW subsystem. The dotnet-trace tool works identically on all platforms, and the same EventSource providers emit the same events. The main difference is that kernel-level events (context switches, disk I/O) aren’t available via EventPipe; you’d need perf or bpftrace for those.
How do I measure the overhead of enabling a provider?
Use the EventSource class’s IsEnabled method to conditionally compute expensive payloads only when a consumer is listening. For runtime providers, benchmark your application with a known workload and the provider enabled at increasing verbosity levels. The OnEventCommand callback in your EventSource can also log the enabled keywords and level so you can audit what is being collected.
What is the difference between a manifest-based and a self-describing EventSource?
A manifest-based provider requires an XML manifest registered with the system, and consumers must have the manifest to decode events. Self-describing events embed the schema in the event metadata, so tools like PerfView can decode them without prior registration. All modern .NET EventSource classes should use self-describing events by calling the EventSource constructor with the appropriate flags or relying on the source generator.