Garbage collection statistics are the first forensic artifact most engineers reach for when a .NET process starts misbehaving in production. Yet most of what gets read from a trace is either the wrong number, the wrong generation, or the wrong conclusion. This article is about reading GC stats from production traces the way an incident responder should: with an eye for allocation pressure, pause pathology, and heap fragmentation, not just a single headline metric. We will work with .NET 6+ runtimes on Windows, Linux containers, Kubernetes, Azure App Service Linux, and serverless runtimes, where the GC behaves differently depending on workstation versus server mode, container CPU limits, and memory ceilings.
Adjacent concepts matter here: ephemeral segments, large object heap, pinned object heap, finalization queue, background GC, workstation GC, server GC, regions in .NET 7+, and the difference between allocation rate and survival rate. If you cannot separate those, you will misread a trace. The goal is not to admire the numbers. The goal is to decide whether the process is healthy, whether it is degrading, and what to change first.
What a Production Trace Actually Contains
A production trace is not a benchmark. It is a recording of a process under real load, often with sampling overhead, missing CPU counters, and a partial view of thread activity. When you collect a trace with dotnet-trace, PerfView, or a memory profiler, you are capturing events emitted by the runtime: GC start and end, allocation ticks, suspension begin and resume, finalization events, and sometimes heap layout snapshots. The GC stats you extract are only as good as the event set you enabled.
For .NET 6 and later, the runtime emits events through Microsoft-Windows-DotNETRuntime on Windows and LTTng on Linux. The event names are stable enough to script against, but the semantics shift between GC modes. A GC/Start event in workstation mode means one thread is doing the collection. In server mode, it means multiple threads are participating. If you do not know which mode the process was in, you cannot interpret pause time or CPU cost correctly.
Event Sequence That Matters
When reading a trace, follow this sequence for each collection:
- GC/Start — generation, reason, and whether it is a background or blocking collection.
- GC/SuspendEEStart and GC/SuspendEEStop — the suspension window, often the largest contributor to pause time.
- GC/GlobalHeapHistory — per-generation sizes, fragmentation, and promotion counts.
- GC/HeapStats — per-heap details, including pinned object counts and finalization queue length.
- GC/RestartEEStart and GC/RestartEEStop — the resume window.
- GC/End — total collection time and heap size after collection.
Most dashboards show only the last number. That is like reading a crash dump by looking at the exit code.
Reading the Numbers That Actually Predict Incidents
Three numbers predict production incidents better than any others: allocation rate, promotion rate, and pause time distribution. Everything else is context.
Allocation Rate
Allocation rate is the volume of bytes allocated per second, not the total heap size. A process can have a stable 2 GB heap and still be allocating 500 MB per second if objects die young. High allocation rate forces frequent Gen0 collections, which are cheap individually but expensive in aggregate. In a trace, look for GC/AllocationTick events. Sum the allocation amounts over a fixed window, then divide by the window length. If the rate is above 100 MB/s in a container with one CPU, you are paying a tax on every request.
Do not confuse allocation rate with memory leak. A leak is a rising heap size after full collections. A high allocation rate with a flat heap is just churn. Both are problems, but the fixes are different.
Promotion Rate
Promotion rate is the number of bytes that survive a Gen0 collection and move to Gen1, or survive Gen1 and move to Gen2. High promotion means objects are living just long enough to escape the cheap generations. That is the worst case for a server application: you pay for the allocation, you pay for the copy, and you pay for the eventual Gen2 collection. In GC/GlobalHeapHistory, look at the Promoted fields. If Gen0 to Gen1 promotion is consistently above 20% of the Gen0 allocation volume, you have a mid-life crisis in your object graph.
Common causes: request-scoped caches, short-lived MemoryStream instances that cross async boundaries, and LINQ chains that materialize intermediate collections. The trace will not tell you the cause. It will tell you the size of the problem.
Pause Time Distribution
Average pause time is a lie. A process with 10 ms average pauses can still have 500 ms outliers that time out clients. Read the distribution. In PerfView, use the GCStats view and look at the pause time histogram. In dotnet-trace output, sort the GC/End events by duration and look at the 95th and 99th percentiles. If the 99th percentile is more than 10x the median, you have a pause pathology, not a GC tuning problem.
Pause time outliers come from a few places: blocking Gen2 collections, suspension waits on threads in unmanaged code, and finalization storms. Each has a different signature in the trace.
Gen2, LOH, and POH: The Long Game
Gen2 collections are the only collections that can compact the large object heap, and they are the most expensive. In .NET 6+, the large object heap is not collected as often as Gen0 and Gen1, but it is still part of Gen2. A full blocking Gen2 collection will pause all managed threads, walk the entire heap, and potentially compact LOH segments. That is a multi-hundred-millisecond event on a large heap.
The pinned object heap, introduced in .NET 5, is a separate segment for objects that are pinned. Pinning is the enemy of compaction. If your trace shows a high number of pinned objects in GC/HeapStats, the GC cannot move those objects, which fragments the heap and forces more frequent Gen2 collections. Look for the PinnedObjectCount field. If it is in the thousands, find the pinning code. Usually it is a byte[] passed to native code, a Memory<byte> over a pinned buffer, or a GCHandle that was never freed.
Fragmentation as a Leading Indicator
Fragmentation is the gap between the heap size and the amount of live data. In GC/GlobalHeapHistory, compare FinalYoungestDesired and TotalHeapSize to the live object count. If the heap is 4 GB but live data is 1 GB, you have 3 GB of fragmentation. That is not a leak. That is a compaction failure. The fix is not more memory. The fix is fewer pins, fewer long-lived arrays of varying sizes, or a different GC mode.
On Linux containers, fragmentation is worse because the runtime cannot always return memory to the OS. The heap may look stable in the container, but the cgroup memory limit is being hit because the runtime is holding onto segments. Read the GC/HeapStats TotalCommittedBytes and TotalReservedBytes fields. If committed is close to reserved, the runtime is using what it asked for. If committed is far below reserved, the runtime is holding memory it does not need.
Workstation vs. Server GC in Production Traces
The GC mode changes the meaning of every number. Workstation GC uses one thread for collections and is optimized for low pause time on interactive workloads. Server GC uses one thread per logical processor and is optimized for throughput on multi-core servers. In a container with a CPU limit, server GC can be a disaster: the runtime sees the host’s CPU count, not the container’s limit, and spawns too many GC threads. That causes thread contention, longer suspension windows, and higher CPU usage during collections.
In .NET 6+, the runtime respects DOTNET_GCHeapCount and DOTNET_PROCESSOR_COUNT to limit GC threads. But if those are not set, the trace will show server GC threads fighting for CPU. Look at the GC/Start event’s Depth and Type fields. If Type is BackgroundGC and the process is on a two-CPU container, you are paying for a mode that does not fit the environment.
Azure App Service Linux and serverless runtimes often default to workstation GC because the platform knows the CPU limit. But if you override the GC mode in runtimeconfig.json or an environment variable, you can break that assumption. The trace will show the result: longer suspension windows and more time in GC/SuspendEE than in the actual collection.
Reading a Real Incident: The Case of the 900 ms Pause
A .NET 7 API on Kubernetes started timing out at the 99th percentile. The team collected a trace with dotnet-trace and saw a 900 ms pause every 45 seconds. The average pause was 12 ms. The dashboard showed a healthy process. The trace showed a different story.
The GC/Start events showed a blocking Gen2 collection every 45 seconds, triggered by AllocSmall. The GC/GlobalHeapHistory showed 2.1 GB of Gen2 heap with 1.4 GB of fragmentation. The GC/HeapStats showed 12,000 pinned objects. The pinning came from a logging library that pinned every log message buffer for asynchronous I/O. The fix was to change the logging library’s buffer strategy, not to tune the GC. The trace did not name the library. It named the symptom: 12,000 pins, 1.4 GB of fragmentation, and a 900 ms pause every 45 seconds.
That is the art of reading GC stats. You do not read the number. You read the relationship between the numbers.
Tools and Commands That Produce Usable Stats
You do not need a commercial profiler to read GC stats from a production trace. You need the right event set and the right post-processing.
dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x1FFFF:5— the0x1FFFFmask enables GC, loader, and exception events. The:5is the verbosity level. Higher verbosity gives more detail but more overhead.dotnet-trace convert --format speedscope— converts the trace to a format that shows pause time as a flame graph.- PerfView on Windows — the
GCStatsview is the fastest way to see pause time distribution and per-generation sizes. dotnet-counters monitor --process-id <pid> --counters System.Runtime— live counters for allocation rate, GC count, and heap size. Not a trace, but a good first signal.
For Linux containers, dotnet-trace uses LTTng under the hood. You need liblttng-ust installed in the container. If it is missing, the trace will be empty. That is a common failure in distroless images. The fix is to add the package or use a sidecar collector.
What the Numbers Do Not Tell You
A trace shows what the GC did. It does not show why the application allocated the objects. It does not show which method created the pinned buffer. It does not show which request path caused the allocation spike. For that, you need a memory profiler with allocation call stacks, or you need to correlate the trace with application logs and distributed traces.
Do not fall into the trap of tuning the GC to fix an allocation problem. If the trace shows high allocation rate and high promotion, the fix is in the code, not in DOTNET_gcServer or DOTNET_GCConserveMemory. The GC is the messenger. The trace is the message. Read it accordingly.
FAQ
What is the difference between allocation rate and heap size in a GC trace?
Allocation rate is the number of bytes allocated per second, measured from GC/AllocationTick events. Heap size is the total managed heap after a collection, measured from GC/HeapStats. A process can have a stable heap size and a very high allocation rate if objects die young. High allocation rate forces frequent Gen0 collections, which cost CPU even if the heap does not grow.
How do I know if a Gen2 collection is blocking or background from a trace?
Look at the GC/Start event’s Type field. A BackgroundGC type means the collection runs concurrently with application threads for most of its duration. A BlockingGC type means all managed threads are suspended for the entire collection. Blocking Gen2 collections are the ones that cause multi-hundred-millisecond pauses. Background Gen2 collections still have a short suspension window at the beginning and end, but the bulk of the work happens concurrently.
Why does my .NET 6 container show high GC pause times when the host has plenty of CPU?
The container likely has a CPU limit that is lower than the host’s CPU count. Server GC spawns one GC thread per logical processor it sees, which may be the host’s count, not the container’s limit. Those threads contend for the limited CPU, which lengthens suspension windows and collection time. Set DOTNET_GCHeapCount or DOTNET_PROCESSOR_COUNT to match the container’s CPU limit, or switch to workstation GC if the workload is latency-sensitive.
What does a high pinned object count mean in a GC trace?
A high PinnedObjectCount in GC/HeapStats means many objects cannot be moved during compaction. Pinned objects fragment the heap, which reduces the effectiveness of Gen2 collections and can lead to a rising heap size even when live data is stable. The usual sources are pinned buffers for native I/O, GCHandle instances that are not freed, and Memory<byte> over pinned arrays. The fix is to reduce pinning, not to increase the heap size.
