An Out-of-Memory exception in .NET doesn’t arrive with a polite warning. The stack trace usually fingers something harmless—a small byte array, a string concat, a plain object init. The message is direct: “Exception of type ‘System.OutOfMemoryException’ was thrown.” But the real cause sat quietly in the corner for hours before the crash. I’ve spent years pulling apart these failures in production systems, and what follows is the method I trust.
Interpreting the Exception: Memory Pressure vs. Allocation Failure
Before you launch WinDbg or PerfView, get clear on what the CLR actually means by “out of memory.” It isn’t always a shortage of virtual address space. In .NET Framework, the GC heap sits inside a contiguous chunk of virtual memory. If the runtime can’t reserve or commit enough room to grow a gen2 segment, the allocation bails out. .NET Core and .NET 5+ use non-contiguous segments, but the OS can still say no to a commit request when the process bumps into its virtual memory ceiling or system-wide commit charge runs dry.
Then there’s Large Object Heap fragmentation. Objects over 85,000 bytes land on the LOH, and by default the LOH doesn’t compact. Over time, free space wedges between pinned or long-lived large objects, and suddenly an allocation fails because no single free block is big enough. The same mess can hit the generational heap when pinned buffers—often from async socket work or interop allocations—chew generation 2 into fragments.
GC Mode and Budget Constraints
Workstation GC and Server GC react differently under load. Workstation GC uses one heap and runs collections on the allocating thread. That can stall responses, but it tends to expose OOM earlier. Server GC spreads per-core heaps with dedicated GC threads. A box with 32 logical processors running Server GC will spin up 32 heaps, each with its own segments. If one heap fills while others still have free space, the runtime forces a full blocking gen2 collection before it throws OOM. That collection can hide the problem for a while, but the underlying imbalance doesn’t go away.
Tuning GC budgets through COMPlus_GCHeapCount, COMPlus_GCLatencyMode, or GCSettings.LatencyMode controls how aggressively segments expand. SustainedLowLatency mode blocks gen2 collections but also restricts segment growth—useful during latency-sensitive windows, but it can fast-track an OOM if the app keeps allocating heavily.
Capturing a Memory Dump at the Right Moment
Timing is everything. Grab a full dump after the exception is caught and the process has already recycled, and you’ve lost the evidence. The dump you want is the one taken during the allocation failure itself. Automate it with procdump: procdump -ma -e 1 -f OutOfMemoryException <pid>. The -e 1 flag triggers on an unhandled first-chance exception. OOM is often handled and rethrown, so -e 2 for second-chance is an option, but by then some heap structures may have already been cleaned up.
If you can’t touch the production environment directly, set up a scheduled task that watches memory counters. When private bytes creep toward the virtual memory limit, take a series of dumps at 30-second intervals. Comparing consecutive dumps surfaces the allocating thread and the delta of objects on the heap.
Using Environment.FailFast as a Diagnostic Trigger
When recovery logic obscures the OOM, you can wrap code paths with a diagnostic flag that calls Environment.FailFast right after catching an OOM. This creates a dump with the exception context intact. The string argument to FailFast writes a message to the Windows Event Log and includes it in the dump—a breadcrumb you can search for. It’s a blunt instrument, but better than chasing ghosts through retry loops.
WinDbg Analysis: Mapping the Real Culprit
With a dump loaded in WinDbg, start with the exception record. !pe prints the managed exception object. Check the _message field and the stack trace. The stack often points to a benign spot—the victim. The actual cause is the state of the GC heap and the virtual memory layout.
Virtual Memory and Commit Size
Run !address -summary to see the process virtual address space breakdown. Focus on MEM_COMMIT and MEM_RESERVE. If committed memory is near the 2 GB limit for 32-bit processes, or near the configured virtual memory cap for 64-bit, you’re dealing with genuine address space exhaustion. Also check Largest Region by Usage for Free. A fragmented address space, even with plenty of free total, can stop the GC from reserving a contiguous segment.
Next, inspect the GC heap with !eeheap -gc. This lists every GC segment, its start and end addresses, and how much is committed. Lots of small segments instead of a few large ones means the GC is struggling to grow. For LOH analysis, !dumpheap -stat with the LOH address range shows the largest objects. Sort by size and hunt for unexpected piles: byte arrays from file reads, cached XML documents, large string dictionaries.
Pinning and Fragmentation
Pinned objects stop the GC from compacting the heap around them. Use !gchandles to list pinned handles. Each entry is an object held in place, usually for async I/O or interop. Cross-reference with !dumpobj to see what’s pinned. Thousands of small pinned buffers carve the heap into unusable gaps. In newer .NET versions, the Pinned Object Heap isolates pinned objects, which reduces generational heap fragmentation, but you have to opt in with DOTNET_GCPinnedObjectHeapBudget or the GCSettings class.
PerfView and ETW: Tracing Allocations Over Time
WinDbg freezes a moment; PerfView plays the whole film. Collect a GC trace with PerfView /GCCollectOnly /AcceptEULA /DataFile:oomtrace.etl and let it run until the OOM hits. Open the trace and go to the GCStats view. The “GC Rollup By Generation” chart shows allocation rates and collection frequencies. A sharp rise in gen2 allocations without matching collections points to a leak or an unbounded cache.
The “Heap Snapshot” diff is gold. Take two snapshots during the trace: one at process start (or after warmup) and another minutes later. The diff lists types whose instance count grew out of proportion. Watch for types with a high “survival rate” from gen0 to gen2—objects that escape ephemeral collections and pile up, slowly starving the process.
Identifying Leaking Threads
Some OOMs don’t come from managed memory at all but from thread stacks. Each thread reserves 1 MB of virtual memory for its stack (adjustable via the Thread constructor). A process that spawns hundreds of threads can exhaust address space long before the GC heap fills. In the dump, !threads lists managed threads. A high count of idle threads—pool threads with no work or abandoned timers—signals a thread management leak. Backtrace a few with ~* k to see why they were created and why they hung around.
Case Study: The Silent Fragmenter
I once debugged a Windows service that crashed every Tuesday at 3:13 AM with an OOM. The dump showed a healthy GC heap size—only 600 MB committed out of a 4 GB address space. But !address -summary revealed 512 MB of MEM_FREE scattered into 1,200 fragments. The offender: a daily batch job that allocated and released thousands of 90 KB temporary buffers during file processing. Each buffer was over 85 KB, so each hit the LOH. After processing, the buffers were freed, but the LOH doesn’t compact, so the free space fragmented. Day after day, the fragments multiplied until a new 90 KB allocation couldn’t find a contiguous block, even though total free space was plenty.
The fix had two parts: enable LOH compaction on a schedule using GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce after the batch job, and pre-allocate a pool of reusable buffers to cut down LOH churn. The OOM never came back.
Preventative Patterns
Once the immediate OOM is sorted, harden the app against future hits.
- ArrayPool and MemoryPool: For temporary buffers, reach for
ArrayPool<byte>.SharedorMemoryPool<byte>.Shared. These pools rent and return large arrays, which slashes LOH allocations. - Stream Pipelines: When processing large streams, lean on
System.IO.Pipelinesto work with memory in slices instead of monolithic byte array allocations. - Bounded Caches: Swap unbounded
ConcurrentDictionarycaches forMemoryCacheor a custom LRU that enforces a size limit and evicts entries under pressure. - GC.RegisterForFullGCNotification: This API pings you when a full GC is approaching. Use it to drop cached data proactively, lowering the chance of an OOM during the next allocation burst.
- ThreadPool Tuning: Cap the maximum number of threads with
ThreadPool.SetMaxThreadsto prevent stack-induced OOM.
FAQ: Common Questions on OOM Investigations
Why does the OOM exception point to a small allocation, like new byte[32]?
The stack trace shows the allocation that happened to trigger the GC’s failure to grow a segment. By that point, the heap is already near its limit or heavily fragmented. The small allocation is the final nudge; the real issue is the accumulated memory pressure from earlier, larger allocations.
How do I distinguish between managed memory leaks and native leaks?
Compare !eeheap -gc output (managed heap size) with the process’s private bytes from !address -summary. If private bytes are significantly larger than the GC heap plus loaded modules, you’ve got a native leak—maybe from P/Invoke handles, COM objects, or a third-party library. Use !heap -s to enumerate native heaps and look for high allocation counts.
Can I force LOH compaction in .NET Framework 4.5.1 or earlier?
No. GCSettings.LargeObjectHeapCompactionMode arrived in .NET Framework 4.5.2. Before that, the only way to compact the LOH was a process restart or a custom object pool that reused large buffers, sidestepping LOH fragmentation entirely. Upgrading to a supported runtime is strongly recommended if LOH fragmentation keeps biting you.
Is it safe to catch OutOfMemoryException and retry the operation?
Usually, no. By the time an OOM is thrown, the process state may be corrupted. The GC may have tried a full compaction and still failed, leaving internal data structures in a questionable state for future allocations. If you must handle OOM, do it at a process boundary—log the failure, flush telemetry, and exit gracefully using Environment.FailFast or a controlled recycle.
Investigating OOM exceptions asks for patience and a methodical eye. Correlate dump analysis with ETW traces and apply memory-efficient patterns, and a mysterious crash turns into a solved case. The tools are there—you just have to use them with precision.