Advanced .NET Debugging

02:14. The pod has restarted four times since the deploy and the log hands you one line: Could not load file or assembly 'Pricing.Contracts, Version=4.2.1.0, Culture=neutral, PublicKeyToken=...'. You exec in, list the directory, and the file is sitting right there. That is the moment this stops being a missing-file problem. On .NET 6 through 9, binding is not a filesystem lookup — it is resolution against state the runtime computed before your code ran, and at least four distinct failure modes print that same sentence. Here is the map, and where to spend the forty minutes you have before the next recycle.

The message describes a request, not a file

In .NET Framework, the Fusion loader probed the disk and fuslogvw showed you every probe path. Fusion is gone. On .NET 6-9 the runtime hands the name to the AssemblyLoadContext that owns the assembly making the reference — the ALC, the isolation boundary that owns load policy; the default ALC backs your main app. The default context checks the Trusted Platform Assemblies list first: the TPA, a closed set of paths that hostpolicy computed from .deps.json before Main. Only if that misses does it fall back to probing the application base and culture subdirectories. If the assembly is not on the TPA and not in a probe path, the copy on disk does not exist as far as binding is concerned.

So the first command is not ls. It is grep 'Pricing.Contracts' *.deps.json inside the container. An entry that is absent means your publish never emitted it — fix the artifact, not the folder. An entry that is present means the failure is in the path, the casing, or the version, and the next checks differ. If the failure names the runtime itself (Microsoft.NETCore.App), you are in hostfxr roll-forward territory and the app’s deps.json is irrelevant. The dependency-loading overview on Microsoft Learn maps the whole stack if you want the reference beside you.

The version in the message is the second trap. There are no binding redirects on .NET 6-9; unification happens once, at restore, where NuGet picks a single winner for each package and publish writes one file. The Version=4.2.1.0 in the error is what some compile-time reference asked for — it may never have existed as a file anywhere in the image. When the winner is older than what the calling code expects, you do not even get a load failure: you get MissingMethodException at a call site, which nobody files under binding. The restore-time NU1605 downgrade warning is the early signal, and most build setups let it scroll past.

Loading is lazy, so the failure is time-shifted

An assembly loads when the JIT first compiles a method that references a type in it, or when something calls Assembly.Load explicitly. The deploy is not the trigger; first execution is. A settlement job or an export endpoint that no canary ever hit detonates weeks later at 03:00, and the stack points at the call site — where the reference was — not at the restore decision that broke it months ago.

Publish modes bend this further. ReadyToRun changes JIT timing, not binding. Trimming deletes assemblies outright, which makes the failure deterministic while keeping the message identical. Single-file inverts the file-exists check: assemblies live inside the bundle, nothing is on disk, and Assembly.Location returns empty for bundled assemblies — a tell, not a bug. I once burned a Friday on a health check that verified a file in the folder; the app was single-file, the folder was irrelevant, and the real problem was a stale .deps.json baked into the layer next to the bundle.

Four signatures, four different root causes

Classify before you touch anything. The exception type is the coarse map onto the failure layers:

• FileNotFoundException (0x80070002). Resolution produced no candidate: missing from deps.json, a casing mismatch on ext4, a RID-specific asset that never shipped, or a plugin directory the custom ALC was never told about.
• BadImageFormatException (0x8007000B). A candidate was found and refused on format: x86 native in an x64 pod, a glibc-linked .so on Alpine’s musl, a truncated COPY in a Docker layer, or a .NET Framework binary loaded into a .NET 8 process.
• MissingMethodException / TypeLoadException. Binding succeeded and the contract did not: unification picked an older winner. This is the one that gets filed as a code bug for three days.
• FileLoadException (0x80131040). The ALC refused a duplicate identity — LoadFromAssemblyPath on a name already loaded in that context. Plugin hosts that reload configuration shims hit this weekly.

One more signature belongs on the list even though it is not a load failure: InvalidCastException between identical type names. That is the same assembly loaded into two ALCs — two runtime type identities that no cast will ever reconcile. The fix is load policy, not casting. The dump-side workflow for that pattern is in our AssemblyLoadContext debugging guide.

The Fusion log is gone — what replaces it, layer by layer

Startup failures: COREHOST_TRACE=1 and COREHOST_TRACE_FILE=/tmp/host.log dump hostfxr and hostpolicy’s work — the TPA they computed, framework roll-forward, additional deps. Host-time only; it goes silent once Main starts.

Managed loads later: nothing logs them by default. Hook AssemblyLoadContext.Default.Resolving at the top of Program.cs — log the requested name and the requesting assembly, return null so the failure still propagates — and every future incident becomes one log line instead of archaeology. AppDomain.CurrentDomain.AssemblyLoad logs successful loads for the audit trail, with the same empty-Location caveat for bundles.

Live evidence: the runtime emits AssemblyLoad events under the Microsoft-Windows-DotNETRuntime Loader keyword (0x8000). Run dotnet-trace collect -p <pid> --providers Microsoft-Windows-DotNETRuntime:0x8000:4 while you replay the failing request, then read the events in PerfView. On Windows Server, PerfView captures the provider by default — filter the Loader events and you have the load sequence with timestamps. The full resolution algorithm is in the managed assembly loading reference, and it is shorter than most people expect.

If it is already crash-looping: dump forensics

When the process dies at startup, you capture rather than attach. On Linux, createdump -f /dumps/app.dmp <pid>. In a pod, set DOTNET_DbgEnableMiniDump=1 and DOTNET_DbgMiniDumpType=2 (heap) or 4 (full), point DOTNET_DbgMiniDumpName at a mounted volume, and let the crash write its own evidence. We keep a fuller walkthrough of collecting crash dumps on Kubernetes that covers the pod-spec side.

Then dotnet-dump analyze — the same SOS commands work in WinDbg if that is your bench:

• pe -nested: the full exception chain, inner exceptions, HRESULTs.
• clrstack: the topmost app frame is the method whose JIT pulled the trigger — your time-bomb location.
• dumpdomain: every ALC and its assemblies. The same simple name in two contexts is the InvalidCastException case, decided in one command.
• dumpassembly <addr> and dumpmodule -mt <addr>: which file backed it; an empty path means bundled.
• name2ee <module> <type>: which copy of the type the runtime will actually bind.

The dump is the only place you can see both sides at once — what loaded, and what the request asked for. The dotnet-dump tool reference covers collection flags, and our dotnet-dump command cheat sheet has the SOS syntax in one page.

Container traps that multiply all of this

Casing: ext4 is case-sensitive, NTFS is not. A Dockerfile COPY that preserves wrong casing builds a layer that works on the Windows dev box and dies in the pod. Verify with ls in the container, not your IDE’s file tree.

Build output masquerading as publish output: copying bin/Release/net8.0 instead of the publish folder ships no processed deps.json, so fallback probing loads whatever it can see. The app half-boots and dies at the first missing reference. Copy the publish output whole; never cherry-pick DLLs into a layer.

RID-specific assets: native and some managed assets land under runtimes/linux-x64/native/ and only resolve when deps.json says so. Publish with -r linux-x64 — linux-musl-x64 on Alpine — or ship the runtimes tree intact.

Base image drift: same tag, different digest — Debian with glibc on one node pool, Alpine with musl on another, and a native BadImageFormatException that looks random across the fleet. Pin digests and match the RID to the base.

Two environment variables can also inject resolution paths you never configured: DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE. Check them in the pod spec before you blame the artifact; I lost most of a night to an inherited DOTNET_ADDITIONAL_DEPS that pointed at a store which existed only on the build agent.

Managed hosts add their own layer. Azure App Service Linux runs what is actually in /home/site/wwwroot — check via Kudu SSH that the deps.json you are reading is the one the platform deployed. On AWS Lambda, layers merge in mount order, so two layers shipping the same assembly at different versions means merge order decides the winner while the error names the loser. Verify what landed in the execution environment, not what you uploaded.

The 40-minute sequence

1. 0-5: classify. Exact message, HRESULT, full stack — kubectl logs --previous, the App Service log stream, or pe -nested from the dump. Match it to one of the four signatures. Touch nothing yet.
2. 5-15: resolve statically. Grep the deps.json for the simple name. Absent → publish artifact problem; stop here. Present → confirm the listed relative path exists in the container with exact casing, including the runtimes/<rid>/ subtree.
3. 15-25: match the failure’s age. Dies at startup → one run with COREHOST_TRACE. Loads fine and fails later → host trace is silent by design; hook Resolving in the next build, or capture live with the Loader keyword while you replay the request.
4. 25-40: dump and decide. dumpdomain: two copies of the simple name → ALC policy fix. One copy plus MissingMethodException → version split; dotnet list package --include-transitive to find who pulled what, then pin.

Decisions that prevent the next one

• Fail the build on NU1605. Downgrades are where unification incidents are born, and they are cheap to stop at restore.
• Pin package versions in one place with Central Package Management so the graph cannot split quietly.
• Add a CI step that verifies every deps.json entry resolves to a file in the publish output with exact casing. Ten lines of shell; it catches the casing and cherry-picked-DLL classes before they ship.
• Register the Resolving hook in your shared bootstrap and log to your normal structured logger. Future you gets one log line instead of a Saturday.
• Publish with an explicit RID and pin base image digests in both Docker stages.

FAQ

Where did the Fusion log go in .NET 8?

Nowhere you can reach. Fusion and fuslogvw are .NET Framework machinery. The replacements are per layer: COREHOST_TRACE for host-time resolution, the Resolving event for managed loads, and the Loader keyword (0x8000) for live load events.

Do binding redirects work in .NET Core or .NET 5+?

No. app.config redirects are ignored. Unification happens once, at restore and publish; if two packages need different versions, one file ships and everything binds to it. The symptom is NU1605 at restore or MissingMethodException at run. Fix the graph instead of trying to redirect at runtime.

Why does it work on my machine but not in the container?

Five usual answers, in order of frequency: file casing, build output copied instead of publish output, a RID-specific asset that never shipped, musl versus glibc in the base image, and a deps.json in the image that does not match what you published. Compare the container’s deps.json against your local one — that diff closes most of these tickets.

How do I tell which AssemblyLoadContext loaded an assembly?

In a dump, dumpdomain lists every context and its assemblies; a duplicate simple name across contexts is your answer. Live, the Resolving event hands you the requesting context directly in its arguments.

Binding failures are not hard because the loader is mysterious; the algorithm is documented and deterministic. They are hard because four root causes print one sentence, and the sentence describes what was asked for, never what was found. Classify the exception first. Everything after that is a command, not a debate.

Garbage collection statistics are the first forensic artifact most engineers reach for when a .NET process starts misbehaving in production. Yet most of what gets read from a trace is either the wrong number, the wrong generation, or the wrong conclusion. This article is about reading GC stats from production traces the way an incident responder should: with an eye for allocation pressure, pause pathology, and heap fragmentation, not just a single headline metric. We will work with .NET 6+ runtimes on Windows, Linux containers, Kubernetes, Azure App Service Linux, and serverless runtimes, where the GC behaves differently depending on workstation versus server mode, container CPU limits, and memory ceilings.

Adjacent concepts matter here: ephemeral segments, large object heap, pinned object heap, finalization queue, background GC, workstation GC, server GC, regions in .NET 7+, and the difference between allocation rate and survival rate. If you cannot separate those, you will misread a trace. The goal is not to admire the numbers. The goal is to decide whether the process is healthy, whether it is degrading, and what to change first.

What a Production Trace Actually Contains

A production trace is not a benchmark. It is a recording of a process under real load, often with sampling overhead, missing CPU counters, and a partial view of thread activity. When you collect a trace with dotnet-trace, PerfView, or a memory profiler, you are capturing events emitted by the runtime: GC start and end, allocation ticks, suspension begin and resume, finalization events, and sometimes heap layout snapshots. The GC stats you extract are only as good as the event set you enabled.

For .NET 6 and later, the runtime emits events through Microsoft-Windows-DotNETRuntime on Windows and LTTng on Linux. The event names are stable enough to script against, but the semantics shift between GC modes. A GC/Start event in workstation mode means one thread is doing the collection. In server mode, it means multiple threads are participating. If you do not know which mode the process was in, you cannot interpret pause time or CPU cost correctly.

Event Sequence That Matters

When reading a trace, follow this sequence for each collection:

• GC/Start — generation, reason, and whether it is a background or blocking collection.
• GC/SuspendEEStart and GC/SuspendEEStop — the suspension window, often the largest contributor to pause time.
• GC/GlobalHeapHistory — per-generation sizes, fragmentation, and promotion counts.
• GC/HeapStats — per-heap details, including pinned object counts and finalization queue length.
• GC/RestartEEStart and GC/RestartEEStop — the resume window.
• GC/End — total collection time and heap size after collection.

Most dashboards show only the last number. That is like reading a crash dump by looking at the exit code.

Reading the Numbers That Actually Predict Incidents

Three numbers predict production incidents better than any others: allocation rate, promotion rate, and pause time distribution. Everything else is context.

Allocation Rate

Allocation rate is the volume of bytes allocated per second, not the total heap size. A process can have a stable 2 GB heap and still be allocating 500 MB per second if objects die young. High allocation rate forces frequent Gen0 collections, which are cheap individually but expensive in aggregate. In a trace, look for GC/AllocationTick events. Sum the allocation amounts over a fixed window, then divide by the window length. If the rate is above 100 MB/s in a container with one CPU, you are paying a tax on every request.

Do not confuse allocation rate with memory leak. A leak is a rising heap size after full collections. A high allocation rate with a flat heap is just churn. Both are problems, but the fixes are different.

Promotion Rate

Promotion rate is the number of bytes that survive a Gen0 collection and move to Gen1, or survive Gen1 and move to Gen2. High promotion means objects are living just long enough to escape the cheap generations. That is the worst case for a server application: you pay for the allocation, you pay for the copy, and you pay for the eventual Gen2 collection. In GC/GlobalHeapHistory, look at the Promoted fields. If Gen0 to Gen1 promotion is consistently above 20% of the Gen0 allocation volume, you have a mid-life crisis in your object graph.

Common causes: request-scoped caches, short-lived MemoryStream instances that cross async boundaries, and LINQ chains that materialize intermediate collections. The trace will not tell you the cause. It will tell you the size of the problem.

Pause Time Distribution

Average pause time is a lie. A process with 10 ms average pauses can still have 500 ms outliers that time out clients. Read the distribution. In PerfView, use the GCStats view and look at the pause time histogram. In dotnet-trace output, sort the GC/End events by duration and look at the 95th and 99th percentiles. If the 99th percentile is more than 10x the median, you have a pause pathology, not a GC tuning problem.

Pause time outliers come from a few places: blocking Gen2 collections, suspension waits on threads in unmanaged code, and finalization storms. Each has a different signature in the trace.

Gen2, LOH, and POH: The Long Game

Gen2 collections are the only collections that can compact the large object heap, and they are the most expensive. In .NET 6+, the large object heap is not collected as often as Gen0 and Gen1, but it is still part of Gen2. A full blocking Gen2 collection will pause all managed threads, walk the entire heap, and potentially compact LOH segments. That is a multi-hundred-millisecond event on a large heap.

The pinned object heap, introduced in .NET 5, is a separate segment for objects that are pinned. Pinning is the enemy of compaction. If your trace shows a high number of pinned objects in GC/HeapStats, the GC cannot move those objects, which fragments the heap and forces more frequent Gen2 collections. Look for the PinnedObjectCount field. If it is in the thousands, find the pinning code. Usually it is a byte[] passed to native code, a Memory<byte> over a pinned buffer, or a GCHandle that was never freed.

Fragmentation as a Leading Indicator

Fragmentation is the gap between the heap size and the amount of live data. In GC/GlobalHeapHistory, compare FinalYoungestDesired and TotalHeapSize to the live object count. If the heap is 4 GB but live data is 1 GB, you have 3 GB of fragmentation. That is not a leak. That is a compaction failure. The fix is not more memory. The fix is fewer pins, fewer long-lived arrays of varying sizes, or a different GC mode.

On Linux containers, fragmentation is worse because the runtime cannot always return memory to the OS. The heap may look stable in the container, but the cgroup memory limit is being hit because the runtime is holding onto segments. Read the GC/HeapStats TotalCommittedBytes and TotalReservedBytes fields. If committed is close to reserved, the runtime is using what it asked for. If committed is far below reserved, the runtime is holding memory it does not need.

Workstation vs. Server GC in Production Traces

The GC mode changes the meaning of every number. Workstation GC uses one thread for collections and is optimized for low pause time on interactive workloads. Server GC uses one thread per logical processor and is optimized for throughput on multi-core servers. In a container with a CPU limit, server GC can be a disaster: the runtime sees the host’s CPU count, not the container’s limit, and spawns too many GC threads. That causes thread contention, longer suspension windows, and higher CPU usage during collections.

In .NET 6+, the runtime respects DOTNET_GCHeapCount and DOTNET_PROCESSOR_COUNT to limit GC threads. But if those are not set, the trace will show server GC threads fighting for CPU. Look at the GC/Start event’s Depth and Type fields. If Type is BackgroundGC and the process is on a two-CPU container, you are paying for a mode that does not fit the environment.

Azure App Service Linux and serverless runtimes often default to workstation GC because the platform knows the CPU limit. But if you override the GC mode in runtimeconfig.json or an environment variable, you can break that assumption. The trace will show the result: longer suspension windows and more time in GC/SuspendEE than in the actual collection.

Reading a Real Incident: The Case of the 900 ms Pause

A .NET 7 API on Kubernetes started timing out at the 99th percentile. The team collected a trace with dotnet-trace and saw a 900 ms pause every 45 seconds. The average pause was 12 ms. The dashboard showed a healthy process. The trace showed a different story.

The GC/Start events showed a blocking Gen2 collection every 45 seconds, triggered by AllocSmall. The GC/GlobalHeapHistory showed 2.1 GB of Gen2 heap with 1.4 GB of fragmentation. The GC/HeapStats showed 12,000 pinned objects. The pinning came from a logging library that pinned every log message buffer for asynchronous I/O. The fix was to change the logging library’s buffer strategy, not to tune the GC. The trace did not name the library. It named the symptom: 12,000 pins, 1.4 GB of fragmentation, and a 900 ms pause every 45 seconds.

That is the art of reading GC stats. You do not read the number. You read the relationship between the numbers.

Tools and Commands That Produce Usable Stats

You do not need a commercial profiler to read GC stats from a production trace. You need the right event set and the right post-processing.

• dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x1FFFF:5 — the 0x1FFFF mask enables GC, loader, and exception events. The :5 is the verbosity level. Higher verbosity gives more detail but more overhead.
• dotnet-trace convert --format speedscope — converts the trace to a format that shows pause time as a flame graph.
• PerfView on Windows — the GCStats view is the fastest way to see pause time distribution and per-generation sizes.
• dotnet-counters monitor --process-id <pid> --counters System.Runtime — live counters for allocation rate, GC count, and heap size. Not a trace, but a good first signal.

For Linux containers, dotnet-trace uses LTTng under the hood. You need liblttng-ust installed in the container. If it is missing, the trace will be empty. That is a common failure in distroless images. The fix is to add the package or use a sidecar collector.

What the Numbers Do Not Tell You

A trace shows what the GC did. It does not show why the application allocated the objects. It does not show which method created the pinned buffer. It does not show which request path caused the allocation spike. For that, you need a memory profiler with allocation call stacks, or you need to correlate the trace with application logs and distributed traces.

Do not fall into the trap of tuning the GC to fix an allocation problem. If the trace shows high allocation rate and high promotion, the fix is in the code, not in DOTNET_gcServer or DOTNET_GCConserveMemory. The GC is the messenger. The trace is the message. Read it accordingly.

FAQ

What is the difference between allocation rate and heap size in a GC trace?

Allocation rate is the number of bytes allocated per second, measured from GC/AllocationTick events. Heap size is the total managed heap after a collection, measured from GC/HeapStats. A process can have a stable heap size and a very high allocation rate if objects die young. High allocation rate forces frequent Gen0 collections, which cost CPU even if the heap does not grow.

How do I know if a Gen2 collection is blocking or background from a trace?

Look at the GC/Start event’s Type field. A BackgroundGC type means the collection runs concurrently with application threads for most of its duration. A BlockingGC type means all managed threads are suspended for the entire collection. Blocking Gen2 collections are the ones that cause multi-hundred-millisecond pauses. Background Gen2 collections still have a short suspension window at the beginning and end, but the bulk of the work happens concurrently.

Why does my .NET 6 container show high GC pause times when the host has plenty of CPU?

The container likely has a CPU limit that is lower than the host’s CPU count. Server GC spawns one GC thread per logical processor it sees, which may be the host’s count, not the container’s limit. Those threads contend for the limited CPU, which lengthens suspension windows and collection time. Set DOTNET_GCHeapCount or DOTNET_PROCESSOR_COUNT to match the container’s CPU limit, or switch to workstation GC if the workload is latency-sensitive.

What does a high pinned object count mean in a GC trace?

A high PinnedObjectCount in GC/HeapStats means many objects cannot be moved during compaction. Pinned objects fragment the heap, which reduces the effectiveness of Gen2 collections and can lead to a rising heap size even when live data is stable. The usual sources are pinned buffers for native I/O, GCHandle instances that are not freed, and Memory<byte> over pinned arrays. The fix is to reduce pinning, not to increase the heap size.

When a production .NET 6+ service stops responding, leaks memory, or spikes CPU, the first question is not “what changed?” It is “what evidence do we have?” Two tools produce that evidence without a debugger attached: dotnet-dump and dotnet-gcdump. They are part of the .NET diagnostics CLI family, alongside dotnet-trace, dotnet-counters, and dotnet-stack. This article is for engineers who must collect a dump from a Linux container, inspect managed heaps, and decide whether the problem is a leak, a pin, a finalizer queue, or a workload pattern. It assumes you already know that a memory dump is not a profiler and that a GC dump is not a heap dump.

The audience here runs .NET 6+ on Windows, Linux containers, Kubernetes, Azure App Service Linux, and serverless runtimes. The constraints are the same across those environments: no Visual Studio on the box, no interactive debugger, and often no shell history after the pod dies. The workflow that matters is capture, transfer, analyze, and preserve. Everything else is noise.

What dotnet-dump and dotnet-gcdump Actually Capture

dotnet-dump collects a full process dump: managed heap, native heap, thread stacks, module list, and environment. On Linux it uses the runtime’s diagnostics IPC channel, not gcore. On Windows it can produce a minidump or a full dump. The file is not portable across OS families by default; a dump taken on Linux must be analyzed on Linux or with a tool that understands Linux dumps. dotnet-gcdump collects a compact, GC-focused snapshot: object graph, roots, finalization queue, and GC heap segments. It is not a full memory dump. It is smaller, faster, and often enough for managed memory investigations.

Adjacent concepts you will meet immediately: EEHeap, GCHeap, LOH, POH, finalization queue, freachable queue, pinned objects, generation 2, memory pressure, and workstation vs server GC. If those terms are not familiar, the dump will not explain them to you. The tooling assumes you know what a root is and why a pinned byte[] on the LOH behaves differently from a short-lived string.

Installing the Tools Without Breaking the Box

Install as a global tool, not as a project dependency:

dotnet tool install --global dotnet-dump
dotnet tool install --global dotnet-gcdump

On a locked-down production image, install into a sidecar or a diagnostics pod. Do not modify the running application image to add diagnostic tools. On Kubernetes, run a one-shot pod in the same namespace with the same UID and the same /tmp and /diag volume mounts. On Azure App Service Linux, use the SSH console and install into /tmp or a persistent storage mount. On serverless, you usually cannot attach; you must rely on startup hooks or pre-instrumented images.

Version alignment matters. The tool version must be compatible with the runtime version of the target process. A dotnet-dump built for .NET 8 can usually inspect a .NET 6 process, but the reverse is not guaranteed. Check the tool’s --version output and the target process’s runtime version before capture.

Capturing a Full Dump with dotnet-dump

Find the process ID first. On Linux, pgrep -f MyApp or ps aux | grep dotnet. On Windows, tasklist | findstr MyApp. Then:

dotnet-dump collect -p 1234 -o /diag/app_20250321_1410.dmp

Add --type Full on Windows if you need native memory. On Linux the default is a full dump. The process pauses briefly during collection. For a large heap, that pause can be seconds. Do not collect during a rolling deployment or while a health probe is about to fail. If the process is already unresponsive, the pause is irrelevant; collect immediately.

For a crashing process, use dotnet-dump collect --crashreport or configure DOTNET_DbgEnableMiniDump and DOTNET_DbgMiniDumpType to capture automatically on unhandled exceptions. That is the only reliable way to get a dump from a process that dies before you can attach.

Capturing a GC Dump with dotnet-gcdump

dotnet-gcdump collect -p 1234 -o /diag/app_20250321_1410.gcdump

The GC dump is the better first move for managed memory issues. It is smaller by an order of magnitude, it does not require the same pause, and it can be analyzed in PerfView, Visual Studio, or dotnet-gcdump report. If the GC dump does not answer the question, then take a full dump. That order saves time and disk.

One common mistake: taking a GC dump after the process has already been restarted. The GC dump is a point-in-time snapshot. If the pod restarted, the evidence is gone. Automate capture on memory thresholds, not after the incident.

Analyzing a Full Dump with dotnet-dump analyze

Open the dump:

dotnet-dump analyze /diag/app_20250321_1410.dmp

The first commands are always the same:

clrstack
pe
dumpheap -stat

clrstack shows the managed stack of the current thread. pe prints the current exception, if any. dumpheap -stat gives a type-by-type object count and total size. That output is the triage. Look for the type with the largest total size that should not be there. A System.String at 40% of the heap is normal for a logging-heavy service. A MyApp.Cache.CacheEntry at 40% is a finding.

Then narrow the heap:

dumpheap -type MyApp.Cache.CacheEntry

Pick an address and trace roots:

gcroot 00007f8b1c00a1e0

If the root chain ends in a static field, a timer, or an event handler, you have the leak. If it ends in a pinned byte[], you have a pinning problem, not a managed leak. If it ends in the finalization queue, you have a finalizer that is not completing.

For large object heap issues:

dumpheap -stat -type System.Byte[]
dumpheap -type System.Byte[] -min 85000

The LOH threshold is 85,000 bytes. Objects above that are not compacted by default. Repeated LOH allocations and frees fragment the heap. The dump shows the fragmentation, not the cause. The cause is in the allocation pattern, which a GC dump or a trace can reveal.

Analyzing a GC Dump

Open the GC dump in PerfView or Visual Studio. The key views are Heap Stacks, GC Heap Alloc Ignore Free, and Finalizable Objects. Heap Stacks shows the allocation call stacks for the objects still on the heap. That is the direct answer to “what allocated this?” GC Heap Alloc Ignore Free shows the same for all allocations, including freed ones. Finalizable Objects shows objects waiting for finalization.

If the GC dump shows a single call stack responsible for 80% of the heap, the fix is in that code path. If it shows a flat distribution across many call stacks, the problem is likely a cache, a static collection, or a retention policy. The GC dump does not tell you which objects are reachable from where; it tells you what was allocated and by whom. For reachability, you need the full dump and gcroot.

Linux Container Specifics

On Linux, the diagnostics IPC channel is a Unix domain socket in /tmp. If the container runs as non-root and /tmp is not writable, the tools cannot connect. Set DOTNET_DiagnosticPorts or mount a writable /tmp. The default socket name includes the process ID and a random component. If you run the tool in a sidecar, it must share the same /tmp volume.

Memory limits matter. A container with a 512 MB limit and a 400 MB heap will produce a dump larger than the limit. Write the dump to a mounted volume, not the container’s writable layer. If the dump fills the layer, the kubelet will evict the pod and you lose the evidence.

For Kubernetes, the cleanest pattern is a diagnostics DaemonSet or a one-shot Job with hostPID: true and the same /tmp mount. The Job runs dotnet-dump collect against the target PID, writes to a PVC, and exits. The alternative is kubectl exec into the pod, but that requires the tool to be present in the image.

Windows and Azure App Service Linux

On Windows, run the tool from an elevated prompt. The process must be running under the same user or a user with debug privileges. For IIS-hosted apps, the process is w3wp.exe; identify the right instance by the application pool name. For Windows services, the process is the service executable.

On Azure App Service Linux, the SSH console gives you a shell. Install the tools into /tmp, find the dotnet process, and collect. The dump must be downloaded via FTP or the Kudu API before the instance is recycled. App Service recycles instances on configuration changes, scale operations, and platform maintenance. Do not assume the dump will be there tomorrow.

Common Failure Patterns and What They Look Like

Event Handler Leak

The heap is dominated by a type that should be short-lived. gcroot shows a chain ending in a static event. The fix is to unsubscribe or use a weak event pattern. The dump proves the root; it does not fix the code.

Pinned Object Fragmentation

The LOH is fragmented. dumpheap -stat shows many byte[] objects above 85,000 bytes. gcroot shows them pinned by a GCHandle or an async operation. The fix is to pool buffers or use ArrayPool<byte>. The dump shows the pinning; the allocation trace shows the source.

Finalizer Queue Backup

The finalization queue is long. dumpheap -type System.Object with the finalization queue root shows objects waiting. The fix is to implement IDisposable correctly and call Dispose, or to remove the finalizer if it is not needed. The dump shows the queue; the code review shows the missing using.

Large Object Heap Growth Without Leak

The LOH grows but the object count is stable. This is fragmentation, not a leak. The fix is to reduce LOH allocations or enable GCSettings.LargeObjectHeapCompactionMode for a one-time compaction. The dump shows the fragmentation; the GC settings show the tradeoff.

When Not to Use These Tools

If the process is CPU-bound, use dotnet-trace or dotnet-counters first. A dump is a point-in-time snapshot; it may catch the CPU spike or it may not. If the process is leaking slowly, a single dump may not show the trend. Take two dumps an hour apart and compare dumpheap -stat output. If the process is crashing on startup, use DOTNET_DbgEnableMiniDump and collect the crash dump automatically. If the problem is a deadlock, a dump is the right tool; clrstack on all threads shows the wait chain.

Preserving Evidence

Dumps are evidence. Store them with the incident record. Include the runtime version, the OS version, the container image digest, the environment variables, and the exact command used to collect. A dump without that context is a file, not evidence. If the dump contains sensitive data, treat it as sensitive. Do not upload it to a public issue tracker. Do not email it. Use a private storage account with access logging.

FAQ

What is the difference between dotnet-dump and dotnet-gcdump?

dotnet-dump captures a full process dump including native memory, thread stacks, and the managed heap. dotnet-gcdump captures only the GC-relevant data: object graph, roots, finalization queue, and heap segments. The GC dump is smaller and faster, but it cannot answer questions about native memory, thread stacks, or module state.

Can I analyze a Linux dump on Windows?

Not reliably. A dump taken on Linux must be analyzed on Linux or with a tool that explicitly supports Linux dumps. The reverse is also true. If you must analyze on a different OS, use a GC dump, which is platform-neutral, or collect the dump on the same OS family as the analysis machine.

How do I collect a dump from a crashing process?

Set DOTNET_DbgEnableMiniDump=1 and DOTNET_DbgMiniDumpType=4 in the process environment. On an unhandled exception, the runtime writes a dump to the current directory or the path in DOTNET_DbgMiniDumpName. This is the only reliable method for processes that die before you can attach.

Why does my dump show a huge System.String count?

Strings are the most common managed object in a logging-heavy service. A high string count is not automatically a leak. Compare the string count and total size across two dumps taken an hour apart. If the count grows without bound, trace the roots. If it is stable, it is a workload pattern, not a leak.

Next Step for This Site

This article is the first in a series on production .NET diagnostics. The next article will cover dotnet-trace and dotnet-counters for CPU and thread pool investigations. If you have a dump that resists analysis, send the dumpheap -stat output and the gcroot chain for the top type. I will use it as a case study in a future post.

Socket exhaustion is the point where a process can no longer create new outbound or inbound network connections because it has run out of available sockets, ephemeral ports, or file descriptors. In .NET, this usually shows up as SocketException with messages like “Only one usage of each socket address (protocol/network address/port) is normally permitted” or “An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.” Adjacent concepts include ephemeral port range exhaustion, TIME_WAIT accumulation, handle leaks, connection pool starvation, and async socket misuse. For production .NET services handling thousands of requests per second, socket exhaustion is rarely a single smoking gun. It is a slow-burn failure that starts with latency spikes, then intermittent timeouts, then cascading dependency failures. This article is for the engineer who is already seeing those symptoms and needs a methodical path to root cause.

First, Separate Client-Side from Server-Side Exhaustion

Socket exhaustion is not one failure mode. The first diagnostic split is whether the process is failing to accept inbound connections or failing to create outbound connections. Server-side exhaustion usually means the listening socket’s backlog is full or the process has hit its file descriptor limit. Client-side exhaustion usually means the process has consumed the entire ephemeral port range for a given destination IP and port, or it has leaked sockets through undisposed HttpClient instances, raw Socket objects, or broken connection pooling.

In high-throughput .NET services, the most common variant is client-side exhaustion caused by outbound HTTP calls. A service that calls ten downstream APIs per request at 2,000 requests per second is creating 20,000 outbound connections per second at peak. If any of those connections are not returned to the pool, the process burns through the ephemeral port range in minutes.

The Ephemeral Port Range Is a Hard Limit

Windows assigns outbound connections an ephemeral port from a configurable range. The default range on modern Windows Server is roughly 49,152 to 65,535, which gives about 16,384 ports per destination IP and port. Linux uses a different default range, often 32,768 to 60,999, controlled by net.ipv4.ip_local_port_range. When a .NET process opens a socket to a specific remote endpoint, the operating system binds a local ephemeral port. That port cannot be reused for the same remote endpoint until the socket is fully closed and the TIME_WAIT period has elapsed, unless SO_REUSEADDR or connection pooling is in play.

This is not a .NET-specific limit. It is a TCP stack constraint. But .NET applications make it easy to hit because HttpClient, HttpWebRequest, and raw Socket all have different lifetime rules. A single misused HttpClient per request is enough to exhaust the range under load.

Common .NET Causes of Socket Exhaustion

1. HttpClient Created Per Request

The classic failure. Creating a new HttpClient for every outbound call means each instance gets its own connection pool. The pool is discarded when the client is garbage collected, but the underlying sockets are not immediately closed. Under sustained load, the process accumulates sockets in TIME_WAIT or CLOSE_WAIT until the ephemeral port range is empty. The fix is to use a single static or long-lived HttpClient, or better, IHttpClientFactory with named or typed clients.

2. Sockets Not Disposed After Exceptions

Raw Socket and TcpClient usage is still common in high-performance services. If an exception is thrown between Connect and Close, and the code does not use a finally block or using statement, the socket leaks. One leaked socket per failed request is enough to kill a service during a dependency outage, because the failure rate spikes at exactly the moment the service is under the most stress.

3. Connection Pool Starvation

This is not the same as socket exhaustion, but it produces identical symptoms. HttpClient pools connections to the same endpoint. The default limit is ServicePointManager.DefaultConnectionLimit, which is 2 in some legacy .NET Framework configurations and 10 in others. If a service makes 100 concurrent calls to the same downstream API, only 10 connections are available. The other 90 requests queue. The queue grows, timeouts fire, and the service looks like it is out of sockets when it is actually out of pooled connections. In .NET Core and .NET 5+, SocketsHttpHandler.MaxConnectionsPerServer controls this, and the default is int.MaxValue, which is a different problem: unbounded connection growth.

4. Firewall or Load Balancer Idle Timeout Mismatch

A connection pooled by HttpClient can be silently closed by an intermediate network device. The client does not know the connection is dead until it tries to write to it. The write fails, the socket is discarded, and a new one is created. If the idle timeout on the load balancer is shorter than the client’s keep-alive interval, the client is constantly creating new connections. This is not a leak, but it looks like one in the metrics.

Diagnostic Sequence for a Production Incident

When a service starts throwing SocketException under load, do not guess. Run the sequence below in order. Each step either confirms or eliminates a cause.

Step 1: Capture the Exact Exception and Stack Trace

The exception message tells you which side is failing. “Only one usage of each socket address” is client-side ephemeral port exhaustion. “An operation on a socket could not be performed because the system lacked sufficient buffer space” is often a non-paged pool or buffer issue, not a port issue. “No connection could be made because the target machine actively refused it” is a server-side backlog or listener failure. The stack trace tells you which code path is creating the socket. If the stack trace points to HttpClient.SendAsync, the problem is in the outbound call path. If it points to Socket.Accept, the problem is inbound.

Step 2: Check the Process Handle Count

On Windows, use Performance Monitor or Get-Process -Name YourService | Select-Object Handles. On Linux, use ls /proc/<pid>/fd | wc -l. A process that is leaking sockets will show a steadily increasing handle count. A process that is simply hitting the ephemeral port limit may have a stable handle count but a high number of sockets in TIME_WAIT. The distinction matters: handle leak means undisposed objects; TIME_WAIT accumulation means high connection churn.

Step 3: Inspect Socket States

On Windows, netstat -ano | findstr <pid> shows all sockets owned by the process. Count the states. A large number of sockets in CLOSE_WAIT means the remote side closed the connection and the local process never called Close. That is a bug in the .NET code. A large number in TIME_WAIT is normal for high-churn services, but if the count is in the tens of thousands, the service is creating and destroying connections too quickly. On Linux, ss -tanp | grep <pid> gives the same view.

Step 4: Check the Ephemeral Port Range and Current Usage

On Windows, netsh int ipv4 show dynamicport tcp shows the current range. On Linux, cat /proc/sys/net/ipv4/ip_local_port_range. Then compare the number of outbound connections in netstat or ss to the range size. If the count is near the range limit, the service is either leaking sockets or churning connections too fast. Widening the range is a temporary mitigation, not a fix. The fix is to stop the leak or reduce churn.

Step 5: Capture a Dump and Inspect Socket Objects

If the handle count is rising but netstat does not show a matching number of sockets, the leak may be in a different handle type. If the handle count matches the socket count, capture a memory dump with dotnet-dump collect or procdump -ma. Load it in WinDbg or dotnet-dump and run !dumpheap -type System.Net.Sockets.Socket. The number of live Socket objects tells you whether the leak is managed or native. If the managed count is low but the native handle count is high, the leak is in a native library or a P/Invoke path.

Fixing the Root Cause, Not the Symptom

Once the diagnostic sequence identifies the cause, the fix is usually small. The hard part is resisting the urge to widen the ephemeral port range and move on. That buys hours, not days. The real fix is in the code.

Use IHttpClientFactory Correctly

In ASP.NET Core, register IHttpClientFactory and inject it into services. The factory manages handler lifetimes and connection pooling. For typed clients, use AddHttpClient<TClient>(). For ad-hoc calls, use CreateClient() and dispose the client when done. The factory reuses the underlying handler, so disposing the client does not close the pooled connections. This is the single highest-impact change for most .NET services.

Set Explicit Connection Limits

In .NET Core and .NET 5+, configure SocketsHttpHandler.MaxConnectionsPerServer to a value that matches the downstream service’s capacity. If the downstream API can handle 200 concurrent connections, set the limit to 200. This prevents the client from opening unbounded connections during a traffic spike and protects both sides. In .NET Framework, set ServicePointManager.DefaultConnectionLimit early in the process lifetime, before any outbound call is made.

Dispose Sockets in Finally Blocks

For raw socket code, wrap every Connect, Send, and Receive in a try/catch/finally that closes the socket. Better, use using declarations or await using for IAsyncDisposable socket wrappers. A socket that is not closed in an exception path is a leak that will only appear under failure conditions, which is exactly when you cannot afford it.

Align Keep-Alive and Idle Timeouts

If the service sits behind a load balancer or firewall, set the client’s keep-alive interval to less than the network device’s idle timeout. In SocketsHttpHandler, set PooledConnectionIdleTimeout and KeepAlivePingDelay. The goal is for the client to detect dead connections before the network device silently drops them. This reduces the number of failed writes and the resulting connection churn.

Monitoring for Early Warning

Socket exhaustion is predictable if you monitor the right counters. The key metrics are:

• Current outbound connections per process, compared to the ephemeral port range size.
• Socket count by state, especially CLOSE_WAIT and TIME_WAIT.
• Handle count for the process, watched for monotonic increase.
• Connection pool wait time from HttpClient or SocketsHttpHandler telemetry.
• SocketException rate broken down by message and stack trace.

In .NET, System.Net.Http and System.Net.Sockets emit events through EventSource. The System.Net.Http source includes connection pool metrics. The System.Net.Sockets source includes socket connect and accept events. Wire these into your existing telemetry pipeline. The data is already there; most teams just do not collect it.

When Widening the Port Range Is Acceptable

There are legitimate cases where the ephemeral port range is too small for the workload. A service that makes 50,000 outbound connections per minute to a single destination will exhaust a 16,384-port range even with perfect connection pooling, because the pool cannot reuse a connection that is still in TIME_WAIT. In that case, widening the range on Windows with netsh int ipv4 set dynamicport tcp start=1025 num=64510 or on Linux with sysctl -w net.ipv4.ip_local_port_range="1024 65535" is a reasonable mitigation. But do this only after confirming that the connection churn is inherent to the workload, not caused by a leak or a pooling misconfiguration. Otherwise you are masking the bug.

Case Study: The 3 AM Pager Storm

A .NET 6 service handling payment callbacks started throwing SocketException every night at 3 AM. The on-call engineer restarted the service, and the errors stopped for 24 hours. The pattern repeated for a week. The team assumed a nightly batch job was overloading the service. The real cause was a scheduled database maintenance window that closed all idle connections to the downstream payment API. The service’s HttpClient pool held connections that the database server had closed. The next wave of requests tried to write to dead connections, failed, and created new ones. The connection churn spiked, the ephemeral port range filled, and the service started throwing SocketException. The fix was to set PooledConnectionIdleTimeout to 5 minutes and KeepAlivePingDelay to 1 minute. The service now detects dead connections before the write fails, and the 3 AM pager storm stopped.

FAQ

What is the difference between socket exhaustion and connection pool starvation?

Socket exhaustion means the operating system cannot allocate a new socket because the process has used all available ephemeral ports or file descriptors. Connection pool starvation means the HttpClient or SocketsHttpHandler has reached its configured maximum connections per server and is queueing requests. Both produce timeouts and SocketException, but the fixes are different. Pool starvation is fixed by raising the connection limit or reducing concurrency. Socket exhaustion is fixed by eliminating leaks or reducing connection churn.

How do I know if my .NET service is leaking sockets?

Watch the process handle count over time. If it increases monotonically under constant load and never drops, you likely have a leak. Then check netstat -ano | findstr <pid> on Windows or ss -tanp | grep <pid> on Linux. If the number of sockets in CLOSE_WAIT is high and growing, the remote side closed the connection and your code never called Close. That is a managed code bug. If the socket count is stable but the handle count is rising, the leak is in a different handle type.

Can I just increase the ephemeral port range to fix socket exhaustion?

You can, but it is a mitigation, not a fix. If the service is leaking sockets, a wider range only delays the failure. If the service is churning connections too fast, a wider range may hide the problem until traffic grows again. The correct sequence is to diagnose the cause, fix the leak or pooling issue, and then decide whether the workload genuinely needs a wider range. In high-throughput services, a wider range is sometimes necessary even with perfect code, but it should be the last step, not the first.

What is the best way to prevent socket exhaustion in .NET Core and .NET 5+?

Use IHttpClientFactory for all outbound HTTP calls. Configure SocketsHttpHandler.MaxConnectionsPerServer to a value that matches the downstream service’s capacity. Set PooledConnectionIdleTimeout and KeepAlivePingDelay to detect dead connections before writes fail. Dispose raw sockets in finally blocks or with using declarations. Monitor socket count by state and handle count per process. These practices eliminate the vast majority of socket exhaustion incidents in production .NET services.

Next up on this site: a deep look at CLOSE_WAIT vs TIME_WAIT in .NET services, including how to read netstat output without guessing and what each state tells you about the code that owns the socket.

When the Spinner Never Stops

A hang is a silent failure. No crash dump. No exception. No stack trace. Just a frozen UI, a load balancer timeout, and a support queue that won’t stop growing. In production .NET environments, hangs are often the most expensive class of defect. They degrade slowly, slip past traditional logging, and refuse to reproduce on demand. The threadpool is starved. A lock is held forever. An async operation never finishes. The common thread: the application is still running, but it has stopped making progress. This article is about proving what happened during a hang using memory dumps—no guesswork, no restarting the process before you have answers.

What a Hang Actually Means in .NET

A hang is a liveness failure. The process is alive—memory is allocated, threads exist—but one or more critical execution paths are blocked. In .NET, this usually shows up in three patterns: synchronous deadlocks, async/await deadlocks, and threadpool exhaustion. Each leaves a distinct signature in a memory dump. The diagnostic challenge isn’t collecting the dump; it’s knowing which threads to examine, which synchronization primitives to inspect, and how to reconstruct the causal chain from raw bytes to a blocked call stack.

The Windows Debugger (WinDbg) and its cross-platform successor, WinDbg Preview, remain the primary tools for this work. The SOS extension (Son of Strike) provides managed debugging commands that understand the CLR’s internal structures. For Linux and containerized workloads, dotnet-dump and LLDB with SOS offer equivalent capabilities. The workflow is the same: capture a dump during the hang, load it with the correct version of SOS, and interrogate the runtime state.

The Dump Collection Decision

A full memory dump is ideal but often impractical in production—too big, and the suspension time hurts. A minidump with heap is the pragmatic compromise. It contains the thread list, call stacks, and the managed heap. Enough to diagnose the vast majority of hangs. Use ProcDump with the -ma flag for a full dump or -mp for a miniplus dump that includes heap data. The critical flag is -h: trigger on an unresponsive window, or use -tc to trigger when a thread consumes excessive CPU without completing work.

For containerized .NET applications, the dotnet-dump tool is the standard approach. Install it as a global tool and invoke dotnet-dump collect -p PID. The resulting file is cross-platform and can be analyzed on any machine with the matching .NET SDK and SOS extension. Always collect two or three dumps spaced 30 seconds apart. A single dump is a snapshot; multiple dumps reveal movement—or the lack of it.

Pattern 1: The Sync-Block Deadlock

The classic deadlock involves two or more threads each holding a lock the other needs. In .NET, this often involves Monitor.Enter or the lock statement. The dump reveals threads in a WaitSleepJoin state, sitting inside Monitor.Enter or WaitHandle.WaitOne. The SOS command !syncblk lists all sync blocks and their owning threads. Cross-reference with !clrstack to see which thread holds which lock and which lock each thread is waiting to acquire.

A real-world example: a high-traffic ASP.NET application hangs every few days under peak load. Dump analysis shows Thread 12 holding a lock on object A and waiting on object B, while Thread 27 holds B and waits on A. The lock graph is a perfect cycle. The fix is not to increase timeouts; it’s to enforce a consistent lock acquisition order or replace the locking strategy with a non-blocking alternative like ConcurrentDictionary or SemaphoreSlim with WaitAsync.

Async/Await Deadlocks: The SynchronizationContext Trap

This pattern is well-documented but still pervasive. An asynchronous method awaits a task, and the continuation attempts to resume on a captured SynchronizationContext that is blocked by a synchronous call further up the stack. The classic example: calling Task.Result or Task.Wait() on an async method from a UI thread or an ASP.NET request context. The dump shows a thread blocked on WaitHandle.WaitOne inside Task.Result, while the task it is waiting for has completed but cannot resume because the same thread is blocked.

In the dump, use !dumpheap -type Task to find the relevant task objects, then !mdt to inspect their state. A task stuck in WaitingForActivation with a non-null m_continuationObject that points to a StandardTaskContinuation is a strong indicator. The !dso command on the blocked thread reveals the captured SynchronizationContext. The fix is architectural: use ConfigureAwait(false) in library code, and never block on async code from a context-sensitive thread.

Pattern 2: Threadpool Starvation

Threadpool starvation is insidious because it mimics a deadlock but has a different root cause. The threadpool has a limited injection rate; if all threads are busy with long-running or blocking work, new work items queue up indefinitely. The application appears hung because no thread is available to process incoming requests or scheduled continuations. This is common in ASP.NET Core when synchronous blocking calls are made on threadpool threads, or when fire-and-forget tasks consume all available workers.

In a dump, start with !threadpool to see the number of active threads, the queue length, and the completion port status. A high number of work items in the queue with all threads busy is a red flag. Use !dumpheap -type ThreadPoolWorkQueue to inspect queued items. The !clrstack command on multiple threads often reveals a common pattern: many threads blocked on I/O or locks, preventing the threadpool from injecting new threads fast enough. The solution is to identify and remove the blocking calls, or to use dedicated threads for long-running work instead of consuming threadpool resources.

Pattern 3: The Silent Async Halt

Not all hangs involve locks. A fire-and-forget async operation that fails silently can leave the application in an inconsistent state. The UI is waiting for a completion signal that never arrives. A message is never dequeued. A timer callback stops firing. These hangs are harder to spot because no thread is visibly blocked. Instead, the application simply stops doing the next thing.

In the dump, look for Task objects with a Status of Faulted or WaitingForActivation that are not being observed. Use !dumpheap -type Task and filter for those with a non-zero m_stateFlags indicating a fault. The !pe command on the task’s exception holder reveals the swallowed exception. The fix is to ensure all fire-and-forget tasks have a continuation that logs failures, or to use a library like System.Threading.Channels for producer-consumer patterns that surface errors explicitly.

Proving Causality with Multiple Dumps

A single dump is a photograph; two dumps are a short film. When a hang is intermittent or the root cause is ambiguous, collect two or three dumps 30-60 seconds apart. Compare the thread stacks. Threads that are making progress will show different call stacks or different instruction pointers. Threads that are truly hung will be frozen in the same location. This technique is especially useful for distinguishing a deadlock from a slow-running operation. If the same thread is stuck in WaitForMultipleObjects across all dumps, you have a blocking problem. If it moves, you have a performance problem.

For async hangs, compare the state of suspect Task objects across dumps. A task that remains in WaitingForActivation across multiple snapshots is a strong candidate for the root cause. Use the task ID to track it. The !dumpheap output includes the address; use that address in subsequent dumps to confirm the task has not transitioned.

Common Diagnostic Commands Cheat Sheet

These are the commands I reach for first in any hang investigation. They assume you have already loaded the correct version of SOS (.loadby sos coreclr for .NET Core, .loadby sos clr for .NET Framework).

• !threads — Lists all managed threads, their OS IDs, and their current apartment state. Look for threads with a non-zero Lock Count.
• !syncblk — Shows all Monitor locks. The owner thread ID and the number of waiters tell you where contention lives.
• !dumpheap -type Task — Enumerates all Task objects. Combine with -mt to filter by MethodTable for performance.
• !mdt <address> — Dumps the managed object at the given address. Use on Task objects to see their status, exception, and continuation.
• !clrstack -a — Shows the managed call stack with parameter and local variable values. Essential for understanding why a thread is blocked.
• !dso — Dumps all managed objects referenced from the current thread’s stack. Reveals captured SynchronizationContexts and other root objects.
• !threadpool — Displays threadpool statistics, including active threads, queue length, and completion port threads.

FAQ

Why does my application hang only under heavy load?

Heavy load exposes threadpool starvation and lock contention. Under light load, threads are available to process work quickly, and lock contention is rare. As load increases, the threadpool may become saturated, causing work items to queue. If those queued items hold locks that other threads need, a deadlock can form. Additionally, the threadpool’s hill-climbing algorithm may not inject new threads fast enough to keep up with demand, especially if existing threads are blocked on I/O or locks. Use !threadpool to check queue depth and active threads during the hang.

How do I capture a dump when the application is hung but not crashed?

For Windows, ProcDump is the standard tool. Use procdump -ma -h <PID> to capture a full dump when the target process’s window is hung. For a console or service application, use procdump -ma <PID> and trigger it manually, or use the -tc flag to trigger on a specific thread consuming CPU without completing. For Linux containers, use the dotnet-dump global tool: dotnet-dump collect -p <PID>. Always collect at least two dumps to confirm the hang is persistent.

What is the difference between a deadlock and a hang?

A deadlock is a specific type of hang where two or more threads are each waiting for a resource held by another, creating a cycle. A hang is a broader term: the application is unresponsive, but the cause could be a deadlock, threadpool starvation, an infinite loop, or a blocked async operation. Deadlocks are a subset of hangs. Dump analysis can distinguish them: a deadlock shows threads waiting on locks held by each other, while a hang from threadpool starvation shows all threads busy or waiting with no available workers.

Can I prevent hangs by using async/await everywhere?

Async/await reduces the risk of threadpool starvation by not blocking threads during I/O, but it introduces its own class of hangs: async deadlocks from captured SynchronizationContext and fire-and-forget tasks that fail silently. Async code must still be written carefully. Use ConfigureAwait(false) in library code, avoid async void except for event handlers, and always observe task exceptions. Async is not a silver bullet; it changes the failure modes.

Building a Hang-Resilient Diagnostic Practice

The ability to prove a hang with dumps is a skill that compounds. Each investigation teaches you a new failure signature. You start to recognize the shape of a sync-block deadlock from the thread list alone. You develop a library of scripts and breakpoints. More importantly, you start designing systems that fail loudly: timeouts that throw, health checks that detect stalled pipelines, and structured logging that captures the state of synchronization primitives. The dump is your last resort, but it should never be your first surprise.

The next time your application hangs, resist the urge to restart it. Capture the evidence. The dump contains the truth, and with the right technique, you can extract it.

The Silent Killer of Production Systems

Let’s be honest: a hung application is worse than a crash. A crash is loud. It leaves a corpse you can autopsy. A hang is a zombie—still breathing, still burning resources, but utterly brain-dead. Your users stare at a spinning wheel. Your monitoring dashboard shows a flatline. No exceptions. No logs. Just silence. In the .NET world, a hang almost always comes down to one of two things: threads that are blocked waiting for something that will never happen, or threads that are running in a circle they can’t escape. This guide is about cutting through the noise and finding the smoking gun in a memory dump.

Capturing the State of a Hung Process

You can’t fix what you can’t see. A single memory dump is a snapshot, but a hang is a story about time—specifically, the lack of progress over time. You need at least two dumps, several seconds apart, to prove that nothing is moving. For a CPU-bound hang, where the process is pegged at 90% or higher but not actually finishing work, ProcDump is your best friend. The command below grabs three dumps, ten seconds apart, when the CPU threshold is breached for five consecutive seconds. That gives you a timeline of the spinning thread.

procdump -ma -c 90 -s 5 -n 3 -o w3wp.exe

For a quiet hang—low CPU, but requests are piling up—a manual dump or a simple count-based trigger works fine. The rule is simple: capture at least two dumps. A single dump is a photograph. A series is a short film. You need the film to see who’s stuck and who’s still moving.

Proving a Deadlock: The Wait-Chain Trail

Deadlocks are oddly satisfying to diagnose because the evidence is absolute. Two threads, each holding a lock the other wants. In managed code, the monitor lock is the usual suspect, but you’ll also see deadlocks on reader-writer locks or even hybrid messes involving unmanaged sync objects. Your first move in WinDbg is the SOS extension’s !dlk command (or !deadlock in some versions).

Run !dlk against your dump. If it spits out a deadlock cycle, you’re halfway home. The output names the threads, the objects they’re waiting on, and the threads holding those objects. Thread A holds lock X and wants lock Y; Thread B holds lock Y and wants lock X. It’s right there in black and white. But what if !dlk comes up empty? The hang is still real. Now you trace the wait chain by hand. Look for threads in a WaitSleepJoin state. Dump the object they’re waiting on with !do and check its sync block index. Then use !syncblk to find the owner thread. If that owner is also waiting, follow the chain. A cycle is a deadlock. A long chain that ends in a thread doing unmanaged I/O or waiting on a GC is a different animal—a resource contention hang, which we’ll get to later.

Thread Pool Starvation: The Silent Throttle

Most hangs aren’t deadlocks. In high-throughput ASP.NET apps, the real villain is thread pool starvation. The .NET thread pool has a fixed number of threads to process work. When all of them are busy, new work gets queued. If those busy threads are themselves waiting on something that needs a thread pool thread to complete—the classic sync-over-async antipattern—the queue backs up forever. Requests time out. The app looks dead.

To prove starvation, check the pool’s internal state. The SOS command !threadpool gives you the high-level view. Look at the “Work Request in Queue” counter. If it’s in the thousands and not dropping across your dump series, you’ve got a problem. Next, find the culprits. Use !eestack -ee to list all managed threads and their call stacks. Filter for threads that are running but not completing. In an ASP.NET scenario, you’ll often see dozens of threads stuck inside Task.Result, Task.Wait(), or .GetAwaiter().GetResult(). These are blocking calls on thread pool threads. The fix is to make the whole call chain async, but the immediate diagnostic proof is the combination of a growing queue and a saturated pool of blocked threads.

Finalizer and GC Hangs: When Cleanup Blocks Everything

Here’s a less obvious but devastating scenario: the finalizer thread gets stuck. The .NET runtime has exactly one finalizer thread. If it blocks—maybe waiting on a lock, or doing a blocking I/O call inside a finalizer—the entire finalization queue stalls. The GC can’t reclaim objects that are ready for finalization until that thread runs, so memory pressure builds. Eventually, every thread that tries to allocate memory gets blocked waiting for a GC, which is itself waiting for the finalizer thread. The whole app hangs.

To spot this, first check the finalizer thread’s state. Use !threads and find the thread marked “(Finalizer).” Note its OS thread ID, switch to it, and examine its call stack with !clrstack. If it’s stuck in a WaitSleepJoin state, you’ve found the bottleneck. Next, check the finalization queue with !finalizequeue. A large number of objects “Ready for finalization” confirms the pressure. The root cause is the code inside the finalizer of the object at the head of the queue. This is a design flaw: finalizers must never block. The proof is in the dump: a single blocked thread, a growing queue, and a process-wide stall.

CPU-Bound Hangs: The Infinite Loop

Not every hang is a waiting game. A thread stuck in an infinite loop will eat 100% of a CPU core and never yield. The process might seem responsive if other cores are free, but if the loop is in a critical path or you have multiple such threads, the application grinds to a halt. The diagnostic approach here is different. You’re not looking for wait reasons; you’re looking for the thread that’s running and never changes its instruction pointer.

Capture a series of three to five dumps a few seconds apart. In each dump, run !runaway to identify the thread consuming the most CPU time. Note its OS thread ID. Then, in each dump, switch to that thread and examine its managed call stack with !clrstack. If the top frames are identical across all dumps, you’ve found your infinite loop. The next step is to examine the local variables and the loop condition to understand why it never exits. This often comes down to a subtle bug in a while loop or a recursive method that never reaches its base case.

Practical Dump Analysis Workflow

When you’re paged at 3 a.m. for a hung production server, you need a repeatable, efficient workflow. Here’s the sequence I follow, refined over hundreds of incidents:

1. Capture the right data. Use ProcDump to take at least two full dumps 10–15 seconds apart. If the process is using high CPU, use the CPU threshold trigger. Otherwise, a simple procdump -ma -n 2 -s 15 w3wp.exe will do.
2. Open the first dump in WinDbg. Load SOS with .loadby sos clr (or .loadby sos coreclr for .NET Core). Set the symbol path to the public Microsoft symbol server.
3. Run !dlk. If it finds a deadlock, you’re done. Identify the involved locks and the owning threads. Correlate with source code.
4. If no deadlock, run !threadpool. Check the work queue depth. If it’s high and growing, you likely have thread pool starvation. Use !eestack -ee to find the blocking calls.
5. Check the finalizer thread. Use !threads to find it, then !clrstack to see what it’s doing. If it’s blocked, check the finalization queue with !finalizequeue.
6. If CPU is high, use !runaway. Find the top CPU consumer and check its stack across multiple dumps for a repeating pattern.
7. Correlate with the second dump. Confirm that the problematic threads are still in the same state. A hang is defined by a lack of progress.

This workflow covers the vast majority of production hangs. The key is to move quickly from the general (is it a deadlock, starvation, or a CPU loop?) to the specific (which lock, which method, which line of code?).

FAQ: Common Questions About Hang Analysis

Why didn’t my application log any errors during the hang?

Hangs are not exceptions. A deadlocked thread isn’t throwing; it’s waiting. A starved thread pool isn’t failing; it’s queuing. The application is in a state of suspended animation, not a crash. That’s why memory dumps are essential—they’re the only way to observe the internal state of the runtime when no external signals are being emitted. Your logging framework is likely also blocked, waiting for a thread to write the log entry.

Can I use a tool other than WinDbg to analyze these dumps?

Yes, but with tradeoffs. Visual Studio’s memory dump analyzer can open managed dumps and has a friendlier interface for inspecting threads and call stacks. However, it lacks the specialized SOS commands like !dlk and !syncblk that make deadlock detection trivial. For quick, targeted analysis, WinDbg with SOS remains the most powerful option. For a more guided experience, consider the Debug Diagnostics Tool (DebugDiag) from Microsoft, which can automate hang analysis and generate a report identifying common patterns like deadlocks and finalizer hangs.

What if the hang is intermittent and I cannot capture a dump at the right moment?

Intermittent hangs are the hardest to diagnose. You need to set up a proactive monitoring strategy. Use ProcDump’s -tc (thread count) trigger to capture a dump when the number of threads exceeds a healthy baseline, which often correlates with thread pool starvation. Alternatively, use the -h (hang) trigger with a watchdog timer if your application has a health-check endpoint. The goal is to automate dump collection so you’re not relying on manual intervention during a transient event.

How do I differentiate between a managed deadlock and an unmanaged one?

The !dlk command only detects deadlocks involving managed monitor locks. If your application uses unmanaged synchronization objects like Mutex, Event, or Semaphore via P/Invoke, or if the deadlock involves a mix of managed and unmanaged code, you’ll need to use the native debugging commands. Switch to the thread of interest and use k to view the native call stack. Look for calls into WaitForSingleObject or WaitForMultipleObjects. Use !handle to inspect the handle being waited on. This is a more manual process but follows the same logical chain: find who is waiting, what they’re waiting for, and who holds that resource.

Next Steps: Building a Diagnostic Runbook

This article has given you the forensic techniques to prove a hang. The next step is to integrate this knowledge into your team’s operational practices. Create a runbook that maps specific symptoms (high CPU, zero CPU, growing request queues) to the appropriate ProcDump triggers and WinDbg commands. Pre-configure your symbol server access and ensure every developer has a local cache. The goal is to reduce the time from “the site is down” to “here is the offending line of code” to under fifteen minutes. In a future article, we’ll tackle the related problem of high memory pressure and how to use dump analysis to identify memory leaks before they cause an outage.

When a production .NET app keels over with an access violation or locks up with a deadlocked thread, the stack trace is usually the first—and sometimes the only—clue you get. But a stack trace is a summary. It tells you where execution stopped, not why the stack looked the way it did when the process fell apart. To move from symptom to root cause, you need to understand the physical layout of a managed thread’s stack, how the CLR carves out space for frames, and what the runtime does when things go sideways. This article dissects the stack segment, frame construction, and the interplay between managed and unmanaged code on the same call stack. If you’re staring at a crash dump right now, this is the mental model you need.

What a Thread Stack Actually Is

A thread stack is a contiguous range of virtual memory the operating system reserves when a thread is created. On Windows, the default stack size for a managed thread is 1 MB, though the CLR may tweak this for its own internal threads. The stack grows downward from higher addresses to lower addresses. The top of the stack—the lowest address currently in use—is pointed to by the stack pointer register (ESP on x86, RSP on x64). The base of the current frame is held in the base pointer register (EBP or RBP), though frame-pointer omission (FPO) can muddy this in optimized code.

In a managed process, the stack holds a mix of native frames (CLR internals, OS transitions, P/Invoke calls) and managed frames (your C# methods). The CLR’s just-in-time (JIT) compiler emits code that respects the standard calling conventions of the platform, but it also injects metadata and unwind information the runtime uses for garbage collection and exception handling. When you open a memory dump in WinDbg with the SOS extension, the !clrstack command reconstructs the managed call stack by walking this metadata. The native stack, seen with the k command, shows the raw return addresses. The two views are complementary, and discrepancies between them often point straight to the root cause of a crash.

Stack Frame Anatomy

Each method call creates a stack frame. On x64, the frame typically contains the return address, the saved base pointer (if frame pointers are enabled), any callee-saved registers, local variables, and space for outgoing arguments. The CLR’s JIT compiler may omit the base pointer for leaf frames, using only the stack pointer with offsets to reference locals. This optimization shrinks frame size but makes manual stack reconstruction in a debugger more of a headache. When you run !clrstack -a in WinDbg, the SOS extension reads the JIT’s unwind data to locate managed locals and arguments, bypassing the raw pointer arithmetic.

For crash analysis, the critical detail is the boundary between managed and unmanaged frames. A managed method that calls into native code via P/Invoke or COM interop will have a managed frame sitting on top of a native frame. If the native code corrupts the stack—by overflowing a buffer or writing to a stray pointer—the managed frame’s return address can be overwritten. When the CPU tries to return from the managed method, it jumps to an invalid address, producing an access violation. The dump’s exception record will show the faulting instruction pointer, but the stack trace may be truncated or nonsensical. You need to manually inspect the raw stack memory to find the transition point.

Managed-to-Unmanaged Transitions

The CLR uses a mechanism called “stack crawling” to report managed frames. When a thread transitions from managed to native code, the runtime records the managed frame’s location so the garbage collector can find roots. This record is stored in the thread’s Thread object and in the stack itself. The transition sequence on x64 typically looks like this:

1. Managed code calls a P/Invoke stub generated by the CLR.
2. The stub marshals arguments, sets up the native calling convention, and calls the target native function.
3. Upon return, the stub unmarshals any out parameters and restores the managed context.

If the native function never returns—because it hangs, crashes, or corrupts the stack—the managed debugger commands may fail to walk past the transition. In that case, use !dumpstack to see the raw stack contents, then manually resolve symbols with !ip2md or ln. Look for the InlinedCallFrame or HelperMethodFrame structures that the CLR embeds on the stack. These frames contain the managed return address and the MethodDesc pointer, which you can use to identify the calling managed method even when !clrstack fails.

Stack Corruption Patterns

Stack corruption in .NET applications usually falls into one of three categories: buffer overruns in unsafe code, mismatched calling conventions in P/Invoke declarations, or premature garbage collection of delegates passed to native code. Each leaves a distinct signature on the stack.

Buffer Overruns in Unsafe Code

When you use unsafe blocks or stackalloc, the JIT allocates space on the stack for your buffers. A write past the end of a stackalloc buffer will overwrite adjacent stack slots—often the return address of the current method. The result is a crash when the method attempts to return, with the instruction pointer pointing to garbage. In the dump, you will see the faulting instruction address is not in any loaded module. Use !analyze -v to get the exception context, then dump the raw stack around the stack pointer. If you see recognizable data patterns (like strings or repeated bytes) in the return address slot, you have found the corruption source.

To confirm, check the managed method that owns the frame. Use !clrstack -a on the thread that crashed, if possible, or examine neighboring threads that may have corrupted this one. Look for methods that use stackalloc or call native functions with unsafe buffers. The corrupted return address often points into the heap or to an unmapped page, which is a strong indicator of a buffer overflow.

P/Invoke Mismatches

A mismatched P/Invoke signature—wrong calling convention, incorrect parameter size, or missing [Out] attribute—can cause the native function to write beyond its allotted stack space. On x64, the first four parameters are passed in registers, but any additional parameters go on the stack. If the managed declaration specifies fewer parameters than the native function expects, the native code may read garbage from the stack. If it specifies more, the native code may overwrite the caller’s local variables or return address.

To diagnose this, compare the managed P/Invoke signature with the native function’s documentation. Pay attention to the CallingConvention field. On x64, the default is Winapi, which maps to the Microsoft x64 calling convention. Mismatches here can cause stack imbalance. After the call, the CLR may try to unwind the stack using incorrect metadata, leading to a corrupted stack pointer and a subsequent crash in unrelated code. In the dump, you will often see the crash occur in a method far removed from the actual P/Invoke call, because the stack corruption went undetected until the method returned.

Delegate Lifetimes and Native Callbacks

When you pass a managed delegate to native code via P/Invoke, the CLR creates a thunk—a small native-to-managed transition stub—and pins the delegate to prevent garbage collection. If the delegate is collected prematurely, the thunk becomes a dangling pointer. The next time the native code invokes the callback, the process crashes with an access violation in the thunk. The stack trace will show a transition from native code directly into unmapped memory, with no managed frames above it.

To prevent this, ensure the delegate is rooted for the entire duration of the native call. Use GCHandle.Alloc to pin the delegate if the native code stores the callback asynchronously. In the dump, you can identify this pattern by the absence of a managed frame above the native-to-managed transition. The !clrstack command will show a truncated stack, while !dumpstack reveals a native frame calling into an address that is not a valid managed method. Use !u to disassemble the faulting address; if it looks like a thunk but the surrounding memory is freed, you have a collected delegate.

Reading the Stack in a Dump

When the managed debugger commands fail, you need to go raw. Start with !threads to identify the crashing thread, then switch to it with ~<thread>s. Dump the native stack using k or kb to see return addresses. If the stack is completely corrupted, use dps @rsp L200 to dump 200 pointer-sized values from the stack pointer. Look for addresses that resolve to managed code (use !ip2md on each) or to known CLR helper functions. This manual reconstruction can reveal the sequence of calls leading to the crash.

Pay attention to the RBP chain. On x64, the base pointer is not always used as a frame pointer, but when it is, you can follow the chain of saved RBP values to walk the stack manually. Each frame’s saved RBP points to the previous frame’s RBP, and the return address is stored just above it. This technique is fragile but can be the only way to reconstruct a stack when unwind data is missing or corrupted.

Stack Guard Pages and StackOverflowException

The CLR places a guard page at the end of each thread’s stack. When the stack grows into this page, the OS raises a guard-page exception, which the CLR translates into a StackOverflowException. This exception cannot be caught in managed code because the stack is exhausted—there is no room to push a handler frame. The process terminates immediately. In the dump, you will see the thread’s stack pointer very close to the stack limit, and the call stack will show deep recursion or an infinite loop.

Diagnosing a StackOverflowException requires examining the raw stack memory. Use !threads to find the thread with the exception, then !pe to view the exception details. The managed stack may be unavailable, but the native stack often contains repeated calls to the same method, indicating recursion. Check the method’s local variable sizes; large structs or arrays allocated on the stack can accelerate exhaustion. The fix is either to convert the recursion to iteration or to move large locals to the heap.

FAQ

Why does !clrstack show fewer frames than the native stack?

The CLR stack walker only reports managed frames and the native transitions between them. Native frames that do not involve managed code—such as OS kernel transitions, CLR internal helper functions, or frames from other runtimes loaded in-process—are omitted. If you see a native frame that you expect to have a managed counterpart, the JIT may have inlined the managed method, or the unwind information may be missing. Use !dumpstack to see the full raw stack and cross-reference addresses with !ip2md to identify managed code.

How can I tell if a crash is caused by stack corruption versus heap corruption?

Stack corruption typically manifests as an access violation with the instruction pointer pointing to an invalid or unmapped address, often during a return instruction. The call stack may be truncated or contain nonsensical frames. Heap corruption, by contrast, usually causes a crash inside the heap manager (e.g., ntdll!RtlpHeapCorruption) or during a subsequent allocation/free. If the faulting instruction is a ret and the stack pointer points to an address that looks like data rather than code, suspect stack corruption. Use !analyze -v to check the exception context and the raw stack contents.

What tools beyond WinDbg can I use to inspect the managed stack?

For live debugging, Visual Studio’s diagnostic tools and the SOS extension for Visual Studio provide a more accessible interface to the same CLR debugging APIs. For post-mortem analysis, the dotnet-dump tool offers cross-platform support and a subset of SOS commands. On Linux, LLDB with the SOS plugin can analyze .NET Core dumps. Each tool has limitations in stack reconstruction, especially for optimized or partially corrupted stacks, so familiarity with raw stack inspection in WinDbg remains essential for production crash analysis.

How does the CLR handle stack walking for dynamically emitted code?

Dynamically generated methods, such as those created via System.Reflection.Emit or expression trees, have associated unwind data registered with the CLR’s code manager. The stack walker uses this data just as it does for JIT-compiled methods. However, if the dynamic assembly is collected or the unwind data is not properly registered, !clrstack may show a broken frame or fail to walk past it. In such cases, the raw stack will show the transition, and you can use !dumpmt on the method table pointer found in the frame to identify the dynamic type.