Why Your Application Hangs and How to Prove It With Dumps

When the Spinner Never Stops

A hang is a silent failure. No crash dump. No exception. No stack trace. Just a frozen UI, a load balancer timeout, and a support queue that won’t stop growing. In production .NET environments, hangs are often the most expensive class of defect. They degrade slowly, slip past traditional logging, and refuse to reproduce on demand. The threadpool is starved. A lock is held forever. An async operation never finishes. The common thread: the application is still running, but it has stopped making progress. This article is about proving what happened during a hang using memory dumps—no guesswork, no restarting the process before you have answers.

What a Hang Actually Means in .NET

A hang is a liveness failure. The process is alive—memory is allocated, threads exist—but one or more critical execution paths are blocked. In .NET, this usually shows up in three patterns: synchronous deadlocks, async/await deadlocks, and threadpool exhaustion. Each leaves a distinct signature in a memory dump. The diagnostic challenge isn’t collecting the dump; it’s knowing which threads to examine, which synchronization primitives to inspect, and how to reconstruct the causal chain from raw bytes to a blocked call stack.

The Windows Debugger (WinDbg) and its cross-platform successor, WinDbg Preview, remain the primary tools for this work. The SOS extension (Son of Strike) provides managed debugging commands that understand the CLR’s internal structures. For Linux and containerized workloads, dotnet-dump and LLDB with SOS offer equivalent capabilities. The workflow is the same: capture a dump during the hang, load it with the correct version of SOS, and interrogate the runtime state.

The Dump Collection Decision

A full memory dump is ideal but often impractical in production—too big, and the suspension time hurts. A minidump with heap is the pragmatic compromise. It contains the thread list, call stacks, and the managed heap. Enough to diagnose the vast majority of hangs. Use ProcDump with the -ma flag for a full dump or -mp for a miniplus dump that includes heap data. The critical flag is -h: trigger on an unresponsive window, or use -tc to trigger when a thread consumes excessive CPU without completing work.

For containerized .NET applications, the dotnet-dump tool is the standard approach. Install it as a global tool and invoke dotnet-dump collect -p PID. The resulting file is cross-platform and can be analyzed on any machine with the matching .NET SDK and SOS extension. Always collect two or three dumps spaced 30 seconds apart. A single dump is a snapshot; multiple dumps reveal movement—or the lack of it.

Pattern 1: The Sync-Block Deadlock

The classic deadlock involves two or more threads each holding a lock the other needs. In .NET, this often involves Monitor.Enter or the lock statement. The dump reveals threads in a WaitSleepJoin state, sitting inside Monitor.Enter or WaitHandle.WaitOne. The SOS command !syncblk lists all sync blocks and their owning threads. Cross-reference with !clrstack to see which thread holds which lock and which lock each thread is waiting to acquire.

A real-world example: a high-traffic ASP.NET application hangs every few days under peak load. Dump analysis shows Thread 12 holding a lock on object A and waiting on object B, while Thread 27 holds B and waits on A. The lock graph is a perfect cycle. The fix is not to increase timeouts; it’s to enforce a consistent lock acquisition order or replace the locking strategy with a non-blocking alternative like ConcurrentDictionary or SemaphoreSlim with WaitAsync.

Async/Await Deadlocks: The SynchronizationContext Trap

This pattern is well-documented but still pervasive. An asynchronous method awaits a task, and the continuation attempts to resume on a captured SynchronizationContext that is blocked by a synchronous call further up the stack. The classic example: calling Task.Result or Task.Wait() on an async method from a UI thread or an ASP.NET request context. The dump shows a thread blocked on WaitHandle.WaitOne inside Task.Result, while the task it is waiting for has completed but cannot resume because the same thread is blocked.

In the dump, use !dumpheap -type Task to find the relevant task objects, then !mdt to inspect their state. A task stuck in WaitingForActivation with a non-null m_continuationObject that points to a StandardTaskContinuation is a strong indicator. The !dso command on the blocked thread reveals the captured SynchronizationContext. The fix is architectural: use ConfigureAwait(false) in library code, and never block on async code from a context-sensitive thread.

Pattern 2: Threadpool Starvation

Threadpool starvation is insidious because it mimics a deadlock but has a different root cause. The threadpool has a limited injection rate; if all threads are busy with long-running or blocking work, new work items queue up indefinitely. The application appears hung because no thread is available to process incoming requests or scheduled continuations. This is common in ASP.NET Core when synchronous blocking calls are made on threadpool threads, or when fire-and-forget tasks consume all available workers.

In a dump, start with !threadpool to see the number of active threads, the queue length, and the completion port status. A high number of work items in the queue with all threads busy is a red flag. Use !dumpheap -type ThreadPoolWorkQueue to inspect queued items. The !clrstack command on multiple threads often reveals a common pattern: many threads blocked on I/O or locks, preventing the threadpool from injecting new threads fast enough. The solution is to identify and remove the blocking calls, or to use dedicated threads for long-running work instead of consuming threadpool resources.

Pattern 3: The Silent Async Halt

Not all hangs involve locks. A fire-and-forget async operation that fails silently can leave the application in an inconsistent state. The UI is waiting for a completion signal that never arrives. A message is never dequeued. A timer callback stops firing. These hangs are harder to spot because no thread is visibly blocked. Instead, the application simply stops doing the next thing.

In the dump, look for Task objects with a Status of Faulted or WaitingForActivation that are not being observed. Use !dumpheap -type Task and filter for those with a non-zero m_stateFlags indicating a fault. The !pe command on the task’s exception holder reveals the swallowed exception. The fix is to ensure all fire-and-forget tasks have a continuation that logs failures, or to use a library like System.Threading.Channels for producer-consumer patterns that surface errors explicitly.

Proving Causality with Multiple Dumps

A single dump is a photograph; two dumps are a short film. When a hang is intermittent or the root cause is ambiguous, collect two or three dumps 30-60 seconds apart. Compare the thread stacks. Threads that are making progress will show different call stacks or different instruction pointers. Threads that are truly hung will be frozen in the same location. This technique is especially useful for distinguishing a deadlock from a slow-running operation. If the same thread is stuck in WaitForMultipleObjects across all dumps, you have a blocking problem. If it moves, you have a performance problem.

For async hangs, compare the state of suspect Task objects across dumps. A task that remains in WaitingForActivation across multiple snapshots is a strong candidate for the root cause. Use the task ID to track it. The !dumpheap output includes the address; use that address in subsequent dumps to confirm the task has not transitioned.

Common Diagnostic Commands Cheat Sheet

These are the commands I reach for first in any hang investigation. They assume you have already loaded the correct version of SOS (.loadby sos coreclr for .NET Core, .loadby sos clr for .NET Framework).

• !threads — Lists all managed threads, their OS IDs, and their current apartment state. Look for threads with a non-zero Lock Count.
• !syncblk — Shows all Monitor locks. The owner thread ID and the number of waiters tell you where contention lives.
• !dumpheap -type Task — Enumerates all Task objects. Combine with -mt to filter by MethodTable for performance.
• !mdt <address> — Dumps the managed object at the given address. Use on Task objects to see their status, exception, and continuation.
• !clrstack -a — Shows the managed call stack with parameter and local variable values. Essential for understanding why a thread is blocked.
• !dso — Dumps all managed objects referenced from the current thread’s stack. Reveals captured SynchronizationContexts and other root objects.
• !threadpool — Displays threadpool statistics, including active threads, queue length, and completion port threads.

FAQ

Why does my application hang only under heavy load?

Heavy load exposes threadpool starvation and lock contention. Under light load, threads are available to process work quickly, and lock contention is rare. As load increases, the threadpool may become saturated, causing work items to queue. If those queued items hold locks that other threads need, a deadlock can form. Additionally, the threadpool’s hill-climbing algorithm may not inject new threads fast enough to keep up with demand, especially if existing threads are blocked on I/O or locks. Use !threadpool to check queue depth and active threads during the hang.

How do I capture a dump when the application is hung but not crashed?

For Windows, ProcDump is the standard tool. Use procdump -ma -h <PID> to capture a full dump when the target process’s window is hung. For a console or service application, use procdump -ma <PID> and trigger it manually, or use the -tc flag to trigger on a specific thread consuming CPU without completing. For Linux containers, use the dotnet-dump global tool: dotnet-dump collect -p <PID>. Always collect at least two dumps to confirm the hang is persistent.

What is the difference between a deadlock and a hang?

A deadlock is a specific type of hang where two or more threads are each waiting for a resource held by another, creating a cycle. A hang is a broader term: the application is unresponsive, but the cause could be a deadlock, threadpool starvation, an infinite loop, or a blocked async operation. Deadlocks are a subset of hangs. Dump analysis can distinguish them: a deadlock shows threads waiting on locks held by each other, while a hang from threadpool starvation shows all threads busy or waiting with no available workers.

Can I prevent hangs by using async/await everywhere?

Async/await reduces the risk of threadpool starvation by not blocking threads during I/O, but it introduces its own class of hangs: async deadlocks from captured SynchronizationContext and fire-and-forget tasks that fail silently. Async code must still be written carefully. Use ConfigureAwait(false) in library code, avoid async void except for event handlers, and always observe task exceptions. Async is not a silver bullet; it changes the failure modes.

Building a Hang-Resilient Diagnostic Practice

The ability to prove a hang with dumps is a skill that compounds. Each investigation teaches you a new failure signature. You start to recognize the shape of a sync-block deadlock from the thread list alone. You develop a library of scripts and breakpoints. More importantly, you start designing systems that fail loudly: timeouts that throw, health checks that detect stalled pipelines, and structured logging that captures the state of synchronization primitives. The dump is your last resort, but it should never be your first surprise.

The next time your application hangs, resist the urge to restart it. Capture the evidence. The dump contains the truth, and with the right technique, you can extract it.