Why Your App Hangs and How to Prove It with Production Dumps

The Silent Killer of Production Systems

Let’s be honest: a hung application is worse than a crash. A crash is loud. It leaves a corpse you can autopsy. A hang is a zombie—still breathing, still burning resources, but utterly brain-dead. Your users stare at a spinning wheel. Your monitoring dashboard shows a flatline. No exceptions. No logs. Just silence. In the .NET world, a hang almost always comes down to one of two things: threads that are blocked waiting for something that will never happen, or threads that are running in a circle they can’t escape. This guide is about cutting through the noise and finding the smoking gun in a memory dump.

Capturing the State of a Hung Process

You can’t fix what you can’t see. A single memory dump is a snapshot, but a hang is a story about time—specifically, the lack of progress over time. You need at least two dumps, several seconds apart, to prove that nothing is moving. For a CPU-bound hang, where the process is pegged at 90% or higher but not actually finishing work, ProcDump is your best friend. The command below grabs three dumps, ten seconds apart, when the CPU threshold is breached for five consecutive seconds. That gives you a timeline of the spinning thread.

procdump -ma -c 90 -s 5 -n 3 -o w3wp.exe

For a quiet hang—low CPU, but requests are piling up—a manual dump or a simple count-based trigger works fine. The rule is simple: capture at least two dumps. A single dump is a photograph. A series is a short film. You need the film to see who’s stuck and who’s still moving.

Proving a Deadlock: The Wait-Chain Trail

Deadlocks are oddly satisfying to diagnose because the evidence is absolute. Two threads, each holding a lock the other wants. In managed code, the monitor lock is the usual suspect, but you’ll also see deadlocks on reader-writer locks or even hybrid messes involving unmanaged sync objects. Your first move in WinDbg is the SOS extension’s !dlk command (or !deadlock in some versions).

Run !dlk against your dump. If it spits out a deadlock cycle, you’re halfway home. The output names the threads, the objects they’re waiting on, and the threads holding those objects. Thread A holds lock X and wants lock Y; Thread B holds lock Y and wants lock X. It’s right there in black and white. But what if !dlk comes up empty? The hang is still real. Now you trace the wait chain by hand. Look for threads in a WaitSleepJoin state. Dump the object they’re waiting on with !do and check its sync block index. Then use !syncblk to find the owner thread. If that owner is also waiting, follow the chain. A cycle is a deadlock. A long chain that ends in a thread doing unmanaged I/O or waiting on a GC is a different animal—a resource contention hang, which we’ll get to later.

Thread Pool Starvation: The Silent Throttle

Most hangs aren’t deadlocks. In high-throughput ASP.NET apps, the real villain is thread pool starvation. The .NET thread pool has a fixed number of threads to process work. When all of them are busy, new work gets queued. If those busy threads are themselves waiting on something that needs a thread pool thread to complete—the classic sync-over-async antipattern—the queue backs up forever. Requests time out. The app looks dead.

To prove starvation, check the pool’s internal state. The SOS command !threadpool gives you the high-level view. Look at the “Work Request in Queue” counter. If it’s in the thousands and not dropping across your dump series, you’ve got a problem. Next, find the culprits. Use !eestack -ee to list all managed threads and their call stacks. Filter for threads that are running but not completing. In an ASP.NET scenario, you’ll often see dozens of threads stuck inside Task.Result, Task.Wait(), or .GetAwaiter().GetResult(). These are blocking calls on thread pool threads. The fix is to make the whole call chain async, but the immediate diagnostic proof is the combination of a growing queue and a saturated pool of blocked threads.

Finalizer and GC Hangs: When Cleanup Blocks Everything

Here’s a less obvious but devastating scenario: the finalizer thread gets stuck. The .NET runtime has exactly one finalizer thread. If it blocks—maybe waiting on a lock, or doing a blocking I/O call inside a finalizer—the entire finalization queue stalls. The GC can’t reclaim objects that are ready for finalization until that thread runs, so memory pressure builds. Eventually, every thread that tries to allocate memory gets blocked waiting for a GC, which is itself waiting for the finalizer thread. The whole app hangs.

To spot this, first check the finalizer thread’s state. Use !threads and find the thread marked “(Finalizer).” Note its OS thread ID, switch to it, and examine its call stack with !clrstack. If it’s stuck in a WaitSleepJoin state, you’ve found the bottleneck. Next, check the finalization queue with !finalizequeue. A large number of objects “Ready for finalization” confirms the pressure. The root cause is the code inside the finalizer of the object at the head of the queue. This is a design flaw: finalizers must never block. The proof is in the dump: a single blocked thread, a growing queue, and a process-wide stall.

CPU-Bound Hangs: The Infinite Loop

Not every hang is a waiting game. A thread stuck in an infinite loop will eat 100% of a CPU core and never yield. The process might seem responsive if other cores are free, but if the loop is in a critical path or you have multiple such threads, the application grinds to a halt. The diagnostic approach here is different. You’re not looking for wait reasons; you’re looking for the thread that’s running and never changes its instruction pointer.

Capture a series of three to five dumps a few seconds apart. In each dump, run !runaway to identify the thread consuming the most CPU time. Note its OS thread ID. Then, in each dump, switch to that thread and examine its managed call stack with !clrstack. If the top frames are identical across all dumps, you’ve found your infinite loop. The next step is to examine the local variables and the loop condition to understand why it never exits. This often comes down to a subtle bug in a while loop or a recursive method that never reaches its base case.

Practical Dump Analysis Workflow

When you’re paged at 3 a.m. for a hung production server, you need a repeatable, efficient workflow. Here’s the sequence I follow, refined over hundreds of incidents:

1. Capture the right data. Use ProcDump to take at least two full dumps 10–15 seconds apart. If the process is using high CPU, use the CPU threshold trigger. Otherwise, a simple procdump -ma -n 2 -s 15 w3wp.exe will do.
2. Open the first dump in WinDbg. Load SOS with .loadby sos clr (or .loadby sos coreclr for .NET Core). Set the symbol path to the public Microsoft symbol server.
3. Run !dlk. If it finds a deadlock, you’re done. Identify the involved locks and the owning threads. Correlate with source code.
4. If no deadlock, run !threadpool. Check the work queue depth. If it’s high and growing, you likely have thread pool starvation. Use !eestack -ee to find the blocking calls.
5. Check the finalizer thread. Use !threads to find it, then !clrstack to see what it’s doing. If it’s blocked, check the finalization queue with !finalizequeue.
6. If CPU is high, use !runaway. Find the top CPU consumer and check its stack across multiple dumps for a repeating pattern.
7. Correlate with the second dump. Confirm that the problematic threads are still in the same state. A hang is defined by a lack of progress.

This workflow covers the vast majority of production hangs. The key is to move quickly from the general (is it a deadlock, starvation, or a CPU loop?) to the specific (which lock, which method, which line of code?).

FAQ: Common Questions About Hang Analysis

Why didn’t my application log any errors during the hang?

Hangs are not exceptions. A deadlocked thread isn’t throwing; it’s waiting. A starved thread pool isn’t failing; it’s queuing. The application is in a state of suspended animation, not a crash. That’s why memory dumps are essential—they’re the only way to observe the internal state of the runtime when no external signals are being emitted. Your logging framework is likely also blocked, waiting for a thread to write the log entry.

Can I use a tool other than WinDbg to analyze these dumps?

Yes, but with tradeoffs. Visual Studio’s memory dump analyzer can open managed dumps and has a friendlier interface for inspecting threads and call stacks. However, it lacks the specialized SOS commands like !dlk and !syncblk that make deadlock detection trivial. For quick, targeted analysis, WinDbg with SOS remains the most powerful option. For a more guided experience, consider the Debug Diagnostics Tool (DebugDiag) from Microsoft, which can automate hang analysis and generate a report identifying common patterns like deadlocks and finalizer hangs.

What if the hang is intermittent and I cannot capture a dump at the right moment?

Intermittent hangs are the hardest to diagnose. You need to set up a proactive monitoring strategy. Use ProcDump’s -tc (thread count) trigger to capture a dump when the number of threads exceeds a healthy baseline, which often correlates with thread pool starvation. Alternatively, use the -h (hang) trigger with a watchdog timer if your application has a health-check endpoint. The goal is to automate dump collection so you’re not relying on manual intervention during a transient event.

How do I differentiate between a managed deadlock and an unmanaged one?

The !dlk command only detects deadlocks involving managed monitor locks. If your application uses unmanaged synchronization objects like Mutex, Event, or Semaphore via P/Invoke, or if the deadlock involves a mix of managed and unmanaged code, you’ll need to use the native debugging commands. Switch to the thread of interest and use k to view the native call stack. Look for calls into WaitForSingleObject or WaitForMultipleObjects. Use !handle to inspect the handle being waited on. This is a more manual process but follows the same logical chain: find who is waiting, what they’re waiting for, and who holds that resource.

Next Steps: Building a Diagnostic Runbook

This article has given you the forensic techniques to prove a hang. The next step is to integrate this knowledge into your team’s operational practices. Create a runbook that maps specific symptoms (high CPU, zero CPU, growing request queues) to the appropriate ProcDump triggers and WinDbg commands. Pre-configure your symbol server access and ensure every developer has a local cache. The goal is to reduce the time from “the site is down” to “here is the offending line of code” to under fifteen minutes. In a future article, we’ll tackle the related problem of high memory pressure and how to use dump analysis to identify memory leaks before they cause an outage.