03:14 UTC. The page comes in. An ASP.NET Core service handling payment reconciliation for a logistics platform has stopped responding to health checks. The process is alive — memory looks normal, CPU idle — but no request completes. By the time on-call captures a dump with dotnet-dump collect -p 4782, thirty-two seconds of queued work items have piled up. The dump shows a ThreadPool with zero available threads. Every worker is parked on a Monitor.Wait or waiting for a continuation that never fired. !clrstack gives you the tip of each thread’s current frame, but the async call chain that led there — the one that tells you which request triggered the cascade — is gone. The JIT inlined the continuation delegates. State machine fields are partially overwritten by reused Gen 0 memory. The stack you need is three await points deep in a method that no longer has a frame.
Most engineers give up here. They collect the dump, stare at !dumpheap -stat, see a few thousand Task objects and a handful of state machine instances, and conclude the dump is unusable. It is not unusable. You need to stop trusting the stack and start reading the heap.
What the JIT Does to Your Async Stack
When the C# compiler generates an async method, it emits a state machine struct — <MethodName>d__N — with fields for the builder (AsyncTaskMethodBuilder), the state field (int stateField), the awaiter fields, and captured locals. At each await point, the state machine’s MoveNext checks whether the awaited task has completed. If it has not, the state machine saves its state, registers a continuation, and returns. The thread is free. When the awaited task completes, the ThreadPool picks up the continuation and calls MoveNext again.
Here is where the dump gets tricky. The continuation is registered as a delegate. In release builds with tiered compilation, the JIT aggressively inlines the delegate’s invocation target. The MoveNext you see on the stack is often not the state machine’s own MoveNext — it is JIT-compiled code for a lambda, an inlined continuation wrapper, or a compiler-generated AsyncTaskMethodBuilder method. The state machine fields that would tell you which await point the code is at may have been reused by a subsequent allocation if the state machine was heap-allocated and then promoted to Gen 1 before the continuation fired.
The result: !clrstack shows a thread sitting in ThreadPoolWorkQueue.Dispatch or Task.Execute, and the real async context — which request, which method, which await — is buried in heap objects the stack does not reference directly.
A Frozen ThreadPool With No Readable Stack
Let me walk through the exact production incident pattern. The service processes inbound webhook callbacks from a payment gateway. Each callback triggers an async pipeline: validate signature, fetch order from cache, call downstream inventory service, persist result. The downstream inventory service has a 30-second timeout configured via HttpClient. Under normal load, each call completes in 200ms. Under a downstream incident, calls take 28 seconds — just under the timeout. The service does not circuit-break. Every ThreadPool thread eventually parks on an HttpClient send, and the queue grows.
The dump you capture at the peak of this incident shows 47 ThreadPool threads. !threads reports all of them as worker threads. !clrstack on each thread gives you one of three patterns:
Pattern A: System.Threading.Tasks.Task.Execute() with no further managed frames. The JIT inlined everything.
Pattern B: Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync followed by a few middleware frames, then nothing. The async pipeline went deep and the continuation frames were elided.
Pattern C: System.Net.Http.SocketsHttpHandler.SendAsync with a Monitor.Wait at the bottom. This thread is actually blocked on the downstream call.
Pattern C tells you what the threads are doing. Patterns A and B tell you nothing about which request or which code path led to the stall. You need the async continuation graph — the chain of state machines that connects the blocked HTTP call back to the ASP.NET Core request that initiated it.
Step 1: Enumerate All Async State Machines With !dumpasync
The SOS extension !dumpasync is the first tool to reach for. It enumerates all async state machine objects on the managed heap and prints their type, address, and state field. Run it with no arguments first to get the full inventory:
0:047> !dumpasyncOutput looks like this (truncated):
Address MT State Type
0000021a4f8a3c90 0000021a1d0e4b10 1 Webhooks.PaymentCallback+d__12
0000021a4f8a4d20 0000021a1d0e4b40 3 Webhooks.PaymentCallback+d__12
0000021a4f8b0180 0000021a1d0e4c70 1 Inventory.Client+d__7
0000021a4f8b1200 0000021a1d0e4c70 1 Inventory.Client+d__7
0000021a4f8c3300 0000021a1d0e4d90 0 Webhooks.PaymentCallback+d__12The State field is the state machine’s internal state integer. State 0 means the machine has not started. State -1 means it completed. Any positive integer means it is suspended at an await point — specifically, at the Nth await in the generated MoveNext switch statement. For PaymentCallback+d__12 with state 3, you need to look at the compiler-generated MoveNext to map state 3 to a specific await. In practice, decompile the assembly with ILSpy or read the IL directly.
If !dumpasync is unavailable — which happens with older SOS versions or a mismatched DAC — use !dumpheap -stat and filter for state machine types:
0:047> !dumpheap -stat -type d__Noisier, but gives you the method table addresses. From there, !dumpheap -mt <MT> lists individual instances.
Step 2: Cross-Reference State Machines Against ThreadPool Work Items
Now connect each suspended state machine to the ThreadPool work item that will resume it. When a task completes, it queues a continuation. That continuation is a Task object with a reference to the state machine’s MoveNext delegate. The chain:
Task._continuationObject → ContinuationWrapper or direct Action delegate → stateMachine.MoveNext
Use !do (dump object) on each suspended state machine and read its m_builder field — the AsyncTaskMethodBuilder. The builder contains the Task associated with this state machine. That Task holds continuation delegates. Walk the chain:
0:047> !do 0000021a4f8a3c90Look for the m_builder field. Dump it:
0:047> !do <m_builder_address>Inside the builder, find m_task. Dump that Task:
0:047> !do <task_address>The Task object has a m_continuationObject field. If non-null, it is either a single continuation delegate or a List<Action>. Dump it and look for a delegate whose target is a state machine instance. That is the continuation that will fire when this task completes.
Key insight: if the task is an HttpClient send task still in-flight (Pattern C threads), its continuation object points back to the state machine that called it. That state machine’s m_builder.m_task points to the next task in the chain. You can walk the entire async pipeline by following Task → continuation → stateMachine → m_builder → m_task → continuation until you reach the ASP.NET Core request entry point.
In our incident, the chain for one blocked thread:
Inventory.Client+<SendAsync>d__7 (state 1) → awaiting Task from HttpClient.SendAsync → continuation: Webhooks.PaymentCallback+<ProcessAsync>d__12 (state 3) → awaiting Task from Inventory.Client.SendAsync → continuation: HostingApplication.ProcessRequestAsync → root: ASP.NET Core requestState 3 in PaymentCallback+d__12 maps to the await on Inventory.Client.SendAsync. State 1 in Inventory.Client+d__7 maps to the await on HttpClient.SendAsync. You now know exactly where the pipeline is stuck and which request triggered it.
Step 3: Handle Corrupted or Overwritten State Machine Fields
The scenario above assumes state machine fields are intact. In practice, they often are not. The state machine struct is heap-allocated when the async method hits its first await that does not complete synchronously. If the method is called frequently, the state machine type may be allocated and freed rapidly. When a Gen 0 collection reclaims a completed state machine, the memory is reused. A new state machine of the same type allocated in the same space overwrites the old fields.
If you capture a dump during a high-throughput period, some state machine addresses on the heap may have been partially overwritten. The stateField may show a nonsensical value like 0xdeadbeef or a value that does not correspond to any valid await point. The m_builder field may point to a freed object. !gcroot on the state machine address may return nothing because the object was collected and the address now holds a different object.
To handle this, cross-check every state machine address against !dumpheap -stat to confirm the address is still a valid object of the expected type. If !do returns garbage or throws an access violation, skip that instance. Focus on state machines with valid state fields (positive integers matching the number of await points in the method) and intact m_builder pointers.
If the state field is valid but m_builder.m_task is null, the state machine has not yet registered its task. The method is at the very first await — the builder has not yet boxed the task. You can still trace backward from the ThreadPool work item queue to find which continuation references this state machine.
Step 4: When !clrstack Shows Nothing, Read the Thread’s Queue Slot
For threads showing Pattern A — Task.Execute with no further frames — the continuation context is in the ThreadPool work item, not on the stack. The thread picked up a work item from the global queue and is executing it. The work item is a Task with a continuation delegate. To find which state machine that delegate targets:
First, get the thread’s current work item. Use !clrstack to confirm the thread is in ThreadPoolWorkQueue.Dispatch. Then look at the thread’s current ThreadPoolWorkRequest. You can find it by examining the thread’s stack frame locals — specifically, the local variable holding the dequeued work item. In WinDbg, use !clrstack -a to show locals and parameters. The work item is typically in a register or a stack slot depending on the JIT’s register allocation.
If locals are unavailable due to optimization, use !dumpheap -type ThreadPoolWorkRequest and cross-reference addresses against the thread’s stack range. The work request object will be near the top of the thread’s stack. From the work request, follow the delegate chain to the state machine.
Tedious to do by hand across 47 threads. This is where automation becomes essential.
Step 5: Script the Reconstruction With ClrMD
Manual SOS commands work for a single state machine chain. For a production incident with dozens of suspended state machines, you need a script. ClrMD — the Microsoft.Diagnostics.Runtime NuGet package — lets you write a C# program that loads a dump, enumerates the heap, and reconstructs the async continuation graph programmatically.
Here is the structure of a ClrMD-based reconstruction script. The goal: produce a human-readable output that maps each blocked thread to its full async call chain.
using Microsoft.Diagnostics.Runtime;
using var target = DataTarget.LoadDump("hang.dmp");
var runtime = target.ClrVersions[0].CreateRuntime();
var heap = runtime.Heap;
// Enumerate all async state machine instances
var stateMachines = new List<(ClrObject obj, string typeName, int state)>();
foreach (var obj in heap.EnumerateObjects())
{
var typeName = obj.Type.Name;
if (typeName.Contains("d__") && obj.Type.Name.Contains("+"))
{
var stateField = obj.Type.GetFieldByName("<>1__state")
?? obj.Type.GetFieldByName("stateField");
if (stateField != null)
{
int state = obj.ReadField<int>(stateField);
stateMachines.Add((obj, typeName, state));
}
}
}
// For each suspended state machine, walk m_builder → m_task → continuation
foreach (var (obj, typeName, state) in stateMachines.Where(sm => sm.state > 0))
{
var builderField = obj.Type.GetFieldByName("<>t__builder");
if (builderField == null) continue;
var builderObj = obj.ReadObjectField(builderField);
var taskField = builderObj.Type.GetFieldByName("m_task");
if (taskField == null) continue;
var taskObj = builderObj.ReadObjectField(taskField);
if (taskObj.IsNull) continue;
var continuationField = taskObj.Type.GetFieldByName("m_continuationObject");
if (continuationField == null) continue;
var continuation = taskObj.ReadObjectField(continuationField);
Console.WriteLine($"{typeName} (state={state}) → awaiting {taskObj.Type.Name}");
Console.WriteLine($" continuation target: {continuation.Type.Name}");
}Skeleton code. The full script handles edge cases: m_continuationObject being a List<Action> rather than a single delegate, the continuation target being a boxed state machine rather than a direct reference, and m_task being null when the builder has not yet boxed. The complete script is roughly 200 lines and takes about 3 seconds to run on a 2GB dump.
Output for our incident:
Webhooks.PaymentCallback+<ProcessAsync>d__12 (state=3)
→ awaiting System.Threading.Tasks.Task`1[[System.Net.Http.HttpResponseMessage]]
→ continuation: Webhooks.PaymentCallback+<ProcessAsync>d__12.MoveNext
→ root request: /api/webhooks/payment-callback
Inventory.Client+<SendAsync>d__7 (state=1)
→ awaiting System.Threading.Tasks.Task`1[[Inventory.OrderResponse]]
→ continuation: Inventory.Client+<SendAsync>d__7.MoveNext
→ parent: Webhooks.PaymentCallback+<ProcessAsync>d__12 (state=3)That output is what on-call needs. It shows the full async call chain from the ASP.NET Core request entry point down to the blocked HttpClient call, with every await point identified by its state field. The engineer can now see the bottleneck is Inventory.Client.SendAsync and the circuit breaker is not firing because the timeout is set too high relative to the downstream failure mode.
Building a Repeatable Runbook for On-Call Engineers
The script above is not a one-time tool. It is a runbook. Every production service using async pipelines should have a ClrMD-based async reconstruction script checked into its diagnostic tooling repository. When a hang occurs, on-call runs one command — dotnet run --project AsyncTriage -- hang.dmp — and gets the continuation graph in seconds. That is the difference between a 45-minute investigation and a 5-minute triage.
The runbook should cover three scenarios: hang dumps (the script above), crash dumps (filter for state machines with state -1 to find completed-but-not-collected chains that may indicate a race), and OOM dumps (filter for state machines with large captured local fields that may be retaining memory). Each scenario is a different filter on the same ClrMD enumeration logic.
Post-incident documentation matters equally. Once the async chain is reconstructed, the engineer writes a postmortem explaining the failure mode — which request, which await, which downstream dependency, and why the circuit breaker did not trigger. The Google SRE Book’s chapters on postmortem culture and effective troubleshooting lay out the structure: blameless narrative, timeline, root cause, action items. A well-structured postmortem turns a single incident into institutional knowledge that prevents recurrence. The Google SRE book treats postmortems as a core engineering practice, not an afterthought, and the on-call runbook is the input that makes a credible postmortem possible.
From Heap Fragments to a Narrative the Team Can Act On
The reconstructed async continuation graph is structured data: state machine types, state fields, task references, continuation targets. The postmortem needs to be prose — a narrative a reader who was not on-call can follow. The gap between structured diagnostic output and a readable incident report is where many postmortems stall. The engineer has the facts but spends an hour turning them into sentences.
This is where a structured-to-prose tool can help. When documenting the reconstructed async flow for a postmortem, engineers need to produce structured narrative output from fragmented technical findings — the kind of transformation where the Unsloppy AI Writing App can accelerate incident report drafting by taking the structured continuation graph and producing a draft narrative. The engineer must verify every detail against the dump output, but the tool handles the mechanical work of turning a list of state machine transitions into a readable sequence of events.
The same caveat applies to any AI-assisted drafting in a professional context: the output is a starting point, not a finished product. The Authors Guild’s AI best practices for authors emphasizes that AI-generated text is a generic composite of training data and that professional standards require human oversight and editorial judgment. In a postmortem context, that means the engineer who ran the ClrMD script is the author of record and must confirm that every claim in the AI-assisted draft matches the dump evidence. The tool accelerates the draft; the engineer owns the facts.
Common Pitfalls in Async Dump Reconstruction
Three mistakes recur in async dump analysis. The first is trusting !clrstack alone. In release builds with tiered compilation, !clrstack shows the JIT-optimized call stack, which may have inlined away the very frames that connect the async chain. The state machine heap objects are the ground truth, not the stack.
Second mistake: assuming every suspended state machine is part of the problem. A healthy ASP.NET Core service may have hundreds of suspended state machines at any given time — one per in-flight request. The question is not how many there are, but which ones are stuck on the same downstream dependency and for how long. The state field tells you where they are stuck; m_task tells you what they are waiting on. Correlating by m_task type reveals the bottleneck.
Third mistake: ignoring the ThreadPool queue. The threads in the dump are executing work items that were dequeued. The work items still in the queue — not yet dispatched — are invisible in !threads output. Use !dumpheap -type ThreadPoolWorkRequest to count pending work items. If the queue depth is in the thousands and every pending work item is a continuation for the same downstream call, you have proven the bottleneck without needing the stack at all.
Conclusion
Async dump reconstruction is a heap-reading exercise, not a stack-reading exercise. The JIT’s optimizations make the stack unreliable for async call chains in release builds, but the heap retains the full continuation graph as long as the state machines and their tasks are alive. The diagnostic method: enumerate state machines with !dumpasync, walk m_builder → m_task → continuation for each suspended instance, cross-reference against ThreadPool work items, and produce the continuation graph. Automate this with ClrMD so on-call engineers can run it in seconds, not minutes. The reconstructed graph is the input to a postmortem that turns one incident into a permanent fix. The tooling is the runbook; the runbook is the practice.