Reading the DAC: When SOS Commands Lie and How to Verify Them in .NET 8+ Crash Dumps

03:17 UTC. The pager fires. An ASP.NET Core 8 API on Kubernetes has crashed for the fourth time in 72 hours. The container runtime logs an access violation (0xC0000005) each time. Operations grabs a core dump with dotnet-dump collect inside the pod, copies it to a jump box, and loads it in WinDbg with the .NET 8 SOS extension. Then comes two hours of pure frustration. !dumpheap returns nothing. !threads reports 4,096 threads in a process whose /proc/<pid>/status shows 38. !syncblk lists sync blocks that don’t map to any live heap object. The team is chasing phantoms. The dump isn’t lying about the crash—it’s lying about the runtime state, because the Data Access Component (DAC) tables inside the dump are inconsistent.

This happens more often than the documentation suggests. When you capture a crash dump from a containerized .NET 8+ service, first-pass analysis can yield garbage for reasons that have nothing to do with your debugging skills. The causes are mundane: how the dump was captured, what cgroup limits truncated, and whether the DAC version in the dump matches the SOS DLL in your analysis environment. The Google SRE book’s chapter on effective troubleshooting argues that systematic incident response requires isolating components and validating assumptions before acting on them. That principle maps directly onto this problem. Treat SOS commands as ground truth without cross-checking DAC state, and you’ll waste hours chasing root causes that don’t exist.

What the DAC Actually Does and Why It Breaks

The DAC—mscordacwks.dll on Windows, libmscordaccore.so on Linux—bridges the CLR’s internal data structures and the debugging tools that read them. SOS, WinDbg, dotnet-dump, ClrMD: all depend on the DAC to interpret the runtime’s memory layout. GC heap segments, method tables, thread objects, sync blocks, the EE (Execution Engine) state. The DAC reads these structures from the dump file and translates them into the managed abstractions that SOS commands surface.

The DAC is version-specific. Built alongside the runtime, it expects exact layout compatibility with the CLR version it was compiled for. Load SOS in WinDbg or dotnet-dump, and the tool tries to locate the matching DAC for the target runtime. In .NET 8+, the DAC sits in the shared framework directory (e.g., /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.x/libmscordaccore.so). If your analysis environment has a different .NET SDK patch version installed, or if the DAC file wasn’t captured alongside the dump, the tool may load a mismatched DAC. The output will be silently, maddeningly wrong.

DAC corruption manifests in three patterns in containerized environments. First: incomplete captures. When dotnet-dump collect or gcore gets interrupted by a container shutdown or an OOM killer event, the core file may be truncated. Critical memory pages the DAC needs to walk the heap are gone. Second: cgroup memory limits. If the container’s memory cgroup limit sits below the process’s working set, the kernel may kill the process before the dump completes. The resulting core file has gaps in the heap regions. Third: DAC version mismatches. Load a .NET 8.0.4 DAC against a .NET 8.0.11 dump, and you’ll get struct offset errors that surface as nonsensical object sizes, impossible thread counts, or heap walks that terminate early.

The Scenario: Access Violations in a Kubernetes-Hosted API

Back to the 03:17 pager. The service was an ASP.NET Core 8 minimal API in a Kubernetes Deployment with three replicas, each limited to 1.5 CPU and 2 GiB memory via resource requests and limits. The crashes were intermittent—every 18–24 hours, no correlation to traffic patterns. The kubelet recorded the access violation in the pod’s termination logs: exited with code 139 (SIGSEGV).

Initial dump capture:

dotnet-dump collect -p <pid> --type Full -o /tmp/crash.dmp

The file was 1.8 GiB, matching the process’s RSS at capture time. Size alone wasn’t suspicious. The first sign of trouble appeared when loading SOS:

dotnet-dump analyze /tmp/crash.dmp
> loadmodule sos
> !eeversion
CLR version: 8.0.11.523.2154 (Release)  <-- matches the pod's runtime
DAC file: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.4/libmscordaccore.so  <-- WRONG VERSION
DAC table version: 2.0.0.0

Runtime version: 8.0.11. DAC loaded: 8.0.4. The analysis environment had an older .NET SDK installed, and dotnet-dump found that DAC first. Every subsequent SOS command was reading the dump through a DAC that expected different struct layouts.

With the wrong DAC, !dumpheap -stat returned a summary with zero objects. !threads reported 4,096 threads—a corrupted read of the thread list. !syncblk showed 15 sync blocks, but !dumpheap -mt <syncblockMT> found no objects matching those method tables. The team concluded the heap was corrupted and began investigating native interop, P/Invoke boundary violations, and unsafe code in a third-party library. They were wrong.

Detecting DAC Corruption Before It Wastes Your Time

First rule of dump analysis in .NET 8+: verify the DAC before trusting any SOS output. The NIST Cybersecurity Framework 2.0 emphasizes that detection and investigation of anomalous runtime behavior should follow a structured framework rather than ad-hoc analysis, and that data integrity concerns extend to diagnostic artifacts. Crash dumps must be validated for completeness and consistency before being trusted as evidence. A corrupted DAC state doesn’t mean the crash didn’t happen. It means your analysis of the crash is built on an unreliable foundation.

Three cross-checks catch most DAC corruption early. First: compare the runtime version reported by !eeversion against the DAC file path it loaded. If they don’t match at the patch level, stop. Fix the DAC before proceeding. Second: run !eeheap -gc and compare the total heap size against the dump file size and the process’s known working set. If !eeheap -gc reports 0 bytes but the dump is 1.8 GiB, the DAC can’t walk the heap—either because of version mismatch or because heap segments are missing from the dump. Third: run !verifyheap. If it reports corruption immediately at the first segment, that’s a DAC problem, not a heap problem. If it runs for several seconds and then reports a specific object with a bad method table, that’s genuine corruption worth investigating.

In our scenario, fixing the DAC resolved the phantom data. The correct DAC was inside the container image at /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/libmscordaccore.so. Copying it to the analysis environment and loading it explicitly:

dotnet-dump analyze /tmp/crash.dmp
> loadmodule /path/to/libmscordaccore.so
> setsostid
> !eeversion
CLR version: 8.0.11.523.2154 (Release)
DAC file: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/libmscordaccore.so
DAC table version: 2.0.0.0

Now !dumpheap -stat returned 412,000 objects. !threads reported 38 threads—matching /proc data. !syncblk showed 3 sync blocks, all with valid object references. The heap wasn’t corrupted. The DAC was.

When the DAC Is Correct but the Dump Is Incomplete

Two weeks later, a separate incident produced a different failure mode. DAC version matched perfectly. But !dumpheap -stat still returned only 12,000 objects in a process with 800 MiB of managed heap. !eeheap -gc showed three heap segments, but the largest segment (600 MiB) had a starting address that fell outside the dump’s mapped memory range. The core file was truncated.

This happens when the dump capture is interrupted or when the container’s memory cgroup limit causes the OOM killer to fire mid-capture. On Linux, gcore and dotnet-dump collect both use ptrace to attach to the process and read its memory. If the process dies during capture, the core file will be incomplete. The symptom: the DAC loads correctly, !eeversion matches, !threads looks reasonable, but large heap segments are missing.

To detect this, compare !eeheap -gc segment addresses against the dump’s memory map. In WinDbg, run !address and look for the segment addresses reported by !eeheap -gc. If a segment address isn’t in the address map, that segment wasn’t captured. In dotnet-dump, there’s no direct equivalent of !address, but you can check by attempting !dumpheap -start <segmentAddress> -end <segmentEndAddress>. If the DAC returns an error reading memory at those addresses, the segment is missing from the dump.

An incomplete dump isn’t always salvageable through SOS. If the missing segment contains the objects you need to investigate, recapture is the only option. But if the missing segment is Gen 2 and the crash is in Gen 0 allocation or the thread pool, the dump may still be useful for thread state and stack analysis.

Falling Back to ClrMD When SOS Misfires

When SOS commands produce inconsistent results even with a correct DAC, ClrMD (Microsoft.Diagnostics.Runtime) provides a programmatic alternative. It reads the DAC directly without going through the SOS command layer. ClrMD is particularly useful for extracting specific data—thread pool state, exception objects, GC heap statistics—when you need to validate SOS output or when SOS commands are failing on a specific dump.

The following ClrMD script loads a .NET 8 dump, extracts thread pool state and all exception objects, and prints them without relying on any SOS command. Useful as a cross-check when !threads or !dumpheap -type Exception produce suspicious results.

using Microsoft.Diagnostics.Runtime;

using var dataTarget = DataTarget.LoadDump("/tmp/crash.dmp");
var runtime = dataTarget.GetClrRuntime();

// Verify DAC version
Console.WriteLine($"Runtime: {runtime.ClrInfo.Version}");
Console.WriteLine($"DAC: {runtime.ClrInfo.DacInfo.FileName}");
Console.WriteLine($"DAC version: {runtime.ClrInfo.DacInfo.Version}");

// Thread pool state
var threadPool = runtime.ThreadPool;
Console.WriteLine($"Min threads: {threadPool.MinThreads}");
Console.WriteLine($"Max threads: {threadPool.MaxThreads}");
Console.WriteLine($"Active threads: {threadPool.ActiveThreads}");
Console.WriteLine($"Available: {threadPool.AvailableThreads}");

// All managed threads with exceptions
foreach (var thread in runtime.Threads)
{
    var ex = thread.CurrentException;
    if (ex != null)
    {
        Console.WriteLine($"Thread {thread.OSThreadId} (Managed {thread.ManagedId}): {ex.Type.Name}");
        Console.WriteLine($"  Message: {ex.Message}");
        foreach (var frame in ex.StackTrace)
        {
            Console.WriteLine($"    at {frame}");
        }
    }
}

// Count exception objects on heap
int exCount = 0;
foreach (var obj in runtime.Heap.EnumerateObjects())
{
    var type = obj.Type;
    if (type != null && type.Name.Contains("Exception"))
    {
        exCount++;
    }
}
Console.WriteLine($"Total exception objects on heap: {exCount}");

ClrMD reads the DAC through the same libmscordaccore / mscordacwks interface. If the DAC is mismatched, ClrMD will also produce incorrect data. The difference: ClrMD gives you programmatic control to cross-check values against each other and against known process state. Compare runtime.Threads.Count() against the thread count in /proc/<pid>/status you captured alongside the dump. If they disagree, the DAC is wrong.

In our scenario, the ClrMD script with the correct DAC confirmed 38 threads (matching /proc), 3 active exceptions on the heap, and a thread pool with 12 min threads and 200 max threads. The access violation was occurring in a native library call related to TLS handshake, not in managed heap corruption. The team had been chasing heap corruption because the wrong DAC made it look like the heap was empty.

Diagnostic Procedure: Verifying DAC Integrity in .NET 8+ Dumps

The following procedure works for both Windows minidumps and Linux ELF core dumps analyzed with dotnet-dump or WinDbg. Run these steps before any substantive SOS analysis.

Step 1: Load the dump and verify the DAC version match.

dotnet-dump analyze /tmp/crash.dmp
> loadmodule sos
> !eeversion

Expected output (healthy):

CLR version: 8.0.11.523.2154 (Release)
DAC file: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.11/libmscordaccore.so
DAC table version: 2.0.0.0

Expected output (corrupted):

CLR version: 8.0.11.523.2154 (Release)
DAC file: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.4/libmscordaccore.so
DAC table version: 2.0.0.0

If the patch versions differ (8.0.11 vs 8.0.4), locate the correct DAC from the container image and reload: loadmodule /path/to/correct/libmscordaccore.so.

Step 2: Cross-check heap size against dump size.

> !eeheap -gc

Expected output (healthy):

GC Heap Size: 823,456,789 (0x310ab445 bytes)
Total GC Heap Size: 823 MB

Expected output (corrupted):

GC Heap Size: 0 (0x0 bytes)
Total GC Heap Size: 0 MB

If the heap reports 0 bytes but the dump file is >100 MB, either the DAC is mismatched or heap segments are missing from the dump. Proceed to Step 3.

Step 3: Verify heap integrity.

> !verifyheap

Expected output (healthy):

No heap corruption detected.

Expected output (DAC corruption):

Heap corruption detected at 0x7f234a001000
Object at 0x7f234a001000 has invalid method table 0x0

If !verifyheap fails at the very first segment address, the DAC can’t read the heap structure. If it fails at a specific object after walking several segments, that’s genuine corruption.

Step 4: Check for missing heap segments.

> !eeheap -gc
(suppose it reports segment at 0x7f8000000000)
> !dumpheap -start 0x7f8000000000 -end 0x7f8001000000

Expected output (segment present):

Address       MT     Size
0x7f8000000010 0x... 64
0x7f8000000050 0x... 128
...

Expected output (segment missing):

Error reading memory at 0x7f8000000000

If the segment is missing, the dump is incomplete. Recapture with a longer timeout or check for OOM killer interference during capture.

Step 5: Cross-check thread count.

> !threads

Expected output (healthy):

ThreadCount: 38
UnstartedThread: 0
BackgroundThread: 36
PendingThread: 0
DeadThread: 2

Expected output (corrupted):

ThreadCount: 4096
UnstartedThread: 4096
...

Compare ThreadCount against the process’s actual thread count from /proc/<pid>/status (Threads line) or from Task Manager. If they differ by more than 1–2, the DAC is misreading the thread list.

Step 6: Fall back to ClrMD if SOS remains inconsistent.

Install the Microsoft.Diagnostics.Runtime NuGet package and run the ClrMD script from the previous section. Compare ClrMD’s runtime.Threads.Count() and runtime.Heap.EnumerateObjects().Count() against SOS output. If ClrMD and SOS disagree with the same DAC loaded, the DAC itself may be corrupted in the dump—try a different DAC patch version from the same minor (8.0.x) as a last resort.

Root Cause and Remediation

The access violations in the original scenario were caused by a race condition in a native TLS library used by the HTTP client stack. The service made outbound HTTPS calls under high concurrency, and the native library had a known use-after-free bug in its session cache—triggered when connections were recycled under load. The fix: update the native library (bundled in the base image) and cap HttpClientHandler‘s MaxConnectionsPerServer to reduce concurrent TLS handshakes.

The debugging team lost two hours chasing heap corruption that didn’t exist because the DAC was mismatched. The post-incident review produced two action items. First: always capture the DAC file alongside the dump by copying libmscordaccore.so from the container image to the same directory as the dump file. Second: add the DAC version check (!eeversion output comparison) as the first step in the team’s dump analysis runbook.

Documenting the diagnostic procedure in a shared runbook matters as much as the procedure itself. The runbook is where the team’s hard-won knowledge lives—each crash investigation adds a clause, a cross-check, or a fallback path. For teams that want to maintain structured post-incident documentation without scattered wiki pages, an AI writing app that keeps your postmortem drafts organized can standardize the format across incidents so the next on-call engineer doesn’t start from scratch. The runbook structure—symptom, hypothesis, data collection, analysis, root cause, remediation—should be consistent enough that any team member can follow it at 03:00 UTC.

Conclusion

DAC corruption in .NET 8+ crash dumps isn’t a rare edge case. It’s a predictable failure mode in containerized environments where analysis environments diverge from production images, cgroup limits truncate core dumps, and the DAC is rarely captured alongside the dump. The defense is procedural: check the DAC version before trusting any SOS command, cross-check heap size and thread count against known values, run !verifyheap to distinguish DAC errors from genuine heap corruption, and maintain a ClrMD fallback script for when SOS commands produce impossible values. Every dump tells a story. Only a verified DAC can tell you whether the story is true.