The security audit log showed something that should not have been possible. User A, authenticated via bearer token at 14:32:07.118, requested /api/orders/summary. User B, authenticated four seconds later at 14:32:11.402, hit the same endpoint from a different session, a different IP, a different tenant claim. The audit entry for User B’s request recorded User A’s ClaimsPrincipal in the UserId field. In a regulated environment, cross-request identity contamination is not a curiosity. It is an incident. The NIST CSF 2.0 framework applies here directly—not as a checkbox, but as the structural reasoning for why identity-bleed bugs demand forensic investigation rather than a hotfix and a shrug.
The application: an ASP.NET Core 8 service running behind IIS in-process on Windows Server 2022, handling roughly 1,200 concurrent requests at peak. The middleware pipeline included a custom TenantContextMiddleware that resolved tenant identity from request headers and stored it in an AsyncLocal<TenantContext> accessed by downstream handlers, logging components, and the audit infrastructure. The bug appeared only under load. Never in development. Never in staging. Never when the thread pool was idle.
The Symptom: Stale Identity in the Audit Trail
First evidence: a discrepancy between the IIS request log and the application audit log. The IIS log showed User B’s request arriving with User B’s JWT. The application audit log—written by a handler that read tenant identity from AsyncLocal<TenantContext>—recorded User A’s identity. The handler had no caching layer. The middleware set the AsyncLocal value at the start of every request. No obvious shared state.
The on-call engineer’s first hypothesis was a logging bug. Maybe the audit serializer was reading a stale field. That hypothesis collapsed when we found the business logic itself had operated on User A’s tenant context. User B’s order summary query had been filtered by User A’s tenant ID. The contamination was not cosmetic. It was functional.
Second hypothesis: a race condition in the middleware—two requests mutating the same AsyncLocal instance. But AsyncLocal<T> does not share storage across async flows. Each logical call context gets its own copy of the value. That is the entire point of the type. If the middleware was setting the value per-request, the flows should have been isolated. Unless the middleware was not setting it per-request. Unless something was capturing the ExecutionContext at a point where it contained a stale value and propagating that snapshot into a context where it did not belong.
Capturing the Dump During the Contamination Window
Reproducing this in development was not feasible. The bleed required thread-pool pressure sufficient to cause continuation scheduling patterns that exposed the stale context. We needed a dump from production, captured during the contamination window.
The strategy: instrument the audit handler to trigger a dump when it detected a mismatch between the JWT-validated identity (available from HttpContext.User) and the AsyncLocal<TenantContext> value. The instrumentation was straightforward:
// Inside AuditHandler.WriteAuditEntry
var contextTenant = _tenantContext.Value;
var httpContextUser = httpContext.User?.Identity?.Name;
if (contextTenant != null &&
httpContextUser != null &&
contextTenant.UserId != httpContextUser)
{
// Contamination detected — capture a full dump
var dumpPath = $"C:\\dumps\\contamination_{DateTime.UtcNow:yyyyMMdd_HHmmss}_{Guid.NewGuid():N}.dmp";
NativeMethods.MiniDumpWriteDump(
Process.GetCurrentProcess().Handle,
Process.GetCurrentProcess().Id,
File.Create(dumpPath).SafeFileHandle.DangerousGetHandle(),
MiniDumpType.FullMemory, ...);
_logger.LogCritical("AsyncLocal contamination detected. " +
"HttpContext user: {HttpContextUser}, AsyncLocal user: {AsyncLocalUser}. " +
"Dump written to {DumpPath}",
httpContextUser, contextTenant.UserId, dumpPath);
}
Within three hours of deploying this instrumentation to one production node, we had two dumps captured during confirmed contamination events. Both showed the same structural pattern.
Enumerating In-Flight State Machines with !dumpasync
The first dump was 4.2 GB—full memory, which is what you want for ExecutionContext forensics. Loading it in WinDbg with the SOS extension for .NET 8:
0:000> .loadby sos coreclr
0:000> !dumpasync
The !dumpasync command enumerates async state machines currently in-flight—awaiting completion. The output is a table of state machine objects, their types, and current state fields. In a healthy request pipeline, each state machine should be associated with a single ExecutionContext that carries the request-scoped AsyncLocal values.
0:000> !dumpasync
Dumping async state machines...
MT MethodTable State Object Type
00007ff8e1234000 00007ff8e1234050 0 0000025a4f8c1230 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[MyApp.Orders.SummaryResult]
00007ff8e1234200 00007ff8e1234250 2 0000025a4f8c1450 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[MyApp.Orders.SummaryResult]
00007ff8e1234400 00007ff8e1234450 0 0000025a4f8c1670 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[MyApp.TenantContextMiddleware+<Invoke>d__3]
...
42 async state machines found
42 in-flight state machines at the moment of the dump. The interesting ones: the TenantContextMiddleware+<Invoke>d__3 instances—the middleware’s async state machine. In a correctly structured pipeline, each request gets its own middleware state machine, each carrying its own ExecutionContext. What we found instead was the smoking gun.
Inspecting ExecutionContext and AsyncLocal Backing Fields
The AsyncLocal<T> value is not stored in the AsyncLocal instance itself. It lives in the current ExecutionContext, keyed by the AsyncLocal‘s internal value handle. When you set _tenantContext.Value = newTenant, the runtime creates a copy-on-write clone of the current ExecutionContext, adds the value to the clone’s internal dictionary, and makes that clone the active context for the current async flow. When an await suspends, the current ExecutionContext is captured and stored in the state machine. When the continuation resumes, that captured context is restored.
To trace the contamination, we needed to examine the ExecutionContext instances associated with each in-flight state machine. Starting with the middleware state machine:
0:000> !dumpobj 0000025a4f8c1670
Name: MyApp.TenantContextMiddleware+<Invoke>d__3
MethodTable: 00007ff8e1234400
EEClass: 00007ff8e1234380
Size: 96(0x60) bytes
Fields:
MT Field Offset Type VT Attr Value Name
00007ff8e2201000 4000001 40 System.Object 0 instance 0000025a4f8c1700 <>t__builder
00007ff8e2203000 4000002 48 System.Threading.Tasks.Task 0 instance 0000025a4f8c1850 <>1__state
00007ff8e2205000 4000003 50 ...text.ExecutionContext 0 instance 0000025a4f8c1920 <>u__taskId
0:000> !dumpobj 0000025a4f8c1920
Name: System.Threading.ExecutionContext
MethodTable: 00007ff8e2205000
Fields:
MT Field Offset Type VT Attr Value Name
00007ff8e2206000 4000001 8 ...ions.AsyncLocalValueMap 0 instance 0000025a4f8c1a00 m_localValues
00007ff8e2207000 4000002 10 System.Boolean 1 instance 1 m_isDefault
Now the AsyncLocalValueMap to see what values this ExecutionContext carries:
0:000> !dumpobj 0000025a4f8c1a00
Name: System.Threading.AsyncLocalValueMap
Fields:
MT Field Offset Type VT Attr Value Name
00007ff8e2208000 4000001 8 System.Object[] 0 instance 0000025a4f8c1b00 _array
0:000> !dumpobj 0000025a4f8c1b00
Name: System.Object[]
Size: 48(0x30) bytes
Array: Rank 1, Number of elements 3
Elements:
[0] 0000025a4f8c1c00 (MyApp.TenantContext)
[1] 0000025a4f8c1d00 (MyApp.TenantContext)
[2] null
0:000> !dumpobj 0000025a4f8c1c00
Name: MyApp.TenantContext
Fields:
MT Field Offset Type VT Attr Value Name
00007ff8e2209000 4000001 8 System.String 0 instance 0000025a4f8c1e00 UserId
00007ff8e2209000 4000002 10 System.String 0 instance 0000025a4f8c1f00 TenantId
0:000> !dumpobj 0000025a4f8c1e00
Name: System.String
String: user-A-guid-here
There it was. The ExecutionContext captured in this middleware state machine carried TenantContext.UserId = "user-A-guid-here". But this state machine was associated with User B’s request—the request that triggered the contamination detection. The HttpContext.User for this request contained User B’s identity. The AsyncLocal value contained User A’s identity.
The question was no longer whether the context was contaminated. It was where the contamination originated.
Correlating Thread-Pool Queue Depth with the Bleed
Next step: understand why the contamination appeared only under load. The hypothesis was that thread-pool pressure caused a specific continuation scheduling pattern that exposed the stale context. To test this, we needed to correlate the thread-pool state at the time of the dump with the contamination.
0:000> !threadpool
CPU utilization: 78%
Worker Pool:
Queue Length: 47
Thread Count: 64
Active Threads: 58
Min Threads: 12
Max Threads: 32767
Completion Port:
Thread Count: 8
Active Threads: 6
Queue Length: 3
A queue depth of 47 worker items with 58 active threads out of 64 total tells us the thread pool was saturated. The hill-climbing algorithm had not yet expanded the thread count further—likely because CPU utilization was already at 78%, and the algorithm backs off when adding threads would not improve throughput.
Under this pressure, continuations were being queued and dispatched with minimal delay between request boundaries. The key insight: when a continuation resumes on a thread pool thread, the runtime restores the ExecutionContext captured at the await point. If that ExecutionContext was captured with a stale value, the continuation runs with that stale value—regardless of what any other request has done to any other AsyncLocal in the meantime.
The contamination was not caused by thread-pool scheduling itself. Thread-pool pressure was the trigger condition that made the bug observable. The root cause was elsewhere.
The Root Cause: ExecutionContext Captured at Startup
Examining the middleware source code revealed the pattern that caused the bleed. Here is the problematic implementation, simplified to the essential structure:
public class TenantContextMiddleware
{
private readonly RequestDelegate _next;
private static readonly AsyncLocal<TenantContext> _currentContext = new();
// BUG: This delegate captures ExecutionContext at construction time
private readonly Func<TenantContext> _getContext = () => _currentContext.Value;
public TenantContextMiddleware(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
var tenant = ResolveTenant(context);
_currentContext.Value = tenant;
// The _getContext delegate was created during middleware construction,
// which ran during application startup. Its closure captured the
// ExecutionContext that was active at that moment — an empty context.
// But the delegate itself is shared across all requests.
context.Items["TenantResolver"] = _getContext;
await _next(context);
}
}
The delegate _getContext was constructed once, during middleware pipeline initialization. It captured the ExecutionContext active at construction time—the startup context, which had no tenant. The delegate was then shared across every request. When downstream code invoked _getContext(), it executed within the captured startup ExecutionContext, not the request’s ExecutionContext. Under low load, the timing happened to work out such that the value appeared correct—because the AsyncLocal‘s value handle resolved against whatever context was active on the thread. Under thread-pool pressure, the scheduling patterns exposed the discrepancy: the delegate’s captured context did not contain the per-request tenant value, and the fallback behavior produced stale or cross-thread values.
The fix was to eliminate the captured delegate and access the AsyncLocal directly, or to capture the ExecutionContext per-request:
public async Task Invoke(HttpContext context)
{
var tenant = ResolveTenant(context);
_currentContext.Value = tenant;
// Correct: resolve per-request, no shared captured context
context.Items["TenantResolver"] = new Func<TenantContext>(() => _currentContext.Value);
await _next(context);
}
With this change, each request gets its own delegate, created within the request’s ExecutionContext. The closure captures the correct context. The AsyncLocal value resolves correctly regardless of thread-pool scheduling.
Verifying the Fix with ETW Traces
After deploying the fix, we needed to verify that the contamination was eliminated—not just that it stopped appearing in the audit log, but that the underlying ExecutionContext propagation was correct. PerfView captured ETW events from the CLR’s System.Threading.ExecutionContext provider, and we correlated context switch events with request boundary markers from the ASP.NET Core hosting provider.
The verification methodology followed the structured postmortem approach described in the Google SRE Book, specifically the incident response and postmortem culture chapters. The process of reconstructing the timeline—which middleware ran in what order, against which request, with which ExecutionContext snapshot—is fundamentally an investigative narrative. Writing that narrative as a structured document is part of the forensic methodology, just as a novelist tracks character POV consistency across chapters using AI novel writing software that maintains narrative coherence across complex storylines, a debugging engineer tracks context flow across async boundaries. The structural problem is the same: multiple threads of execution, each carrying state that must remain consistent, and a single point where the wrong state surfaced at the wrong time.
The ETW trace confirmed that after the fix, every continuation resumed with an ExecutionContext that carried the correct tenant value for its originating request. No cross-request contamination appeared in 72 hours of production traffic at peak load.
The Repeatable Diagnostic Workflow
This investigation distilled into a repeatable workflow for any suspected AsyncLocal contamination:
1. Instrument for detection. Add a check at the point where the AsyncLocal value is consumed that compares it against a known-correct source (such as HttpContext.User). Trigger a dump capture on mismatch. Without this, you are guessing at timing.
2. Capture a full memory dump. A minidump without full memory will not contain the ExecutionContext instances you need to inspect. Use MiniDumpType.FullMemory or dotnet-dump collect --full on Linux.
3. Enumerate async state machines with !dumpasync. Identify the state machines associated with the middleware or handler where the contamination was detected. Note their object addresses.
4. Inspect the ExecutionContext on each state machine. Walk the fields: state machine → ExecutionContext → AsyncLocalValueMap → values. Compare the AsyncLocal values against the expected per-request values.
5. Check the thread-pool state with !threadpool. Correlate queue depth and active thread count with the contamination timing. Thread-pool pressure is the trigger condition that makes context-bleed bugs observable; it is not the root cause.
6. Examine the code path for captured ExecutionContext. Look for delegates, Func/Action closures, or callback registrations created during application startup or singleton initialization. Any closure created outside a request scope captures the ExecutionContext active at that moment, which will not contain per-request AsyncLocal values.
7. Fix by eliminating the shared captured context. Move the closure creation into the per-request code path, or eliminate the closure entirely and access the AsyncLocal directly.
8. Verify with ETW. Trace the ExecutionContext propagation through the fixed pipeline and confirm that each continuation resumes with the correct context.
Conclusion
AsyncLocal<T> is safe for per-request context propagation when used correctly. The danger is not in the type itself but in the invisible ExecutionContext snapshots that the runtime captures and restores at every await boundary. When a closure or delegate created at startup captures an ExecutionContext with no per-request state, and that closure is shared across all requests, the AsyncLocal values it resolves will be wrong. Not always. Not predictably. But under exactly the thread-pool pressure conditions that production traffic creates and development environments do not.
The forensic methodology here—dump during the contamination window, enumerate state machines, inspect ExecutionContext fields, correlate with thread-pool state, trace to the captured closure—applies to any AsyncLocal bleed scenario. The specific middleware pattern will vary. The diagnostic workflow will not.
