Why Assembly Binding Failures Are Harder to Diagnose Than You Think

02:14. The pod has restarted four times since the deploy and the log hands you one line: Could not load file or assembly 'Pricing.Contracts, Version=4.2.1.0, Culture=neutral, PublicKeyToken=...'. You exec in, list the directory, and the file is sitting right there. That is the moment this stops being a missing-file problem. On .NET 6 through 9, binding is not a filesystem lookup — it is resolution against state the runtime computed before your code ran, and at least four distinct failure modes print that same sentence. Here is the map, and where to spend the forty minutes you have before the next recycle.

The message describes a request, not a file

In .NET Framework, the Fusion loader probed the disk and fuslogvw showed you every probe path. Fusion is gone. On .NET 6-9 the runtime hands the name to the AssemblyLoadContext that owns the assembly making the reference — the ALC, the isolation boundary that owns load policy; the default ALC backs your main app. The default context checks the Trusted Platform Assemblies list first: the TPA, a closed set of paths that hostpolicy computed from .deps.json before Main. Only if that misses does it fall back to probing the application base and culture subdirectories. If the assembly is not on the TPA and not in a probe path, the copy on disk does not exist as far as binding is concerned.

So the first command is not ls. It is grep 'Pricing.Contracts' *.deps.json inside the container. An entry that is absent means your publish never emitted it — fix the artifact, not the folder. An entry that is present means the failure is in the path, the casing, or the version, and the next checks differ. If the failure names the runtime itself (Microsoft.NETCore.App), you are in hostfxr roll-forward territory and the app’s deps.json is irrelevant. The dependency-loading overview on Microsoft Learn maps the whole stack if you want the reference beside you.

The version in the message is the second trap. There are no binding redirects on .NET 6-9; unification happens once, at restore, where NuGet picks a single winner for each package and publish writes one file. The Version=4.2.1.0 in the error is what some compile-time reference asked for — it may never have existed as a file anywhere in the image. When the winner is older than what the calling code expects, you do not even get a load failure: you get MissingMethodException at a call site, which nobody files under binding. The restore-time NU1605 downgrade warning is the early signal, and most build setups let it scroll past.

Loading is lazy, so the failure is time-shifted

An assembly loads when the JIT first compiles a method that references a type in it, or when something calls Assembly.Load explicitly. The deploy is not the trigger; first execution is. A settlement job or an export endpoint that no canary ever hit detonates weeks later at 03:00, and the stack points at the call site — where the reference was — not at the restore decision that broke it months ago.

Publish modes bend this further. ReadyToRun changes JIT timing, not binding. Trimming deletes assemblies outright, which makes the failure deterministic while keeping the message identical. Single-file inverts the file-exists check: assemblies live inside the bundle, nothing is on disk, and Assembly.Location returns empty for bundled assemblies — a tell, not a bug. I once burned a Friday on a health check that verified a file in the folder; the app was single-file, the folder was irrelevant, and the real problem was a stale .deps.json baked into the layer next to the bundle.

Four signatures, four different root causes

Classify before you touch anything. The exception type is the coarse map onto the failure layers:

• FileNotFoundException (0x80070002). Resolution produced no candidate: missing from deps.json, a casing mismatch on ext4, a RID-specific asset that never shipped, or a plugin directory the custom ALC was never told about.
• BadImageFormatException (0x8007000B). A candidate was found and refused on format: x86 native in an x64 pod, a glibc-linked .so on Alpine’s musl, a truncated COPY in a Docker layer, or a .NET Framework binary loaded into a .NET 8 process.
• MissingMethodException / TypeLoadException. Binding succeeded and the contract did not: unification picked an older winner. This is the one that gets filed as a code bug for three days.
• FileLoadException (0x80131040). The ALC refused a duplicate identity — LoadFromAssemblyPath on a name already loaded in that context. Plugin hosts that reload configuration shims hit this weekly.

One more signature belongs on the list even though it is not a load failure: InvalidCastException between identical type names. That is the same assembly loaded into two ALCs — two runtime type identities that no cast will ever reconcile. The fix is load policy, not casting. The dump-side workflow for that pattern is in our AssemblyLoadContext debugging guide.

The Fusion log is gone — what replaces it, layer by layer

Startup failures: COREHOST_TRACE=1 and COREHOST_TRACE_FILE=/tmp/host.log dump hostfxr and hostpolicy’s work — the TPA they computed, framework roll-forward, additional deps. Host-time only; it goes silent once Main starts.

Managed loads later: nothing logs them by default. Hook AssemblyLoadContext.Default.Resolving at the top of Program.cs — log the requested name and the requesting assembly, return null so the failure still propagates — and every future incident becomes one log line instead of archaeology. AppDomain.CurrentDomain.AssemblyLoad logs successful loads for the audit trail, with the same empty-Location caveat for bundles.

Live evidence: the runtime emits AssemblyLoad events under the Microsoft-Windows-DotNETRuntime Loader keyword (0x8000). Run dotnet-trace collect -p <pid> --providers Microsoft-Windows-DotNETRuntime:0x8000:4 while you replay the failing request, then read the events in PerfView. On Windows Server, PerfView captures the provider by default — filter the Loader events and you have the load sequence with timestamps. The full resolution algorithm is in the managed assembly loading reference, and it is shorter than most people expect.

If it is already crash-looping: dump forensics

When the process dies at startup, you capture rather than attach. On Linux, createdump -f /dumps/app.dmp <pid>. In a pod, set DOTNET_DbgEnableMiniDump=1 and DOTNET_DbgMiniDumpType=2 (heap) or 4 (full), point DOTNET_DbgMiniDumpName at a mounted volume, and let the crash write its own evidence. We keep a fuller walkthrough of collecting crash dumps on Kubernetes that covers the pod-spec side.

Then dotnet-dump analyze — the same SOS commands work in WinDbg if that is your bench:

• pe -nested: the full exception chain, inner exceptions, HRESULTs.
• clrstack: the topmost app frame is the method whose JIT pulled the trigger — your time-bomb location.
• dumpdomain: every ALC and its assemblies. The same simple name in two contexts is the InvalidCastException case, decided in one command.
• dumpassembly <addr> and dumpmodule -mt <addr>: which file backed it; an empty path means bundled.
• name2ee <module> <type>: which copy of the type the runtime will actually bind.

The dump is the only place you can see both sides at once — what loaded, and what the request asked for. The dotnet-dump tool reference covers collection flags, and our dotnet-dump command cheat sheet has the SOS syntax in one page.

Container traps that multiply all of this

Casing: ext4 is case-sensitive, NTFS is not. A Dockerfile COPY that preserves wrong casing builds a layer that works on the Windows dev box and dies in the pod. Verify with ls in the container, not your IDE’s file tree.

Build output masquerading as publish output: copying bin/Release/net8.0 instead of the publish folder ships no processed deps.json, so fallback probing loads whatever it can see. The app half-boots and dies at the first missing reference. Copy the publish output whole; never cherry-pick DLLs into a layer.

RID-specific assets: native and some managed assets land under runtimes/linux-x64/native/ and only resolve when deps.json says so. Publish with -r linux-x64 — linux-musl-x64 on Alpine — or ship the runtimes tree intact.

Base image drift: same tag, different digest — Debian with glibc on one node pool, Alpine with musl on another, and a native BadImageFormatException that looks random across the fleet. Pin digests and match the RID to the base.

Two environment variables can also inject resolution paths you never configured: DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE. Check them in the pod spec before you blame the artifact; I lost most of a night to an inherited DOTNET_ADDITIONAL_DEPS that pointed at a store which existed only on the build agent.

Managed hosts add their own layer. Azure App Service Linux runs what is actually in /home/site/wwwroot — check via Kudu SSH that the deps.json you are reading is the one the platform deployed. On AWS Lambda, layers merge in mount order, so two layers shipping the same assembly at different versions means merge order decides the winner while the error names the loser. Verify what landed in the execution environment, not what you uploaded.

The 40-minute sequence

1. 0-5: classify. Exact message, HRESULT, full stack — kubectl logs --previous, the App Service log stream, or pe -nested from the dump. Match it to one of the four signatures. Touch nothing yet.
2. 5-15: resolve statically. Grep the deps.json for the simple name. Absent → publish artifact problem; stop here. Present → confirm the listed relative path exists in the container with exact casing, including the runtimes/<rid>/ subtree.
3. 15-25: match the failure’s age. Dies at startup → one run with COREHOST_TRACE. Loads fine and fails later → host trace is silent by design; hook Resolving in the next build, or capture live with the Loader keyword while you replay the request.
4. 25-40: dump and decide. dumpdomain: two copies of the simple name → ALC policy fix. One copy plus MissingMethodException → version split; dotnet list package --include-transitive to find who pulled what, then pin.

Decisions that prevent the next one

• Fail the build on NU1605. Downgrades are where unification incidents are born, and they are cheap to stop at restore.
• Pin package versions in one place with Central Package Management so the graph cannot split quietly.
• Add a CI step that verifies every deps.json entry resolves to a file in the publish output with exact casing. Ten lines of shell; it catches the casing and cherry-picked-DLL classes before they ship.
• Register the Resolving hook in your shared bootstrap and log to your normal structured logger. Future you gets one log line instead of a Saturday.
• Publish with an explicit RID and pin base image digests in both Docker stages.

FAQ

Where did the Fusion log go in .NET 8?

Nowhere you can reach. Fusion and fuslogvw are .NET Framework machinery. The replacements are per layer: COREHOST_TRACE for host-time resolution, the Resolving event for managed loads, and the Loader keyword (0x8000) for live load events.

Do binding redirects work in .NET Core or .NET 5+?

No. app.config redirects are ignored. Unification happens once, at restore and publish; if two packages need different versions, one file ships and everything binds to it. The symptom is NU1605 at restore or MissingMethodException at run. Fix the graph instead of trying to redirect at runtime.

Why does it work on my machine but not in the container?

Five usual answers, in order of frequency: file casing, build output copied instead of publish output, a RID-specific asset that never shipped, musl versus glibc in the base image, and a deps.json in the image that does not match what you published. Compare the container’s deps.json against your local one — that diff closes most of these tickets.

How do I tell which AssemblyLoadContext loaded an assembly?

In a dump, dumpdomain lists every context and its assemblies; a duplicate simple name across contexts is your answer. Live, the Resolving event hands you the requesting context directly in its arguments.

Binding failures are not hard because the loader is mysterious; the algorithm is documented and deterministic. They are hard because four root causes print one sentence, and the sentence describes what was asked for, never what was found. Classify the exception first. Everything after that is a command, not a debate.