DOTNET_GCHeapHardLimitPercent vs the cgroup OOM Killer: Which Number Wins When a .NET 8 Container Dies at 512MB

The question is malformed, and the malformation is why the pod keeps dying. DOTNET_GCHeapHardLimitPercent and the cgroup OOM killer are not two thresholds racing to the same finish line. One bounds the commit size of the GC heap and GC bookkeeping. The other fires on cgroup-level memory usage that includes memory the GC limit never promised to bound. They do not compete. They measure different things, and only one of them can kill your process.

What follows is a triage order for a 512MB container that died. Forty minutes, one pod, no guessing.

What the GC limit actually bounds

Microsoft defines the heap hard limit precisely: “the maximum commit size, in bytes, for the GC heap and GC bookkeeping.” That is the whole contract. It is not a process memory limit, not an RSS limit, and not a cgroup limit.

In a container, the runtime treats the cgroup memory limit as total physical memory. The documentation states: “If the process runs inside an environment with a set memory limit (for example, a container), that limit is treated as the total physical memory. In such environments, the default value is 75%.” At 512MB, the default managed budget is therefore roughly 384MB of GC heap plus bookkeeping. The remaining ~128MB is expected to cover the runtime, native allocations, mapped images, thread stacks, and whatever else the process touches.

That expectation is the failure mode. The GC will happily keep its heap under 384MB while the process crosses 512MB in native memory, and the kernel kills the container. No managed OutOfMemoryException. No GC log entry. Just a SIGKILL and a restart count.

Which number is actually in effect

Before you tune anything, establish what the runtime read at startup. The precedence rules are documented and they are not intuitive:

HeapHardLimitPercent is ignored if per-object-heap hard limits or the byte-based heap hard limit are configured.
Per-object-heap percent settings are ignored if DOTNET_GCHeapHardLimitSOH, DOTNET_GCHeapHardLimitLOH, and DOTNET_GCHeapHardLimitPOH are specified.
If you set any per-object-heap byte limit, you must set SOH and LOH together or the runtime fails to initialize.

Then the format trap. The docs are explicit: “For number values, use decimal notation for settings in the runtimeconfig.json or runtimeconfig.template.json file and hexadecimal notation for environment variable settings.” A value of 30 in runtimeconfig.json means 30 percent. A value of 30 in DOTNET_GCHeapHardLimitPercent means 0x30, which is 48 percent. If your post-mortem says “we set 30” and the effective limit looks wrong, this is why.

And the timing trap: “These configurations are only read by the runtime when the GC is initialized (usually this means during the process startup time). If you change an environment variable when a process is already running, the change won’t be reflected in that process.” A live kubectl exec that exports a new value changes nothing. The pod must be recreated.

Record the effective configuration as the runtime saw it, not as the manifest claims. The docs point to GC.GetConfigurationVariables() as the way to inspect values in use at runtime. Put that output in the incident record.

Step 1: Did the process kill itself, or was it killed?

This is the fork in the road. Everything downstream depends on the answer.

If the process self-terminated with a managed OOM, a dump collected on crash will show it. Enable dump generation with DOTNET_DbgEnableMiniDump=1 and select the type with DOTNET_DbgMiniDumpType; the documented default is 2 (Heap). Then open the dump with dotnet-dump analyze and run analyzeoom, which “displays the info of the last OOM that occurred on an allocation request to the GC heap.” For an unhandled exception that killed the app, pe -lines prints the exception object and stack.

If the dump shows no managed OOM and the process simply vanished, you are looking at an external kill. The GC heap limit is not the prime suspect. Stop tuning it.

One caveat that has burned people: collecting the dump can itself trigger the kill. The dotnet-dump documentation warns that “collecting a full or heap dump may cause the OS to page in substantial virtual memory for the target process. If the target process is running in a container with an enforced memory limit, the increased memory usage may cause the OS to terminate the container if the limit was exceeded.” At 512MB, a Heap dump is a large fraction of the budget. If the pod dies during dump collection, you have learned something about headroom, not about the original fault.

Step 2: Was the managed heap near its limit?

You need counters from before the death, not after. dotnet-counters is the first-level tool here. The relevant signals are dotnet.gc.last_collection.heap.size by generation, dotnet.gc.last_collection.memory.committed_size, dotnet.gc.heap.total_allocated, and dotnet.process.memory.working_set.

Read them together. Two patterns matter:

Managed heap at or near the configured limit, committed size climbing, working set tracking it. The GC budget is the binding constraint. Tuning the limit is a legitimate response, but only after you understand why the workload needs more managed memory than 75% of 512MB allows.
Managed heap flat or well below the limit while working set and cgroup usage climb. The growth is outside the GC heap. Lowering DOTNET_GCHeapHardLimitPercent shrinks the managed budget without touching the actual growth. That is the wrong knob, and turning it will make the next incident arrive sooner.

Version note: “If the app uses .NET version 8 or lower, the System.Runtime Meter doesn’t exist in those versions and dotnet-counters will fall back to display the older System.Runtime EventCounters instead.” On .NET 8 you will see the EventCounter layout, not the Meter layout. The numbers are still there; the names differ.

Step 3: Account for memory the GC limit does not cover

This is where most 512MB container deaths actually live. The GC limit bounds managed heap commit. It says nothing about:

Native allocations from interop, cryptography, networking stacks, or third-party libraries.
Mapped images and the runtime itself.
Thread stacks, which multiply with thread pool growth.
File cache, which cgroup accounting includes.

The file cache point is documented in a G-Research write-up of a GC issue they traced and fixed: “On Linux, Kubernetes uses Docker to run our application. Docker limits memory usage via a cgroup. Cgroup memory limits (and usage figures) include the file cache.” That post describes a .NET 5-era problem where the GC misread cgroup usage that included file cache and entered aggressive GC mode. The fix landed for .NET 6 preview and as a .NET 5 servicing patch. Do not assume the same mechanism is your .NET 8 fault; the point that survives is the accounting one. Cgroup usage includes memory the GC does not manage.

So when managed heap is flat and cgroup usage climbs, the investigation moves to native memory, mapped images, file cache, dump collection overhead, and co-tenant processes. Not to the GC percent.

The multi-process trap

If your container runs more than one .NET process, the arithmetic changes. Each process applies its own percentage to the same cgroup limit. Two .NET processes at the default 75% of 512MB each budget ~384MB of managed heap. The sum exceeds the container limit before either process has done anything unusual.

A runtime collaborator addressed exactly this in a dotnet/runtime discussion: “With 3.0, the heap limits were introduced which cause a single process to nicely stay within the limits. If you’re running multiple .NET apps in a container, they will have the same issues as the single process containers before 3.0.” The same thread contains a reported case where a single-process container produced a GC OutOfMemoryException at the 75% limit, while a multi-process container was terminated by the OOM killer or the scheduler. That is the signature: managed OOM in one case, external kill in the other, same limit.

If you cannot split the processes into separate containers, you must divide the budget deliberately and record the division. The default assumes one process per cgroup.

The decision procedure

Given a 512MB container death, in order:

Collect the dump on crash. Set DOTNET_DbgEnableMiniDump=1 and choose a dump type that fits your headroom. If dump collection kills the pod, note that and reduce the dump type or raise the limit temporarily.
Run analyzeoom and pe. If there is a managed OOM, the GC heap limit is in play. If not, it is not.
Pull the counter timeline. Compare managed heap size and committed size against working set. Divergence means the growth is outside the GC heap.
Check the effective configuration. Confirm which of the hard-limit settings actually applied, in the correct number base, as read at startup.
Count the .NET processes in the cgroup. Multiply the per-process budget by that count and compare to the container limit.
Only then tune. If managed heap was at the limit, raise or lower the limit with a stated reason. If managed heap was flat, leave the GC settings alone and pursue native memory, file cache, or co-tenants.

What goes in the post-mortem

So the next incident is diagnosable in minutes rather than hours, record:

The effective GC configuration as the runtime reported it, including which hard-limit setting won and the number base used.
The cgroup limit and cgroup usage at the moment of death, not the average over the hour.
The dump type collected and whether collection completed before termination.
The counter timeline for managed heap size, committed size, and working set, with timestamps aligned to the death.
The count of .NET processes sharing the cgroup.

The number that wins is not the GC percent or the OOM killer threshold. It is whichever accounting boundary the process actually crossed. Find that boundary first. The rest is configuration.

FAQ

Does `DOTNET_GCHeapHardLimitPercent` prevent cgroup OOM kills?

No. It bounds the GC heap and bookkeeping commit size. The cgroup OOM killer responds to total cgroup memory usage, which includes native memory, mapped images, and file cache. A process can honor its GC limit and still be killed.

Why did my environment variable change have no effect?

GC configuration is read when the GC initializes, typically at process startup. Changing the variable in a running process does nothing. Recreate the process.

I set 30 and got 48 percent. Why?

Environment variable values are hexadecimal. 30 is 0x30, which is 48. Use decimal in runtimeconfig.json and hexadecimal in environment variables, or convert deliberately.

Can collecting a dump kill the container?

Yes. The dotnet-dump documentation warns that full or heap dump collection can page in substantial virtual memory and push a memory-limited container over its limit. Test your dump configuration against your actual headroom.

Two .NET processes in one 512MB container. What is the managed budget?

Each process applies the percentage independently to the same cgroup limit. At the default 75%, each budgets roughly 384MB of managed heap. The sum exceeds the container limit. Split the processes or divide the budget explicitly.

Where are the primary sources?

The GC configuration semantics, precedence, and number-base rules are in the runtime configuration options for garbage collection. Dump collection and analysis commands are in dotnet-dump and Collect dumps on crash. Counter names and the .NET 8 fallback behavior are in dotnet-counters. The multi-process container discussion is dotnet/runtime discussion 84828. The cgroup file-cache accounting point is from G-Research.