A CPU profile lands on your desk. The top frame is gc_heap::mark_phase. The pod is at 90% CPU and the next recycle is in forty minutes. The temptation is to reach for a GC knob. Resist it. A mark-phase sample tells you the GC spent CPU finding live objects during the capture window. It does not tell you whether the fix is fewer allocations, less survival, or a different heap configuration. Those are three different levers, and the artifact in front of you cannot choose between them.
This is a workflow for making that choice from evidence, in the order the evidence becomes available.
What the sample actually establishes
The mark phase’s job is to find all live objects (.NET GC design notes). When gc_heap::mark_phase dominates a sampled-thread profile, the runtime was spending wall-clock time traversing the object graph. That is the whole claim. It is consistent with a high allocation rate producing frequent collections, with a large surviving heap producing long collections, and with memory pressure tightening the allocation budget so collections happen more often than the workload alone would require. The sample cannot distinguish these.
Two properties of the tooling matter before you interpret anything. First, dotnet-trace collects traces through EventPipe without a native profiler, and its dotnet-sampled-thread-time profile samples .NET thread stacks at roughly 1000 Hz to estimate wall-clock time used by code (dotnet-trace documentation). That is a sampling estimate, not an accounting of CPU cycles. Second, the old cpu-sampling profile was removed because its name was misleading: it sampled all threads regardless of CPU usage (same page). If your runbook still references it, the profile name is wrong and the mental model behind it is worse.
So the first verdict is narrow: marking consumed CPU during the capture window. Everything after that is a question about why marking is doing so much work, and the answer lives in counters and heap state, not in the flame graph.
Get the counters before you get clever
dotnet-counters is built for exactly this stage: ad-hoc health monitoring and first-level performance investigation before deeper work with PerfView or dotnet-trace (dotnet-counters documentation). It is cheap enough to run against a live pod and it gives you the numbers that separate the hypotheses.
You need four things from the same window as the CPU samples:
- Allocation volume.
dotnet.gc.heap.total_allocatedgives cumulative allocated bytes; watch its delta per refresh interval to get a rate. - Collection counts by generation.
dotnet.gc.collectionsreports gen0, gen1, and gen2 counts separately. The ratio matters more than the absolute numbers. - Heap size after the last collection, by generation.
dotnet.gc.last_collection.heap.sizebreaks out gen0, gen1, gen2, LOH, and POH. This is where survival shows up. - Pause time and process CPU time.
dotnet.gc.pause.timeanddotnet.process.cpu.timelet you check whether GC pauses track the CPU spike or whether the CPU is going somewhere else entirely.
A minimal collection command:
dotnet-counters collect --process-id <PID> \
--counters System.Runtime[dotnet.gc.collections,dotnet.gc.heap.total_allocated,dotnet.gc.last_collection.heap.size,dotnet.gc.pause.time,dotnet.process.cpu.time] \
--refresh-interval 3 --format csv -o counters.csvRun it for at least several minutes, long enough to cover a load cycle. A thirty-second window during a traffic spike will tell you the GC is busy; it will not tell you whether the busy-ness is normal for that load.
Read the allocation budget, not just the allocation rate
The reason counters beat the flame graph is that the GC’s own trigger logic is visible in them. The allocator triggers a collection when the allocation budget for a generation is exceeded or when it can no longer allocate on a segment (GC design notes). The budget is a size limit set mostly from that generation’s survival rate (same document).
That single design fact drives the whole decision. High allocation with low survival produces frequent, cheap gen0 collections: the budget stays generous because most objects die young, and marking work per collection is small. High survival produces the opposite: the budget grows, collections become less frequent but each one marks more live objects, and gen2 growth eventually makes full or background collections expensive. The same mark-phase sample appears in both cases. The counters do not.
There is a third case that the counters catch and the profile cannot. In a production trace discussed in the dotnet/runtime repository, the GC maintainer attributed high percent time in GC to collections happening too often because the process was operating near 85% memory load, at which point the GC tightens the gen0 allocation budget (dotnet/runtime discussion #94411). The workload had not changed. The memory load had. If your counters show gen0 collection counts climbing while allocation rate is flat, look at committed size and the container limit before you look at the code.
Two related claims from that discussion are worth internalizing because they kill common misdiagnoses. There is no mechanism that stops a GC after one second (same discussion). And a non-full collection does not collect the full heap by design (same discussion). If your incident notes say “the GC timed out” or “the GC only collected part of the heap,” those notes describe a misunderstanding, not a runtime behavior.
When the question is survival, open a dump
Counters tell you that survival is high. They do not tell you what is surviving. That is a heap question, and the right instrument is a dump analyzed with SOS. dotnet-dump runs SOS commands to analyze crashes and the GC, but it is not a native debugger and does not display native stack frames (dotnet-dump documentation). For a managed heap question, that limitation is irrelevant.
Start with the summary sorted by total size:
dotnet-dump collect --process-id <PID> --type Heap -o heap.dmp
dotnet-dump analyze heap.dmp -c "dumpheap -stat" -c "exit"dumpheap -stat shows a summary of objects on the heap sorted by total size (dotnet-dump documentation). Read it before you read anything else. If the top entries are request-scoped buffers and strings with counts in the millions, you have an allocation-volume problem that happens to be surviving long enough to reach gen1 or gen2. If the top entries are a handful of large caches, dictionaries, or arrays that persist across requests, you have a retention problem and allocation cuts will not touch it.
One operational warning. Collecting a full or heap dump may cause the OS to page in substantial virtual memory, and in a container with an enforced memory limit the increased usage may cause the OS to terminate the container (dotnet-dump documentation). In Kubernetes or App Service Linux, check the limit against current committed size before you collect. If the margin is thin, collect from a replica that is already being drained, or raise the limit for the collection window if your platform allows it.
The decision: allocation cuts or heap tuning
With counters and a heap summary in hand, the choice is usually obvious. The table below is a decision aid, not a rule set; the thresholds are workload-specific and you should calibrate against your own baseline.
| Evidence pattern | Likely cause | First lever |
|---|---|---|
| High allocation rate, low gen2 growth, gen0 counts high but pause time low | Allocation volume | Allocation cuts |
| High allocation rate, gen2 growing, gen2 collections frequent | Allocation plus survival | Allocation cuts, then retention review |
| Flat allocation rate, gen0 counts climbing, committed size near limit | Memory-pressure budget tightening | Heap sizing and limit review |
| Low allocation rate, large gen2, long mark phases on background collections | Surviving heap size | Retention review, then GC flavor |
| High CPU with low GC pause time and low collection counts | Not GC | Re-collect the trace; the sample attribution is wrong |
The last row deserves emphasis. If dotnet.gc.pause.time is flat while CPU is pegged, the mark-phase frames in your profile may be a small fraction of samples that happened to land in the GC during a brief window, or the profile may have captured a different process. Confirm the attribution before you change anything.
Allocation cuts
Allocation reduction is the more direct lever when the evidence shows volume. It attacks the trigger itself: fewer bytes allocated means the allocation budget is reached less often, which means fewer collections, which means less total marking work. It does not require a restart, it does not interact with container limits, and it composes with every other fix.
The dump tells you where to cut. If dumpheap -stat shows a dominant array or string type, trace its allocation site with the gc-verbose profile, which tracks collections and samples object allocations (dotnet-trace documentation). Allocation sampling is statistical; treat the counts as a ranking, not a census.
Heap tuning
Configuration is the coarser lever, and it has a hard constraint: GC settings are read by the runtime when the GC is initialized, usually at process startup, and changing an environment variable on a running process will not be reflected in that process (runtime configuration for garbage collection). You cannot trial a GC setting on a live pod. You change it, restart, and observe. That makes it a poor first move when a faster, reversible lever exists.
When configuration is the right move, know which knob you are turning.
Workstation versus server GC. Workstation GC is the default; server GC is enabled through System.GC.Server or DOTNET_gcServer (runtime configuration for garbage collection). Server GC runs collection on multiple dedicated threads, with a heap and a dedicated thread per logical CPU, and collects the heaps at the same time (workstation and server GC). That parallelism is why server GC can shorten wall-clock mark time on a large heap. It is also why server GC is resource-intensive: the same documentation notes that many server-GC processes on few logical CPUs interfere with each other. In a Kubernetes cluster where a dozen pods share a node’s cores, server GC can turn a per-pod CPU problem into a node-level scheduling problem. Check the CPU limit before you assume server GC is the answer.
Background GC. Background (concurrent) GC is enabled by default and controlled by System.GC.Concurrent or DOTNET_gcConcurrent (runtime configuration for garbage collection). Disabling it makes gen2 collections blocking. That can reduce total CPU spent on concurrent marking in some workloads, at the cost of longer pauses. If your SLO is throughput and not tail latency, this is a legitimate trade. If your SLO is p99 latency, it is usually the wrong direction.
Heap hard limits. In an environment with a set memory limit, such as a container, the heap hard limit percent defaults to 75% of that limit (runtime configuration for garbage collection). This is the setting that interacts with the memory-pressure case above. If committed size is running near the limit and gen0 budgets are tightening, the fix may be to give the process more headroom, not to tune the GC. Raising a container memory limit is a configuration change too, and it is often cheaper than a code change.
LOH threshold. The large object heap threshold defaults to 85,000 bytes and is configurable through System.GC.LOHThreshold or DOTNET_GCLOHThreshold (runtime configuration for garbage collection). Raising it moves medium-sized objects onto the small object heap, where they are collected with ephemeral generations. That can reduce LOH growth and gen2 marking work, but it also increases gen0 and gen1 pressure. Do not change it without measuring both sides.
A worked sequence
Suppose the counters show a high allocation rate, gen0 collections in the hundreds per minute, gen2 growing steadily, and pause time climbing. The profile shows mark-phase samples. The sequence:
- Confirm the window. Re-collect counters and a trace over the same interval. If the mark-phase share of samples does not track the pause-time delta, stop and re-examine the attribution.
- Collect a heap dump from a replica with memory headroom. Run
dumpheap -stat. Identify whether the dominant bytes are short-lived types that survived, or long-lived retained structures. - If short-lived types dominate, the problem is allocation volume plus promotion. Cut allocations at the sites the dump identifies. Re-measure after deploy. Do not touch GC configuration yet.
- If long-lived structures dominate, the problem is retention. Review the cache or buffer that holds them. If retention is intentional and the heap is simply large, then and only then evaluate server GC, background GC settings, and heap limits against the container’s CPU and memory budget.
- If allocation is flat but gen0 counts are climbing, check committed size against the container limit. The GC may be tightening budgets under memory pressure. The fix is headroom, not code.
The order matters because each step is cheaper and more reversible than the next. Allocation cuts ship as a normal deploy. GC configuration requires a restart and a new observation window. Container limit changes require coordination with whoever owns the platform.
What not to conclude
Do not conclude that mark-phase samples imply allocation rate. They do not; they imply marking work, and marking work is a function of both collection frequency and live-object count. Do not conclude that a specific GC setting is correct for a workload without measuring that workload. Do not conclude that server GC is faster; it is parallel, and parallelism has a cost when cores are shared. And do not carry forward the one-second-timeout story: the runtime does not abort collections partway through, and a non-full collection not covering the full heap is by design, not a failure.
The GC being the hot path is a fact about where CPU went. The decision about what to do is a fact about why it went there. Get the counters, get the heap, then choose.
FAQ
Can I change GC settings without restarting the process?
No. GC configuration is read when the GC initializes, usually at process startup, and environment variable changes on a running process are not reflected (runtime configuration for garbage collection). Plan a restart and a fresh observation window.
Does a mark-phase sample mean the GC is the bottleneck?
It means marking consumed CPU during the capture window. Whether that CPU was necessary depends on allocation rate, survival, and memory pressure, which the sample does not show. Check dotnet.gc.pause.time against process CPU time; if pause time is flat, the attribution is suspect.
Why does the GC collect more often when memory is tight?
The allocation budget is set mostly from survival rate, and memory load influences how aggressively the GC collects. In the production trace discussed in dotnet/runtime, operating near 85% memory load caused the GC to tighten the gen0 allocation budget, producing high percent time in GC without a workload change (dotnet/runtime discussion #94411).
Is a heap dump safe to collect in a memory-limited container?
Not always. Collecting a full or heap dump may page in substantial virtual memory, and in a container with an enforced limit the increased usage may cause the OS to terminate the container (dotnet-dump documentation). Check headroom first, or collect from a draining replica.
What is the difference between the dotnet-sampled-thread-time and gc-collect profiles?
dotnet-sampled-thread-time samples thread stacks at roughly 1000 Hz to estimate wall-clock time used by code; gc-collect tracks GC collections only, at very low overhead (dotnet-trace documentation). Use the first to attribute CPU, the second to characterize collection behavior without the sampling overhead.