Advanced .NET Debugging

The shirt reads I HAD POTENTIAL in white capitals. Beneath the words, a white curved ramp descends from an upper-left ledge to a lower-right flat. At the top of the ramp sits a dashed, empty circle. At the bottom rests a solid white ball. The ramp is curved, not a straight diagonal. That curvature is the first thing to notice, because it changes what the diagram is saying.

This is not a generic motivational slogan with a decorative swoosh. It is a small physics diagram, and the joke depends on reading it as one. The dashed circle marks where the ball was. The solid ball marks where it is. The curve is the path between them. The words above supply the verdict: the object had potential, and the potential is now spent.

I want to argue that the design works because it treats potential in both of its senses at once — the everyday sense of promise, and the physics sense of stored energy available to do work — and then shows the second sense emptying out. The shirt is funny because the diagram is accurate enough to be legible and bleak enough to be true.

What the drawing actually shows

Look at the geometry before the joke. A ball at the top of a curved ramp has gravitational potential energy relative to the bottom. Release it, and that potential converts to kinetic energy as it descends. At the bottom, on the flat, it has given up the height and keeps only whatever motion friction and the flat have left it. The dashed circle at the upper ledge is the ghost of the starting position. The solid ball at the lower ledge is the resting state.

The curve matters. A straight diagonal would read as a generic downward trend — a line going the wrong way. A curve reads as a track, a chute, a path something actually travels. The eye follows it from the empty dashed circle to the solid ball, and the direction of travel is downhill. The design does not need an arrow. The ramp is the arrow.

I am describing what the artwork shows, not making a claim about the physics education of whoever drew it. The diagram is a cartoon, not a free-body diagram. But it is specific enough that the joke lands only if you read the two circles as the same object at two times. That is the whole mechanism.

The double meaning is the argument

In ordinary speech, potential is a compliment you give someone young or promising: you have potential, meaning you have not yet done the thing but might. In physics, potential is a quantity — energy stored by virtue of position, available to be converted. The shirt collapses the two. The dashed circle is the promise. The solid ball is the outcome. The distance between them is the curve, and the curve is the whole story.

What makes it a good object rather than a cheap one is that it does not editorialize. There is no frowning face, no sigh, no extra caption. The words state a fact in the past tense — had — and the diagram supplies the evidence. The reader does the rest. That restraint is why the shirt can sit in a drawer for years and still read correctly; the joke is structural, not topical.

It also avoids the usual failure of sarcastic slogan shirts, which is over-explaining. A shirt that says I had potential and then adds a winking emoji or a second punchline is telling you it is funny. This one shows you a ball that has stopped moving and lets you notice.

Why this reads as a work-culture object

The people most likely to find this funny are the ones who have sat in a review and heard the word potential used as a softener. Potential is what you are told you have when the raise is not available, when the promotion is deferred, when the project is cancelled but the feedback is positive. It is a word that describes a future that has not been funded.

The shirt’s past tense is the joke: the potential was real, and it has been converted into a resting state. That is not a bitter message so much as a resigned one. It reads less like a complaint and more like a lab note. The ball is where it is. The dashed circle is where it was. Draw your own conclusions about the ramp.

I would not overclaim this as a statement about any particular employer or industry. It is a shirt. But the reason it works as an off-duty object for technical readers is that it treats a soft word with diagrammatic precision. Engineers tend to like that move — taking a vague term and pinning it to a measurable quantity, even as a joke.

The design’s real constraint: it has to be readable at a distance

A diagram joke has a harder job than a text joke. It has to survive being seen across a room, at an angle, in bad light, on fabric that moves. The shirt solves this by keeping the palette to two colors — white print on the garment — and by using large, simple shapes. The capitals are heavy enough to read before the diagram resolves. The dashed circle and solid ball are distinct silhouettes, not fine line art. The curve is a single confident stroke.

That is a design decision worth naming, because it is the difference between a shirt that works and a shirt that only works in a product photo. The joke is carried by the arrangement of shapes, not by small print. If you can see the two circles and the curve, you can reconstruct the meaning even if the words are partly obscured. That redundancy is what makes it a wearable object rather than a poster.

The shirt is listed as the I Had Potential Funny Physics Pun Sarcastic Minimalist Quote tee from Quirky Cotton Collective. The retail title calls it a physics pun and a minimalist quote; the artwork itself is the ramp-and-ball diagram described above. I am reading the drawing, not the marketing copy.

What the shirt is not

It is not a demonstration of anything. The diagram does not prove a physical result, and the shirt does not claim to. It is a cartoon that borrows the visual grammar of physics to make a point about a word. That distinction matters if you are the kind of reader who wants the joke to be technically correct. It is correct enough: a ball at the top of a ramp does have potential energy, and a ball at the bottom on a flat has less of it. The shirt does not need to be more rigorous than that.

It is also not a slogan about self-improvement. There is no invitation to climb back up the ramp. The dashed circle is empty. The ball is solid and still. The shirt’s mood is acceptance, not aspiration, and that is what keeps it from being another motivational tee with a downbeat font.

A note on the object

This publication and Quirky Cotton Collective share an owner. This is an editorial column about the artwork, not an independent review.

Diagnostic Procedure

The procedure here is reading the object, not debugging a runtime. Still, the method is the same one I use on a dump: state what you see, separate observation from interpretation, and check the claim against the artifact.

1. Open the listing image at full size. Confirm the printed elements: white capitals reading I HAD POTENTIAL; a white curved ramp descending left to right; a dashed empty circle at the upper-left ledge; a solid white ball on the lower-right flat. Expected output: four distinct elements, one curve, two circle states. If the ramp reads as a straight diagonal, you are looking at a different design or a distorted preview.
2. Identify the two circle states as the same object at two times. The dashed circle is the prior position; the solid ball is the current position. Expected output: a single implied trajectory from upper left to lower right. If you cannot pair the circles, the diagram does not resolve and the joke does not land.
3. Read the words in the past tense. The verb is had, not have. Expected output: the sentence describes a completed state, matching the ball at rest. If you read it as present tense, you have misread the shirt.
4. Check the palette and contrast. The print is white on the garment; the shapes are large and solid. Expected output: legible at conversational distance. If the dashed circle disappears at a few feet, the design is not doing its job.
5. Separate observation from interpretation. Observation: two circles, one curve, past-tense words. Interpretation: potential energy converted to a resting state; a joke about promise that has been spent. Expected output: you can state both without mixing them. If your reading requires a claim the artwork does not support, discard it.
6. Verification. Re-read the shirt as a stranger would: words first, then diagram. If the diagram changes the meaning of the words — if potential now reads as a quantity that has been used up — the design has worked. If the diagram is merely decorative, it has not.

The ball is at the bottom of the ramp. The dashed circle is empty. The shirt is correct about both.

The shirt prints a white rounded notification banner with a bell icon. Inside the banner, in plain sentence case: There is motion at your front door. Below the banner, a circular, green-tinted doorbell-camera view shows a wide-eyed orange tabby very close to the lens, porch siding and lights behind it. That is the whole design. No caption, no punchline, no second line of text. The joke is the gap between the alert and the image.

I keep coming back to the wording. It is not Someone is at your door. It is not Motion detected. It is There is motion at your front door — a full declarative sentence, grammatically complete, semantically empty. The notification tells you a category of event occurred at a location. It does not tell you what the event is, who or what caused it, or what you should do about it. That is the exact shape of a bad alert.

Anyone who has been on call recognizes the pattern. A page fires. The subject line names a symptom and a hostname. The body contains a threshold crossing. Nothing in the message tells you whether this is a cat or a burglar. You open the dashboard, pull the trace, and spend the first two minutes of the incident just establishing what you are looking at. The shirt is funny because the cat is the payload, but the banner is the setup — and the setup is a notification that technically satisfies its contract while failing its purpose.

Look at the cat’s position in the circular frame. It is not at the edge, not mid-stride, not doing anything. It is pressed close to the lens, eyes wide, filling the view. The camera has done its job: it detected motion and captured an image. The image is useless for classification. A human looking at this frame can tell it is a cat, but only after the fact. The notification arrived first, and the notification said nothing useful. The design puts the useless alert above the ambiguous image, which is the correct order of operations for the joke and the correct order of operations for a real incident.

The green tint is worth noting. Doorbell cameras often render night vision in green-tinted monochrome, and the artwork reproduces that convention. The porch siding and lights behind the cat are visible but indistinct. This is not a daylight capture. The motion happened at night, or in low light, which is when you least want an ambiguous alert. The visual language says: this is a real camera view, not a cartoon. The cat is a cat, but the frame is the kind of frame a real system produces.

I am not going to claim the shirt is about software. It is about a doorbell camera and a cat. But the reason it lands for technical readers is that the notification text is a perfect specimen of a message that reports an event without conveying information. The banner has a bell icon, which signals urgency. The sentence has a subject and a predicate, which signals completeness. Neither signal is true. The bell is not urgent. The sentence is not complete. The cat is the only thing in the design that carries actual information, and it arrives below the alert, in a frame that requires interpretation.

If you have ever written an alert that fired on a threshold without a runbook link, you have written this banner. If you have ever received a page that said elevated error rate without naming the service, the endpoint, or the error class, you have received this banner. The shirt is a portrait of that failure mode, rendered as a cat.

The product itself is a standard t-shirt from Quirky Cotton Collective, listed as There Is Motion At Your Front Door Funny Cat Doorbell Meme. The design is the argument. The banner is the alert. The cat is the event. The gap between them is the joke, and the gap is also the lesson.

Editorial note: This publication and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The garment preview shows one word, set in large bold white lowercase letters on a dark shirt: meh. The period is part of the design. There is no animal illustration, no secondary slogan, no visual gag competing for attention. The typography is the entire argument, and the argument is a shrug.

That is worth reading closely, because the design does something that most slogan shirts get backwards. It takes the least emphatic word available and renders it at maximum volume. The result is not a contradiction so much as a calibration problem the wearer gets to enjoy: the shirt shouts a word that means I decline to shout.

What the artwork actually shows

Stick to what is visible. Large, bold, lowercase, white, on a dark ground. The word is meh, followed by a period. No animal, no flag, no pride variant, no mascot. The lowercase choice matters: MEH would read as a command or a brand; meh. reads as a verdict delivered quietly and then filed.

The period is the sharpest element. A period is how you end a sentence you have already decided. It converts a filler word into a complete statement. Without it, the shirt is a mood. With it, the shirt is a ruling.

I am describing the design as printed, not claiming anything about the wearer’s personality, politics, or emotional availability. A shirt is not a diagnostic instrument. It is a piece of visual language, and this one is unusually efficient at it.

Why the loud/quiet mismatch works

Most sarcastic shirts try to be funny by adding. They stack a setup and a punchline, or pair a phrase with an illustration that explains the phrase. This one subtracts. It removes the joke’s scaffolding and leaves the punchline standing alone in a large font.

That is a real design decision, and it maps onto something technical readers deal with constantly: the difference between a message and a usable promise. A slogan is a claim you can print. A promise is a claim someone can hold you to. meh. is deliberately neither. It is a boundary marker with no enforcement mechanism, which is exactly why it reads as honest rather than aggressive.

If you have ever written a status update that says investigating and meant I have no idea yet but I am not panicking in public, you already understand the register. The shirt is the wearable version of that update.

The typography as a boundary statement

Large type on a dark shirt is a visibility choice. The wearer is not hiding the word; they are making it legible from across a room. But the word itself is a withdrawal. That combination is the whole design: I am willing to be seen saying this, and what I am saying is that I am not available for more.

This is different from an introvert slogan that explains introversion. It does not ask for accommodation or offer a personality profile. It sets a tone and stops. For readers who spend their working lives in incident channels where every message is expected to advance a diagnosis, a garment that advances nothing is a genuine off-duty object.

I would not read it as a statement about engineering culture, and I would not pretend it demonstrates anything about software, security, or team performance. It is a shirt. Its job is to be read at a glance and then left alone, which is also what the word is asking for.

Where the object sits

The listing is Meh Funny Sarcastic Saying Introvert Dry Humor, sold under the Quirky Cotton Collective brand. The design is the large bold lowercase meh. typography on a dark shirt, with no animal or pride variant. That is the object as observed; it does not establish a wearing experience, a review history, or a publication origin, and I am not going to invent one.

What I can say is that the design is coherent. The word, the case, the weight, the period, and the dark ground all point the same direction. Nothing on the shirt is trying to be cleverer than the shirt. That is rarer than it sounds in novelty apparel, where the temptation is always to add one more element until the original idea is buried.

A note on the demand signal

The keyword meh shirt shows a reported monthly range of 100 to 1,000 US searches, with flat three-month and year-over-year change and high ad competition, per the supplied Google Keyword Planner extract for September 2025 through August 2026. Related variants such as meh introvert shirt and meh sarcastic shirt are retained as related queries, not summed into an invented total. A steady, non-seasonal query with high competition is consistent with a durable low-key phrase rather than a trend spike, but I am not going to overread a search volume into a cultural thesis.

The more interesting observation is structural: the phrase is short enough to survive being printed at scale, and the design does not depend on a reference that will expire. A meme-adjacent shirt ages badly when the reference dies. meh. has no reference to die.

What to take from it

If you are evaluating the design rather than buying it, the useful question is not is this funny but does the visual treatment match the message. Here it does, in an inverted way: the message is low-energy and the treatment is high-visibility, and the tension between them is the joke. Remove either half and the design collapses. A small meh is just a word. A large MEH is a different, angrier shirt. The lowercase period is load-bearing.

That is the whole review. The shirt says one thing, says it loudly, and then stops talking. For a word that means no strong opinion, it has a surprisingly clear one about how to present itself.

You have a .NET 8 dump open. !dumpheap -stat shows a block of System.String and System.RuntimeType objects that do not sit in any generation you can name. !gcwhere on one of them returns something that does not look like gen0, gen1, or gen2. Before you call it a leak, decide which heap the object lives on. That decision is separate from whether the object is reachable, and conflating the two is what produces the false verdict.

What the NonGC heap is, precisely

The .NET 8 runtime introduced a heap that is not managed by the garbage collector. The design document states it plainly: “a specialized heap that is not managed by the Garbage Collector (GC) and is designed to store immortal objects with certain benefits for GC and code generation (codegen)” (NonGC Heap design doc).

The name is new. The mechanism is not. The same document notes the feature “is based on pre-existing Frozen Segments which were added long time ago to serve a similar purpose and were used on CoreRT/NativeAOT.” What changed in .NET 8 is that the functionality was “exposed to users through public profiling and debugging APIs” and is now “heavily leveraged for several codegen optimizations in RyuJIT.”

The rename was deliberate. The diagnostics issue that tracked the work records the decision: “we landed on disclosing the Frozen Object Heap as a new memory region that holds managed objects, but is distinct from the GC entirely. It has also been proposed that we stop calling it the Frozen Object Heap because it does support dynamic allocations at runtime. The current suggestion is to call it the Non-GC heap” (Diagnostic support for NonGC heap).

That rename has a consequence you will feel in every dump triage. The same issue states: “previously terms ‘GC heap’, ‘Managed heap’, ‘.NET heap’, ‘.NET Managed heap’ all meant the same thing and that is no longer true.” When a command says “GC heap,” it now means the GC-managed portion only. The NonGC heap is managed memory that is not in it.

Which objects qualify

The design doc lists what is commonly found on the NonGC heap: string literal objects; RuntimeType objects (CoreCLR-only at the time of writing); boxed statics; and certain static readonly fields that can be pre-initialized on NativeAOT or recognized by CoreCLR, such as Array<>.Empty.

The exclusions matter more for triage, because they tell you what will not be there and therefore what you should still expect to find on the GC heap:

• Non-immortal objects. The doc is explicit: “Only immortal objects should be allocated on NonGC heap to avoid ‘dead weight’ in that heap.”
• Objects from unloadable contexts. “NonGC heap should never be used for objects belonging to unloadable contexts. Doing so could lead to potential memory leaks. As a consequence, all string literals associated with unloadable Assembly Load Contexts (ALCs) won’t be placed in the NonGC Heap.”
• Objects that reference GC-heap objects. “NonGC heap must never contain references to GC heaps’ objects. However, it may contain references to other NonGC objects and GC heaps are allowed to reference NonGC objects.” The reference direction is one-way.
• Large objects and unusual alignment. “The current design of CoreCLR’s FrozenObjectHeapManager has certain design limitations such as: Large objects are not supported. Objects requiring unusual data alignment are not supported.”

The runtime source confirms the size limit at the allocator. FrozenObjectHeapManager::TryAllocateObject carries the comment: “May return nullptr if object is too large (larger than FOH_COMMIT_SIZE) in such cases caller is responsible to find a more appropriate heap to allocate it” (frozenobjectheap.cpp). The same file defines the commit chunk as 64 KB and the default reserved segment size as 4 MB, and doubles the reserved size of each new segment “to reduce the number of frozen segments in apps with lots of frozen objects.”

Why they never move

Three properties follow from the heap not being GC-managed, and each one changes what you see in a dump.

No mark-and-sweep participation. The design doc: “Since the objects in the NonGC Heap are always alive and the heap itself is not managed by the GC, there’s no need for the heap to be involved in the mark-and-sweep procedure. This also eliminates the need to root objects to a specific AppDomain or Assembly Load Context (ALC) to maintain their persistence.”

No write barriers. “Since the NonGC heap is not managed by the GC we no longer need to emit the write-barriers for objects in it.” The doc shows the before and after for a field assignment: pre-.NET 8 codegen loads a pinned handle, then the object, then calls CORINFO_HELP_ASSIGN_REF for the write barrier; the .NET 8 version moves a direct object reference into the field and returns.

Address embedding in RyuJIT. “The JIT compiler can ‘bake’ direct addresses into the code generation, removing the need for redundant indirect loads.” For a string literal, the pre-.NET 8 sequence loads the address of a pinned handle and then dereferences it; the .NET 8 sequence loads the object address directly. For typeof(int), the pre-.NET 8 sequence calls CORINFO_HELP_TYPEHANDLE_TO_RUNTIMETYPE; the .NET 8 sequence moves the RuntimeType address directly.

That is the mechanism behind “never move.” The JIT has embedded the address in generated code. If the object moved, the code would be wrong. The heap exists in part so the object cannot move.

Reading it in a dump

dotnet-dump analyze gives you an interactive shell that accepts SOS commands. The tool is not a native debugger: the documentation states it “allows you to run SOS commands to analyze crashes and the garbage collector (GC), but it isn’t a native debugger so things like displaying native stack frames aren’t supported” (dotnet-dump). For heap work that limitation is usually irrelevant.

The SOS command surface you already use is documented in the SOS debugging extension reference. DumpHeap “displays information about the garbage-collected heap and collection statistics about objects.” EEHeap “displays information about process memory consumed by internal runtime data structures,” with -gc limiting output to garbage collector data structures.

On the diagnostics issue, a contributor reported the state of tooling support: “As of now, SOS (in the main branch of dotnet/diagnostics) should fully support frozen objects in all commands. Feel free to raise a specific issue there if a command is misbehaving. For example, !dumpheap will enumerate those objects, !eeheap reports frozen segments, !gcwhere will report frozen objects, etc.”

Treat that as a statement about a specific branch at a specific time, not a permanent guarantee. Verify against the SOS build in front of you.

One caveat is worth carrying into every triage. In the same issue thread, a runtime engineer noted: “!eeheap -gc does include non-GC heap segments currently, just because they are threaded onto gen2 (for GC bookkeeping purposes for segments; for regions I’m thinking of making that a no op so it wouldn’t show up in sos by accident and instead will be tracked on the VM side which has this knowledge anyway).”

So a segment appearing under a GC-oriented view is not by itself proof that the objects in it are GC-managed. Cross-check with a command that reports the object’s location before drawing a conclusion. The GC design document also describes the physical layout: “There is also a NonGC heap that contains ro (readonly) segments” (Garbage Collection Design).

Telling NonGC from POH

Both heaps hold long-lived data. They are not interchangeable, and the design doc gives the reason the POH was rejected as the home for interned literals.

“The POH has a public API, allowing users to use it for both long- and short-living objects. This flexibility can create complications when these objects are mixed with immortal ones.” The doc walks through an interleaving of user buffers and JIT-created string and RuntimeType objects, then notes that because immortal objects are not all allocated at once, the arrangement “leads to fragmentation in case if e.g. buffer3 is no longer needed.”

The second reason is mechanical: “placing the objects in the POH wouldn’t allow us to omit write-barriers, and these objects would still need to be scanned, reducing the efficiency gains the NonGC Heap is designed to provide.”

For dump triage, the practical distinction is this. The POH is a GC heap. Its objects participate in collections, they have generations, and dotnet-counters reports its size alongside the other GC generations. The NonGC heap is not a GC heap. Its objects have no meaningful generation, and the GC counters do not cover it.

You can see the POH in counter output. The dotnet-counters documentation shows dotnet.gc.last_collection.heap.size broken out by gen0, gen1, gen2, loh, and poh. There is no nongc line in that output. The diagnostics issue is explicit about the intent: “All of our existing performance counters for gc-* are not intended to include non-gc heap information.” The same paragraph covers configuration: “All of our GC* environment variables solely control/constrain the behavior of the GC portion of the heap and have no impact on the Non-GC heap.”

The generation sentinel

If you call GC.GetGeneration on a NonGC object, you get a value that is not a generation. The design doc states: “The most visible change that now GC.GetGeneration(object) API returns int.MaxValue for such objects. This change serves to emphasize that these objects are not managed by the garbage collector.”

The issue that tracked the design proposed a different value: “we propose to return -1 to indicate that these objects do not belong to any generation.” The two sources disagree on the exact sentinel. Do not hard-code either value into a triage script. Test the return value against the runtime you are actually running, or avoid the API in diagnostic code and use the heap-location commands instead.

What matters for the dump is the concept: generation is a GC-heap property, not a universal object property. An object with no generation is not an object with a broken generation.

A triage checklist

Two passes, in order.

Pass one: which heap. For each suspicious object, establish its location before you reason about its lifetime. If it is enumerated by heap-walking commands but has no generation and does not move between collections, treat it as NonGC. Do not assume that a segment listed under a GC-oriented view contains only GC-managed objects; the bookkeeping caveat above applies.

Pass two: is it reachable. Only after you have placed the object on a heap should you ask whether anything roots it. For NonGC objects, the design doc adds a warning that changes the answer: “these objects should never be considered as ‘dead’ if no one references them since it’s possible that they’re actually kept alive by unmanaged data structures in VM itself.” Absence of a managed root is not evidence of a leak for these objects.

Record the runtime version and the SOS or ClrMD version alongside the dump. Diagnostic support for this heap landed incrementally, and the contributor statement above is scoped to a branch. A command that behaves one way in one SOS build may behave differently in another. The version is part of the evidence.

FAQ

Is the Frozen Object Heap the same thing as the NonGC heap? The NonGC heap is the current name for the concept. The design doc says the feature “is based on pre-existing Frozen Segments,” and the diagnostics issue records the rename because the heap “does support dynamic allocations at runtime.” Older material and some tool output still say FOH.

Can I allocate on it directly? No public allocation API is described in the sources here. The design doc contrasts it with the POH, which “has a public API.” The NonGC heap is populated by the runtime for the object categories listed above.

Will a NonGC object show up in !dumpheap -stat? The contributor report on the diagnostics issue says !dumpheap enumerates those objects. Confirm against your SOS build.

Why does !eeheap -gc show frozen segments? Because they are threaded onto gen2 for GC bookkeeping, per the runtime engineer’s comment in the diagnostics issue. The same comment describes a possible change for regions. Do not read the presence of a segment under -gc as proof of GC management.

Do GC counters include it? No. The diagnostics issue states the existing gc-* counters are not intended to include non-GC heap information, and the dotnet-counters output shows no NonGC line.

Why not just put interned strings in the POH? Because the POH has a public API and mixed lifetimes, which fragments when immortal objects are interleaved with user allocations, and because POH objects would still need to be scanned and would still require write barriers. The design doc gives both reasons.

Four tiles. That is the whole design. I WEAR THIS SHIRT PERIODICALLY runs across the chest, and beneath it sit four periodic-table-style squares: 53 I, 74 W, 22 Ti, 16 S. The tile symbols read I W Ti S. They do not spell the sentence. They spell the word the sentence is built on.

That gap between what the tiles say and what the sentence says is the entire joke, and it is worth slowing down on, because most people who glance at this shirt will read it wrong in a specific and predictable way.

What the tiles actually encode

Take the four symbols in order: I, W, Ti, S. Say them out loud and they collapse into periodically — or close enough that the ear finishes the word before the eye has caught up. The tiles are not a phonetic transcription. They are a set of chemical symbols that happen to sound like the syllables of an adverb when you read them as a sequence rather than as elements.

This is the part I find more interesting than the pun itself. A periodic-table tile is a compressed data format. It carries an atomic number, a symbol, and — to anyone who has spent time with the table — an implied position, group, and period. The design borrows that format and then uses it for something the format was never meant to do: spell a word by sound. The atomic numbers are real. The symbols are real. The word they assemble is not chemistry. It is a sentence about laundry frequency wearing the costume of a reference table.

I am describing what I see in the artwork, not making a claim about how the design was built or where the joke came from. What I can say from the tiles themselves is that the numbers and symbols are internally consistent with the real table — 53 is iodine, 74 is tungsten, 22 is titanium, 16 is sulfur — and that consistency is what makes the substitution land. If the numbers were wrong, the joke would still be readable but the format would be broken, and the format is doing half the work.

Why the sentence and the tiles disagree on purpose

Read the shirt literally and you get a contradiction. The sentence says the shirt is worn periodically. The tiles, read as elements, say iodine-tungsten-titanium-sulfur, which is not a material anyone wears. The resolution is that the tiles are not meant to be read as elements at all. They are meant to be read as syllables, and the periodic-table styling is a visual cue that tells you to expect a chemistry joke before you have parsed a single symbol.

That cue is the load-bearing element. Strip the tile borders and the atomic numbers and you have four letters in boxes. Keep them and you have a signal: this is a chemistry shirt, adjust your reading accordingly. The design trusts the reader to make that adjustment. It does not explain itself. There is no caption, no arrow, no parenthetical. The shirt assumes you will either get it in the first second or spend the next ten wondering why iodine is on a t-shirt.

I think that assumption is the honest part of the design. A lot of slogan shirts over-explain. They add a second line, a small-print punchline, a visual nudge. This one does not. It commits to the bit and lets the reader close the loop.

The difference between a slogan and a usable promise

Here is where the shirt connects to something I think about more than I probably should. In engineering work, we are surrounded by terse messages that look like slogans but are actually promises. A status page that says operational is a promise. A dashboard tile that says healthy is a promise. A runbook step that says verify is a promise. The words are short. The obligation behind them is not.

This shirt is the opposite kind of object. It is a slogan that makes no promise at all. It does not claim the wearer is a chemist. It does not claim the shirt is worn on a schedule. It claims only that the sentence and the tiles are in on the same joke, and that is a claim the design can actually keep. There is nothing to verify. There is no runtime behavior to check. The shirt does exactly what it appears to do, which is sit there and be read.

I find that restful, in the way a well-formed error message is restful. It says one thing. It means that thing. It does not require a follow-up investigation.

What the design does not do

It does not teach chemistry. The tiles are not arranged to show a reaction, a group trend, or a periodic property. They are arranged to spell a word. Anyone hoping the shirt doubles as a study aid will be disappointed; the atomic numbers are decoration in service of a pun, not a lesson.

It also does not resolve the ambiguity for a reader who does not know the symbols. If you do not recognize Ti as titanium, the tile still reads as a tile, and the joke still works phonetically. The chemistry knowledge is a bonus, not a prerequisite. That is a more forgiving design than it first appears, and I suspect it is why the phrase travels as well as it does — the pun is available to anyone who reads it aloud, and the format is available to anyone who has seen the table once.

Reading it as an object

The shirt is listed as I Wear This Shirt Periodically Funny Chemistry Science Pun under the Quirky Cotton Collective brand. The artwork is the four-tile arrangement described above. That is the whole object. There is no hidden second design, no back-print described in the listing, no additional text beyond the sentence and the tiles.

What I keep coming back to is how much the design asks of a glance. Most shirts in this category are read once and forgotten. This one has a small puzzle in it — not a hard one, but a real one — and the puzzle is the reason the shirt is more interesting than its sentence. The sentence is a statement. The tiles are a question. Together they are a joke that only works if you notice both.

If you wear it, you are wearing a claim about frequency and a set of element tiles that quietly disagree with it. That tension is the design. I do not think it needs to be resolved. I think it is the point.

Editorial note: This publication and Quirky Cotton Collective share an owner.

Editorial note: This publication and Quirky Cotton Collective share an owner.

The artwork is a closed loop of text and four smiling root vegetables. The words LET’S ROOT FOR EACH OTHER AND WATCH EACH OTHER GROW wrap around a carrot, a radish, a beet, and an onion. Each vegetable is drawn with its root visible in soil, and each has a face. The roots are not hidden; they are the point. The design is a gardening pun, but the pun has two meanings, and the second one is where the useful argument lives.

Rooting for someone is a common phrase. It means offering support, encouragement, a wish for their success. Rooting, in the botanical sense, is what a plant does underground before anything visible happens. The shirt puts those two meanings in the same sentence. The first half is social: let’s root for each other. The second half is biological: and watch each other grow. The conjunction and is doing quiet work. It suggests that the support and the growth are linked, that one enables the other, and that both are mutual.

That is a pleasant sentiment. It is also a slogan, and slogans are not promises. A slogan is a compressed argument. It asks the reader to fill in the rest. The question for a technical reader — someone who spends their days reading stack traces and verifying claims — is whether the argument holds up when you unpack it. Does root for each other mean anything specific? Does watch each other grow describe a healthy relationship or an expectation of perpetual improvement? The artwork does not answer those questions. It presents the loop and leaves the interpretation to the viewer.

The visual composition reinforces the loop. The text surrounds the vegetables, so the eye travels from word to word and back to the start. There is no beginning or end. The four vegetables are arranged as a group, not a hierarchy. The carrot is not larger than the radish. The beet is not more central than the onion. They are all in the soil, all smiling, all rooted. The design treats mutual support as a flat structure, not a top-down one. That is a visual choice, and it is consistent with the wording.

But the second half of the sentence introduces a subtle demand. Watch each other grow implies observation and progress. Growth is measurable. It is the kind of word that appears in performance reviews and quarterly goals. In a work culture, watch each other grow can become monitor each other’s improvement. The shirt does not say that, but the phrase is available for that reading. The smiling vegetables do not look stressed, but they are also not in a position to object. They are illustrations.

This is the difference between a slogan and a usable promise. A usable promise is specific. It names what will be provided, by whom, under what conditions, and what happens if it is not. A slogan is a mood. The shirt is a mood. That is fine for a t-shirt. It is not fine for a team charter. The mistake would be to treat the shirt’s sentiment as a policy. Let’s root for each other is not a substitute for a blameless postmortem process. Watch each other grow is not a substitute for a career development plan. The artwork does not claim otherwise. It is the viewer who might.

For technical readers, there is a familiar pattern here. A slogan is like a log message that says something went wrong. It is not a diagnosis. It is a prompt to investigate. The shirt’s prompt is about mutual support and growth. The investigation is whether the words describe a practice or just a feeling. The artwork itself does not provide the evidence. It provides the question.

The design is available as a standard t-shirt from Quirky Cotton Collective on Amazon. The listing title is Let’s Root For Each Other And Watch Each Other Grow Garden. The visual observation is the one described above: four smiling root vegetables with visible roots, surrounded by the text. That observation establishes the object. It does not establish a wearing experience, a review, or a popularity claim. The shirt is a printed garment. The argument is in the reading.

If the shirt were a diagram, it would be a cycle, not a pipeline. A pipeline has inputs and outputs, stages and failure points. A cycle has no exit. The mutual rooting and watching is a cycle. That is appealing in a garden. It is less appealing in a team that never ships. The artwork does not distinguish between the two. It simply presents the loop. The viewer brings the context.

That is the useful argument: the shirt is not a promise of mutual growth. It is a visual pun that borrows the language of growth and support. It works as a gardening shirt because the pun is literal — the roots are visible. It works as a work-culture shirt only if the viewer supplies the missing specifics. The design does not do that work. It cannot. It is a t-shirt.

The smiling vegetables are the tell. They are not growing in the sense of changing. They are already grown. The carrot is a carrot. The beet is a beet. The growth they represent is the growth that already happened underground, before the visible part. The shirt celebrates that hidden phase. That is a more precise reading than watch each other grow as a continuous demand. The roots are the evidence. The visible part is the result. The mutual part is the soil.

So the shirt is best read as a reminder, not a mandate. It says: support is underground work. Growth is what becomes visible later. The each other is the shared soil. That is a defensible interpretation of the artwork. It is not a historical claim about the phrase’s origin, and it is not a scientific claim about plant physiology. It is a reading of the design as presented.

For a technical reader, the takeaway is not about gardening or teamwork. It is about the difference between a slogan and a specification. The shirt is a slogan. It is a good slogan because it is compact, visual, and open to interpretation. But if you want a usable promise, you have to write it yourself. The shirt will not do it for you. It will just smile back.

A 2GB .etl is not a file you open. It is a file you stream. PerfView will load it, but the UI is a liability when the incident clock is running: symbol resolution, tree building, and the WPF heap all compete with the pod that is about to recycle. The TraceEvent library is the same engine PerfView uses, and it is scriptable. This is the procedure for turning a multi-gigabyte ETW or EventPipe trace into a top-allocators report keyed by type and time interval, from a console process, with no UI in the loop.

What the trace actually contains

Allocation data in a .NET trace comes from the CLR runtime provider Microsoft-Windows-DotNETRuntime. The relevant keyword is gcsampledobjectallocationhigh (hex 0x200000) for the high-detail sampled allocation events, and allocationsampling (hex 0x80000000000) for the newer allocation sampling keyword. The gc-verbose profile in dotnet-trace is documented as tracking GC collections and sampling object allocations; the gc-collect profile tracks collections only at very low overhead. If your trace was collected with gc-collect, there are no allocation events to aggregate, and no script will conjure them. Confirm the profile before you write a line of code.

Two event shapes matter:

• GCAllocationTick — emitted by the runtime on the allocation sampling path. It carries an allocation amount and an allocation kind (Small, Large, or Pinned), plus a type name when the type-name keyword is enabled.
• Type-name resolution — the type name on the tick is a TypeID. Resolving it to a string requires the type-name events that the runtime emits when gcheapandtypenames (hex 0x1000000) is enabled. Without that keyword, you get a TypeID and a number, not a class name.

The dotnet-trace documentation lists both keywords explicitly in the CLR provider keyword table, so you can verify what was requested at collection time by reading the collection command or the profile name recorded in the trace metadata.

Choose the right entry point for the format

TraceEvent exposes different front doors for the two container formats. Consult the TraceEvent documentation for the current reader API for your version of the library.

• .etl — Windows ETW. Use the TraceEvent reader for .etl files and drive the callback loop as shown in the library’s samples.
• .nettrace — EventPipe. Use the EventPipe reader over a Stream, or the EventPipe dispatcher for live sessions. dotnet-trace is built on EventPipe, and its default output format is NetTrace.

If you have a .nettrace and you want the .etl path, dotnet-trace convert exists, but it is a conversion, not a free lunch: it rewrites the file and can change event ordering or drop events the converter does not understand. Prefer the native reader for the format you actually collected. If the trace came from dotnet-trace collect-linux, note the documentation’s warning that the .nettrace format there is updated and that convert and report may not work with it yet. Read it with the EventPipe reader and do not round-trip it.

The streaming pattern

Do not materialize events. The TraceEvent callback model is a push loop: you register handlers, then call Process(), and the library reads the file in blocks and invokes your callbacks. Your job is to keep the callback cheap and the state small.

using Microsoft.Diagnostics.Tracing;
using Microsoft.Diagnostics.Tracing.EventPipe;
using Microsoft.Diagnostics.Tracing.Parsers;
using Microsoft.Diagnostics.Tracing.Parsers.Clr;

// .etl path
using var log = TraceLog.CreateFromEventTraceLogFile(etlPath);

// .nettrace path
// using var src = new EventPipeEventSource(nettracePath);

var clr = new ClrTraceEventParser(log);

// Bucket state: (intervalIndex, typeName) -> bytes
var buckets = new Dictionary<(int, string), long>();
var typeNames = new Dictionary<long, string>();

clr.TypeBulkType += data =>
{
    for (int i = 0; i < data.Count; i++)
    {
        var t = data.ClrInstanceID; // placeholder; see note below
    }
};

clr.GCAllocationTick += data =>
{
    var typeName = data.TypeName ?? $"TypeID:{data.TypeID}";
    var interval = (int)((data.TimeStampRelativeMSec - startMs) / intervalMs);
    var key = (interval, typeName);
    buckets.TryGetValue(key, out var cur);
    buckets[key] = cur + data.AllocationAmount64;
};

log.Process();

Three things in that sketch are load-bearing and worth stating plainly:

1. Use TimeStampRelativeMSec, not wall clock. ETW timestamps are QPC-based and the library normalizes them to milliseconds relative to the trace start. Wall-clock conversion is a separate step and is not needed for interval bucketing.
2. Resolve TypeID lazily. The type-name events arrive interleaved with allocation ticks. If you resolve eagerly you will miss names that arrive later; if you resolve at the end you need to keep the TypeID-to-name map alive. The map is small compared to the event stream, so keep it and resolve at report time.
3. Do not call data.TypeName if the keyword was not enabled. It will be null or empty, and you will silently bucket everything under one key. Check once at startup whether the trace contains type-name events; if not, fail fast with a message that names the missing keyword.

Memory: the 2GB problem is not the file, it is your dictionary

A 2GB .etl with allocation sampling at the default rate can contain tens of millions of ticks. If you key your dictionary on (interval, typeName) and the type name is a fresh string per event, you will allocate a string per tick and the process will die before the report prints. Two mitigations:

• Intern type names. Keep a Dictionary<long, string> from TypeID to a single canonical string, and key buckets on the TypeID, not the string. Resolve to strings only when you emit the report.
• Bound the interval count. A 10-minute trace at 1-second intervals is 600 buckets per type. That is fine. A 10-minute trace at 1-millisecond intervals is 600,000 buckets per type, and that is not fine. Pick the interval from the question you are answering, not from the resolution of the trace.

The dotnet-trace documentation notes that the in-memory buffer defaults to 256 MB and that if the target emits events faster than they can be written to disk, the buffer may overflow and events will be dropped. That warning applies to collection, but the same arithmetic applies to your reader: if your callback is slow, the library’s internal queues grow. Keep the callback to a dictionary increment and nothing else.

Intervals and session boundaries

An .etl can contain multiple trace sessions if it was merged. The TraceLog exposes session boundaries; if you bucket purely on relative milliseconds you will splice two sessions into one timeline and produce a report that looks like an allocation spike at the seam. Check the session count before you process, and if it is greater than one, either split the file or offset the second session’s timestamps. The same applies to .nettrace files that were concatenated.

For a single-session trace, the interval math is trivial: interval = floor((TimeStampRelativeMSec - t0) / intervalMs). Emit the report as interval start time in relative milliseconds, plus the type name and the summed allocation bytes. If you need wall-clock labels, convert once at the end using the trace’s start time, not per event.

What the report should say

A top-allocators report that is useful during an incident has three columns and one sort:

1. Interval — relative start, in milliseconds or seconds.
2. Type — the resolved class name, or TypeID:<n> if the keyword was missing.
3. Allocated bytes — the sum of AllocationAmount64 for that interval and type.

Sort by allocated bytes descending within each interval. Do not sort globally; the point of the interval dimension is to see the type that dominates now, not the type that dominated the whole trace. A type that allocates 4GB over ten minutes but nothing in the last thirty seconds is not your incident. A type that allocates 200MB in the last thirty seconds is.

If the trace has the type-name keyword, also emit a second view: top types by total allocated bytes across the whole trace, with the interval of their peak. That is the view that tells you whether the spike is a new type or an existing type that changed rate.

When the script is the wrong tool

Two cases where you should stop scripting and use something else:

• The trace has no allocation events. If it was collected with gc-collect or without the allocation keywords, the script has nothing to aggregate. Recollect with --profile gc-verbose or with explicit --providers "Microsoft-Windows-DotNETRuntime:0x200000:4" and accept the overhead. The dotnet-trace docs list the keyword hex values so you can construct the provider string without guessing.
• The trace is truncated. The dotnet-trace documentation states that an unhandled exception during collection results in an incomplete trace, that the trace is truncated when the runtime shuts down, and that Rundown information will be missing, so stacks may be unresolved. Allocation ticks near the end of such a trace may be present but type names may not be. The script should detect the missing type-name events and report the trace as incomplete rather than emitting a report that silently attributes bytes to TypeID:0.

If you need stacks rather than types, TraceEvent can give you them, but the cost is different: stack events are larger, and the aggregation key becomes a frame list rather than a TypeID. That is a separate report and a separate memory budget. Do not try to produce both from one pass unless you have measured the working set.

Verification before you trust the numbers

Cross-check the script’s total against a counter. dotnet-counters exposes dotnet.gc.heap.total_allocated from the System.Runtime meter, and the documentation shows it as a cumulative byte counter. If your script’s total allocated bytes for the trace window is within the same order of magnitude as the delta of that counter over the same window, the script is reading the right events. If it is off by 100x, you are reading the wrong keyword or double-counting a merged session.

One more check: allocation sampling is sampling. The runtime does not emit a tick for every allocation. The report is a distribution, not an accounting ledger. Use it to rank types and intervals, not to assert that a type allocated exactly N bytes.

FAQ

Can I read a .nettrace with TraceLog? No. TraceLog is the ETW reader. Use the EventPipe reader for .nettrace, or convert with dotnet-trace convert and accept the conversion’s limitations. The dotnet-trace docs warn that convert and report may not work with the updated .nettrace format produced by collect-linux.

Why is my type name null? The trace was collected without the gcheapandtypenames keyword. The allocation tick carries a TypeID; the name comes from a separate event family. Recollect with the keyword enabled, or accept TypeID-only output.

How do I handle a trace with multiple sessions? Detect the session count before processing. If it is greater than one, split the file or offset timestamps at the seam. Bucketing on relative milliseconds across sessions produces a false spike.

Is there a command-line alternative to scripting? PerfView has command-line modes, and dotnet-trace report exists, but the report verbs are limited and the documentation notes they may not work with newer .nettrace variants. For a custom top-allocators-by-interval view, the TraceEvent library is the supported path.

Sources

• Microsoft, dotnet-trace performance analysis utility, https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace — CLR provider keyword table, gc-verbose and gc-collect profile descriptions, buffer overflow warning, truncated-trace and missing-Rundown behavior, collect-linux format caveat.
• Microsoft, Investigate performance counters (dotnet-counters), https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters — dotnet.gc.heap.total_allocated counter and System.Runtime meter.
• Microsoft, Dump collection and analysis utility (dotnet-dump), https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump — dump types and SOS command surface, for the cases where the trace does not answer the question and a heap dump does.

Alert at 03:47 UTC: PagerDuty fires — pod api-gateway-7d4f6b-x2k9 in the prod-east-2 cluster hit OOMKilled. Kubernetes restarted the pod. The previous 30 minutes show a steady climb in memory from 800 MB to the 2 Gi cgroup limit. Three more pods trending the same way. The service is an ASP.NET Core 8.0 API running on Linux containers with Server GC enabled, handling roughly 4,000 RPS of authenticated API traffic. No deploy in the last 48 hours. No traffic spike. Something is accumulating.

Restarting and moving on is tempting. But the pattern — gradual, monotonic memory growth across multiple pods — points to a managed heap leak, not a transient spike. What follows is a hypothesis-driven methodology: observe, hypothesize, collect data, confirm or kill each hypothesis, refine, repeat until root cause. This is the structured troubleshooting approach described in the Google SRE Book (Chapter 12, Effective Troubleshooting), and it is the discipline that separates a 20-minute fix from a 4-hour war room.

Hypothesis 1: LOH Fragmentation or Native Memory Pressure

First assumption to test: is large object heap fragmentation causing GC to fail reclaiming memory, or is native memory (unmanaged buffers, memory-mapped files, stack) responsible? On .NET 8 with Server GC, the runtime uses segmented heap layouts (regions on .NET 9, segments on .NET 8), and LOH fragmentation can mimic a managed leak if large arrays are allocated and pinned. The way to distinguish: check dotnet-counters for GC heap size versus working set, and look at the GC pause time pattern.

Before the next pod OOMs, capture live counters from a trending pod:

dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
    gc-heap-size                : 1,847 MB
    gen-2-gc-count              : 342
    gen-2-size                  : 1,612 MB
    loh-size                    : 89 MB
    working-set                 : 1,983 MB
    gc-pause-time               : 287 ms

What to notice: gc-heap-size is 1,847 MB out of a 1,983 MB working set. The managed heap dominates. LOH is only 89 MB — fragmentation there cannot explain 1.6 GB of Gen 2 size. Gen 2 GC count is 342 and climbing, with pause times of 287 ms. This is a managed leak promoting objects to Gen 2, not LOH fragmentation or native pressure.

Verdict: Hypothesis 1 killed. LOH is small. Working set tracks heap size. Native memory is not the driver. The leak is in the managed heap, and it is accumulating in Gen 2.

Hypothesis 2: Unbounded ConcurrentDictionary in a DI Singleton

Gen 2 accumulation with high GC frequency and long pause times means objects are being promoted to Gen 2 and retained there. The most common pattern: a static or singleton cache with unbounded key cardinality. Before dumping, check which pods are still alive and grab a gcdump from the one closest to OOM:

kubectl exec -it api-gateway-7d4f6b-x2k9 -- dotnet-gcdump collect -o /tmp/leak.gcdump
kubectl cp prod-east-2/api-gateway-7d4f6b-x2k9:/tmp/leak.gcdump ./leak.gcdump

Open leak.gcdump in PerfView or Visual Studio’s memory profiler. The type summary shows:

Type                                  Count       Size (bytes)
System.String                         1,847,293   312,894,016
System.Linq.Expressions.Expression`1  923,641     147,782,560
System.Byte[]                         923,641     88,672,816
System.Object[]                       923,641     58,952,904
System.Func`3                         923,641     47,289,152
... (thousands of unique types)

What to notice: nearly 1.8 million strings and 923,641 expression-related objects. The ratio is roughly 2:1 strings to expressions — consistent with a dictionary where each entry has a key (string) and a value (compiled expression delegate plus its captured closures). The total here is ~655 MB in counted types, but the gcdump only shows object sizes, not the aggregate retained size including references. The actual retained footprint is much larger.

Click into System.Linq.Expressions.Expression<TDelegate> and look at the roots. PerfView shows the retention path:

Root: System.ConcurrentDictionary`2[[System.String, mscorlib],[System.Func`3[...]]]
  └─ static field: ApiGateway.Services.ExpressionCache._cache
       └─ ConcurrentDictionary`2.Node[]
            └─ ConcurrentDictionary`2.Node
                 └─ key: "user:tenant=acme:query=SELECT * FROM orders WHERE status='{userInput}' AND region='{region}'"
                 └─ value: Func`3 (compiled expression delegate)

There it is. A static ConcurrentDictionary<string, Func<...>> keyed by an interpolated string that includes user-supplied query input. Every unique user query produces a unique cache key, a new expression tree, a new compiled delegate, and a new string. The cache grows without bound because the key space is unbounded.

Verdict: Hypothesis 2 confirmed. The retention path is clear in the gcdump. But a gcdump shows aggregated roots — to prove the exact retention chain and rule out other roots, we need a full core dump with SOS.

Collecting the Full Dump Before the Pod Dies

The pod that served the gcdump OOM-killed 4 minutes later. We need a dump from a pod that is trending but still alive. Set up a watch on memory and trigger a dump at 1.5 Gi:

kubectl exec -it api-gateway-7d4f6b-x5m2 -- bash -c 'while true; do rss=$(grep VmRSS /proc/1/status | awk "{print \$2}"); if [ $rss -gt 1572864 ]; then dotnet-dump collect -p 1 -o /tmp/full.dmp && break; fi; sleep 5; done'

This watches RSS via /proc/1/status and triggers dotnet-dump collect when it exceeds 1.5 Gi (1,572,864 KB). The dump lands at /tmp/full.dmp. Copy it out:

kubectl cp prod-east-2/api-gateway-7d4f6b-x5m2:/tmp/full.dmp ./full.dmp

The dump is 1.9 GB. On Linux, analyze it with dotnet-dump analyze (or copy to a Windows machine with WinDbg + SOS — both work on .NET 8 Linux dumps). I will use dotnet-dump analyze here since it runs on the same Linux jump box:

dotnet-dump analyze ./full.dmp

Dump Analysis: Confirming the Retention Path with SOS

First, verify the runtime version and heap state:

> clrstack -l
OS Thread Id: 0x1 (1)
        Child SP               IP Call Site
00007F8B5BFFC7A0 00007f8b6a123456 [HelperMethodFrame: 00007f8b5bffc7a0]
00007F8B5BFFC7B0 00007f8b6a0f8c23 System.Threading.ConcurrentDictionary`2[[System.String, System.Private.CoreLib],[System.Func`3[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]].GetOrAdd(...)

What to notice: a thread is actively inside ConcurrentDictionary.GetOrAdd at the moment of the dump. This is the cache being written to under load.

Check the heap statistics for the dominant types:

> dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
00007f8b6a0d4a20  923641   147782560 System.Linq.Expressions.Expression`1[[System.Func`3...]]
00007f8b6a0d3e80 1847293   312894016 System.String
00007f8b6a0d2c10  923641    88672816 System.Byte[]
00007f8b6a0d1f50  923641    58952904 System.Object[]
00007f8b6a0d0e90  923641    47289152 System.Func`3[[System.String, System.Private.CoreLib]...]
...
00007f8b6a0d4a20  923641   147782560 System.Linq.Expressions.Expression`1
Total 4,847,223 objects, Total size: 1,612,894,016 bytes

What to notice: the counts match the gcdump exactly — 923,641 expression trees, 1,847,293 strings. The total managed heap is 1.6 GB, almost all in Gen 2. The 2:1 string-to-expression ratio confirms each cache entry creates one expression and at least one string key (the string count is higher because expression trees themselves contain string literals for parameter names and method names).

Now, pick one expression object and trace its root:

> dumpheap -type System.Linq.Expressions.Expression`1 -short
00007f8b4a123450
00007f8b4a1235f0
00007f8b4a123790
... (923,641 entries)

Grab the first address and run !gcroot:

> gcroot 00007f8b4a123450
HandleTable:
    00007f8b6b001200(pinned handle)
    -> 00007f8b4a200000 System.Object[]
    -> 00007f8b4a201100 ApiGateway.Services.ExpressionCache
    -> 00007f8b4a201210 System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, System.Private.CoreLib],[System.Func`3...]]
    -> 00007f8b4a205000 System.Collections.Concurrent.ConcurrentDictionary`2+Node[]
    -> 00007f8b4a206100 System.Collections.Concurrent.ConcurrentDictionary`2+Node
    -> 00007f8b4a123450 System.Linq.Expressions.Expression`1[[System.Func`3...]]

What to notice: the root is a pinned handle pointing to an object array, which holds the ExpressionCache singleton instance, which holds the ConcurrentDictionary, which holds the node array, which holds the individual node, which holds the expression. This is a strong reference chain from a static root through DI to the cache. The GC cannot collect any of these objects.

Examine one of the string keys to confirm the cardinality problem:

> dumpobj 00007f8b4a206120
Name:        System.String
MethodTable: 00007f8b6a0d3e80
EEClass:     00007f8b6a0d3d80
Size:        164(0xa4) bytes
String:      user:tenant=acme:query=SELECT * FROM orders WHERE status='pending_shipment' AND region='us-east-1'

> dumpobj 00007f8b4a206220
Name:        System.String
MethodTable: 00007f8b6a0d3e80
EEClass:     00007f8b6a0d3d80
Size:        172(0xac) bytes
String:      user:tenant=acme:query=SELECT * FROM orders WHERE status='pending_refund' AND region='us-east-1'

What to notice: the only difference between these keys is the status value, which comes from user input. Every distinct status value creates a new cache entry. With dozens of status values, multiple tenants, multiple regions, and free-text query variations, the key space is effectively unbounded.

Root cause confirmed. The ExpressionCache singleton, registered in DI as AddSingleton<IExpressionCache, ExpressionCache>(), wraps a ConcurrentDictionary<string, Func<...>> with no eviction policy, no size limit, and a key composed of user-supplied input. Under production load with diverse query patterns, the cache grows without bound, promoting everything to Gen 2, increasing GC pause times, and eventually triggering OOM when the cgroup limit is exceeded.

The Code That Caused It

The cache implementation, simplified from the production code:

public class ExpressionCache
{
    private readonly ConcurrentDictionary<string, Func<string, object, object>> _cache = new();

    public Func<string, object, object> GetOrCompile(string tenant, string query, string region)
    {
        var key = $"user:tenant={tenant}:query={query}:region={region}";
        return _cache.GetOrAdd(key, k =>
        {
            var parameter = Expression.Parameter(typeof(string), "input");
            // ... expression building logic ...
            return Expression.Lambda<Func<string, object, object>>(body, parameter).Compile();
        });
    }
}

The intent was to cache compiled expression delegates to avoid the cost of repeated Expression.Compile() calls. The assumption was that the key space — tenant × query template × region — would be small and bounded. In practice, query includes user-supplied filter values, making every unique query a unique key. The cache was never bounded because the team assumed the cardinality was low. The fix requires two changes: bound the cache and normalize the key.

Remediation: Bounded MemoryCache with Size Limits

Replace the unbounded ConcurrentDictionary with Microsoft.Extensions.Caching.Memory.MemoryCache configured with a size limit and compaction policy:

public class BoundedExpressionCache : IExpressionCache
{
    private readonly MemoryCache _cache;
    private readonly CacheEntryOptions _options;

    public BoundedExpressionCache()
    {
        _cache = new MemoryCache(new MemoryCacheOptions
        {
            SizeLimit = 10_000,
            CompactionPercentage = 0.25,
            ExpirationScanFrequency = TimeSpan.FromMinutes(5)
        });

        _options = new CacheEntryOptions
        {
            Size = 1,
            SlidingExpiration = TimeSpan.FromMinutes(30)
        };
    }

    public Func<string, object, object> GetOrCompile(string tenant, string queryTemplate, string region)
    {
        // Normalize: hash the query template, exclude user-supplied values from the key
        var normalizedQuery = QueryNormalizer.ExtractTemplate(query);
        var key = $"expr:{tenant}:{normalizedQuery}:{region}";

        return _cache.GetOrCreate(key, entry =>
        {
            entry.SetSize(_options.Size);
            entry.SlidingExpiration = _options.SlidingExpiration;
            var parameter = Expression.Parameter(typeof(string), "input");
            // ... expression building logic ...
            return Expression.Lambda<Func<string, object, object>>(body, parameter).Compile();
        });
    }
}

The critical changes: SizeLimit = 10_000 caps the cache at 10,000 entries. CompactionPercentage = 0.25 means when the limit is hit, the cache evicts 25% of entries (LRU-ish, based on last access). SlidingExpiration ensures stale entries are removed even without pressure. The key is normalized — QueryNormalizer.ExtractTemplate strips user-supplied values and reduces the query to its structural template, so SELECT * FROM orders WHERE status='pending' AND region='us-east' and SELECT * FROM orders WHERE status='shipped' AND region='us-east' produce the same cache key SELECT * FROM orders WHERE status=@p0 AND region=@p1.

On .NET 8, MemoryCache with SizeLimit uses an LRU eviction strategy implemented in Microsoft.Extensions.Caching.Memory (source: dotnet/extensions repo, MemoryCache.cs). The compaction algorithm selects entries for removal based on priority and last access timestamp. Set entry.Priority = CacheItemPriority.NeverRemove only for entries that must survive compaction — do not use it for user-facing caches.

Verification: Confirming the Fix with dotnet-counters

Deploy the fix to one pod in the canary pool and monitor dotnet-counters for 30 minutes under production load:

dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]
[System.Runtime]
    gc-heap-size                : 412 MB
    gen-2-gc-count              : 8
    gen-2-size                  : 47 MB
    loh-size                    : 23 MB
    working-set                 : 598 MB
    gc-pause-time               : 12 ms

What to notice: gc-heap-size dropped from 1,847 MB to 412 MB. gen-2-gc-count is 8 over 30 minutes versus 342 in the same window before the fix. gen-2-size is 47 MB — the bounded cache keeps most entries in Gen 0 or Gen 1, and sliding expiration ensures they are collected before promotion. GC pause time dropped from 287 ms to 12 ms.

Verification here is not just "memory went down." The proof is that Gen 2 GC frequency and pause time returned to baseline, and that the cache size stays bounded under sustained load. Run the canary for 2 hours at production RPS and confirm gen-2-gc-count grows linearly with time (not quadratically with traffic) and that gc-heap-size plateaus rather than climbing. If gc-heap-size still climbs, there is a second leak — return to hypothesis 1 with the new baseline.

This verification posture aligns with the risk-management principle that remediated controls require continuous monitoring to confirm effectiveness — a practice formalized in the NIST Cybersecurity Framework 2.0 under its Detect and Recover functions. A one-time memory reading is not verification; a sustained counter trend under load is.

Postmortem and Team Practice

The postmortem for this incident produced three action items beyond the code fix:

1. Cache audit. Every static ConcurrentDictionary, Dictionary, and MemoryCache in the codebase must be reviewed for boundedness. The team created a Roslyn analyzer that flags any ConcurrentDictionary or Dictionary declared as a static field or in a singleton, requiring either a [BoundedCache] attribute or a suppression with justification.

2. Dump collection automation. The manual dump capture during this incident took 12 minutes — too long under OOM pressure. The team deployed dotnet-monitor as a sidecar in the Kubernetes deployment, configured to collect a full dump automatically when gc-heap-size exceeds 80% of the cgroup limit. The configuration in dotnet-monitor.yaml:

rules:
  - name: high-memory-dump
    selectors:
      - processName: ApiGateway
    triggers:
      - type: gcHeapSize
        thresholdMb: 1600
    actions:
      - type: collectDump
        options:
          type: full
          egress: kubernetes-pvc

3. Postmortem documentation. The incident timeline, hypothesis chain, and dump analysis were written up in a blameless postmortem following a structured format that treats every incident as a learning artifact. For teams that want to streamline postmortem writing and ensure consistent narrative structure across incidents, having an AI story generator like Unsloppy in the documentation workflow can help standardize timeline reconstruction from fragmented Slack messages and PagerDuty alerts into a coherent draft. The key is that the tool supports the investigation narrative — symptom, hypothesis, evidence, verdict — not replaces the engineering judgment behind it.

Diagnostic Procedure

1. Capture live counters from a trending pod:

   dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]

   Confirm gc-heap-size is close to working-set (managed heap dominates) and gen-2-size is the largest generation. If LOH size is disproportionate, pivot to LOH fragmentation investigation.

2. Collect a gcdump from the trending pod:

   kubectl exec -it <pod> -- dotnet-gcdump collect -o /tmp/leak.gcdump
   kubectl cp <namespace>/<pod>:/tmp/leak.gcdump ./leak.gcdump

   Open in PerfView. Sort by size. Identify the dominant type and click into its roots. Look for static fields, singletons, or ConcurrentDictionary as the root.

3. Collect a full core dump before OOM:

   kubectl exec -it <pod> -- bash -c 'while true; do rss=$(grep VmRSS /proc/1/status | awk "{print \$2}"); if [ $rss -gt 1572864 ]; then dotnet-dump collect -p 1 -o /tmp/full.dmp && break; fi; sleep 5; done'

   This watches RSS and triggers a dump at 1.5 Gi. Adjust threshold based on your cgroup limit.

4. Analyze the dump with SOS:

   dotnet-dump analyze ./full.dmp
   > clrstack -l          # confirm no thread is stuck in GC
   > dumpheap -stat       # find dominant type by count and size
   > dumpheap -type <Type> -short  # get object addresses
   > gcroot <address>     # trace retention path to root

   Walk the gcroot output from the root to the leaked object. Identify the static field, singleton, or handle that prevents collection.

5. Examine the cache key strings:

   > dumpobj <string-address>

   Confirm the key contains user-supplied input. If keys are unique per request, the cache is unbounded.

6. Remediate with a bounded cache:
   Replace ConcurrentDictionary with MemoryCache configured with SizeLimit, CompactionPercentage, and SlidingExpiration. Normalize the key to exclude user-supplied values.
7. Deploy to canary and verify:

   dotnet-counters monitor -n ApiGateway --counters System.Runtime[System.Runtime]

   Confirm over 30+ minutes at production RPS:

   • gc-heap-size plateaus (does not climb monotonically).
   • gen-2-gc-count grows linearly with time, not with traffic volume.
   • gc-pause-time returns to baseline (under 50 ms for most workloads).
   • gen-2-size remains small relative to total heap.

   If any metric fails to stabilize, return to step 1 with the new baseline — there may be a second leak.