Output Stream of Consciousness

Few things are more dreadful for an application than a recurring "out of memory" error.

You ask for memory graphs for the past hours or days and scan them... and you don't find anything untoward. Now, you're really confused.

One thing about the memory management in .NET - you do not actually have to be using all the memory to get a System.OutOfMemoryException.

We had a server application blowing up with just around 1 Gb of memory used, when we have seen much more than that with load testing with no issues.

In our particular case, the culprit was AppDomains. We create them to safely host external libraries written for .NET. AppDomains are finalized, and it seemed reasonable - based on that and the fact that they do not figure prominently in memory tools - to let .NET take care of them as they go out of scope.

The out of memory exceptions were happening inside CreateDomain almost exclusively. This might be due to the memory being "reserved" somehow, or it could just be that CreateDomain looks for some other resource - or space on a list for a resource - that the .NET runtime can no longer provide.

We cannot easily tell which, since the error occurs inside nCreateDomain. It's a [MethodImpl(MethodImplOptions.InternalCall)] - meaning part of the CLR itself.

So, we used AppDomain.Unload(...) with those application domains when the sessions ended, and the out of memory issue - fingers crossed - has seemingly disappeared.