C# Entity Framework high memory usage, memory leak?

Question

I've got a small MVC webapplication running using Entity Framework 6. When starting the application by browwsing to the homepage (eg. www.mywebsite.dev) on my dev. machine the applicationpool get's started and the page get loaded as expected.

Despithe the fact that the homepage is pretty ligthweight and only get's a few things from the database (2 menu's, 2 paragraphs with text, and a collection with 3-4 objects) the application pool is already > 200 MB (!) after just loading the homepage once..

Using this and this article i've managed to figure out how to profile the manage memory, and I also removed a few static properties blocking the disposal of the context. The DbContext has lazy loading disabled,

public class MyContext: DbContext
    {
        private readonly Dictionary<Type, EntitySetBase> _mappingCache = new Dictionary<Type, EntitySetBase>();

        #region dbset properties
        //Membership sets
        public IDbSet<UserProfile> UserProfiles { get; set; }
        public IDbSet<Project> Project { get; set; }
        public IDbSet<Portfolio> Portfolio { get; set; }
        public IDbSet<Menu> Menu { get; set; }
        public IDbSet<MenuItem> MenuItem { get; set; }
        public IDbSet<Page> Page { get; set; }
        public IDbSet<Component> Component { get; set; }
        public IDbSet<ComponentType> ComponentType { get; set; }
        public IDbSet<BlogCategory> BlogCategory { get; set; }
        public IDbSet<Blog> Blog { get; set; }
        public IDbSet<Caroussel> Carousel { get; set; }
        public IDbSet<CarouselItem> CarouselItem { get; set; }
        public IDbSet<Redirect> Redirect { get; set; }
        public IDbSet<TextBlock> TextBlock { get; set; }
        public IDbSet<Image> Image { get; set; }
        public IDbSet<ImageContent> ImageContent { get; set; }
        #endregion

        /// <summary>
        /// The constructor, we provide the connectionstring to be used to it's base class.
        /// </summary>
        public MyContext() : base("name=MyConnectionstring")
        {
            //Disable lazy loading by default!
            Configuration.LazyLoadingEnabled = false;

            Database.SetInitializer<BorloContext>(null);
        }

        //SOME OTHER CODE
}

I still see a lot of objects in memory from which I expect they're related to entity framework's lazy loading.

I've setup the website with a few layers;

1. Controller - The usual stuff
2. Service - Trought a using statement used in the controllers. The services are disposable and contain a UnitOfWork. The UnitOfWork is initialized in the Constructor of the service and disposed when the service itsself get's disposed.
3. UnitOfWOrk - The UnitOfWork class contains a readonly private variable containing the context, together with a set of properties instantiating a Generic Repository of type T. Again, the UnitOfWork is disposable and it disposes the context when the Dispose method is called.
4. The Generic Repository matches an interface, takes the DbContext trought it's constructor and offers a basic set of methods trough an interface.

Below an example of how this is used.

PartialController

public class PartialController : BaseController
    {
        //private readonly IGenericService<Menu> _menuService;
        //private readonly UnitOfWork _unitOfWork = new UnitOfWork();
        //private readonly MenuService _menuService;

        public PartialController()
        {
            //_menuService = new GenericService<Menu>();
            //_menuService = new MenuService();
        }

        /// <summary>
        /// Renders the mainmenu based on the correct systemname.
        /// </summary>
        [ChildActionOnly]
        public ActionResult MainMenu()
        {
            var viewModel = new MenuModel { MenuItems = new List<MenuItem>() };

            try
            {
                Menu menu;
                using (var service = ServiceFactory.GetMenuService())
                {
                    menu= service.GetBySystemName("MainMenu");
                }

                //Get the menuItems collection from somewhere
                if (menu.MenuItems != null && menu.MenuItems.Any())
                {
                    viewModel.MenuItems = menu.MenuItems.ToList();
                    return View(viewModel);
                }
            }
            catch (Exception exception)
            {
                //TODO: Make nice function of this and decide throwing or logging.
                if (exception.GetType().IsAssignableFrom(typeof(KeyNotFoundException)))
                {
                    throw;
                }
                else
                {
                    //TODO: Exception handling and logging
                    //TODO: If exception then redirect to 500-error page.
                }

            }

            return View(viewModel);
        }
    }

ServiceFactory

public class ServiceFactory
    {
        public static IService<Menu> GetMenuService()
        {
            return new MenuService();
        }
}

MenuService

public class MenuService : BaseService, IService<Menu>
{
private readonly UnitOfWork _unitOfWork;
private bool _disposed;

public MenuService()
{
    if (_unitOfWork == null)
    {
        _unitOfWork = new UnitOfWork();
    }
}

/// <summary>
/// Retrieves the menu by the provided systemname.
/// </summary>
/// <param name="systemName">The systemname of the menu.</param>
/// <returns>The menu if found. Otherwise null</returns>
public Menu GetBySystemName(string systemName)
{
    var menu = new Menu();

    if (String.IsNullOrWhiteSpace(systemName)) throw new ArgumentNullException("systemName","Parameter is required.");

    if (Cache.HasItem(systemName))
    {
        menu = Cache.GetItem(systemName) as Menu;
    }
    else
    {
        var retrievedMenu = _unitOfWork.MenuRepository.GetSingle(m => m.SystemName.Equals(systemName), "MenuItems,MenuItems.Page");

        if (retrievedMenu == null) return menu;

        try
        {
            var exp = GenericRepository<CORE.Entities.MenuItem>.IsPublished();
            var menuItems = (exp != null) ?
                retrievedMenu.MenuItems.AsQueryable().Where(exp).Select(MenuTranslator.Translate).OrderBy(mi => mi.SortOrder).ToList() :
                retrievedMenu.MenuItems.Select(MenuTranslator.Translate).OrderBy(mi => mi.SortOrder).ToList();

            menu.MenuItems = menuItems;
        }
        catch (Exception)
        {
            //TODO: Logging
        }

        Cache.AddItem(systemName, menu, CachePriority.Default, CacheDuration.Short);
    }

    return menu;
}

public IEnumerable<Menu> Get()
{
    throw new NotImplementedException();
}

~MenuService()
{
    Dispose(false);
}

protected virtual void Dispose(bool disposing)
{
    if (!_disposed)
    {
        if (disposing)
        {
            _unitOfWork.Dispose();
        }
    }
    _disposed = true;
}

public void Dispose()
{
    Dispose(true);
    GC.SuppressFinalize(this);
}

}

GenericRepository

public class GenericRepository<TEntity> : IGenericRepository<TEntity> where TEntity : class, IEntityObject

{ /// /// The database context used. /// internal MyContext Context;

/// <summary>
/// The loaded set of entities.
/// </summary>
internal DbSet<TEntity> DbSet;

/// <summary>
/// The constructor taking the databasecontext.
/// </summary>
/// <param name="context">The databasecontext to use.</param>
public GenericRepository(MyContext context)
{
    //Apply the context
    Context = context;

    //Set the entity type for the current dbset.
    DbSet = context.Set<TEntity>();
}
public IQueryable<TEntity> AsQueryable(bool publishedItemsOnly = true)
{
    if (!publishedItemsOnly) return DbSet;
    try
    {
        return DbSet.Where(IsPublished());
    }
    catch (Exception)
    {
        //TODO: Logging
    }

    return DbSet;
}

/// <summary>
/// Gets a list of items matching the specified filter, order by and included properties.
/// </summary>
/// <param name="filter">The filter to apply.</param>
/// <param name="includeProperties">The properties to include to apply eager loading.</param>
/// <param name="publishedItemsOnly">True if only publish and active items should be included, otherwise false.</param>
/// <returns>A collection of entities matching the condition.</returns>
public virtual IQueryable<TEntity> Get(Expression<Func<TEntity, bool>> filter, string includeProperties, bool publishedItemsOnly)
{
    var query = AsQueryable(publishedItemsOnly);

    if (filter != null)
    {
        query = query.Where(filter);
    }


    if (String.IsNullOrWhiteSpace(includeProperties))
        return query;

    //Include all properties to the dbset to enable eager loading.
    query = includeProperties.Split(new[] {','}, StringSplitOptions.RemoveEmptyEntries).Aggregate(query, (current, includeProperty) => current.Include(includeProperty));

    return query;
}

}

To make a very long story short. What in my code / situation could possibly cause the problem that when loading just the homepage a amazing 200 MB or more is used? One strange thing i've noticed is that just before the page is loaded the memory jumps from 111 mb to 232 MB in the example below;

EDIT Result from trace with dotMemory

EDIT 2 Below the results after i loaded the homepage. The homepage is now empty and in the global asax a single call to one service is made. I've kept the page open for a while and then refreshed, resulting in all the peaks.

Below a more detailed result, with apperently a lot of strings taking a lot of memory..?

EDIT 3 Different view from dotMemory

Solution

So, image is much more clearer now. dotMemory displays, that your app takes only 9Mb of memory, we can see this on the Snapshot view. This is also confirmed by Memory Traffic view. ~73Mb was allocated from the beginning of profiling and ~65Mb was already collected to the Snapshot #1 point.

What about total memory usage displayed on the real time data chart, sorry I did not realized earlier then the most of your app memory usage is generation 0 heap. (And also I missed that your app uses only ~8Mb on the snapshot tile on this screen).

Gen 0 heap size displays the maximum bytes that can be allocated in generation 0; it does not indicate the current number of bytes allocated in generation 0. http://msdn.microsoft.com/en-us/library/x2tyfybc(v=vs.110).aspx

Gen 0 heap size looks abnormally big for my taste, but it is an internal details of .net garbage collector, and it has a right to do that.

I have ventured to suggest that your app is running on the computer with big amount of RAM and/or with big CPU cache. But it can be also special aspects of ASP server implementation.

Conclusion - there is no problem with your app memory usage :) At least on loading just the homepage.

P.S. I would recommend to watch dotMemory video tutorials, in order to learn how to use it

Comments

[Uwe Keim]

For memory leaks, I usually use ANTS Memory Profiler.

[Rob]

@UweKeim I just tried that one, but I couldn't get anything usefull from the result (or it is my knowledge..;-)) I tried dotMemory, from which I attached the results in the question

[Uwe Keim]

Performance profiling was always much more straightforward to me. Still memory profiling needs to be done. Usually I plan several hours (if not a whole day) when it comes to find memory issues.

[Rob]

@UweKeim, hmm good point. I'll do a bit more debugging and profilling tonight. But still if anyone has a suggestion, i'll be more than open to it:)

[Rob]

OKay i've figured out that the usage of the service is most likely causing the problem, only I don't know why..

[Ed Pavlov]

"when loading just the homepage a amazing 200 MB or more is used" - get snapshot, open snapshot overview, look objects of what types take the major part of memory. Look what objects exclusively retain the major part of memory. "before the page is loaded the memory jumps from 111 mb to 232 MB" get a snaphot before jump and after, see what are "new" objects (click on a cell in "new" column of "all" row). Then you will be able to investigate them in any way you want.

[Rob]

Good one! Will try that for sure:)

[Rob]

@Ed.ward i've updated the question with some results. Apperently, there is a large set of strings taking a lot of memory. Also there is a lot of unmanaged memory in use..? any ideas?

[Keith Payne]

One thing that will hurt is assigning context/UoW objects to class fields/props. Those guys are heavy and are really meant for local var use.

[Ed Pavlov]

@Rob It is a "Memory traffic" view on the second screenshot. It shows all objects created from the beginning of profiling to the time you got the snapshot. As you can see on this view ~5Mb of strings were created and ~3.5 were collected (deleted). You need not "Memory traffic" view, you need "Snapshot" view where dotMemory shows summary information about live objects on the moment of getting snapshot. To open it click "Snapshot #1" link on the white tile representing snapshot in "Memory Snapshots" area.

[Rob]

@Ed.ward Thanks, think I got it now and i've updated the question with new information. Also I've refactored A bit ad the UnitOfWork is now entirely removed. The Service are now directly instantiating a Repository of type T. Please note that the profiling was done in the same situation, so still with an empty homepage and one call to a service class in the global asax.

[Rob]

@Keith, never knew that! I'll keep it mind while resolving the issue!

[Rob]

Thanks for the good response! So if I understand you correctly it's just the maximum amount of memory that can be allocated? How come then that it's growing a small bit on each page refresh? With one or two resfreshed the homepage (this time including content) takes over 200 MB Again, see this (imgur.com/5FydyhD) for details.. The machine indeed is running a lot of memory. It's a heavy duty laptop with 16GB of RAM and a core i7.. So because of that it's just allocating a large amount of memory.

[Ed Pavlov]

@Rob "just the maximum amount of memory that can be allocated" - can be allocated (in Gen 0) before garbage collecting occurs. Garbage collector starts its work, when Gen 0 heap is full, it deletes all garbage and promotes all survived objects to the Gen 1. Since you application is running on the laptop, not on the server hardware, I guess such big Gen 0 heap's size is implementation details of ASP server but I'm not very familiar with ASP to proof that.

[Rob]

well for now i'll accept that the problem is solved. dotMemory pointed me to a few other things which definatly improved things. I'll continue the development and see how it goes on the testenvironment. Thans for all your help!