How are people unit testing with Entity Framework 6, should you bother?

Question

I am just starting out with Unit testings and TDD in general. I have dabbled before but now I am determined to add it to my workflow and write better software.

I asked a question yesterday that kind of included this, but it seems to be a question on its own. I have sat down to start implementing a service class that I will use to abstract away the business logic from the controllers and map to specific models and data interactions using EF6.

The issue is I have roadblocked myself already because I didn't want to abstract EF away in a repository (it will still be available outside the services for specific queries, etc) and would like to test my services (EF Context will be used).

Here I guess is the question, is there a point to doing this? If so, how are people doing it in the wild in light of the leaky abstractions caused by IQueryable and the many great posts by Ladislav Mrnka on the subject of unit testing not being straightforward because of the differences in Linq providers when working with an in memory implementation as apposed to a specific database.

The code I want to test seems pretty simple. (this is just dummy code to try and understand what i am doing, I want to drive the creation using TDD)

Context

public interface IContext
{
    IDbSet<Product> Products { get; set; }
    IDbSet<Category> Categories { get; set; }
    int SaveChanges();
}

public class DataContext : DbContext, IContext
{
    public IDbSet<Product> Products { get; set; }
    public IDbSet<Category> Categories { get; set; }

    public DataContext(string connectionString)
                : base(connectionString)
    {

    }
}

Service

public class ProductService : IProductService
{
    private IContext _context;

    public ProductService(IContext dbContext)
    {
        _context = dbContext;
    }

    public IEnumerable<Product> GetAll()
    {
        var query = from p in _context.Products
                    select p;

        return query;
    }
}

Currently I am in the mindset of doing a few things:

1. Mocking EF Context with something like this approach- Mocking EF When Unit Testing or directly using a mocking framework on the interface like moq - taking the pain that the unit tests may pass but not necessarily work end to end and back them up with Integration tests?
2. Maybe using something like Effort to mock EF - I have never used it and not sure if anyone else is using it in the wild?
3. Not bother testing anything that simply calls back to EF - so essentially service methods that call EF directly (getAll etc) are not unit tested but just integration tested?

Anyone out there actually doing this out there without a Repo and having success?

Solution 1

This is a topic I'm very interested in. There are many purists who say that you shouldn't test technologies such as EF and NHibernate. They are right, they're already very stringently tested and as a previous answer stated it's often pointless to spend vast amounts of time testing what you don't own.

However, you do own the database underneath! This is where this approach in my opinion breaks down, you don't need to test that EF/NH are doing their jobs correctly. You need to test that your mappings/implementations are working with your database. In my opinion this is one of the most important parts of a system you can test.

Strictly speaking however we're moving out of the domain of unit testing and into integration testing but the principles remain the same.

The first thing you need to do is to be able to mock your DAL so your BLL can be tested independently of EF and SQL. These are your unit tests. Next you need to design your Integration Tests to prove your DAL, in my opinion these are every bit as important.

There are a couple of things to consider:

1. Your database needs to be in a known state with each test. Most systems use either a backup or create scripts for this.
2. Each test must be repeatable
3. Each test must be atomic

There are two main approaches to setting up your database, the first is to run a UnitTest create DB script. This ensures that your unit test database will always be in the same state at the beginning of each test (you may either reset this or run each test in a transaction to ensure this).

Your other option is what I do, run specific setups for each individual test. I believe this is the best approach for two main reasons:

• Your database is simpler, you don't need an entire schema for each test
• Each test is safer, if you change one value in your create script it doesn't invalidate dozens of other tests.

Unfortunately your compromise here is speed. It takes time to run all these tests, to run all these setup/tear down scripts.

One final point, it can be very hard work to write such a large amount of SQL to test your ORM. This is where I take a very nasty approach (the purists here will disagree with me). I use my ORM to create my test! Rather than having a separate script for every DAL test in my system I have a test setup phase which creates the objects, attaches them to the context and saves them. I then run my test.

This is far from the ideal solution however in practice I find it's a LOT easier to manage (especially when you have several thousand tests), otherwise you're creating massive numbers of scripts. Practicality over purity.

I will no doubt look back at this answer in a few years (months/days) and disagree with myself as my approaches have changed - however this is my current approach.

To try and sum up everything I've said above this is my typical DB integration test:

[Test]
public void LoadUser()
{
  this.RunTest(session => // the NH/EF session to attach the objects to
  {
    var user = new UserAccount("Mr", "Joe", "Bloggs");
    session.Save(user);
    return user.UserID;
  }, id => // the ID of the entity we need to load
  {
     var user = LoadMyUser(id); // load the entity
     Assert.AreEqual("Mr", user.Title); // test your properties
     Assert.AreEqual("Joe", user.Firstname);
     Assert.AreEqual("Bloggs", user.Lastname);
  }
}

The key thing to notice here is that the sessions of the two loops are completely independent. In your implementation of RunTest you must ensure that the context is committed and destroyed and your data can only come from your database for the second part.

Edit 13/10/2014

I did say that I'd probably revise this model over the upcoming months. While I largely stand by the approach I advocated above I've updated my testing mechanism slightly. I now tend to create the entities in in the TestSetup and TestTearDown.

[SetUp]
public void Setup()
{
  this.SetupTest(session => // the NH/EF session to attach the objects to
  {
    var user = new UserAccount("Mr", "Joe", "Bloggs");
    session.Save(user);
    this.UserID =  user.UserID;
  });
}

[TearDown]
public void TearDown()
{
   this.TearDownDatabase();
}

Then test each property individually

[Test]
public void TestTitle()
{
     var user = LoadMyUser(this.UserID); // load the entity
     Assert.AreEqual("Mr", user.Title);
}

[Test]
public void TestFirstname()
{
     var user = LoadMyUser(this.UserID);
     Assert.AreEqual("Joe", user.Firstname);
}

[Test]
public void TestLastname()
{
     var user = LoadMyUser(this.UserID);
     Assert.AreEqual("Bloggs", user.Lastname);
}

There are several reasons for this approach:

• There are no additional database calls (one setup, one teardown)
• The tests are far more granular, each test verifies one property
• Setup/TearDown logic is removed from the Test methods themselves

I feel this makes the test class simpler and the tests more granular (single asserts are good)

Edit 5/3/2015

Another revision on this approach. While class level setups are very helpful for tests such as loading properties they are less useful where the different setups are required. In this case setting up a new class for each case is overkill.

To help with this I now tend to have two base classes SetupPerTest and SingleSetup. These two classes expose the framework as required.

In the SingleSetup we have a very similar mechanism as described in my first edit. An example would be

public TestProperties : SingleSetup
{
  public int UserID {get;set;}

  public override DoSetup(ISession session)
  {
    var user = new User("Joe", "Bloggs");
    session.Save(user);
    this.UserID = user.UserID;
  }

  [Test]
  public void TestLastname()
  {
     var user = LoadMyUser(this.UserID); // load the entity
     Assert.AreEqual("Bloggs", user.Lastname);
  }

  [Test]
  public void TestFirstname()
  {
       var user = LoadMyUser(this.UserID);
       Assert.AreEqual("Joe", user.Firstname);
  }
}

However references which ensure that only the correct entites are loaded may use a SetupPerTest approach

public TestProperties : SetupPerTest
{
   [Test]
   public void EnsureCorrectReferenceIsLoaded()
   {
      int friendID = 0;
      this.RunTest(session =>
      {
         var user = CreateUserWithFriend();
         session.Save(user);
         friendID = user.Friends.Single().FriendID;
      } () =>
      {
         var user = GetUser();
         Assert.AreEqual(friendID, user.Friends.Single().FriendID);
      });
   }
   [Test]
   public void EnsureOnlyCorrectFriendsAreLoaded()
   {
      int userID = 0;
      this.RunTest(session =>
      {
         var user = CreateUserWithFriends(2);
         var user2 = CreateUserWithFriends(5);
         session.Save(user);
         session.Save(user2);
         userID = user.UserID;
      } () =>
      {
         var user = GetUser(userID);
         Assert.AreEqual(2, user.Friends.Count());
      });
   }
}

In summary both approaches work depending on what you are trying to test.

Solution 2

Effort Experience Feedback here

After a lot of reading I have been using Effort in my tests: during the tests the Context is built by a factory that returns a in memory version, which lets me test against a blank slate each time. Outside of the tests, the factory is resolved to one that returns the whole Context.

However i have a feeling that testing against a full featured mock of the database tends to drag the tests down; you realize you have to take care of setting up a whole bunch of dependencies in order to test one part of the system. You also tend to drift towards organizing together tests that may not be related, just because there is only one huge object that handles everything. If you don't pay attention, you may find yourself doing integration testing instead of unit testing

I would have prefered testing against something more abstract rather than a huge DBContext but i couldn't find the sweet spot between meaningful tests and bare-bone tests. Chalk it up to my inexperience.

So i find Effort interesting; if you need to hit the ground running it is a good tool to quickly get started and get results. However i think that something a bit more elegant and abstract should be the next step and that is what I am going to investigate next. Favoriting this post to see where it goes next :)

Edit to add: Effort do take some time to warm up, so you're looking at approx. 5 seconds at test start up. This may be a problem for you if you need your test suite to be very efficient.

---

Edited for clarification:

I used Effort to test a webservice app. Each message M that enters is routed to a IHandlerOf<M> via Windsor. Castle.Windsor resolves the IHandlerOf<M> which resovles the dependencies of the component. One of these dependencies is the DataContextFactory, which lets the handler ask for the factory

In my tests I instantiate the IHandlerOf component directly, mock all the sub-components of the SUT and handles the Effort-wrapped DataContextFactory to the handler.

It means that I don't unit test in a strict sense, since the DB is hit by my tests. However as I said above it let me hit the ground running and I could quickly test some points in the application

Solution 3

If you want to unit test code then you need to isolate your code you want to test (in this case your service) from external resources (e.g. databases). You could probably do this with some sort of in-memory EF provider, however a much more common way is to abstract away your EF implementation e.g. with some sort of repository pattern. Without this isolation any tests you write will be integration tests, not unit tests.

As for testing EF code - I write automated integration tests for my repositories that write various rows to the database during their initialization, and then call my repository implementations to make sure that they behave as expected (e.g. making sure that results are filtered correctly, or that they are sorted in the correct order).

These are integration tests not unit tests, as the tests rely on having a database connection present, and that the target database already has the latest up-to-date schema installed.

Solution 4

I have fumbled around sometime to reach these considerations:

1- If my application access the database, why the test should not? What if there is something wrong with data access? The tests must know it beforehand and alert myself about the problem.

2- The Repository Pattern is somewhat hard and time consuming.

So I came up with this approach, that I don't think is the best, but fulfilled my expectations:

Use TransactionScope in the tests methods to avoid changes in the database.

To do it it's necessary:

1- Install the EntityFramework into the Test Project. 2- Put the connection string into the app.config file of Test Project. 3- Reference the dll System.Transactions in Test Project.

The unique side effect is that identity seed will increment when trying to insert, even when the transaction is aborted. But since the tests are made against a development database, this should be no problem.

Sample code:

[TestClass]
public class NameValueTest
{
    [TestMethod]
    public void Edit()
    {
        NameValueController controller = new NameValueController();

        using(var ts = new TransactionScope()) {
            Assert.IsNotNull(controller.Edit(new Models.NameValue()
            {
                NameValueId = 1,
                name1 = "1",
                name2 = "2",
                name3 = "3",
                name4 = "4"
            }));

            //no complete, automatically abort
            //ts.Complete();
        }
    }

    [TestMethod]
    public void Create()
    {
        NameValueController controller = new NameValueController();

        using (var ts = new TransactionScope())
        {
            Assert.IsNotNull(controller.Create(new Models.NameValue()
            {
                name1 = "1",
                name2 = "2",
                name3 = "3",
                name4 = "4"
            }));

            //no complete, automatically abort
            //ts.Complete();
        }
    }
}

Solution 5

I would not unit test code I don't own. What are you testing here, that the MSFT compiler works?

That said, to make this code testable, you almost HAVE to make your data access layer separate from your business logic code. What I do is take all of my EF stuff and put it in a (or multiple) DAO or DAL class which also has a corresponding interface. Then I write my service which will have the DAO or DAL object injected in as a dependency (constructor injection preferably) referenced as the interface. Now the part that needs to be tested (your code) can easily be tested by mocking out the DAO interface and injecting that into your service instance inside your unit test.

//this is testable just inject a mock of IProductDAO during unit testing
public class ProductService : IProductService
{
    private IProductDAO _productDAO;

    public ProductService(IProductDAO productDAO)
    {
        _productDAO = productDAO;
    }

    public List<Product> GetAllProducts()
    {
        return _productDAO.GetAll();
    }

    ...
}

I would consider live Data Access Layers to be part of integration testing, not unit testing. I have seen guys run verifications on how many trips to the database hibernate makes before, but they were on a project that involved billions of records in their datastore and those extra trips really mattered.

View more solutions

Comments

[samy]

Hey Modika, I was thinking about this recently (because of this question: stackoverflow.com/questions/25977388/…) In it I try to describe a bit more formally how I work at the moment, but I'd love to hear how you are doing it.

[Modika]

Hi @samy, the way we decided to do it was not unit test anything that touched EF directly. Queries were tested but as integration test, not unit tests. Mocking EF feels a little dirty, but this project was small-ish so the performance impact of having loads of tests hitting a database wasn't really a concern so we could be a bit more pragmatic about it. I am still not 100% sure what the best approach is to be completely truthful with you, at some point you are going to hit EF (and your DB) and unit testing doesn't feel right to me here.

[Modika]

Thanks for the answer, but what would the difference of this be to say a Repository where you are hiding the internals of EF behind it at this level? I don't really want to abstract EF, although i may still be doing that with the IContext interface? I am new to this, be gentle :)

[Jonathan Henson]

@Modika A Repo is fine too. Whatever pattern you want. "I don't really want to abstract EF" Do you want testable code or not?

[Jonathan Henson]

@Modika I would argue that you don't want your interfaces that are being passed around to have ANY semblance of how they are actually implemented. I should be able to swap out a plain ole MySQL ADO.NET layer and the rest of the application should never know that I did it. As long as your interface has EF (e.g. IDBSet) stuff in it, it will never be testable or flexible.

[Modika]

but what happens in your DAO, do you just integration test that section? This is part of my issue i guess, at some point i am going to have to hit something?

[Jonathan Henson]

To clarify, I agree with all of this, and do it myself. I use an in memory database with hibernate though. And I use localdb (creating a new instance on each test run) for EF. I like to know that my HQL and Linq-to-sql are correct (and this does fall in the realm of unit testing).

[Modika]

Thanks @justin i know about the Repository pattern, but reading things like ayende.com/blog/4784/… and lostechies.com/jimmybogard/2009/09/11/wither-the-repository among others have made me think that i don't want this abstraction layer, but then again these talk more about a Query approach as well which gets very confusing.

[Jonathan Henson]

@Modika You can "unit test" the daos if you want, that is a separate concern altogether. I like to know that my linq-to-sql is right which testing helps with. But I do not mock out the context. I just use localdb or an in-memory provider. Then I will run insert on the dao, then use ef to do a get and make sure the record was inserted etc... If you do this during unit testing, make sure you use a localdb instance or an in-memory provider. If at integration testing, it is ok to use a database somewhere.

[Jonathan Henson]

@Modika my point is you won't have ANY testable code if you do not separate your concerns. Data Access and Business Logic MUST be in separate layers to pull off good maintainable tests.

[Modika]

i just didn't feel it nesessary to wrap EF in a repository abstraction as essentially the IDbSets are repo's and the context the UOW, i will update my question a bit as that may be misleading. The issue comes with any abstraction and the main point is what exactly am i testing becuase my queiries will not run in the same boundaries (linq-to-entities vs linq-to-objects) so if i am just testing that my service makes a call that seems a bit wasteful or am i well off here?

[Justin]

@Modika Ayende has chosen a poor implementation of the repository pattern to critique, and as a result is 100% right - its over engineered and doesn't offer any benefits. A good implementation isolates the unit-testable portions of your code from the DAL implementation. Using NHibernate and EF directly makes code difficult (if not impossible) to unit test and leads to a rigid monolithic codebase. I'm still somewhat skeptical of the repository pattern, however I'm 100% convinced that you need to isolate your DAL implementation somehow and the repository is the best thing I've found so far.

[guillaume31]

@Modika Read the second article again. "I don't want this abstraction layer" is not what he says. Plus, read about the original Repository pattern from Fowler (martinfowler.com/eaaCatalog/repository.html) or DDD (dddcommunity.org/resources/ddd_terms). Don't believe naysayers without fully understanding the original concept. What they really criticize is a recent misuse of the pattern, not the pattern itself (although they probably don't know this).

[Jonathan Henson]

@Modika IDbSets are not esentially repo's. The repository pattern as well as all Data access layer classes are concerned with HOW you get the data. Linq-to-SQL on a db set is still part of the how. The service should not be remotely concerned with the HOW, it should only be concerned with the business logic that operates on the data. You are essentially tightly-coupling your code to a database technology, and even worse you are coupling it directly to EF (which is not testable).

[Modika]

,Whilst i agree with your general points, DbContext is a unit of work and IDbSets are definitely some for of repository implementation, and i am not the only one to think that. I can mock EF, and at some layer i will need to run integration tests, does that really matter if i do it in a Repository or further up in a Service? Being tightly coupled to a DB is not really a concern, i am sure it happens but i am not going to plan for something that may not occur.

[Modika]

Thanks for the input, what i may do as i have to get this project running as it is a bonafide paying job is start with some repos and see how i get on, but Effort is very interesting. Out of interest at what layer have you been using effort in your applications?

[Modika]

@guillaume31 i am not against the repository pattern (i do understand it) i am simply trying to figure out if i need it to abstract what is already an abstraction at that level, and if i can omit it and test against EF directly by mocking and use it in my tests at a layer higher up in my application. Additionally, if i don't use a repo i get the benefit of EF extended feature set, with an repo i may not get that.

[Modika]

Accepted as this probably closely follows the question in terms of what i was asking even though it's not 100% what i was after. I am going update the question with what i decided to do a bit later.

[Sedat Kapanoglu]

only if Effort had supported transactions properly

[Gert Arnold]

Here's a different approach to integration testing. TL;DR - Use the application itself to setup test data, rollback a transaction per test.

[Fred]

Bless you for not calling your DAO a "Repository" :)

[Jeffrey A. Gochin]

@Liath, great response. You've confirm my suspicions about testing EF. My question is this; your example is for a very concrete case, which is fine. However, as you noted you may need to test hundreds of entities. In keeping with the DRY principle (Do not Repeat Yourself) how do you scale your solution, without repeating the same basic code pattern every time?

[Liath]

@JeffreyA.Gochin let me answer quickly in comments and then give it some proper thought and try to update my answer more fully in the future. DRY is one rule, as are the test guidelines of Isolated, Atomic, Independent etc - you have to balance both. Try to minimise setup/assertion code in your tests, put them in helper methods. This helps to keep your code DRY but also keeps the tests light and to the point.

[Savage]

Yes to pragmatism. I still argue that the quality of your unit tests is inferior to the quality of your original code. Of course there's value in using TDD to improve your coding practice, and also to enhance maintainability, but TDD can have diminishing value. We run all our tests against the database, because it gives us confidence that our usage of EF and of the tables themselves is sound. The tests do take longer to run, but they're more reliable.

[slopapa]

Actually, I like this solution a lot. Super simple to implement and more realistic testing scenarios. Thanks!

[Sinaesthetic]

I have to disagree with this because it completely sidesteps the issue. Unit testing is about testing the logic of the function. In the OP example, the logic has a dependency on a data store. You're right when you say not to test EF, but that's not the issue. The issue is testing your code in isolation from the datastore. Testing your mapping is a totally different topic imo. In order to test that the logic is interacting with data correctly, you need to be able to control the store.

[Sinaesthetic]

^ this is also why I roll my eyes when people get all uppity about imlementing a repository around EF because EF is a repository -- it has nothing to do with it.

[The Muffin Man]

No one is on the fence about whether you should be unit testing Entity Framework by itself. What happens is that you need to test some method that does some stuff and also happens to make an EF call to the database. The goal is to mock EF so that you can test this method without requiring a database on your build server.

[SwissCoder]

with EF 6, you would use DbContext.Database.BeginTransaction, wouldn't you?

[Sam]

and effort has a bug for strings with csv loader, when we use '' instead of null in strings.

[Tom Leys]

I really like the journey. Thanks for adding edits over time - it's like reading source control and understanding how your thinking has evolved. I really appreciate the functional (with EF) and unit (mocked EF) distinction too.

[Modika]

Thanks for the answer, its a great description of the QueryObject Pattern using a Mediator, and something i am starting to push in my projects as well. I may have to update the question but i am no longer unit testing EF, the abstractions are too leaky (SqlLite might be ok though) so i just integration test my things that query the database and unit test business rules and other logic.

[Modika]

Hi andrew, the issue was never getting the context, you can factory out the context which is what we were doing, abstracting the context and having it built by the factory. The biggest issue was the consistency of what was in memory vs what Linq4Entities does, they are not the same which can lead to misleading tests. Currently, we just integration test database stuff, may not be the best process for everyone mind you.

[andrew pate]

This Moq helper works (codeproject.com/Tips/1045590/…) if you have a context to mock out. If your backing the mocked-out context with a list its not going to behave like a context backed by a sql database might.

[Kevin Burton]

Once I have isolated the DAL with a repository I need someway to "Mock" the database (EF). So far mocking the context and various async extensions (ToListAsync(), FirstOrDefaultAsync(), etc.) have result in frustration for me.

[XRaycat]

Clever solution