asp.net core: "Operations that change non-concurrent collections must have exclusive access."

Question

My server uses MySqlConnector and communicates with a MySQL database on AWS. I store 5 minute chunks of API counters in MySQL. Those API counters are incrementing with each API call, and are handled by ConcurrentDictionary code (which doesn't appear to be a problem). An exception was recently raised by this line of code, which is part of a linq query through MySqlConnector to access a MySQL database table:

await _context.ApiCounts.Where(c => c.ApiName == apiName && c.StartTime >= startTime).ToListAsync();

I've never seen this line fail before, but suddenly one of my servers started throwing the following exception at the line above:

InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported
at System.Collections.Generic.Dictionary`2.FindEntry
at System.Collections.Generic.Dictionary`2.TryGetValue
at Remotion.Linq.Parsing.Structure.NodeTypeProviders.MethodInfoBasedNodeTypeRegistry.GetNodeType
at Remotion.Linq.Parsing.Structure.NodeTypeProviders.MethodInfoBasedNodeTypeRegistry.IsRegistered
at System.Linq.Enumerable.Any
at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.GetQueryOperatorExpression
at Remotion.Linq.Parsing.ExpressionVisitors.SubQueryFindingExpressionVisitor.Visit
at System.Linq.Expressions.ExpressionVisitor.VisitBinary
at System.Linq.Expressions.BinaryExpression.Accept
at System.Linq.Expressions.ExpressionVisitor.VisitBinary
at System.Linq.Expressions.BinaryExpression.Accept
at System.Linq.Expressions.ExpressionVisitor.VisitLambda
at System.Linq.Expressions.Expression`1.Accept
at System.Linq.Enumerable+SelectListPartitionIterator`2.ToArray
at System.Linq.Enumerable.ToArray
at Remotion.Linq.Parsing.Structure.MethodCallExpressionParser.Parse
at Remotion.Linq.Parsing.Structure.ExpressionTreeParser.ParseMethodCallExpression
at Remotion.Linq.Parsing.Structure.QueryParser.GetParsedQuery
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQueryCore
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler+<>c__DisplayClass24_0`1.<CompileAsyncQuery>b__0
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.Generic.IAsyncEnumerable<TResult>.GetEnumerator
at System.Linq.AsyncEnumerable+<Aggregate_>d__6`3.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at BlayFap.Controllers.ServerController+<GetCachedApiCount>d__13.MoveNext (E:\Projects\BlayFap\BlayFap\Controllers\ServerController.cs:342)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at BlayFap.Controllers.ServerController+<GetApiCount>d__14.MoveNext (E:\Projects\BlayFap\BlayFap\Controllers\ServerController.cs:378)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__14.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextResourceFilter>d__22.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__17.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__15.MoveNext
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw

All of the other tables can be accessed fine, but from here on any attempt to use the ApiCounts table results in the above exception. I had to actually reboot the affected server to clear the error. It looks like this exception has to do with concurrent edits on a Dictionary, and that Dictionary appears to be in linq code. I'm guessing that the server may have gotten caught in a situation where EF was both updating and writing some sort of data while executing a linq statement, and then got the Dictionary stuck in a weird state, but I'm not sure how to protect against this and where the real issue is. Is this a .NET bug, or my own?

More info: I do not modify the result from ApiCounts in this function, and I do not call SaveChanges. It is, however, possible for code running async to this (in another REST query) to update ApiCounts and call SaveChanges.

Solution

The solution is to update to EF 2.1.5 or newer.

I was using EF 2.0.3 when this error occurred. It looks like there was a thread unsafe singleton, which could result in the same error I saw. Here's the issue in github: https://github.com/dotnet/efcore/issues/12682

Comments

[Valentin]

I think the solution is to upgrade to at least EF Core 2.1.3: github.com/dotnet/efcore/commit/…

[Gert Arnold]

Restarting the application pool never fixes anything. The exception will come back soon.

[Ramesh R]

Hello @GertArnold, Okay. May be. let me wait for the issue to come back. i'll update if issue reopens