mongodb impossible (?) E11000 duplicate key error dup key when upserting

Question

My understanding is that update with upsert:true on a single document is an atomic operation so this should never result in a duplicate key error, especially not on the primary _id key, when the collection has no unique-ly indexed fields:

Order.update({ _id: order._id }, query, { upsert: true }, cb) // with mongoose

But this appears in the mongod.log:

    2015-03-27T09:39:10.349-0400 I WRITE    [conn258236] update xyz.orders 
query: { _id: "6353f880-c6a7-4260-809f-98e0af27b9a2" } update: { $set: { ... 
} keyUpdates:0 writeConflicts:0 **exception: E11000 duplicate key error dup 
key: { : "6353f880-c6a7-4260-809f-98e0af27b9a2" } code:11000** numYields:1 
locks:{} 138ms


    2015-03-27T09:39:10.349-0400 I COMMAND  [conn258236] command xyz.$cmd 
command: update { update: "orders", writeConcern: { w: 1 }, ordered: true, 
updates: [ { q: { _id: "6353f880-c6a7-4260-809f-98e0af27b9a2" }, u: { $set: { 
... } }, multi: false, upsert: true } ] } keyUpdates:0 writeConflicts:0 
numYields:0 reslen:235 locks:{} 139ms

Here is the output from db.orders.getIndexes():

{
    "v" : 1,
    "key" : {
        "_id" : 1
    },
    "name" : "_id_",
    "ns" : "xyz.orders"
},

We are using MongoDB version 3.0.0 with WiredTiger.

Solution

I am afraid that this is an ongoing problem. I had the same problem and I found a jira ticket about this:

https://jira.mongodb.org/browse/SERVER-14322

It is possible that two updates come in with upsert:true, resulting in neither finding a document and both inserting new documents which conflict on unique index violations of the query predicate.

The "solution" here is to add a retry code into the client.