Avoid mongodb bulk insert duplicate key error

Question

How can I execute a bulk insert and continue in case of duplicate key error?

I have a collection with an unique index on the id field (not _id) and some data in it. Then I get more data and I want to add only the non-present documents to the collection.

I have the following code:

let opts = {
  continueOnError: true, // Neither
  ContinueOnError: true, // of
  keepGoing: true,       // this
  KeepGoing: true,       // works
};
let bulk = collection.initializeUnorderedBulkOp( opts );
bulk.insert( d1 );
bulk.insert( d2 );
bulk.insert( d3 );
...
bulk.insert( dN );
let result = yield bulk.execute( opts ); // this keep throwing duplicate key error

And I just want to ignore the errors and let the bulk finish with all the queued operations.

I searched in npm module api and in the MongoDB api for Bulk, initializeUnorderedBulkOp and the docs for Bulk write with no luck.

---

Also in the docs for Unordered Operations they say:

Error Handling

If an error occurs during the processing of one of the write operations, MongoDB will continue to process remaining write operations in the list.

Which is not true (at least in my case)

Solution 1

You can use db.collection.insertMany(), (new in version 3.2.) with:

ordered:false

With ordered to false, and in case of duplicate key error, the insert operation would continue with any remaining documents.

Here is link to documentation: https://docs.mongodb.com/v3.2/reference/method/db.collection.insertMany/

Solution 2

Ordered Insert in MongoDB

db.hobbies.insertMany([{_id: "yoga", name: "Yoga"}, {_id: "cooking", name: "Cooking"}, {_id: "hiking", name: "Hiking"}], {ordered: true})

{ordered: true} is the default behaviour of insert statements

Unordered Insert in MongoDB

If you want mongodb to continue trying to insert other documents even after one or more failing due to any reason, you must set ordered to false. See example below:

db.hobbies.insertMany([{_id: "yoga", name: "Yoga"}, {_id: "cooking", name: "Cooking"}, {_id: "hiking", name: "Hiking"}], {ordered: false})

Comments

[Blakes Seven]

It is misleading and the error case has actually changed in recent releases, as previously an UnOrderdedBukOp construct would never produce an error that "thows", but only produce a "list of errors" in the response. Not the first to complain about this. The general advice is "ignore" the error and inspect the result object yourself, as the result will always continue to the end of the batch anyway. So your are not correct as it actually did write all operations in the list ( that worked ), but it just threw an error, when I think it should not have.

[Volox]

Thank you for the clarifications, but my problem is that the execute method, using promises, raises an exception and the promise will be rejected only with the error; losing the BulkWriteResult object.

[Blakes Seven]

Yes. This is exactly my response. Prior drivers did not raise the exception in this case, now they do. This however has no effect on how "UnOrdered" ops are processed. The batch is "still" executed as a whole, however, rather than just returning "errors" in the response object the result "throws" and error as at least one error occured in execution. Understand now?

[Markus W Mahlberg]

You could simply use bulk.find().upsert().replaceOne() instead of insert. This way, if a document with said id is found, it will be replaced with the new document, otherwise it will be created. No duplicate key errors, consistent state.

[SET001]

It actually hase nothing to do with error on dupes. As it's name and manual say what is does is - A boolean specifying whether the mongod instance should perform an ordered or unordered insert.

[Stanislav Prusac]

On sam page of documentation you can find this: "With ordered to false, the insert operation would continue with any remaining documents." and this "Excluding Write Concern errors, ordered operations stop after an error, while unordered operations continue to process any remaining write operations in the queue. Ordered operations display the single error encountered while unordered operations display each error in an array. Therefore, with this strange setting (ordered:false) you will continue to process remaining write operations in the list.