Bulk insert in MongoDB using mongoose

Question

I currently have a collection in Mongodb say "Collection1". I have the following array of objects that need to be into inserted into MongoDB. I am using Mongoose API. For now, I am iterating through the array and inserting each of them into mongo. This is ok for now, but will be a problem when the data is too big. I need a way of inserting the data in bulk into MongoDB without repetition. I am not sure how to do this. I could not find a bulk option in Mongoose.

My code below

myData = [Obj1,Obj2,Obj3.......]

myData.forEach(function(ele){
      //console.log(ele)
     saveToMongo(ele);
    });
function saveToMongo(obj){
    (new Collection1(obj)).save(function (err, response) {
          if (err) {
             // console.log('Error while inserting: ' + obj.name + " " +err);
          } else {
            // console.log('Data successfully inserted');
          }
      });

      return Collection1(obj);
  }

Solution 1

You might want to use the insertMany() method here if you're using the latest Mongoose version 4.4.X and greater, which essentially uses Model.collection.insertMany() under the hood and the driver might handle parallelizing >= 1000 docs for you.

myData = [Obj1, Obj2, Obj3.......];
Collection1.insertMany(myData, function(error, docs) {});

or using Promises for better error handling

Collection1.insertMany(myData)
    .then(function(docs) {
         // do something with docs
    })
    .catch(function(err) {
        // error handling here
    });

It works by creating a bunch of documents, calls .validate() on them in parallel, and then calls the underlying driver's insertMany() on the result of toObject({ virtuals: false }); of each doc. Although insertMany() doesn't trigger pre-save hooks, it has better performance because it only makes 1 round-trip to the server rather than 1 for each document.

---

For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows

var bulk = Collection1.collection.initializeOrderedBulkOp(),
    counter = 0;

myData.forEach(function(doc) {
    bulk.insert(doc);

    counter++;
    if (counter % 500 == 0) {
        bulk.execute(function(err, r) {
           // do something with the result
           bulk = Collection1.collection.initializeOrderedBulkOp();
           counter = 0;
        });
    }
});

// Catch any docs in the queue under or over the 500's
if (counter > 0) {
    bulk.execute(function(err,result) {
       // do something with the result here
    });
}

Solution 2

you can pass an array of objects to mongoose model create function

var Collection1 = mongoose.model('Collection1');

Collection1.create(myData,function(err){
    if(err) ...
});

Comments

[John]

Hi, I'm trying your way to add lot of data with bulk mongoose (I have 409 584 data to add) but I have just 273001 data added. Do you know why ?

[chridam]

What's your MongoDB server version?

[John]

I have MongoDB versio 3.2.9 and mongoose 4.7.2. If I use insertMany I got FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory then I try to use bulk method.

[Aseer KT Miqdad]

can we preserve the order in which they created