multi document insert using mongoengine into mongodb

Question

In my flask app I am using MongoeEgine. I am trying to insert multiple documents into my places collection in my MongoDB.

My document class is defined as

class places(db.Document):

  name = db.StringField(max_length=200, required=True)    
  loc = db.GeoPointField(required=True)

  def __unicode__(self):
    return self.name

    a=[]
    a.append({"name" : 'test' , "loc":[-87,101]})
    a.append({"name" : 'test' , "loc":[-88,101]})
    x= places(a)

The last statement fails

x= places(a)
TypeError: __init__() takes exactly 1 argument (2 given)

I also tried to save this to my instance

places.insert(x)
places.save(x)

both fail. Please help.

Solution 1

Places.objects.insert doesn't take a list of dictionaries it has to be Places instances. Normal operations would be to create individual instances of Places and save or insert eg:

Places(name="test", loc=[-87, 101]).save()
Places(name="test 2", loc=[-87, 101]).save()

However if you want to do a bulk insert you can pass a list of Places instances and call insert on the objects queryset eg:

Places.objects.insert([Places(name="test", loc=[-87, 101]), 
                       Places(name="test 2", loc=[-87, 101])])

Solution 2

You try to initialize Document object for multiple documents at once. If you look at mongoengine's BaseDocument class, you'll see, that its __init__ method takes a dictionary of keyword arguments, which relate to fields of one single document.

If you want to do a bulk save, you have to make a list of places instances and pass it to insert() method.

a = []
a.append(places(**{"name": 'test', "loc": [-87,101]}))
a.append(places(**{"name": 'test', "loc": [-88,101]}))
x = places.objects.insert(a)

Comments

[user1340513]

Thanks. How could i use this to do an upsert . I want to insert these rows if they are new , else update the existing row. Is there a Places.objects.upsert or Places.object.insert with upsert=True flag?

[giliev]

@Ross is there some simple way to avoid NotUniqueError? I read somewhere about using ordered in PyMongo bulk insert, but I would prefer to use Mongoengine. I know of _getCollection(), but still I would prefer to avoid not accessing the collection directly. I prefer using the Mongoengine syntax :)

[Lohith]

@Ross Which one of the above operations is an efficient one i mean whether it is a save() operation or bulk insert operation.

[Michael Currin]

This does not work for me. I get an error even when bulk inserting into an empty DB. "mongoengine.errors.NotUniqueError: Document must not have _id value before bulk write (batch op errors occurred)". A comment in the mongoengine source says "inserting documents that already have an _id field will give huge performance debt or raise". I can't see how to get around this as the id attributes of the objects are None.