Meteor observe changes added callback on server fires on all item

Question

Tracker.autorun(function() {
  DATA.find().observeChanges({
    added: function(id, doc) {
       console.log(doc);
    }
  });
});

This code is being called on the server. Every time the meteor server starts, the added function fires for every single item in the database. Is there a way to have the added callback fire only when new items are added?

Solution 1

added will be called for every document in the result set when observeChanges is first run. The trick is to ignore the callback during this initialization period. I have an expanded example in my answer to this question, but this code should work for you:

(function() {
  var initializing = true;
  DATA.find().observeChanges({
    added: function(id, doc) {
      if (!initializing) {
        console.log(doc);
      }
    }
  });
  initializing = false;
})();

Note that Tracker.autorun is a client-only function. On the server I think it only ever executes once.

Solution 2

I struggled with this for a long time. For some reason, David's answer did not work for me - it was firing after the initializing variable was set to false.

This pattern from Avi was successful for me:

var usersLoaded = false;
Meteor.subscribe("profiles", function () {
    // at this point all new users sent down are legitimately new ones
    usersLoaded = true;
});

Meteor.users.find().observe({
    added: function(user) {
        if (usersLoaded) {
            console.log("New user created: ", user);
        }
    }
});

Solution 3

Since it is initialization issue, you can do this.

var observerOfMessages = Messages.find({}).observe({
    added: function(doc){
        if(!observerOfMessages) return;
        console.log(doc)
    }
});

This is more elegant actually.

Solution 4

Provide a selector for the query which does not match old items. If using mongo ObjectID as _id you could query for items that have _id greater than the latest item's:

const latest = DATA.findOne({}, {sort: {_id: -1}})
DATA.find({_id: {$gt: latest._id}}).observeChanges({
  added: function() { ... }
})

Or with createdAt timestamp:

const currentTime = new Date()
DATA.find({createdAt: {$gt: currentTime}}).observeChanges({
  added: function() { ... }
})

Solution 5

Here's another way to solve this:

Meteor.subscribe('messages', function() {
    var messages = Messages.find();
    var msgCount = messages.count();

    messages.observe({
        addedAt: function(doc, atIndex) {
            if(atIndex > (msgCount - 1)) console.log('added');
        }
    });
});

Should only fire for docs added after the existing amount is delivered. It's important that this goes in an onReady callback for Meteor.subscribe so that the msgCount changes as your subscription does... if for example, you're paginating your subscriptions.

cursor.observe() documentation

View more solutions

Comments

[Bads]

This is so simple and I couldn't figure it out for days. Thankyou! How stupid of me to not realize that Deps.autorun is client only. Now I have one more question...if you could shed some light. If I run this function on a very large collection and never call stop, now that Meteor uses oplog, would it tax my system badly.

[Peppe L-G]

@Bads You want the after and before hooks functionality? github.com/matb33/meteor-collection-hooks

[Bads]

@PeppeL-G So you would say that using the after and before hooks is better than observe changes? Lets' say my use case is every time a document is added into a collection, modify it, then add that into another collection. Also I just looked through the entire document. It looks exactly like observeChanges. What is the difference?

[Peppe L-G]

@Bads Just mentioned it as an alternative. I don't think that package works with cursors, so I don't think that solution will have the memory drawback you mention in your solution (if it exists).

[Bads]

@PeppeL-G. I see. Thanks. I will experiment with both the solution.

[David Weldon]

@Bads I'll ask the core devs who work on this, but my gut tells me this isn't the right approach - if for no other reason than your startup time may be prohibitive if DATA is indeed a large collection. I think collection-hooks is a good suggestion to try out. I have not used it, but the code doesn't seem to contain the text "observe". An alternative is to only do inserts to your collection via a method call. That way your server code can always do the additional updates for you.

[Bads]

@DavidWeldon Thanks, I think you are right. For now I will go with before after hooks. It's super easy. edit: The reason I did not want to include this after insert function in meteor method is because the data is coming from multiple method, and it would be nice if I just have one place to fun this function.

[Brett McLain]

@Bads can you post your solution? I have tried using before and after but before actually doesn't get called before in many cases...

[Minderov]

@chaintng profiles is probably how John named his publication where he publishes the Meteor.users collection