Meteor publish/subscribe strategies for unique client-side collections

Question

Using Meteor, I'm wondering how best to handle different client-side collections that share the same server-side database collection. Consider the following example: I have a User collection, and on my client-side I have a list of users that are friends and I have a search feature that performs a query on the entire users database, returning a list of usernames that match the query.

On the Publish server-side method, I have two queries against the same collection that return different sets of documents. Should this data go into two separate collections on the client-side? Or should all of the User documents that match both queries end up in the same collection? If the latter, would I then duplicate code used for both the server-side and client-side query?

On the server:

Meteor.publish('searchResults', function(query){
  var re = new RegExp(query, 'i')
  return Users.find({ 'name' : {$regex: re}})
})

On the client:

Session.set('searchQuery', null)

Meteor.autosubscribe(function(){
  Meteor.subscribe('searchResults', Session.get('searchQuery'))
})

Template.search.events = {
  'keyup #user-search' : function(e){
    Session.set('searchQuery', e.target.value)
  }
}

_.extend(Template.search, {

  searchResults: function() {
    var re = new RegExp(Session.get('searchQuery'), 'i')
    return Users.find({ 'name' : {$regex: re}})
  }
})

This seems like a plausible solution, but not an optimal one. What if I wanted to create a new client-side collection that consisted of search results from multiple server-side Collections?

Solution 1

In a shared area:

function getSearchUsers(query) {
  var re = new RegExp(query, "i");
  return Users.find({name: {$regex: re}});
}

function getFriendUsers() {
  return Users.find({friend: true});    // or however you want this to work
}

On the server:

Meteor.publish("searchUsers", getSearchUsers);
Meteor.publish("friendUsers", getFriendUsers);

On the client:

Template.search.onCreated(function () {
   var self = this;
   self.autorun(function () {
     self.subscribe("searchUsers", Session.get("searchQuery"));
   });
});

Template.friends.onCreated(function () {
  this.subscribe("friendUsers");
});

Template.search.helpers({
  searchResults: function () {
    return getSearchUsers(Session.get("searchQuery"));
  }
});

Template.friends.helpers({
  results: function () {
    return getFriendUsers();
  }
});

The key takeaway from this is that what happens behind the scenes when the data is getting transferred over the wire isn't obvious. Meteor appears to combine the records that were matched in the various queries on the server and send this down to the client. It's then up the client to run the same query again to split them apart.

For example, say you have 20 records in a server-side collection. You then have two publishes: the first matches 5 records, the second matches 6, of which 2 are the same. Meteor will send down 9 records. On the client, you then run the exact same queries you performed on the server and you should end up with 5 and 6 records respectively.

Solution 2

I'm a little bit late to the party, but there is a way to actually have separate collections on the client for subsets of one server collection. In this example i have a server collection called entities which holds information about polygonsand rectangles.
Shared code (lib folder):

// main collection (in this example only needed on the server
Entities = new Meteor.Collection('entities');
// partial collections
RectEntities = new Mongo.Collection('rectEntities');
PolyEntities = new Mongo.Collection('polyEntities');

Client code:

// this will fill your collections with entries from the Entities collection
Meteor.subscribe('rectEntities');
Meteor.subscribe('polyEntities');

Remember that the name of the subscription needs to match the name of the publication (but not the name of the collection itself)
Server code:

Meteor.publish('rectEntities', function(){
    Mongo.Collection._publishCursor( Entities.find({shapeType: 'rectangle'}), this, 'rectEntities'); 
    this.ready();
});

Meteor.publish('polyEntities', function(){
    Mongo.Collection._publishCursor( Entities.find({shapeType: 'polygon'}), this, 'polyEntities'); 
    this.ready();
});

Thanks to user728291 for the much simpler solution using _publishCursor()!
The third argument of the _publishCursor() function is the name of your new collection.
Source: http://docs.meteor.com/#/full/publish_added

Comments

[matb33]

Just want to make a note that my information on how Meteor "combines" the records may not be accurate or true. If a Meteor dev or someone who knows better can confirm, please do. My assumption is based on my own personal observations.

[matb33]

It looks like my observations were correct concerning the combining. See the "Merge Box" section in this answer: stackoverflow.com/a/13867122/962223

[cramhead]

When I put the search functions in a shared folder, e.g. lib the publish functions can find them. When I put them in the same file as the publish functions they are found by the publish functions, but aren't visible on the client. Ideas?

[Thomas]

put them into a Meteor.startup() function!

[THEtheChad]

Ohhhhhh. This makes so much more sense. What you're saying is that, regardless of how many pub/sub functions you have, the Users collection will contain ALL of the records from every subscription and you have to query those records to extract the ones you want.

[Florian]

For future reference, this article explains the merging of multiple subscriptions into collections: meteorpedia.com/read/Understanding_Meteor_Publish_and_Subscr‌​ibe

[user728291]

Your code can be a lot shorter using the undocumented _publishCursor. For example Meteor.publish('polyEntities', function(){ Mongo.Collection._publishCursor( Entities.find({shapeType: 'polygon'}), this, 'polyEntities'); this.ready();});

[PhilippSpo]

@user728291 how would you handle updates on the PolyEntities collection?

[user728291]

You would have to write custom Meteor.methods. The client stub would update/insert to PolyEntities (and possibly RectEntities) and the server would update/insert to Entities.

[PhilippSpo]

i found another easy way: just update the local copy of the collection via PolyEntities._collection_.update() and then have this the observeChanges() method watching those changes and publish them to the Entities collection. Sadly the performance of this is not good it seems.. the changes i make to my local collection get overwritten later when the update from the subscription comes .. the m meteor-method way should be faster though

[PhilippSpo]

yup the meteor method way has no performance issues, the only drawback is, that you than have to call the method manually all the time, and cannot trigger it once something updates the collection..

[Samba]

@PhilippSpo - In my product, we need to make collections non-reactive as we have sharded mongo with no single op log for meteor. So, can we use the technique suggested by you to make client collection different than server collections. And also make client collections as reactive = false.

[user2649601]

How to "consume" this on the client?

[user2649601]

Quite counterintuitive to have to define the selection basically twice, on client and on server. But thanks for explaining this part missing from most meteor docs.