Understanding Meteor Publish / Subscribe

Question

I've got a simple app set up that shows a list of Projects. I've removed the autopublish package so that I'm not sending everything to the client.

 <template name="projectsIndex">    
   {{#each projects}}      
     {{name}}
   {{/each}}
 </template>

When autopublish was turned on, this would display all the projects:

if Meteor.isClient
  Template.projectsIndex.projects = Projects.find()

With it removed, I have to additionally do:

 if Meteor.isServer
   Meteor.publish "projects", ->
     Projects.find()
 if Meteor.isClient
   Meteor.subscribe "projects"
   Template.projectsIndex.projects = Projects.find()

So, is it accurate to say that the client-side find() method only searches records which have been published from the server-side? It's been tripping me up because I felt like I should only be calling find() once.

Solution 1

Collections, publications and subscriptions are a tricky area of Meteor, that the documentation could discuss in more detail, so as to avoid frequent confusion, which sometimes get amplified by confusing terminology.

Here's Sacha Greif (co-author of DiscoverMeteor) explaining publications and subscriptions in one slide:

To properly understand why you need to call find() more than once, you need to understand how collections, publications and subscriptions work in Meteor:

1. You define collections in MongoDB. No Meteor involved yet. These collections contain database records (also called "documents" by both Mongo and Meteor, but a "document" is more general than a database record; for instance, an update specification or a query selector are documents too - JavaScript objects containing field: value pairs).

2. Then you define collections on the Meteor server with

   MyCollection = new Mongo.Collection('collection-name-in-mongo')
   

   These collections contain all the data from the MongoDB collections, and you can run MyCollection.find({...}) on them, which will return a cursor (a set of records, with methods to iterate through them and return them).

3. This cursor is (most of the time) used to publish (send) a set of records (called a "record set"). You can optionally publish only some fields from those records. It is record sets (not collections) that clients subscribe to. Publishing is done by a publish function, which is called every time a new client subscribes, and which can take parameters to manage which records to return (e.g. a user id, to return only that user's documents).

4. On the client, you have Minimongo collections that partially mirror some of the records from the server. "Partially" because they may contain only some of the fields, and "some of the records" because you usually want to send to the client only the records it needs, to speed up page load, and only those it needs and has permission to access.

   Minimongo is essentially an in-memory, non-persistent implementation of Mongo in pure JavaScript. It serves as a local cache that stores just the subset of the database that this client is working with. Queries on the client (find) are served directly out of this cache, without talking to the server.

   These Minimongo collections are initially empty. They are filled by

   Meteor.subscribe('record-set-name')
   

   calls. Note that the parameter to subscribe isn't a collection name; it's the name of a record set that the server used in the publish call. The subscribe() call subscribes the client to a record set - a subset of records from the server collection (e.g. most recent 100 blog posts), with all or a subset of the fields in each record (e.g. only title and date). How does Minimongo know into which collection to place the incoming records? The name of the collection will be the collection argument used in the publish handler's added, changed, and removed callbacks, or if those are missing (which is the case most of the time), it will be the name of the MongoDB collection on the server.

Modifying records

This is where Meteor makes things very convenient: when you modify a record (document) in the Minimongo collection on the client, Meteor will instantly update all templates that depend on it, and will also send the changes back to the server, which in turn will store the changes in MongoDB and will send them to the appropriate clients that have subscribed to a record set including that document. This is called latency compensation and is one of the seven core principles of Meteor.

Multiple subscriptions

You can have a bunch of subscriptions that pull in different records, but they'll all end up in the same collection on the client if the came from the same collection on the server, based on their _id. This is not explained clearly, but implied by the Meteor docs:

When you subscribe to a record set, it tells the server to send records to the client. The client stores these records in local Minimongo collections, with the same name as the collection argument used in the publish handler's added, changed, and removed callbacks. Meteor will queue incoming attributes until you declare the Mongo.Collection on the client with the matching collection name.

What's not explained is what happens when you don't explicitly use added, changed and removed, or publish handlers at all - which is most of the time. In this most common case, the collection argument is (unsurprisingly) taken from the name of the MongoDB collection you declared on the server at step 1. But what this means is that you can have different publications and subscriptions with different names, and all the records will end up in the same collection on the client. Down to the level of top level fields, Meteor takes care to perform a set union among documents, such that subscriptions can overlap - publish functions that ship different top level fields to the client work side by side and on the client, the document in the collection will be the union of the two sets of fields.

Example: multiple subscriptions filling the same collection on the client

You have a BlogPosts collection, which you declare the same way on both the server and the client, even though it does different things:

BlogPosts = new Mongo.Collection('posts');

On the client, BlogPosts can get records from:

1. a subscription to the most recent 10 blog posts

   // server
   Meteor.publish('posts-recent', function publishFunction() {
     return BlogPosts.find({}, {sort: {date: -1}, limit: 10});
   }
   // client
   Meteor.subscribe('posts-recent');
   

2. a subscription to the current user's posts

   // server
   Meteor.publish('posts-current-user', function publishFunction() {
     return BlogPosts.find({author: this.userId}, {sort: {date: -1}, limit: 10});
     // this.userId is provided by Meteor - http://docs.meteor.com/#publish_userId
   }
   Meteor.publish('posts-by-user', function publishFunction(who) {
     return BlogPosts.find({authorId: who._id}, {sort: {date: -1}, limit: 10});
   }
   
   // client
   Meteor.subscribe('posts-current-user');
   Meteor.subscribe('posts-by-user', someUser);
   

3. a subscription to the most popular posts

4. etc.

All these documents come from the posts collection in MongoDB, via the BlogPosts collection on the server, and end up in the BlogPosts collection on the client.

Now we can understand why you need to call find() more than once - the second time being on the client, because documents from all subscriptions will end up in the same collection, and you need to fetch only those you care about. For example, to get the most recent posts on the client, you simply mirror the query from the server:

var recentPosts = BlogPosts.find({}, {sort: {date: -1}, limit: 10});

This will return a cursor to all documents/records that the client has received so far, both the top posts and the user's posts. (thanks Geoffrey).

Solution 2

Yes, the client-side find() only returns documents that are on the client in Minimongo. From docs:

On the client, a Minimongo instance is created. Minimongo is essentially an in-memory, non-persistent implementation of Mongo in pure JavaScript. It serves as a local cache that stores just the subset of the database that this client is working with. Queries on the client (find) are served directly out of this cache, without talking to the server.

As you say, publish() specifies which documents the client will have.

Solution 3

Basic thumb rule here is publish and subscribed variable names should be same on client and server side.

Collections names on Mongo DB and client side should be same.

Assume that I'm using publish and subscribe for my collection named employees then code would look like

---

server side

Here usage of var keyword is optional (use this keyword to make collection local to this file).

CollectionNameOnServerSide = new Mongo.Collection('employees');   

Meteor.publish('employeesPubSub', function() { 
    return CollectionNameOnServerSide.find({});     
});

---

client side .js file

CollectionNameOnClientSide = new Mongo.Collection('employees');
var employeesData = Meteor.subscribe('employeesPubSub');

Template.templateName.helpers({
  'subcribedDataNotAvailable' : function(){
        return !employeesData.ready();
    },
   'employeeNumbers' : () =>{
       CollectionNameOnClientSide.find({'empId':1});
  }
});

---

client side .html file

Here we can use subcribedDataNotAvailable helper method to know if data is ready on client side, if data is ready then print the employee numbers using employeeNumbers helper method.

<TEMPLATE name="templateName">
{{#if subcribedDataNotAvailable}}
   <h1> data loading ... </h1>
 {{else}}
  {{#each employeeNumbers }}
     {{this}}
  {{/each}}
 {{/if}}
<TEMPLATE>

Comments

[Miro]

Very good explanation, @Dan. The only thing I'm missing is where's Posts defined?

[Dan Dascalescu]

@Miro: thanks and good catch. I've fixed the typo (Posts -> BlogPosts).

[Geoffrey Booth]

This is great. Maybe worth mentioning is what happens if you do BlogPosts.find({}) on the client after subscribing to both publications—i.e. it will return a cursor of all documents/records currently on the client, both the top posts and the user's posts. I've seen other questions on SO where the questioner was confused by this.

[mirageglobe]

This is great. thanks. In addition, the Meteor.users() collection gets a tad confusing as it is auto published on the client side. Can a bit be added to the above answer to state the users() collection?

[physiocoder]

Even if much more than originally asked for I think @DVG should mark this great writeup as the accepted answer. Thanks Dan.

[UXE]

Thanks @DanDascalescu, Great explanation that cleared much for me, the only thing that when following meteor docs about "collections" after reading your explanation I think BlogPosts isn't a collection, its the returned object that has methods like "insert", "update" ..etc, and the real collection is posts in the client and the server as well.

[Dan Dascalescu]

@UXE: BlogPosts is a Meteor.Collection object, and yes, it has methods like insert and update. The name of the Minimongo collection on the client, and the name of the MongoDB collection, is indeed posts. Let me know if I can explain better.

[UXE]

@DanDascalescu: thanks, I got confused I think, here is what I'm on now and please tell me if I got it wrong!... posts is just the real collection name inside MongoDB store, BlogPosts is the object used by the mongo client to refer to postscollection!

[Jimmy Knoot]

Is it possible to call for only the record set you subscribed to? As in, is it possible to directly get the record set in my javascript, instead of querying the Minimongo db locally?

[Dan Dascalescu]

@JimmyKnoot - if I understand the question correctly, as in, "Is it possible to obtain the entire record set sent by the server, without having to run the query again on the client's Minimongo", then the answer is no.

[Jimmy Knoot]

@DanDascalescu Yes, that's exactly what I meant. I see, that's a shame, thanks anyway :)

[2016rshah]

I understand why you need to call find() twice (once on the server, once on the client) but if you know that you are going to need a specific subset multiple times on the client, how can you declare a variable for that specific subset? Whenever I try using a callback on the subscribe to assign the variable, the code that uses the variable runs async and breaks everything because the variable hasn't been declared yet. Any suggestions?

[Dean Radcliffe]

Is it possible to 'get the latest' from a particular subscription ? Do I have to repeat the publication query to query the client's collection ? That doesn't seem very DRY, if so. It feels like once records from a subscription make their way into a collection, their relation to the subscription disappears, I'm hoping that is not so.

[Dean Radcliffe]

@JimmyKnoot, I've been looking at the same question! My workaround is to declare a module of named queries, living in lib/ since its loaded before client/ and server/, which is used both to implement the publication, and query the collection that the subscription dumps its data into. Feels hacky, but it's what I've got so far..

[Alexander Mills]

It is my humble opinion that Meteor.subscribe('x') should RETURN something and that we all should know what that is.

[Dan Dascalescu]

@Dean, 2016, Jimmy: this is indeed a problem with Meteor pub/sub: you have to repeat the query on the client. I raised the question on meteor-core and as far as I know, there's no good solution for it in core.

[Dean Radcliffe]

@AlexMills, I think it should return a promise for the subscription's readiness, though I think some of the core team members disagree. Discussion here: github.com/meteor/meteor/issues/3776

[Andriy Drozdyuk]

The whole looks a bit like "projection" from Event Sourcing, or a "read model" from CQRS. In fact, meteor seems to favor separation of Commands and Queries (after removal of "autopublish").

[Cos]

However this doesn't explain what happens when the two published subsets contain different fields. Can someone elaborate on that?

[Salman Hasni]

Nice Explanation (Y)

[Gordon Sun]

Thank you very much @DanDascalescu!

[Codious-JR]

Great Explanation. The whole concept was unclear to me before I read your post.

[evolross]

Does anyone know if you can have two subscriptions on the same document, but have one subscription return a few fields reactively and have another subscription return a certain field(s) using polling - like a counter field? So the polling subscription would use disableOplog: true, pollingIntervalMs: 3000?