Using Multiple Mongodb Databases with Meteor.js

Question

Is it possible for 2 Meteor.Collections to be retrieving data from 2 different mongdb database servers?

Dogs = Meteor.Collection('dogs')        // mongodb://192.168.1.123:27017/dogs
Cats = Meteor.Collection('cats')        // mongodb://192.168.1.124:27017/cats

Solution 1

Update

It is now possible to connect to remote/multiple databases:

var database = new MongoInternals.RemoteCollectionDriver("<mongo url>");
MyCollection = new Mongo.Collection("collection_name", { _driver: database });

Where <mongo_url> is a mongodb url such as mongodb://127.0.0.1:27017/meteor (with the database name)

There is one disadvantage with this at the moment: No Oplog

Old Answer

At the moment this is not possible. Each meteor app is bound to one database.

There are a few ways you can get around this but it may be more complicated that its worth:

One option - Use a separate Meteor App

In your other meteor app (example running at port 6000 on same machine). You can still have reactivity but you need to proxy inserts, removes and updates through a method call

Server:

Cats = Meteor.Collection('cats')

Meteor.publish("cats", function() {
    return Cats.find();
});

Meteor.methods('updateCat, function(id, changes) {
    Cats.update({_id: id}, {$set:changes});
});

Your current Meteor app:

var connection = DDP.connect("http://localhost:6000");

connection.subscribe("cats");
Cats = Meteor.Collection('cats', {connection: connection});

//To update a collection
Cats.call("updateCat", <cat_id>, <changes);

Another option - custom mongodb connection

This uses the node js mongodb native driver.

This is connecting to the database as if you would do in any other node js app.

There is no reactivity available and you can't use the new Meteor.Collection type collections.

var mongodb = Npm.require("mongodb"); //or var mongodb = Meteor.require("mongodb") //if you use npm package on atmosphere

var db = mongodb.Db;
var mongoclient = mongodb.MongoClient;
var Server = mongodb.Server;

var db_connection = new Db('cats', new Server("127.0.0.1", 27017, {auto_reconnect: false, poolSize: 4}), {w:0, native_parser: false});

db.open(function(err, db) {
    //Connected to db 'cats'

    db.authenticate('<db username>', '<db password>', function(err, result) {
      //Can do queries here
      db.close();
   });
});

Solution 2

The answer is YES: it is possible set up multiple Meteor.Collections to be retrieving data from different mongdb database servers.

As the answer from @Akshat, you can initialize your own MongoInternals.RemoteCollectionDriver instance, through which Mongo.Collections can be created.

But here's something more to talk about. Being contrary to @Akshat answer, I find that Oplog support is still available under such circumstance.

When initializing the custom MongoInternals.RemoteCollectionDriver, DO NOT forget to specify the Oplog url:

var driver = new MongoInternals.RemoteCollectionDriver(
    "mongodb://localhost:27017/db", 
    {
      oplogUrl: "mongodb://localhost:27017/local"
    });
var collection = new Mongo.Collection("Coll", {_driver: driver});

Under the hood

As described above, it is fairly simple to activate Oplog support. If you do want to know what happened beneath those two lines of codes, you can continue reading the rest of the post.

In the constructor of RemoteCollectionDriver, an underlying MongoConnection will be created:

MongoInternals.RemoteCollectionDriver = function (
  mongo_url, options) {
  var self = this;
  self.mongo = new MongoConnection(mongo_url, options);
}; 

The tricky part is: if MongoConnection is created with oplogUrl provided, an OplogHandle will be initialized, and starts to tail the Oplog (source code):

if (options.oplogUrl && ! Package['disable-oplog']) {  
  self._oplogHandle = new OplogHandle(options.oplogUrl, self.db.databaseName);
  self._docFetcher = new DocFetcher(self);
}

As this blog has described: Meteor.publish internally calls Cursor.observeChanges to create an ObserveHandle instance, which automatically tracks any future changes occurred in the database.

Currently there are two kinds of observer drivers: the legacy PollingObserveDriver which takes a poll-and-diff strategy, and the OplogObseveDriver, which effectively use Oplog-tailing to monitor data changes. To decide which one to apply, observeChanges takes the following procedure (source code):

var driverClass = canUseOplog ? OplogObserveDriver : PollingObserveDriver;
observeDriver = new driverClass({
  cursorDescription: cursorDescription,
  mongoHandle: self,
  multiplexer: multiplexer,
  ordered: ordered,
  matcher: matcher,  // ignored by polling
  sorter: sorter,  // ignored by polling
  _testOnlyPollCallback: callbacks._testOnlyPollCallback
});

In order to make canUseOplog true, several requirements should be met. A bare minimal one is: the underlying MongoConnection instance should have a valid OplogHandle. This is the exact reason why we need to specify oplogUrl while creating MongoConnection

Solution 3

This is actually possible, using an internal interface:

var d = new MongoInternals.RemoteCollectionDriver("<mongo url>");
C = new Mongo.Collection("<collection name>", { _driver: d });

Comments

[Emily]

It is actually possible to have a collection backed by a different database server; see my answer below.

[Tarang]

@Emily excellent, I've been waiting a long time for this. On the client we use the same ordinary collection yes?

[JoeTidee]

Where do you put these two lines of code? In the if (Meteor.isServer) {} block, in the Meteor.startup(function () {}); block ?

[JoeTidee]

Where do you put these two lines of code? In the "if (Meteor.isServer) {}" block, in the "Meteor.startup(function () {});" block ? I keep getting "ReferenceError: C is not defined".

[JoeTidee]

Note that this should ONLY be run on the server, i.e. inside the "if (Meteor.isServer) {" block. You will also have to add "C = new Mongo.Collection("<collection name>");" to your "if (Meteor.isClient) {" block.

[Tarang]

@JoePrivett anywhere on the server.

[Shwaydogg]

Quick Tip if you find your collections empty: make sure to include the db name at the end of the mongodb url, it will fail silently and your collections will just appear empty. If you want to connect to another local meteor mongodb instance that you ran on say port 4444 and mongo is on 4445 the url would look like this: mongodb://127.0.0.1:4445/meteor

[ajamardo]

This works if the databases have different collection names, but not if you have the same collection names in both databases. Does anyone know how to do that?

[redcap3000]

Any idea how to get this working in react? The client side errors out when exporting the database (like you would a normal collection in react/meteor) Wrapping in Meteor.isServer() doesn't help.

[kroe]

this need to be marked as correct answer! i have flagged this post in hope that an admin or OP can switch the answer!