Nginx-rtmp-module How edge nodes pull streams from multi Master nodes

Question

I have one master ingest server and multiple edge nodes that each pull from the ingest server. But now i have 10000 cameras push streams to master ingest server , i m afraid it can't handle it. So i plan to add more master ingest server ( with LoadBalancer LVS to handle which master ingest server those cameras will push stream to). But with more than one master ingest server how edge nodes pull all those stream . I also test pull option in edge node like this

application live {
   notify_method get;
   idle_streams off;
   pull rtmp://master_server_1/live live=1;
   pull rtmp://master_server_2/live live=1;
}

But it only pull stream from master_server_1. Any advice ? Thanks

Btw, I think i dont need more master server , just upgrade bandwidth . Is it right ? Thanks

Solution

You can have a DNS record resolve to multiple origin servers -- not only could you use this for simple DNS round-robin load balancing, but this would be the way you'd have nginx-rtmp pull from multiple origins.

After a certain point, it makes sense to have more than one ingest server. Other practical considerations aside, it gives you more reliability. With just one server, you have a single point of failure.

Source: https://groups.google.com/forum/#!topic/nginx-rtmp/B2QqkwXxASk

NOTE #1: NGINX caches DNS resolutions until it is restarted. If you expect IPs to change (I'm using an AWS ELB, so they change a lot), you'll either need to mess with the DNS resolver in NGINX or use the exec_pull directive in combination with ffmpeg instead.

NOTE #2: All the nodes being pulled from must have the same streams. It has no way of knowing whether a certain stream exists on the origin or not. It will use the first available (connection wise) origin.

If you want to have origins with separate sets of streams (makes sense for scaling), you will have to manage that on your end. I've solved this using a consistent hashing algorithm to map streams to different video clusters, spreading the load around. e.g.: Streams published to origin "1" can only be played on edges pulling from origin "1". The implementation of such a system ought to be in your application layer.