System architecture of Cam Chat Websites (tinychat, omegle, cam4)

Question

Do anybody know what kind of system architecture is behind Websites like tinychat, omegle, cam4, chatroulette? And how much bandwith is needed per connection?

If I'm not wrong they use some video streaming Server like Flash Media Server, Wowza Media Server, Red5, Erlyvideo but this would eat for 20'000 users around 1 Gb/s bandwith? Or do they use some software who is wroking peer2peer? I found some kind of such Software but they all requires some UDP ports to be opened.

Do anybody know how much users 1 server instance with some of the above server software can handle?

Im happy for any background information.

Solution

At least chatroulette is using RTMFP and stratus to push video from client to client, without passing it to a server (no big bandwidth usage). This goes through most simple home-networks thanks to hole punching.

The session setup and metadata passes through a server but that's simple message passing one could do with almost any technology today. Afaik they're using FMS for that.

See http://labs.adobe.com/technologies/stratus/ and http://blogs.adobe.com/collabmethods/2008/12/try_rtmfp_and_clienttoclient_d.html

The rough flow is as follows:

1. user fires up browser to the site
2. Flash client connects to the server (outbound connection to server, could be rtmp or anything)
3. Flash client gets special permission to connect beyond the policies built in through the stratus api
4. Flash client executes UDP hole punching at the same time as the other person's client is. A direct UDP link is estabelished and they exchange audio/video data

I'd have to research it a bit to figure out what they do for fallback, but most likely it's a simple server relay of data, with the throughput requirements attached.

Comments

[Prix]

And how much bandwith is needed per connection? this information depends on what encode is in use it can vary from a small size to a huge size. Do anybody know how much users 1 server instance with some of the above server software can handle? That does not depend entirely on the Server you are using but aswell on the stream type you are using since before considering the server you must consider your ethernet capacity aswell as your bandwidth limits ... for example if you are streaming 128K/s to 100 users youre then delivering 100mbps per second.

[Nenad]

I assume the mentioned websites stream with around 32K/s, but this would mean for 10k users 2500mbps bandwith is needed, above sites hold above 100k users this will result in 25'000mbps this is really much bandwith. And the server can handle per 1Gbit port around 4'000 users @ 32 K/s. Do anybody know what system spec is required to handle this number of users? Can the above software use multiple nic ports (4x 1Gbit)?

[Prix]

yes it is possible you could use different exits for each link basicly it would be load balance that would redirect users to another server/link once A has hit the max usage or so. A good example of this would be rapidsahre, megaupload and even youtube.

[Antoine Benkemoun]

+1 very good question :)

[Joris]

As written below, the bandwidth (mostly) does not go through the server. Also 32K will get you nowhere, quality-wise. Think 128Kbit/s at very least. Serving live content (no disk) is really cheap, on modern machines think at least 2x 10G

[Nenad]

Youre right, i try to connect to chatroulette without any firewall/nat routing and see it uses RTMFP if i block incoming connections it fall back to RTMP. But i think today most people dont allow such incoming connections or I'm wrong?