How to implement 3-way conference call video chat with WebRTC Native Code for Android?

Question

I'm trying to implement 3-way video chat inside an Android app using the WebRTC Native Code package for Android (i.e. not using a WebView). I've written a signalling server using node.js and used the Gottox socket.io java client library inside the client app to connect to the server, exchange SDP packets and establish a 2-way video chat connection.

However now I'm having problems going beyond that to a 3-way call. The AppRTCDemo app that comes with the WebRTC native code package demonstrates 2-way calls only (if a 3rd party attempts to join a room a "room full" message is returned).

According to this answer (which doesn't relate to Android specifically), I'm supposed to do it by creating multiple PeerConnections, so each chat participant will connect to the 2 other participants.

However, when I create more than one PeerConnectionClient (a Java class which wraps a PeerConection, which is implemented on the native side in libjingle_peerconnection_so.so), there is an exception thrown from inside the library resulting from a conflict with both of them trying to access the camera:

E/VideoCapturerAndroid(21170): startCapture failed
E/VideoCapturerAndroid(21170): java.lang.RuntimeException: Fail to connect to camera service
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.native_setup(Native Method)
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.<init>(Camera.java:548)
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.open(Camera.java:389)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid.startCaptureOnCameraThread(VideoCapturerAndroid.java:528)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid.access$11(VideoCapturerAndroid.java:520)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid$6.run(VideoCapturerAndroid.java:514)
E/VideoCapturerAndroid(21170):  at android.os.Handler.handleCallback(Handler.java:733)
E/VideoCapturerAndroid(21170):  at android.os.Handler.dispatchMessage(Handler.java:95)
E/VideoCapturerAndroid(21170):  at android.os.Looper.loop(Looper.java:136)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid$CameraThread.run(VideoCapturerAndroid.java:484)

This happens when initializing the local client even before attempting to establish a connection so it's not related to node.js, socket.io or any of the signalling server stuff.

How do I get multiple PeerConnections to share the camera so that I can send the same video to more than one peer?

One idea I had was to implement some kind of singleton camera class to replace VideoCapturerAndroid that could be shared between multiple connections, but I'm not even sure that would work and I'd like to know if there is a way to do 3-way calls using the API before I start hacking around inside the library.

Is it possible and if so, how?

Update:

I tried sharing a VideoCapturerAndroid object between multiple PeerConnectionClients, creating it for the first connection only and passing it into the initialization function for the subsequent ones, but that resulted in this "Capturer can only be taken once!" exception when creating a second VideoTrack from the VideoCapturer object for the second peer connection:

E/AndroidRuntime(18956): FATAL EXCEPTION: Thread-1397
E/AndroidRuntime(18956): java.lang.RuntimeException: Capturer can only be taken once!
E/AndroidRuntime(18956):    at org.webrtc.VideoCapturer.takeNativeVideoCapturer(VideoCapturer.java:52)
E/AndroidRuntime(18956):    at org.webrtc.PeerConnectionFactory.createVideoSource(PeerConnectionFactory.java:113)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.createVideoTrack(PeerConnectionClient.java:720)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.createPeerConnectionInternal(PeerConnectionClient.java:482)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.access$20(PeerConnectionClient.java:433)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient$2.run(PeerConnectionClient.java:280)
E/AndroidRuntime(18956):    at android.os.Handler.handleCallback(Handler.java:733)
E/AndroidRuntime(18956):    at android.os.Handler.dispatchMessage(Handler.java:95)
E/AndroidRuntime(18956):    at android.os.Looper.loop(Looper.java:136)
E/AndroidRuntime(18956):    at com.example.rtcapp.LooperExecutor.run(LooperExecutor.java:56)

Attempting to share the VideoTrack object between PeerConnectionClients resulted in this error from the native code:

E/libjingle(19884): Local fingerprint does not match identity.
E/libjingle(19884): P2PTransportChannel::Connect: The ice_ufrag_ and the ice_pwd_ are not set.
E/libjingle(19884): Local fingerprint does not match identity.
E/libjingle(19884): Failed to set local offer sdp: Failed to push down transport description: Local fingerprint does not match identity.

Sharing the MediaStream between PeerConnectionClients results in the app abruptly closing, without any error message appearing in the Logcat.

Solution 1

The problem you are having is that PeerConnectionClient is not a wrapper around PeerConnection it contains a PeerConnection.

I noticed this question wasn't answered so I wanted to see if I could help out a bit. I looked into the source code and PeerConnectionClient is very much hard coded for a single remote peer. You would need to create a collection of PeerConnection objects rather then this line:

private PeerConnection peerConnection;

If you look around a bit more you would notice that it gets a bit more complicated then that.

The mediaStream logic in createPeerConnectionInternal should only be done once and you need to share the stream between your PeerConnection objects like this:

peerConnection.addStream(mediaStream);

You can consult the WebRTC spec or take a look at this stackoverflow question to confirm that the PeerConnection type was designed to only handle one peer. It is also somewhat vaguely implied here.

So you only maintain one mediaStream object:

private MediaStream mediaStream;

So again the main idea is one MediaStream object and as many PeerConnection objects as you have peers you want to connect to. So you will not be using multiple PeerConnectionClient objects, but rather modify the single PeerConnectionClient to encapsulate the multi-client handling. If you do want to go with a design of multiple PeerConnectionClient objects for whatever reason you would just have to abstract the media stream logic (and any support types that should only be created once) out of it.

You will also need to maintain multiple remote video tracks rather then the existing one:

private VideoTrack remoteVideoTrack;

You would obviously only care to render the one local camera and create multiple renderers for the remote connections.

I hope this is enough information to get you back on track.

Solution 2

With the help of Matthew Sanders' answer I managed to get it working, so in this answer I'm going to describe in more detail one way of adapting the sample code to support video conference calling:

Most of the changes need to be made in PeerConnectionClient, but also in the class that uses PeerConnectionClient, which is where you communicate with the signalling server and set up the connections.

Inside PeerConnectionClient, the following member variables need to be stored per-connection:

private VideoRenderer.Callbacks remoteRender;
private final PCObserver pcObserver = new PCObserver();
private final SDPObserver sdpObserver = new SDPObserver();
private PeerConnection peerConnection;
private LinkedList<IceCandidate> queuedRemoteCandidates;
private boolean isInitiator;
private SessionDescription localSdp;
private VideoTrack remoteVideoTrack;

In my application I needed a maximum of 3 connections (for a 4-way chat), so I just stored an array of each, but you could put them all inside an object and have an array of objects.

private static final int MAX_CONNECTIONS = 3;
private VideoRenderer.Callbacks[] remoteRenders;
private final PCObserver[] pcObservers = new PCObserver[MAX_CONNECTIONS];
private final SDPObserver[] sdpObservers = new SDPObserver[MAX_CONNECTIONS];
private PeerConnection[] peerConnections = new PeerConnection[MAX_CONNECTIONS];
private LinkedList<IceCandidate>[] queuedRemoteCandidateLists = new LinkedList[MAX_CONNECTIONS];
private boolean[] isConnectionInitiator = new boolean[MAX_CONNECTIONS];
private SessionDescription[] localSdps = new SessionDescription[MAX_CONNECTIONS];
private VideoTrack[] remoteVideoTracks = new VideoTrack[MAX_CONNECTIONS];

I added a connectionId field to the PCObserver and SDPObserver classes, and inside the PeerConnectionClient constructor I allocated the observer objects in the array and set the connectionId field for each observer object to its index in the array. All the methods of PCObserver and SDPObserver that reference the member variables listed above should be changed to index into the appropriate array using the connectionId field.

The PeerConnectionClient callbacks need to be changed:

public static interface PeerConnectionEvents {
    public void onLocalDescription(final SessionDescription sdp, int connectionId);
    public void onIceCandidate(final IceCandidate candidate, int connectionId);
    public void onIceConnected(int connectionId);
    public void onIceDisconnected(int connectionId);
    public void onPeerConnectionClosed(int connectionId);
    public void onPeerConnectionStatsReady(final StatsReport[] reports);
    public void onPeerConnectionError(final String description);
}

And also the following PeerConnectionClient methods:

private void createPeerConnectionInternal(int connectionId)
private void closeConnectionInternal(int connectionId)
private void getStats(int connectionId)
public void createOffer(final int connectionId)
public void createAnswer(final int connectionId)
public void addRemoteIceCandidate(final IceCandidate candidate, final int connectionId)
public void setRemoteDescription(final SessionDescription sdp, final int connectionId)
private void drainCandidates(int connectionId)

As with the methods in the observer classes, all these functions need to be changed to use the connectionId to index into the appropriate array of per-connection objects, instead of referencing the single objects they were previously. Any invocations of callback functions need to also be changed to pass the connectionId back.

I replaced createPeerConnection with a new function called createMultiPeerConnection, which is passed an array of VideoRenderer.Callbacks objects for displaying the remote video stream, instead of a single one. The function calls createMediaConstraintsInternal() once and createPeerConnectionInternal() for each of the PeerConnections, looping from 0 to MAX_CONNECTIONS - 1. The mediaStream object is created only on the first call to createPeerConnectionInternal(), just by wrapping the initialization code in an if(mediaStream == null) check.

One complication I encountered was when when the app shuts down and the PeerConnection instances are closed and the MediaStream disposed of. In the sample code the mediaStream is added to a PeerConnection using addStream(mediaStream), but the corresponding removeStream(mediaStream) function is never called (dispose() is called instead). However this creates problems (a ref count assert in MediaStreamInterface in the native code) when there are more than one PeerConnection sharing a MediaStream object because dispose() finalizes the MediaStream, which should only happen when the last PeerConnection is closed. Calling removeStream() and close() is not enough either, because it doesn't fully shut down the PeerConnection and this leads to an assert crash when disposing the PeerConnectionFactory object. The only fix I could find was to add the following code to the PeerConnection class:

public void freeConnection()
{
    localStreams.clear();
    freePeerConnection(nativePeerConnection);
    freeObserver(nativeObserver);
}

And then calling these functions when finalizing each PeerConnection except the last:

peerConnections[connectionId].removeStream(mediaStream);
peerConnections[connectionId].close();
peerConnections[connectionId].freeConnection();
peerConnections[connectionId] = null;

and shutting down the last one like this:

peerConnections[connectionId].dispose();
peerConnections[connectionId] = null;

After modifying PeerConnectionClient, it's necessary to change the signalling code to set up the connections in the right order, passing in the correct connection index to each of the functions and handling the callbacks appropriately. I did this by maintaining a hash between socket.io socket ids and a connection id. When a new client joins the room, each of the existing members sends an offer to the new client and receives an answer in turn. It's also necessary to initialize multiple VideoRenderer.Callbacks objects, pass them in to the PeerConnectionClient instance, and divide up the screen however you want for a conference call.