Low-latency streaming using ffmpeg

Question

Recently, I'm trying to stream a remote desktop using ffmpeg. The goal is to being able to send what is shown on one of the computer's screen to another computer while being able to choose settings like the output resolution, etc.

It is a small nearly working proof of concept. When done, I'll be able to replace VNC by using ffmpeg streaming with x2x or synergy for forwarding keyboard and mouse events.

Now, I can start streaming using:

ffmpeg -f x11grab -s "1600x1200" -i ":0" \
    -f alsa -i pulse \
    -s 800x600 -b 200k -f mpegts - \
    | mplayer -cache 1024 -
#
# I have pulse audio configured so that `-i pulse` will
# The output can sent through for example netcat to another host
#

The latency depends on the encoding options. With these options, the delay between the screen update and the video update is arround 800ms.

Thing I'm trying to achieve:

• Reach a latency of 100ms.

• When mplayer is executed it complains that it cannot seek in lear streams and there is no audio. I still have the sound when I'm saving the output to a file and play it.

  EDIT: After adding -cache, messages to seek in linear streams no longer appear. Changing output format to -f mpegts makes the audio work.

• It would be great if the encoding wouldn't take 100% of one of the cpu cores (secondary goal).

After some reseach on the internet, I think that these problems are related to the codecs/options I should use. Yet I don't know the differences between the existing possibilities. Could you give me some options that would solve my problems? Also, is VLC a good alternative and if so what would be the equivalent commands to stream from one desktop to another?

Solution

Streaming from desktopA to desktopB is an interesting problem. If you don't need video/audio, then you might want to check into the NX protocol instead. It is much more efficient than VNC. FreeNX is a server and NoMachine and others make NX clients. There are other servers - free and commercial.

There is also work being performed on a high performance remote desktop, SPICE, capability in as a F/LOSS project. I think Redhat is taking the lead. http://spice-space.org/

I don't know if these are useful answers, but perhaps just leads for something more useful?

For streaming video, avoiding the overhead of a full desktop protocol is probably needed.

Comments

[mtone]

I wonder if a CUDA-based solution for both encoding-decoding wouldn't help tremendously with latency. Sounds like an ideal application. But I don't know if there are streaming capable CUDA encoders yet.

[Wayne Jhukie]

Why do you want to replace VNC, which is optimised for this sort of usage?

[Kru]

Since then, I have found xpra which does even more than I initially needed.

[rogerdpack]

ffmpeg.org/trac/ffmpeg/wiki/StreamingGuide#Latency has some notes on latency for streaming

[Cestarian]

Spice seems to be aimed at streaming virtual desktops, not physical ones... does it even work with physical ones?