MJPEG streaming and decoding

Question

I want to receive JPEG images from an IP camera (over RTSP). For this, I tried cvCreateFileCapture_FFMPEG in OpenCV. But ffmpeg seems to have some problem with the MJPEG format of the streaming (since it automatically tries to detect the streaming info) and I end up with the following error

mjpeg: unsupported coding type

I, then, decided to use live555 for streaming. Till now, I can successfully establish streaming and capture (non-decoded) images through openRTSP.

The question is how can I do this in my application, e.g., in OpenCV. How can I use openRTSP in OpenCV to get images and save them in JPEG format?

I have heard that the data from openRTSP can be sent to a buffer (or a named pipe) and then read in OpenCV's IplImage. But I don't know how to do this.

I will really appreciate any help/suggestion in about this problem. I need answers of either of the following questions:

1. How can I disable ffmpeg's automatic stream information detection and specify my own format (mjpeg), or
2. How can I use openRTSP in OpenCV?

Regards,

Solution

Is this an Axis IP camera? Either way, most IP cameras that provide MPEG4 RTSP stream that can be decoded using OpenCV using cvCreateFileCapture_FFMPEG. However, ffmpeg decoder's MJPEG codec has a widely known unresolved issues. I am sure you would have received an error similar to

[ingenient @ 0x97d20c0]Could not find codec parameters (Video: mjpeg)

Option1 : Using opencv, libcurl and libjpeg

To view mjpeg stream in opencv take a look at the following implementation

http://www.eecs.ucf.edu/~rpatrick/code/onelinksys.c or http://cse.unl.edu/~rpatrick/code/onelinksys.c

Option2: Using gstreamer (no opencv)

I would recommend looking at gstreamer if your goal is to just view or save jpeg images

To view MJPEG stream one may execute media pipeline string as follows

gst-launch -v souphttpsrc location="http://[ip]:[port]/[dir]/xxx.cgi" do-timestamp=true is_live=true ! multipartdemux ! jpegdec ! ffmpegcolorspace ! autovideosink

For RTSP

gst-launch -v rtspsrc location="rtsp://[user]:[pass]@[ip]:[port]/[dir]/xxx.amp" debug=1 ! rtpmp4vdepay ! mpeg4videoparse ! ffdec_mpeg4 ! ffmpegcolorspace! autovideosink

To work with C API see

http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Using_Multimedia_Components/Camera_API_Usage

For a simple example take a look at my other post on rtsp for constructing gstreamer C API media pipeline (This is same as gst-launch string but rather implemented as a C API)

Playing RTSP with python-gstreamer

To save MJPEG stream as multiple images the pipeline (Let us put a vertical flip BIN and connect the PADS to the previous and the next BINS to make it fancier)

gst-launch souphttpsrc location="http://[ip]:[port]/[dir]/xxx.cgi" do-timestamp=true is_live=true ! multipartdemux ! jpegdec !  videoflip method=vertical-flip ! jpegenc !  multifilesink location=image-out-%05d.jpg

Also maybe worthwhile have a look at gst-opencv

UPDATE:

Option3: Using gstreamer, Named Pipe and opencv

On Linux one may get mjpeg stream and convert it to mpeg4 and feed it to a named pipe. Then read the data from the named pipe in opencv

Step 1. Create Named Pipe

mkfifo stream_fifo

Step 2. Create opencvvideo_test.c

// compile with gcc -ggdb `pkg-config --cflags --libs opencv` opencvvideo_test.c -o opencvvideo_test
#include <stdio.h>
#include "highgui.h"
#include "cv.h"


int main( int argc, char** argv){

IplImage  *frame;
    int       key;

    /* supply the AVI file to play */
    assert( argc == 2 );

    /* load the AVI file */
    CvCapture *capture = cvCreateFileCapture(argv[1]) ;//cvCaptureFromAVI( argv[1] );

    /* always check */
    if( !capture ) return 1;    

    /* get fps, needed to set the delay */
    int fps = ( int )cvGetCaptureProperty( capture, CV_CAP_PROP_FPS );

    int frameH    = (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT);
    int frameW    = (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH);

    /* display video */
    cvNamedWindow( "video", CV_WINDOW_AUTOSIZE );

    while( key != 'q' ) {

    double t1=(double)cvGetTickCount();
    /* get a frame */
    frame = cvQueryFrame( capture );
    double t2=(double)cvGetTickCount();
    printf("time: %gms  fps: %.2g\n",(t2-t1)/(cvGetTickFrequency()*1000.), 1000./((t2-t1)/(cvGetTickFrequency()*1000.)));

    /* always check */
    if( !frame ) break;

    /* display frame */
    cvShowImage( "video", frame );

    /* quit if user press 'q' */
    key = cvWaitKey( 1000 / fps );
    }

    /* free memory */
    cvReleaseCapture( &capture );
    cvDestroyWindow( "video" );

    return 0;
}

Step 3. Prepare To Convert From MJPEG to MPEG4 using gstreamer (rate of incoming frames critical)

gst-launch -v souphttpsrc location="http://<ip>/cgi_bin/<mjpeg>.cgi" do-timestamp=true is_live=true ! multipartdemux ! jpegdec ! queue ! videoscale ! 'video/x-raw-yuv, width=640, height=480'! queue ! videorate ! 'video/x-raw-yuv,framerate=30/1' ! queue ! ffmpegcolorspace ! 'video/x-raw-yuv,format=(fourcc)I420' ! ffenc_mpeg4 ! queue ! filesink location=stream_fifo

Step 4. Display Stream in OpenCV

  ./opencvvideo_test stream_fifo

Comments

[praxmon]

Can I use gsteamer to create a video from the images that are being saved? And does gstreamer give me access to EXIF tags?

[enthusiasticgeek]

@PrakharMohanSrivastava Use multifilesrc plugin to create video from images gstreamer.freedesktop.org/data/doc/gstreamer/head/…

[enthusiasticgeek]

@PrakharMohanSrivastava read more on EXIF gstreamer.freedesktop.org/data/doc/gstreamer/head/…

[enthusiasticgeek]

@tod I think the person took down the link. Please see cse.unl.edu/~rpatrick/code/onelinksys.c