OpenCV: process every frame

Question

I want to write a cross-platform application using OpenCV for video capture. In all the examples, i've found frames from the camera are processed using the grab function and waiting for a while. And i want to process every frame in a sequence. I want to define my own callback function, which will be executed every time, when a new frame is ready to be processed (like in directshow for Windows, when you defining and putting into the graph your own filter for such purposes).

So the question is: how can i do this?

Solution 1

According to the code below, all callbacks would have to follow this definition:

IplImage* custom_callback(IplImage* frame);

This signature means the callback is going to be executed on each frame retrieved by the system. On my example, make_it_gray() allocates a new image to save the result of the grayscale conversion and returns it. This means you must free this frame later on your code. I added comments on the code about it.

Note that if your callback demands a lot of processing, the system might skip a few frames from the camera. Consider the suggestions Paul R and diverscuba23 did.

#include <stdio.h>
#include "cv.h"
#include "highgui.h"


typedef IplImage* (*callback_prototype)(IplImage*);


/* 
 * make_it_gray: our custom callback to convert a colored frame to its grayscale version.
 * Remember that you must deallocate the returned IplImage* yourself after calling this function.
 */
IplImage* make_it_gray(IplImage* frame)
{
    // Allocate space for a new image
    IplImage* gray_frame = 0;
    gray_frame = cvCreateImage(cvSize(frame->width, frame->height), frame->depth, 1);
    if (!gray_frame)
    {
      fprintf(stderr, "!!! cvCreateImage failed!\n" );
      return NULL;
    }

    cvCvtColor(frame, gray_frame, CV_RGB2GRAY);
    return gray_frame; 
}

/*
 * process_video: retrieves frames from camera and executes a callback to do individual frame processing.
 * Keep in mind that if your callback takes too much time to execute, you might loose a few frames from 
 * the camera.
 */
void process_video(callback_prototype custom_cb)
{           
    // Initialize camera
    CvCapture *capture = 0;
    capture = cvCaptureFromCAM(-1);
    if (!capture) 
    {
      fprintf(stderr, "!!! Cannot open initialize webcam!\n" );
      return;
    }

    // Create a window for the video 
    cvNamedWindow("result", CV_WINDOW_AUTOSIZE);

    IplImage* frame = 0;
    char key = 0;
    while (key != 27) // ESC
    {    
      frame = cvQueryFrame(capture);
      if(!frame) 
      {
          fprintf( stderr, "!!! cvQueryFrame failed!\n" );
          break;
      }

      // Execute callback on each frame
      IplImage* processed_frame = (*custom_cb)(frame);

      // Display processed frame
      cvShowImage("result", processed_frame);

      // Release resources
      cvReleaseImage(&processed_frame);

      // Exit when user press ESC
      key = cvWaitKey(10);
    }

    // Free memory
    cvDestroyWindow("result");
    cvReleaseCapture(&capture);
}

int main( int argc, char **argv )
{
    process_video(make_it_gray);

    return 0;
}

EDIT:

I changed the code above so it prints the current framerate and performs a manual grayscale conversion. They are small tweaks on the code and I did it for education purposes so one knows how to perform operations at pixel level.

#include <stdio.h>
#include <time.h>

#include "cv.h"
#include "highgui.h"


typedef IplImage* (*callback_prototype)(IplImage*);


/* 
 * make_it_gray: our custom callback to convert a colored frame to its grayscale version.
 * Remember that you must deallocate the returned IplImage* yourself after calling this function.
 */
IplImage* make_it_gray(IplImage* frame)
{
    // New IplImage* to store the processed image
    IplImage* gray_frame = 0; 

    // Manual grayscale conversion: ugly, but shows how to access each channel of the pixels individually
    gray_frame = cvCreateImage(cvSize(frame->width, frame->height), frame->depth, frame->nChannels);
    if (!gray_frame)
    {
      fprintf(stderr, "!!! cvCreateImage failed!\n" );
      return NULL;
    }

    for (int i = 0; i < frame->width * frame->height * frame->nChannels; i += frame->nChannels)
    {
        gray_frame->imageData[i] = (frame->imageData[i] + frame->imageData[i+1] + frame->imageData[i+2])/3;   //B
        gray_frame->imageData[i+1] = (frame->imageData[i] + frame->imageData[i+1] + frame->imageData[i+2])/3; //G
        gray_frame->imageData[i+2] = (frame->imageData[i] + frame->imageData[i+1] + frame->imageData[i+2])/3; //R
    }

    return gray_frame; 
}

/*
 * process_video: retrieves frames from camera and executes a callback to do individual frame processing.
 * Keep in mind that if your callback takes too much time to execute, you might loose a few frames from 
 * the camera.
 */
void process_video(callback_prototype custom_cb)
{           
    // Initialize camera
    CvCapture *capture = 0;
    capture = cvCaptureFromCAM(-1);
    if (!capture) 
    {
      fprintf(stderr, "!!! Cannot open initialize webcam!\n" );
      return;
    }

    // Create a window for the video 
    cvNamedWindow("result", CV_WINDOW_AUTOSIZE);    

    double elapsed = 0;
    int last_time = 0;
    int num_frames = 0;

    IplImage* frame = 0;
    char key = 0;
    while (key != 27) // ESC
    {    
      frame = cvQueryFrame(capture);
      if(!frame) 
      {
          fprintf( stderr, "!!! cvQueryFrame failed!\n" );
          break;
      }

      // Calculating framerate
      num_frames++;
      elapsed = clock() - last_time;
      int fps = 0;
      if (elapsed > 1)
      {
          fps = floor(num_frames / (float)(1 + (float)elapsed / (float)CLOCKS_PER_SEC));
          num_frames = 0;
          last_time = clock() + 1 * CLOCKS_PER_SEC;
          printf("FPS: %d\n", fps);
      }

      // Execute callback on each frame
      IplImage* processed_frame = (*custom_cb)(frame);  

      // Display processed frame
      cvShowImage("result", processed_frame);

      // Release resources
      cvReleaseImage(&processed_frame);

      // Exit when user press ESC
      key = cvWaitKey(10);
    }

    // Free memory
    cvDestroyWindow("result");
    cvReleaseCapture(&capture);
}

int main( int argc, char **argv )
{
    process_video(make_it_gray);

    return 0;
}

Solution 2

Quick thoughts would be to have 2 threads, the first thread is responsible for grabbing the frames and notifiy the second thread when they are available (places them in a processing queue), the second thread does all your processing in an event loop type manner.

See boost::thread and boost::signals2 as those two together should provide most of the framework (except for the queue) for what I described above.

Comments

[Paul R]

Note however that if your processing is significantly slower than real time then you'll rapidly start eating up available memory and/or paging.

[Andrew]

My processing is real time and i want to process the frame exactly when it's available. In opencv 1.0 there was an ability to define such callback function. But i can't find it in 2.1 version

[Paul R]

@Andrew: if your processing can keep up with real-time capture then why not just poll for each frame ?

[Andrew]

There are two problems of such approach: 1) how to determine if frame was changed? 2) it will be not processed as fast as it could be, if it is processed immediately. And it is critical for my purposes

[Andrew]

Do i understand right, that cvQueryFrame will return the same frame if called twice without any time interval? If yes - then it's bad, because i will process the same frame twice too, and i can not have enough time to process next frame in a sequence and so the frame will be dropped. It's critical for me not to drop any frames. It is possible in directshow (i've done it). But now i need my code to work on linux. The easy way - is to use opencv. Another way is to use v4l api.

[karlphillip]

If you need to process every frame twice you don't need to have 2 callbacks to do the job. cvQueryFrame() will not return the same frame twice so you need to do all the dirty work on a single callback. Since the callback receives one frame, you can copy it and call another function to process it. Remember that you can mess around with process_video() at will. The displaying of frames doesn't need to happen inside it: your callback could do it, for instance.

[karlphillip]

@Andrew Did it helped you at the end?

[Andrew]

No, i was understanding how to make a callback. I needed to process the frame immediately when it is available (like in directshow). This program is for a robot vision. At this moment it works under windows XP (i know it's horrible) and i want to move it to linux. The robot is fast and fast video processing is required. The robot is using cross-platform code, except capturing video and working with a COM-port. There is a way to use v4l api for video capture, but it is a bit harder then to use opencv. But, thank you for help!

[karlphillip]

@Andrew New code shows the current framerate and performs a manual grayscale conversion.

[Andrew]

I'll try it, but you are still taking frames in a cycle.

[karlphillip]

@Andrew All you gotta do is store the frames in a buffer and then make another thread read from it, process and display them.

[Andrew]

the conversation becomes long... let's continue in skype: nitrogen0119882006

[Pranaysharma]

Hi Though this is a very old question but is there a better way to do the Callback? Also Is polling a better way or using a call back?