InitUndistortRectifyMap and Remap

Question

I am currently writing an openCV program for a pair of stereo cameras. Camera calibration as well as Stereo Calibration are done.

The next step is to find a feature's position in space from the 2 images I get. That's why I have to Stereo Rectify the images and make my calculations afterwards.

The problem I am facing with initUndistortRectifyMap is the following:

-If I pass R1 or R2 calculated by stereoRectify() to initUndistortRectifyMap() I get black images after remapping.

-If I pass r (an empty matrix) to initUndistortRectifyMap() I get unrectified images after remapping. The images I get are a little distorted though.

I need to pass R1, and R2, to initUndistortRectifyMap() for rectifying the 2 cameras, otherwise when passing an empty matrix the stereo heads won't be rotated into the same plane.

Following is my code:

stereoRectify(intrinsic[0], distCoeffs[0], intrinsic[1], distCoeffs[1], imageSize, 
    R, T_Stereo, R1, R2, newP1, newP2, Q, CV_CALIB_ZERO_DISPARITY, -1, imageSize);

if (x_Cam.GetSerial()=="4002678487")
{
    initUndistortRectifyMap(intrinsic[0], distCoeffs[0], R1, newP1, imageSize,        
        CV_16SC2 , mapx1,  mapy1);
    remap(x_Image, imageRectified[0],mapx1, mapy1, INTER_LINEAR);

    return imageRectified[0];
}   

if (x_Cam.GetSerial()=="4002702131")
{
    //flip(in, in, -1);
    initUndistortRectifyMap(intrinsic[1], distCoeffs[1], R2, newP2, imageSize, 
        CV_16SC2 , mapx2,  mapy2);
    remap(x_Image, imageRectified[1],mapx2, mapy2, INTER_LINEAR, BORDER_CONSTANT, 0);

    return imageRectified[1];
}

I checked all the matrix values going in to stereoRectify() and they are correct. The rotation matrices R1, and R2 seem to be correct as well. I am just getting black images as output.

I tried passing rubbish values into InitUndistortRectifyMap() for R1 and R2 (R1*R2 for example) to see the effect simply, and I did get weird results but not black images.

Solution 1

Well, I kind of went around the problem and thought I'd share the solution incase anyone can use it.

InitUndistortRectifyMap did output blank images when using the Rotation Matrices R1, and R2 produced by StereoRectify.

Therefore, I tried to use StereoRectifyUncalibrated, which produces 2 Homography matrices H1, and H2, and the I calculated the rotations according to OpenCV documentation:

R1 = inv(CamMatrix1)*H1*CamMatrix1 R2 = inv(CamMatrix2)*H2*CamMatrix2

I passed the new R1, and R2 to InitUndistortRectifyMap and remapped and the results are satisfying.

Solution 2

I'm going to dig up this old post because I've put recently some effort in research about this problem. Namely, I've faced in the same issue - after proper stereo calibration and rectification (all opencv provided stereo vision machinery), finally 'warped' images were black. Let me explain why it that so.

First of all, how remap(src,dst,mapX, mapY) works. If you go to documentation, you can see that

And if one of the values mapX(x,y) or mapY(x,y) is outside of src, dst(x,y) = 0 (black!). That means, all pairs (x,y) have unvalid values in mapX or mapY. For example, in my case, from reasons I can't explain I have all values in mapY negative.

But the reason for this is this only occurs in particular stero vision configurations. I made several examples with cameras aligned in such way that rotation angles between them were small (up to several degrees). In that case, opencv procedures work well. The 'black' output issues I had when two euler angles were the same and one angle (rotation around Y) was 17 degrees. In that case, I made some experiment - I translated mapX and mapY in following way:

for(int i=0;i<_imageSize.width;i++)
        for(int j=0;j<_imageSize.height;j++) {
            _mapXA.at<float>(j, i) -= 1200;
            _mapYA.at<float>(j, i) -= 1200;
            _mapXB.at<float>(j, i) += 2500;
            _mapYB.at<float>(j, i) += 2500;
        }

(beacuse I saw _mapXA was too large and _mapYB had negative values around -1500). Suprisingly, the output after remap() was not black. The image looked exactly like rotated around some angle (17 might be, I did not check exactly) but it wasn't rectified at all.

It might be possible that opencv cannot handle non-parallel cameras in stereo vision well. I mean, it works for some examples and for some does not - but it means this method is not realiable. Also, you can find such information Gary Bradski and Adrian Kaehler Learning OpenCV :

Once again, this is one reason why you will tend to get better results if you arrange your cameras to be as nearly frontal parallel as possible (at least until you become expert at stereo vision)

Comments

[Eric Leschinski]

I assume your question would be: "Why does initUndistortRectifyMap() return black images and not just weird results?" Next time try to make explicit your specific question.

[user1901213]

The single cameras are calibrated as well as both cameras together with StereoCalibrate(). I have checked the R and T matrices out of StereoCalibratoin and they are correct.