Opencv virtually camera rotating/translating for bird's eye view

Question

I've a calibrated camera where I exactly know the intrinsic and extrinsic data. Also the height of the camera is known. Now I want to virtually rotate the camera for getting a Bird's eye view, such that I can build the Homography matrix with the three rotation angles and the translation.

I know that 2 points can be transformed from one image to another via Homography as

x=K*(R-t*n/d)K^-1 * x'

there are a few things I'd like to know now: if I want to bring back the image coordinate in ccs, I have to multiply it with K^-1, right? As Image coordinate I use (x',y',1) ?

Then I need to built a rotation matrix for rotating the ccs...but which convention should I use? And how do I know how to set up my WCS?

The next thing is the normal and the distance. Is it right just to take three points lying on the ground and compute the normal out of them? and is the distance then the camera height?

Also I'd like to know how I can change the height of the virtually looking bird view camera, such that I can say I want to see the ground plane from 3 meters height. How can I use the unit "meter" in the translation and homography Matrix?

So far for now, it would be great if someone could enlighten and help me. And please don't suggest generating the bird view with "getperspective", I ve already tried that but this way is not suitable for me.

Senna

Solution

That is the code i would advise (it's one of mine), to my mind it answers a lot of your questions, If you want the distance, i would precise that it is in the Z matrix, the (4,3) coefficient.

Hope it will help you...

Mat source=imread("Whatyouwant.jpg");
int alpha_=90., beta_=90., gamma_=90.;
int f_ = 500, dist_ = 500;

Mat destination;

string wndname1 = getFormatWindowName("Source: ");
string wndname2 = getFormatWindowName("WarpPerspective: ");
string tbarname1 = "Alpha";
string tbarname2 = "Beta";
string tbarname3 = "Gamma";
string tbarname4 = "f";
string tbarname5 = "Distance";
namedWindow(wndname1, 1);
namedWindow(wndname2, 1);
createTrackbar(tbarname1, wndname2, &alpha_, 180);
createTrackbar(tbarname2, wndname2, &beta_, 180);
createTrackbar(tbarname3, wndname2, &gamma_, 180);
createTrackbar(tbarname4, wndname2, &f_, 2000);
createTrackbar(tbarname5, wndname2, &dist_, 2000);

imshow(wndname1, source);
while(true) {
    double f, dist;
    double alpha, beta, gamma;
    alpha = ((double)alpha_ - 90.)*PI/180;
    beta = ((double)beta_ - 90.)*PI/180;
    gamma = ((double)gamma_ - 90.)*PI/180;
    f = (double) f_;
    dist = (double) dist_;

    Size taille = source.size();
    double w = (double)taille.width, h = (double)taille.height;

    // Projection 2D -> 3D matrix
    Mat A1 = (Mat_<double>(4,3) <<
        1, 0, -w/2,
        0, 1, -h/2,
        0, 0,    0,
        0, 0,    1);

    // Rotation matrices around the X,Y,Z axis
    Mat RX = (Mat_<double>(4, 4) <<
        1,          0,           0, 0,
        0, cos(alpha), -sin(alpha), 0,
        0, sin(alpha),  cos(alpha), 0,
        0,          0,           0, 1);

    Mat RY = (Mat_<double>(4, 4) <<
        cos(beta), 0, -sin(beta), 0,
                0, 1,          0, 0,
        sin(beta), 0,  cos(beta), 0,
                0, 0,          0, 1);

    Mat RZ = (Mat_<double>(4, 4) <<
        cos(gamma), -sin(gamma), 0, 0,
        sin(gamma),  cos(gamma), 0, 0,
        0,          0,           1, 0,
        0,          0,           0, 1);

    // Composed rotation matrix with (RX,RY,RZ)
    Mat R = RX * RY * RZ;

    // Translation matrix on the Z axis change dist will change the height
    Mat T = (Mat_<double>(4, 4) <<
        1, 0, 0, 0,
        0, 1, 0, 0,
        0, 0, 1, dist,
        0, 0, 0, 1);

    // Camera Intrisecs matrix 3D -> 2D
    Mat A2 = (Mat_<double>(3,4) <<
        f, 0, w/2, 0,
        0, f, h/2, 0,
        0, 0,   1, 0);

    // Final and overall transformation matrix
    Mat transfo = A2 * (T * (R * A1));

    // Apply matrix transformation
    warpPerspective(source, destination, transfo, taille, INTER_CUBIC | WARP_INVERSE_MAP);

    imshow(wndname2, destination);
    waitKey(30);
}

Comments

[Johannes]

First, thank you very much, the code was a great help for me. But now I'd like to understand it a little more: - why can't I reproject the 2d->3d projection with the inverse Camera Matrix A2^-1? - about the height: how can I now set my virtual camera to a height? until now I have a value 500 and do not really know how high this is. I know that my camera is for example 0.5 meters above the ground when the image is shot, how can I use this here for shifting Z such that I have a camera 2 meters above the ground? thx very much

[Johannes]

Well I just found out, that it also works with the homography H=k(R+nt/dist)*k^-1, but I still have the problem with the height. for normal calculating I took two image points which lie on the ground, this should be ok, right? or is it better to take n=(0,0,1)? anyway, need to find out somehow to change cameras position about 2 meters above ground

[jmartel]

I am sorry Johannes but could you be more precise in all your formulas, what is k,n,t,R if i guess it's scaling factor/point/translation vec/rotation mat, even if i would love being in your mind to help you, unfortunately i'm not ! Could you specify a lil bit more your question... Moreover if you liked the answer, you can accept it and/or upvote for it. That's the philosophy of stackoverflow i guess, thanks a lot ! Julien,

[Johannes]

so for beeing so unprecisely. So K is my camera intrinsic Matrix: (f,0,cx;0,f,cy;0,0,1) n is the normal of the plane,dist the distance to the camera center(which I assume is 0.54 meters), t is my translation vector, R the rotation matrix. According to Faugeras a Point can be mapped to a point via Homography, which is defined by: H=K*(R+t*n/dist)*K^-1

[Johannes]

So I can do a perspective mapping in order to get a bird's eye view. Now what I know is that my camera has a height of 0.54 meters when capturing the image. How is it now possible to virtually translate my camera to a height of 2 meters above the groundplane? Also in your implementation it seems, that the distance is just a factor. It doesn't say whether you shift in pixles or mm or sth else. But this should be computable somehow. Hope you'll understand me now. Thanks!

[jmartel]

The distance in dist in my example depends on the unit of f it is f which helps you to transform (world to pixel) and (pixel to world)

[Johannes]

hmm...so it could be possible to compute the birdview with a given height in meters if I have the focal length in mm? Or do you have another idea how I can get the desired birdview with the given height?

[jmartel]

the translation is modeled by T in my code. It is what should translate your camera, then it should be set to 2-0,54 (converted in the correct unit) in my formula...

[Johannes]

so which entry of the T matrix you want me to change? I have tried some things but didn't get the desired result. T(3,3) doesn't change anything at all. Anyway, thx for your input

[jmartel]

T is defined like this Mat T = (Mat_<double>(4, 4) << 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, dist, 0, 0, 0, 1); Then it's T[3,4] or T[4,3] (following your writting and code) that you should change (wich is the dist coeff) ...

[Johannes]

yeah ok, but then Ive again the problem with the unit. At the moment I have dist=focallength(442), which seems to be more or less the original height. if I change it now to 2-0.54, the image is very very small. so this needs to be scaled somehow?

[Sam]

This is one of the best answers I've found on the topic of perspective change on the whole internet. Thanks, @jmartel!

[Sam]

@jmartel do you have any references that describe in detail the transform you listed above? I've checked all the wiki articles about camera model, 3d transforms, perspective, etc, but not yet enlightened.