Calculating 'up vector' from transformation matrix in 3D

Question

I just came to strange problem with my project in 3D. Everyone knows algorythm of calculating LookAt vector, but it is not so easly to calculate "up" vector from transformation matrix (or at least maybe I simple missed something).

The problem is following:

"Up" vector is (0, 1, 0) for identity rotation matrix and rotate with matrix, but do not scale nor translate. If you have simple rotation matrix procedure is easy (multiply vector and matrix). BUT if matrix contains also translation and rotation (e.g. it was produced by multiplying several other matrices), this won't work, as vector would be translated and scaled.

My question is how to get this "up" vector from single transformation matrix, presuming vector (0, 1, 0) correspond to identity rotation matrix.

Solution 1

Translation actually does affect it. Let's say in the example the transformation matrix didn't do any scaling or rotation, but did translate it 2 units in the Z direction. Then when you transform (0,1,0) you get (0,1,2), and then normalizing it gives (0,1/sqrt(5), 2/sqrt(5)).

What you want to do is take the difference between the transformation of (0,1,0) and the transformation of (0,0,0), and then normalize the resulting vector. In the above example you would take (0,1,2) minus (0,0,2) (0,0,2 being the transformation of the zero vector) to get (0,1,0) as desired.

Solution 2

Apply your matrix to both endpoints of the up vector -- (0, 0, 0) and (0, 1, 0). Calculate the vector between those two points, and then scale it to get a unit vector. That should take care of the translation concern.

Solution 3

Simply multiply the up vector (0,1,0) with the transformation, and normalize. You'll get the new calculated up vector that way.

Comments

[Richard Dunlap]

Except that this solution doesn't work if the transformation has translation in it. See Alex319's concrete example.

[Janie]

He mentioned rotation, but no translation. He'd obviously have to calculate the inverse of the translation matrix. Since the translation matrix commutes with the concatenated rotations, the inverse can be applied at any time.

[Janie]

(ie. speaking about inverse matrices)

[SteamyThePunk]

I apologize, I have only now gone back over this answer to discover my mistake. The basis vectors orientation (column, or row) depend on how you represent the up vector as either a vector, or the matrix dual of the vector. Essentially if the values are encoded as a row or column. In some cases the basises are on the rows.... So it's better to simply multiply unit vectors by the matrix to acquire the basises relevant to the vector and matrix multiplicative relationships.