I just came to strange problem with my project in 3D. Everyone knows algorythm of calculating LookAt vector, but it is not so easly to calculate "up" vector from transformation matrix (or at least maybe I simple missed something).
The problem is following:
"Up" vector is (0, 1, 0) for identity rotation matrix and rotate with matrix, but do not scale nor translate. If you have simple rotation matrix procedure is easy (multiply vector and matrix). BUT if matrix contains also translation and rotation (e.g. it was produced by multiplying several other matrices), this won't work, as vector would be translated and scaled.
My question is how to get this "up" vector from single transformation matrix, presuming vector (0, 1, 0) correspond to identity rotation matrix.
Best Answer
Translation actually does affect it. Let's say in the example the transformation matrix didn't do any scaling or rotation, but did translate it 2 units in the Z direction. Then when you transform (0,1,0) you get (0,1,2), and then normalizing it gives (0,1/sqrt(5), 2/sqrt(5)).
What you want to do is take the difference between the transformation of (0,1,0) and the transformation of (0,0,0), and then normalize the resulting vector. In the above example you would take (0,1,2) minus (0,0,2) (0,0,2 being the transformation of the zero vector) to get (0,1,0) as desired.