Returns the coordinate transform matrix for the frame determined by the points org, xP, yP (the last one or two may be NIL).
The resulting matrix satisfies m*[0,0,1]^T= org, m*[1,0,1]^T= hor, m*[1,0,1]^T= ver.
The matrix m is optional, and used to store the result if given (saves alloc.)