how arkit tracks position and orientation

one critical piece for arkit(and all graphics libraries) is to keep track of a camera(or objects) position in 3d space, and the angle it's facing.

this is represented with what is called "6 degrees of freedom" or 6dof:

  • forward/back
  • up/down
  • left/right
  • roll
  • pitch
  • yaw

in code, 6dof is stored in a 4x4 matrix:

play around with the different sliders above to see how values change.

each of these columns is a different vector that we use to measure the roll, pitch and yaw of the device.

lets start with the simplest one, position:

the position column is simple. it tracks the devices position in 3d space, relative to where it started(0,0,0). the reference point is the back cameras location.

now onto the vectors. lets start with the bottom vector as an example:

for the camera, the x,y,z values can be between -1 and 1, and measures the direction the bottom of the phone points.

initially, we start with a 0 for the "x" value. this measures how far to the "right" the bottom vector is pointing. y is -1 since it points downwards, and z is zero since the vector is neither forward nor backward from the origin.

when we move the different sliders, we can see the bottom edge move around all the 8 different octants.

now we have the same thing, except for 2 other vectors too:

you may be wondering why do we need all of these measurements? couldn't we find the vector the phone is facing based on two points instead of 3?

while we could technically get away with a 4x3 matrix, we will need a 4x4 matrix for transformations, and not all of these transformations can rely on just 3 values. we don't just use this matrix for the position + vector of a camera, but also the position, vector and size of a virtual object that must be rendered(more on this later).

with the latter, you cannot always recompute the third column accurately, so we need to store all 4.

graphics also have always used 4x4 matrices historically like the one above. this is the same layout that openGL made popular, except instead of the camera orientation, its the objects orientation on screen.

  • arkit will also use this format for virtual objects placed in space.

if we ever need to store thousands of poses, we generally do use a more compact form - just a quaternion(4 numbers) and position(3 numbers) rather than the full 16 number matrix.

how placement + size of a virtual object works:

say we are looking through ar camera on an iPhone and want to place an object:

we have the phones position and orientation represented by the matrix above, so to determine where the cup is, we just need to multiply.

the cup is being placed 2 meters ahead, in the direction the phone is currently facing. its basically just sliding backwards along that "screen" arrow. this can be represented by the cups offset array:

[00−21]\begin{bmatrix} 0 \\ 0 \\ -2 \\ 1 \end{bmatrix}

the -2 represents the opposite direction of the screen arrow. the 1 at the end is a simple binary for whether or not we add the phones position ().

now that we have the cups offset array, we can multiply it by the phones entire matrix to get position of the cup:

[0.560.320.760.34−0.78−0.080.620.180.26−0.940.20−0.470001][00−21]≈[−1.18−1.05−0.881]\begin{bmatrix} 0.56 & 0.32 & 0.76 & 0.34 \\ -0.78 & -0.08 & 0.62 & 0.18 \\ 0.26 & -0.94 & 0.20 & -0.47 \\ 0 & 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 0 \\ 0 \\ -2 \\ 1 \end{bmatrix} \approx \begin{bmatrix} -1.18 \\ -1.05 \\ -0.88 \\ 1 \end{bmatrix}

the product we get here represents the cups position in the room on x,y,z.

technically, the real multiplication would be done with a 4x4 matrix for the cup, that also stores its vectors and sizes:

play around with the sliders to see how the values change depending on the orientation of the cup!

this means we need to represent the matrix with all 4 vectors. since its no longer just a vector, but also length, we cannot easily derive the height value from width/depth!

depending on how we want to orient the matrix, we multiply it by the matrix of the camera.

so now we know why a camera's and an object's 6dof are represented by a 4x4 matrix!