Projection - Onto a Line

I will talk about "Projection" in this section. Here goes an intuitive situation describing the concept of projection. A projection splits a vector into two parts: the part that lies along a line, and the part that sticks out at a right angle to it. Both parts are useful. We'll start from the shadow picture, get the formula from the inner product, and then look at what each of the two parts is used for.

What is the projection of a vector onto a line ?

Let's build the idea from a picture before any formula. A vector v starts on a line l and points away from it. The projection is the piece of the line that v covers when you look straight down onto the line.

Let's look at the following illustration. Here I have an vector labeled as v (Blue arrow) and the starting point of it is attached to a line labeled as l. Now we have a sun (light) shining right above the vector. Then you would have a shadow of the vector v on the line. The shadow would like like the vector labeled as u(Black arrow). This shadow vector is called a 'Projection' of the vector v.

Vector v on line l with its shadow u cast by the sun

Figure 1. The projection as a shadow. The black vector u is the shadow of the blue vector v on the line l.

One detail of the picture matters. The light must fall at a right angle to the line, not straight down from the top of the page. The thin line from the tip of v to the tip of u shows this direction, and it meets l at a right angle. With light at any other angle, the shadow is an oblique projection, which is a different operation.

The projection lies on the line : u has the direction of l, or the opposite direction when v points backward.
The projection is never longer than v : |u| = |v| |cos θ|, where θ is the angle between v and l.
A vector at right angles to the line has no shadow : cos 90 deg = 0, so the projection is the zero vector.

How is the projection calculated ?

The picture tells us what u is, but not how to compute it. The trick is to describe the line by any vector y that lies on it. The projection is then just y stretched by the right amount, and the inner product gives that amount.

Now let's think about how we get this shadow vector (projection) mathematically. The method is as described below.

Helper vector y drawn on line l from the start of v

Figure 2. A helper vector y on the line. The projection u is y stretched by some scalar, and the task is to find that scalar.

The conclusion (final equation) is as follows.

proj L of v equals v dot y over y dot y times y

I would not prove this equation itself. You can refer to Linear Algebra text book if you are interested in how this equation are drawn. But it would be worth understanding each component of this equation as show below.

$The fraction v dot y over y dot y is a scalar and y remains a vector$

Figure 3. The parts of the projection formula. The fraction is a scalar that says how much to stretch y, and y gives the direction.

The derivation is short, and it uses only the right angle from Figure 1. Write the projection as u = cy for an unknown scalar c. The remainder v - cy must be perpendicular to y, so its inner product with y is zero. That gives v ⋅ y - c (y ⋅ y) = 0, and so c = (v ⋅ y) / (y ⋅ y).

Let's try it with numbers. Take the line through the origin in the direction y = [2 1], and the vector v = [3 4].

v . y = 3*2 + 4*1 = 10
y . y = 2*2 + 1*1 = 5
c     = 10 / 5    = 2

u = proj(v) = 2 * [2 1] = [4 2]
w = v - u   = [3 4] - [4 2] = [-1 2]

check : w . y = -1*2 + 2*1 = 0

Any vector on the line works as y : with 3y = [6 3], c becomes 30/45 = 2/3, and c times [6 3] is again [4 2].
A unit vector makes the formula shorter : when |y| = 1, y ⋅ y = 1 and the projection is (v ⋅ y)y.
The formula is a matrix : P = yy^T / (y^Ty) projects every vector onto the line. For y = [2 1], P = [4 2; 2 1] / 5, and Pv = [4 2].
Projecting twice changes nothing : P² = P, because a vector already on the line is its own shadow.

What is the projection used for ?

A projection answers practical questions in two ways. Sometimes you want the shadow u itself. More often you want what is left over, the vector w that joins the line to the tip of v at a right angle.

Then... Let's think about why we need this kind of operation. You can utilize this one to get the shadow of an object on computer graphics.. and more common application in a variety of application it is used to get a vector which connecting a point to a line in the shortest path. Just taking the magnitue of vector w in the following illustration gives you the distance (shortest path) between a point and a line.

Vector w from the tip of u to the tip of v, perpendicular to line l

Figure 4. The perpendicular part w = v - u. Its length is the shortest distance from the tip of v to the line.

In the numeric example, w = [-1 2]. So the point (3, 4) lies sqrt(1 + 4) = sqrt(5) = 2.236 away from the line through the origin in the direction [2 1]. No other point on the line is closer. The reason is the right angle: for any other point on the line, the distance is the hypotenuse of a right triangle with w as one side.

The same idea scales up. In least squares, the best-fit solution is the projection of the data vector onto the space of possible model outputs, and the error vector is the w of that projection. In Gram-Schmidt orthogonalization, each new vector has its projections onto the earlier vectors removed, which leaves only its w part.

|w| is the point to line distance : it is the shortest path because w meets the line at a right angle.
w is found by one subtraction : once u is known, w = v - u.
Least squares is a projection : the fitting error is orthogonal to the model space, exactly as w is orthogonal to l.

Summary

From derivation of Projection vector onto a line as explained above, we can figure out two important vectors as illustrated below. Both of these two vectors are widely applied in many cases. The vector u would be widely used in geometric transformation and the vector w is used in matrix orthogonalization and linear regression.

v split into u along line l and w orthogonal to l

Figure 5. The two results of a projection. v = u + w, with u on the line and w orthogonal to it.

u = ((v ⋅ y) / (y ⋅ y)) y : the projection of v onto the line in the direction y.
w = v - u : the part of v orthogonal to the line, with w ⋅ y = 0.
v = u + w always holds : and because u and w are orthogonal, |v|² = |u|² + |w|². In the example, 25 = 20 + 5.

Two extensions come up often. First, the formula assumes that the line passes through the start of v. For a line through a point p that is not the origin, subtract p first, project, and add p back. Second, the same idea works for a plane or any subspace. Put vectors that span the subspace into the columns of a matrix A. The projection matrix is then P = A(A^TA)^-1A^T, and for a single column y it reduces to the yy^T / (y^Ty) of the previous section.

A = | 1  0 |      v = [1 2 6]^T
    | 0  1 |
    | 1  1 |

P v = [2 3 5]^T         (the part of v in the plane spanned by the columns of A)
w   = v - P v = [-1 -1 1]^T

check : A^T w = [0 0]^T  (w is orthogonal to both columns)

This is the same split as in Figure 5, one dimension higher. The plane takes the role of the line, and w is again the shortest path from the tip of v to it.