Many engineering formulas hide the same small calculation. You multiply two lists of numbers element by element and add up the products. That calculation has a name, the inner product, and it appears in geometry, statistics, signal processing and linear algebra alike.

Inner Product is a mathematical operation for two data set (basically two vector or data set) that performs following

i) multiply two data set element-by-element

ii) sum all the numbers obtained at step i)

This may be one of the most frequently used operation in mathematics (especially in engineering math). Sometimes it is used because the result indicates a specific mathemaatical or physical meaning and sometimes it is used just because it simplifies the mathematical representation. Anyway, it is used a LOT. Basically Innerproduct is fundamental operation for what I explains in Sum of Times (Sum of Multiplication) page.  

• Geometric Meaning of Inner Product
• Can I do Inner Product for any two vectors ?
• Applied meaning of Vector Inner Product
• Examples
• On line Calculators

Geometric Meaning of Inner Product

The geomatrc meaning of Inner Product is as follows. Inner Product is a kind of operation which gives you the idea of angle between the two vectors. Actually the most important application of inner product are

• it gives the angle between the two vectors
• it gives the degree of the correlation between two vectors

For more detailed descriptions of the application, see the last section (Applied meaning of Vector Inner Product) and read the whole pages if you are interested in how the vector inner product can indicate this kind of information.

There are several different ways of representing/calculating the inner product. Equation (1) gives you the geometric meaning of inner product. Equation (2) would not shows you any idea of visualization, but it gives you a way of calculating the inner product with very simple multiplication and sums(Equation (2) would be the most common ways to calculate the inner product in most of the application). Equation (3) is the form representing the inner product as a multiplication of two vectors.

Let's check that Equation (1) and Equation (2) agree with a small example. Take u = (3, 1) and v = (1, 2). Equation (2) gives u ⋅ v = 3 x 1 + 1 x 2 = 5. The lengths are |u| = √10 and |v| = √5, so Equation (1) gives cos(θ) = 5/√50 = 0.707, and θ = 45 deg. The two equations agree for any pair of vectors, because the law of cosines turns one into the other.

The sign of the result carries the direction. The inner product is positive when θ is below 90 deg, zero at 90 deg and negative above 90 deg. Equation (3) writes the same sum as a matrix product of the row vector u and the column vector v^T. The picture asks why the transpose is needed, and the next section answers that question.

• Equation (1) gives the meaning : the inner product is the product of the two lengths, scaled by the cosine of the angle between the vectors.
• Equation (2) gives the calculation : multiply matching elements and add them. This works in any number of dimensions.
• The sign tells you the direction : positive for an acute angle, zero for a right angle and negative for an obtuse angle.

Can I do Inner Product for any two vectors ?

What I explained above is a kind of geometric meaning (graphical meaning) of Inner product. This kind of application can be used in 2D (two element vector) and 3D (three element vector) which can be easily visualized, but it would be difficult to interpret the inner product in this way if the number of the element grow larger. In most of the applications other than computer graphics, the inner product of vectors is done in algebric procedure as shown below.

As a simple example, let's think of a case of taking the inner product of two vectors which is made up of 3 elements as shown below.

In the definition above, the last element of v₂ is printed as a₃₃. It should be a₂₃, as every later picture on this page shows.

Before I ask you to take the inner product of two vectors, I would like to talk first about how to make a proper question of inner product.

First if I ask you to perform the following operation (inner product of two vectors in the following format), does it make sense to you ? Is this a valid question ?

The answer is NO. The question itself is not valid. In any matrix multiplication, there is an important rule as shown below (Vector is also a kind of matrix, so vector inner product should follow this rule as well. 'm x n', 'a x b', 'm x b' represents the dimension of a vector or matrix. If you are not familiar with the concept of dimension, refer to vector/matrix dimension page)

Now let's check if the problem I gave you meets this requirement. As shown below, the problem that I gave you does not meet the requirement, so the problem itself is invalid.

Keep in mind what the NO applies to. It applies to the matrix form, where the dot means matrix multiplication. As a dot product, the sum of element by element products of two row vectors of the same length is still well defined. The dimension rule only decides how you must write that sum as a matrix product.

What if I change the problem as below ? (In this problem, I transposed the v2). Does this make sense ?

Yes, it make sense. It is valid problem since it meets the matrix multiplication rule as shown below. One think you have to notice is that the result of this form of inner product is 1 x 1 vector. 1 x 1 vector (matrix) is same as scalar.

Now let's look how this inner product is calculated. The calculation is as simple as follows. You may have a very long calculation if the size of the vector is large. But it is just long, but it is very simple calculation. Just multiply each elements of two vectors and sum them all.

Now let's think of another example of vector porduct. This may look almost same as the previous example, but it is not exactly same. The difference is that v1 is transposed as opposed to v2 in previous example.

Before you start calculation, let's check if this problem is meaning (valid) question or not. It is valid question because it meets the matrix multiplication rule as shown below. One important thing you need to notice in this case is that the result in this case is 3 x 3 matrix, it is not a scalar value. This product is called the outer product, not the inner product.

The result of calculation is as follows. I recommend you to take some time to take a close look at this.

Compare the two results with numbers. Take v₁ = [1 2 3] and v₂ = [4 5 6]. The inner product v₁v₂^T is 4 + 10 + 18 = 32. The outer product v₁^Tv₂ is the 3 x 3 matrix [4 5 6; 8 10 12; 12 15 18]. Its diagonal holds the three products a₁₁a₂₁, a₁₂a₂₂ and a₁₃a₂₃, so the sum of the diagonal, the trace, is again 32. Every row of the outer product is a multiple of v₂, so the matrix has rank 1.

Now let's try to increase the size of a vector and see if you can figure out the result from the simple example as explained above.

Let's suppose you are give two vectors as shown below.

And you are asked to take the inner product of the two vectors as shown below.

First, check if this problem itself is valid or not. (I will leave this to you. Try on your own if this problem is valid or not). Just to give you the conclusion. the answer is 'Yes, the problem is valid'.

What is the result of the inner product in this case ? It is going to be as follows.

If the size of vector gets very large, it would be too tedious to write the result as above. If you take a little bit closer examination of the expression, you would notice that you can simplify the expressing using 'Sum (Sigma)' notation as shown below.

One more rule applies when the elements are complex numbers, as they are for baseband signals and channel coefficients. The inner product then conjugates one of the two vectors: u ⋅ v = Σ u_iv_i^*, which is uv^H in matrix form. The conjugate keeps u ⋅ u real and equal to |u|². For example, u = [1+j 2] gives uu^H = 2 + 4 = 6, but uu^T = 2j + 4, which is not a length.

• Row times column gives the inner product : a 1 x n row vector times an n x 1 column vector is a 1 x 1 result, a scalar.
• Column times row gives the outer product : an n x 1 column vector times a 1 x n row vector is an n x n matrix of rank 1.
• The two products are linked by the trace : the trace of the outer product equals the inner product.
• Complex vectors need a conjugate : use v^H instead of v^T, so that the inner product of a vector with itself is its squared length.

Applied meaning of Vector Inner Product

The formulas above tell you how to compute an inner product. In practice you compute it because the number means something, so let's look at what the value tells you about the two vectors.

When you see the case of vector inner product in real application, it is very important of the practical meaning of the vector inner product.

I see two major application of the inner product. One is to figure out the angle between the two vectors as illustrated above. (First, you calculate the inner product using Equation (2) and with the result and equation (1), you can figure out the angle).

This angle in turn has additional implication.  If this angle is 90 degree(0.5 pi), it means

• the result of the inner product becomes 0
• the correlation between the two vector is 0.
• the two vectors are independent to each other.

If this angle is 0 degree, it means

• the result of the inner product becomes max (multiplication of length of two vectors)
• the correlation between the two vector is 1.
• the two vectors are totally dependent to each other.

If this angle is between 0 and 90, the value is between min(0) and max value and they are partially dependent to each other.

If this angle is between 90 and 180 degrees, the inner product is negative. At 180 degrees the vectors point in opposite directions, the inner product is -|u||v|, and the normalized correlation is -1. The normalized correlation is cos(θ) = u ⋅ v / (|u||v|). For data vectors with the mean removed, this is the Pearson correlation coefficient.

The words dependent and independent in the lists above describe correlation. In linear algebra, two nonzero vectors are linearly dependent only when they are parallel, at 0 or 180 degrees. Any other angle, including 90 degrees, gives two linearly independent vectors.

Second application is nothing to do with geometric meaning, but it is very handy form of mathematical representation of "Sum of Multiplication/Sume of Times" as shown in "Sum of Times" page.

One important thing you have to remember is that the result of inner product of two vectors is a scalar.

• The inner product measures alignment : its normalized value cos(θ) runs from 1 for the same direction, through 0 for orthogonal vectors, to -1 for opposite directions.
• Zero means orthogonal : an inner product of 0 means that the two vectors share no component along each other.
• The result is always a scalar : however long the vectors are, the inner product reduces them to one number.

Examples

The four examples below move from vectors to functions. The first one computes the angle between two 2D vectors. The other three take the inner product of sampled functions, which is the building block of the Fourier transform and of many detection methods.

Example 1

This examples shows how the angle between two vectors can be calculated by Inner Product. Click on the figure or click here for step by step process.

The red vector (√2/2, √2/2) and the green vector (1, 0) both have length 1. So the inner product is directly cos(45 deg) = √2/2 = 0.707. The two calculations on the right show that the order of the vectors does not matter. Both orders give the same number.

Example 2

This examples shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process.

The red curve is sin(θ). The blue curve is sin(θ - φ) with φ = π/2, which is the same as -cos(θ). Multiply the nine samples point by point and add them, and the sum is 0. The integral over one period gives the same answer. In general, the integral of sin(θ) sin(θ - φ) from 0 to 2π is π cos(φ). So this inner product measures the phase difference between two sinusoids of the same frequency.

Example 3

This examples shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process. This type of operation is fundamental process in Discrete Fourier Transform.

Here the blue curve is sin(2θ), so n = 2. The sampled products again add up to 0. The integral of sin(θ) sin(nθ) over one period is 0 for every integer n other than 1, and it is π for n = 1. So sinusoids of different integer frequencies are orthogonal. This is what lets the DFT pick one frequency out of a mixture.

Example 4

This is another that shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process.

Both curves are sinc functions sampled at integer points. The red one is centred at 3 and the blue one at 4. Each one is 1 at its own centre and 0 at every other integer, so the sampled inner product is 0. The continuous inner product is also 0. The integral of sinc(t) sinc(t - k) over all t equals sinc(k), which is 0 for every nonzero integer k. Because of this orthogonality, shifted sinc pulses can carry independent samples, as in the sampling theorem.

• Functions have inner products too : replace the sum over elements by an integral, or by a sum over samples.
• Orthogonal functions do not interfere : sinusoids of different frequencies and sinc pulses at different integer shifts all give an inner product of 0.

On line Calculators

You can find on-line calculator for your practice. it would be good for you to do calculation with pen and paper first and then check out the answer with the calculator.

• Dot Product Calculator - CALCULATOR Online

Matlab and Octave also work as a calculator here. The function dot(a,b) returns the inner product of two vectors. For row vectors, a*b' gives the same number, and a'*b gives the outer product. Note that b' is the conjugate transpose, so a*b' already applies the conjugate rule for complex vectors. A quick check is a = [1 2 3] and b = [4 5 6], which gives 32.