I think one of the most common components of mathenatical equations being used in various engineering field would be as follows. (you will see specific examples later). I am pretty sure that almost all of you have seen this component in the mathematical models which you are working on now. So I thought it would be worthwhile looking into the practical meaning of this component as the first topic of my course (at least in engineering mathematics).

The formula above multiplies two sequences element by element and adds up the products. The result is one number, and the rest of this page is about what that number tells you.

Everybody would have different ways to understand a concept and you can interpret this in any way which is better for you. Personally, I am such a person who has difficulties to understand any concept that cannot be visualized. So whenever I want to learn something, the first step is to try to visualize the basic concepts. Fortunately, I think most of the concept in engineering can be visualized.

My visualization of this mathematical operation is as follows.

Figure 1. Sum of times drawn as a picture. Each pair x(n), y(n) is multiplied at its own sample, and one adder collects all the products into a single number.

• The red and blue curves are the two sequences : x(n) is the red one and y(n) is the blue one, sampled at the same points.
• Each circled x is one multiplication : it sits between the two samples that it multiplies.
• The green lines all end in one adder : the sum at the bottom is the single output of the operation.

As you see in the mathmatical presentation, it has two component x and y. Judging from the fact that x and y has a subscript (i) which implies that x, y is not a single number. They are a sequence number. In computer programming concept, these x, y are a kind of 1 dimensional arrays.

Can I understand the operation of this mathematical presentation just by looking at the illustration above without any additional explanation ? I strongly hope so.

If I present this into a C language syntax which most of you are familiar with, it would be as follows.

sum = 0;
for (i = 0; i < n; i++) {
     sum += x[i]*y[i];
}

One detail differs between the loop and the formula. The formula runs i from 0 to n, which is n + 1 terms, while the loop stops at i = n - 1, which is n terms. The idea is the same, but the index range must match the array length when you write real code. In Matlab and Octave, the same operation is sum(x.*y), or x*y' for two row vectors.

In short, this operation can be summarized as following two steps.

i) multiply (times) each components of the sequence x, y

ii) sum all of the result of the multiplication.

This is why I put the title of this page as "Sum of Times" meaning "Sum of Multiplication".

Another widely used graphical representation of "Sum of Times" is as follows. You will often see this representation in Neural Network.

Figure 2. Sum of times drawn as a neuron. The inputs x_i enter with weights w_i, and the node outputs the sum of their products.

The weights w_i in this drawing play the role of y_i in the formula. So a neuron computes the same number as the loop above, before it applies any activation function.

Now let's think about why this mathematical operation is so important. I think it is because this mathmatical form is the most essential component of the following engineering tools which are most commonly used mathemtical tools in engineering area.

• Correlation - Correlation Coefficient
• Fourier Transform
• Convolution
• Vector Inner Product
• Neural Network
• Application
• Application - Convolution - FIR - Low Pass Filter
• Application - Convolution - Image Processing

Correlation - Correlation Coefficient

Let's start with the tool where the meaning of sum of times is easiest to see. When two sequences go up and down together, most products x_iy_i are positive and the sum grows. When they move against each other, the products are negative. When they are unrelated, positive and negative products cancel.

There are many different variation of mathematical representation of Correlation Coefficient. One of examples is as follows. It would look intimidatingly complicated, but the most important part of the equation is the part marked in red rectangle. All the other parts are to make the result of the equation became between -1 and +1.

What is the meaning of the correlation coefficient ? Correlation Coefficient is a specific number that represent how closely two sequence (data set) are (related) related.

What does it mean by "related" ?

My intuitive understanding about "being related" is "how can the two graph of the data set look similar". You may ask another question saying "What do you mean by 'being similar' ?".

Let me give you my personal understanding as well for this term. 'Being Similar' can be an indicator to show "How well the graph  of the two data set can be overlapped without shifting any of the data set ?". Pay specific attention to "WITHOUT SHIFTING any of the data set". This is very important.

Now let's try visualize the meaning of the Correlation Coefficient. I want you to try to understand this intuitive meaning as hard as possible since this concept will help you understand most of other tools that uses the 'sum of times' component.

I created 6 different data sequence named as x1, x2, x3, x4, x5, x6 respectively. The graph of each data sequences are as follows.

Figure 3. The six test sequences. Sequences x2 to x5 are shifted, mirrored or scaled copies of the sine wave x1, and x6 is uniform random noise.

• x1 is sin(2πt) : about ten cycles over the 160 samples of the script below.
• x2 is the cosine and x4 a 45 deg shift : x2 = cos(2πt) is x1 shifted by π/2, and x4 = sin(2πt - π/4) is shifted by π/4.
• x3 is mirrored and x5 is half as tall : x3 = -x1, and x5 = 0.5 x1 swings only between -0.5 and 0.5.
• x6 has no period : it is 2 rand - 1, uniform noise between -1 and 1.

Now let me ask you a question.

How similar the sequence x1 and x2 are to each other ? Do you think they are very closely related  or very loosely related ? I will give you 10 seconds to think about the answer.

Now let's figure out the correlation coefficient of x1 and x2 using a matlab function as follows. As you see, the calculated coefficient of x1, x2 is 0.000. What does this mean ? It means these two sequences are not related to each other at all. Is this answer close to what you intuitively thought ? It may be not for somebody. You may think "If you shift x2 a little bit to the right or left, you would see x1 and x2 would exactly overlap each other. How come they are not correlated at all ?". You are right, but as I explained earlier, 'Correlation Coefficient' is an indicator to show "How well the graph  of the two data set can be overlapped without shifting any of the data set ?". Do you see 'without shifting any of the data' ? If you do the mathematical operation of correlation coefficient for x1 and x2 as it is, you get the value 0.000, which means 'no correlation'.

corrcoef(x1,x2) = 0.000

Now let's compare x1 with x3. What do you think about the correlation between these two data sets ? First you would notice that these two data set would never overlap (superimpose) each other. So.. Should Correlation Coefficient be 'zero' ? Let's calculate the correlation coefficient using a matlab function and I got the following result. It's '-1'. How do I interpret this ? Does this mean that there is even less correlation than 'zero correlation' ?

No. This means that if you mirror x3 along the horizontal axis, x1 and x3 overlap (superimpose) each other. This is also an example of high correlation.

corrcoef(x1,x3) = -1.000

Let's move to the next comparison. Let's compare x1 and x4. First just look at the shape of the graph of x1 and x4. Does it look identical ? No, if you look at them carefully, you would notice that x4 is a little bit of shifted version of x1. What would the correlation coefficient of these two data set be like ? The result is as follows. It shows pretty strong correlation, but not as strong as '1' (100% correlation). Compare the set (x1,x4) and (x1,x2). (x1,x2) gave us zero correlation but (x1,x4) gives us a certain degree of correlation. Both x2 and x4 are the shifted version of x1. Then why do we have different result ? It is because the amount of shift is different. In case of (x1,x2), the shift is 90 degree (pi/2), but in case of (x1,x4) the shift is less than 90 degree.

corrcoef(x1,x4) = 0.705

Now let's look into the correlation coefficient of x1 and x5. The result is as follows. Do these two graphs superimpose each other ? NO. Then how come the correlation coefficient become 1.0 (100% correlation) ? Is it because the two graph can superimpose if you change the magnitude of x5. (In this case, if you magnify x5 twice, the two graph superimpose).

corrcoef(x1,x5) = 1.000

Now let's take a look at the last case - x1,x6.  It would look like this set would be the least correlated to each other. Let's take a correlation coefficient. The result is as follows. As we expected, the correlation is very low... but it is still a little higher than the set (x1, x2). Isn't this against your intuition ?

corrcoef(x1,x6) = -0.019

We have gone through several sets of data sets to give you some intuitive understanding of the correlation coefficient. But you would have noticed some of the cases which might go against your intuition. This kind of case can be a risk of using intuition or of just relying on visualization, but I think it still be a better way than totally relying on mathematical equation and gets no clue of the real meaning of it.

Following is the Matlab/Octave script that would give you a series of plots as shown above. Try create any of the data on your own and how the graph and correlation coefficient changes.

t=0:2*pi/100:10;
Nt = size(t,2);
x1=sin(2*pi*t);
x2=cos(2*pi*t);
x3=-sin(2*pi*t);
x4=sin(2*pi*t-pi/4.0);
x5=0.5.*sin(2*pi*t);
x6=2.*rand(1,Nt)-1;
subplot(6,1,1);plot(x1);ylabel('x1');axis([0,Nt,-1.1,1.1]);
subplot(6,1,2);plot(x2);ylabel('x2');axis([0,Nt,-1.1,1.1]);
subplot(6,1,3);plot(x3);ylabel('x3');axis([0,Nt,-1.1,1.1]);
subplot(6,1,4);plot(x4);ylabel('x4');axis([0,Nt,-1.1,1.1]);
subplot(6,1,5);plot(x5);ylabel('x5');axis([0,Nt,-1.1,1.1]);
subplot(6,1,6);plot(x6);ylabel('x6');axis([0,Nt,-1.1,1.1]);
fprintf('corrcoef(x1,x1) = %1.3f\n',corrcoef(x1,x1))
fprintf('corrcoef(x1,x2) = %1.3f\n',corrcoef(x1,x2))
fprintf('corrcoef(x1,x3) = %1.3f\n',corrcoef(x1,x3))
fprintf('corrcoef(x1,x4) = %1.3f\n',corrcoef(x1,x4))
fprintf('corrcoef(x1,x5) = %1.3f\n',corrcoef(x1,x5))
fprintf('corrcoef(x1,x6) = %1.3f\n',corrcoef(x1,x6))

Why does the formula reach exactly these values ? For zero-mean sequences, the correlation coefficient reduces to r = Σx_iy_i / sqrt(Σx_i² x Σy_i²). The denominator only removes the size of the two sequences, so the sum of times in the numerator decides the answer. For two sine waves with a phase difference θ, this ratio is cos θ. So x2 gives cos 90 deg = 0, x3 gives cos 180 deg = -1, and x4 gives cos 45 deg = 0.707. The script prints 0.705 because 160 samples do not hold a whole number of cycles. Scaling does not change the ratio, so x5 gives 1 even though its amplitude is 0.5.

One warning about the script. In current Matlab and Octave, corrcoef(x1,x2) returns a 2 x 2 matrix [1 r; r 1], not a single number. The fprintf line then repeats its format four times and prints 1.000, r, r and 1.000. To print only the coefficient, take the off-diagonal entry, for example r = corrcoef(x1,x2); r(1,2).

• The sign of r shows direction : +1 means the two sequences rise and fall together, and -1 means one is the mirror image of the other.
• r ignores amplitude : x5 is half the size of x1 and still gives r = 1, because the denominator divides the size out.
• r is measured without shifting : x2 is an exact shifted copy of x1 but gives r = 0. Cross-correlation, which repeats the calculation at every shift, is the tool that finds such a copy.
• Random noise gives r near 0 : x6 gives -0.019 rather than 0, because the products of a finite random sample never cancel perfectly.

Fourier Transform

The correlation section gave sum of times one meaning: how much two sequences look alike without shifting. The Discrete Fourier Transform uses exactly that meaning. It compares one input sequence with many reference waves, one frequency at a time.

Now we are going to look into such a famous tool, called "Fourier Transform". Here in this section, I would focus on the mathematical meaning of the Fourier Transform (more specifically, Discrete Fourier Transform). See the Fourier Transform page for its role as an engineering tool.

A typical mathematical presentation of Discrete Fourier Transform looks as follows. It may look a little bit complicated, but if you wrap it up into proper chunks you would notice that this is also a kind of "Sum of Times" format.  

As you learned in Correlation section (explained above), the most intuitive meaning of "Sum of Times" is a kind of indicator showing the correlation between two data sequences.

So Fourier Transform is also based on calculating the two specified data sequence as follows.

x_i = input data sequence for Fourier Transform

y_i = e^{-i 2pi k n/N}

It means that Fourier Transform is indication of correlation between a given data sequence and "e^{-i 2pi k n/N}". If you studied "Exponential Form" or any material about Euler's formula, you would know e^{-i 2pi k n/N} is a cyclic function. It is a cyclic function, the cycle of which is represented by the variable 'k'. Therefore, we can say "Fourier Transform is a tool to show the correlation between a specified data sequence and muliples of cyclic data sequence with different cycle".

If I represent my explanation, Fourier Transform can depicted as follows. (In the following graph, "x1" is the input data you want to do 'fourier transform', and the series of plots on right side is the multiple cyclic data sequence with different cycles. The stem plot at the left bottom is the graph shows the forrelation coefficient of each data pair, i.e (x1, s1), (x1,s2), (x1, s3) and so forth.

Figure 4. The input x1 compared with sine waves of 1 Hz to 10 Hz. The stem plot shows the size of each correlation coefficient, and only 2 Hz, 5 Hz and 7 Hz give a nonzero value.

• The stems match the three components of x1 : x1 = sin(2π2t) + 0.3 sin(2π5t) + 0.7 sin(2π7t), and the stems stand at 2, 5 and 7.
• The stem heights follow the amplitudes : each coefficient is the amplitude divided by sqrt(1 + 0.3² + 0.7²) = 1.257. That gives 0.796, 0.239 and 0.557, which are the heights in the plot.
• All other frequencies give 0 : sine waves of different whole frequencies cancel over a whole number of cycles, so their sum of times is zero.

If you convert this description into a mathematical forumula, Fourier Transform can be depicted as follows.

Figure 5. The DFT as a row of sums of times. Each output X_k is one correlation between the input and a complex wave with k cycles.

The complex exponential does more than the sine waves of Figure 4. By Euler's formula, e^-iθ = cos θ - i sin θ, so each X_k correlates the input with a cosine and a sine of the same frequency. That is why |X_k| does not depend on the phase of a component, while a correlation with a sine alone drops to zero for a cosine, as x2 showed. Note also that k runs from 0 to N-1. The last label in the drawing, X_N, would equal X₀, so the last output is X_N-1.

Following is Matlab/Octave script generating the sequence of graphs shown above. Try change the data for x1 in various different way until you get your own intuition of this concept.

t=0:2*pi/400:3;
Nt = size(t,2);
x1 = sin(2*pi*2*t) + 0.3 * sin(2*pi*5*t) + 0.7*sin(2*pi*7*t);
s1=sin(2*pi*1*t);
s2=sin(2*pi*2*t);
s3=sin(2*pi*3*t);
s4=sin(2*pi*4*t);
s5=sin(2*pi*5*t);
s6=sin(2*pi*6*t);
s7=sin(2*pi*7*t);
s8=sin(2*pi*8*t);
s9=sin(2*pi*9*t);
s10=sin(2*pi*10*t);
corr_vec = [corrcoef(x1,s1) corrcoef(x1,s2) corrcoef(x1,s3) corrcoef(x1,s4) corrcoef(x1,s5) corrcoef(x1,s6) corrcoef(x1,s7) corrcoef(x1,s8) corrcoef(x1,s9) corrcoef(x1,s10)];

subplot(10,2,1);plot(x1);ylabel('x1');axis([0,Nt,-2.1,2.1]);

subplot(10,2,1*2);plot(s1);ylabel('s1(1Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,2*2);plot(s2);ylabel('s2(2Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,3*2);plot(s3);ylabel('s3(3Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,4*2);plot(s4);ylabel('s4(4Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,5*2);plot(s5);ylabel('s5(5Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,6*2);plot(s6);ylabel('s6(6Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,7*2);plot(s7);ylabel('s7(7Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,8*2);plot(s8);ylabel('s8(8Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,9*2);plot(s9);ylabel('s9(9Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,10*2);plot(s10);ylabel('s10(10Hz)');axis([0,Nt,-1.1,1.1]);
subplot(10,2,[13 15 17 19]);stem(abs(corr_vec));ylabel('corr');axis([0,10,0,1.0]);

The script builds corr_vec from ten corrcoef calls. That works in an Octave version whose corrcoef(x,y) returns a single number. In current Matlab and Octave, each call returns a 2 x 2 matrix, so corr_vec becomes a 2 x 20 matrix and the stem plot changes. Taking the (1,2) entry of each result, as in the correlation section, restores the plot of Figure 4.

• Each DFT output is one sum of times : X_k = Σ x_n e^-i2πkn/N compares the input with a wave of k cycles.
• A large |X_k| means a strong component at k : the input looks like the reference wave of that frequency.
• Cosine and sine together make the result phase independent : the complex exponential carries both, so a shifted component is not missed.
• N outputs need N sums of N products : that is N² multiplications, which the FFT reduces to about N log₂N.

Convolution

Correlation and the Fourier Transform each compute one sum of times per comparison. Convolution computes a whole sequence of them, sliding one sequence along the other and producing one output sample per position.

Convolution is an tool which is as widely used as Fourier Transform. When we "analyze" or "design" something (e.g, Filter or Control System) we usually use Fourier Transform, but when we "implement" what we designed, we normally use "convolution".

Fourier Transform is a tool which convert a time domain entity into a frequency domain entity. Convolution is a tool which manipulate a time domain entity as it is in time domain.

To be honest, I haven't heard the word 'Convolution' before I first started a filter design product around 20 years ago and I haven't understood the real meaning of this tool for a couple of years even while I was using the tool very heavily and even implemented it in my program. So don't get disappointed if you don't understand this right away. As far as I exprienced, it took me take at least several month to understand the real meaning of any new concept even though it may look simple. The most important thing is to hold on to it without giving up until you grasp the real understanding however long it would take.

Actually it is very difficult to understand the real/intuitive meaning of convolution without animation. (please refer to a couple of links I listed below). But you should not rely on to the animated explation only. You would say "Ah Ha... Now I think I understand this" when you see those animated ones, but your understanding would start fading a couple of days later. At some point, you have to directly attack the math itself and solidify your understanding.

I will try add a couple of more illustration later hoping it help more. For now, if I just describe it the process verbally,

Let's assume we have two data called 'f[m]', 'g[m]'. In convolution, these two data has two different role.. one is the data to be processed and the other one is what we call 'kernel'. Let's assume f[m] is the data to be processed and g[m] as 'kernel'. Let's assume c[n] is the result data of the convolution.  Usually f[] is much longer than g[]. We want to take the convolution of these two data. Overall step goes like this.

i) Take the kernel data sequence (in this case, g[m]) and flip it around (revserse it). This gives you g[-m]

ii) Align f[m] and g[-m] to the left. Now the starting position of f[m] and g[-m] is same.

iii) Perform the "sum of times (sum of multiplication)" between f[m] and g[-m] from the first elements of g[-m] to the last elements of g[-m] and put the result into the first element of c[] data.(It means the result goes into c[0]).

iv) Shift g[-m] data to the right by one step (this is called g[-(m-1)] or g[1-m]). Now the second data of f[] and first data of g[-(m-1)] is aligned.

v) Perform the "sum of times (sum of multiplication)" between f[] and g[] from the first elements of g[-(m-1)] to the last elements of g[-(m-1)] and put the result into the second element of c[] data.(It means the result goes into c[1]).

vi) Shift g[-(m-1)] data to the right by one step (this is called g[-(m-2)] or g[2-m]). Now the third data of f[] and first data of g[-(m-2)] is aligned.

vii) Perform the "sum of times (sum of multiplication)" between f[] and g[] from the first elements of g[-(m-2)] to the last elements of g[-(m-2)] and put the result into the third element of c[] data.(It means the result goes into c[2]).

viii) Now you see any pattern. Do this pattern until g[] reaches to the end of f[].

For clearer understanding, I recommend you to try this whatever programing language (C, Java, Matlab, Octave etc) you are familiar with. If you are not familiar with any of the programming language, Microsoft Excel would be an excellent tool for you to try.

I hope following illustration would help you a little bit understand the mathematical component of the convolution.

Figure 6. The discrete convolution formula. Every output sample c[n] = (f * g)[n] is one sum of times between f[m] and a flipped and shifted copy of g.

The note at the top of the drawing says the result has size n. Read n there as the index of one output sample, not as the length. For a data sequence of length L_f and a kernel of length L_g, the full convolution has L_f + L_g - 1 samples. The drawing below shows the flip and the shift that the four notes describe.

Figure 7. Flip and shift. The blue kernel g[m] is mirrored into the green g[-m], and moving it right by n gives the red g[n-m].

One of the simplest and most common example of the convolution can be found in stock chart. In stock chart, you would see 7 days moving average or 52 days moving everage etc. This kind of moving average is the most common example of convolution.

if you set the data as follows,

f[] = daily stock price data

g[] = [1/7  1/7   1/7   1/7   1/7   1/7   1/7]

If you take the convolution of f[] and g[], the result will give you 7 days moving average of the stock price data.

The seven weights of g[] add up to 1, so a constant price passes through unchanged. Each output sample is the sum of times of the last seven prices with 1/7 each, which is their average. A kernel with unequal weights, such as a weighted moving average, works the same way.

• Convolution is a sliding sum of times : one output sample per shift of the flipped kernel.
• The kernel is flipped, correlation is not : for a symmetric kernel, such as the moving average, the two give the same result.
• The order of f and g does not matter : f * g = g * f, which is why the FIR section below can shift the data instead of the kernel.
• The full output is longer than the input : L_f + L_g - 1 samples, because the kernel enters and leaves the data gradually.

Following is a couple of links that you give you a couple of intuitive understanding of the convolution process.

• Continous Convolution (http://jhu.edu/~signals/convolve/index.html)
• Discrete Convolution(http://jhu.edu/~signals/discreteconv2/index.html)
• Interactive Lecture Module: Continuous-Time LTI Systems and Convolution (http://jhu.edu/~signals/lecture1/frames.html)

Vector Inner Product

Linear algebra has its own name for sum of times: the inner product, or dot product. Once you recognize it, many formulas in textbooks become short. The correlation coefficient and the DFT above can both be written as inner products.

I think you might have learned about a vector. (you may have learned this in high school in some country or learned it in Linear Algebra course in University). In the course, you may have learned an mathematical operator called 'inner product' of vectors. '

If you define a two vector as follows,

The inner product of these two vectors are defined as follows. (What is practical meaning of "Inner Product"? I would try to explain on this in Vector page).

You would notice this is a straightforward example of "sum of times". This is why in many text book many authors likes to this kind of 'vector inner product' notation for various "sum of times" operation. Once you see those process expressed as this kind of purely mathematical operation, you start feel dizzy and close the book. But it is more like a psychogical effect. Just try to convert it into a form which gives you less psychological burden like "sum of times" form and you can continue to read the book -:)

The drawing writes a as a row and b as a column. With that shape, the inner product is also the matrix product of a 1 x 5 matrix and a 5 x 1 matrix, which gives a 1 x 1 result. For example, [1 2 3 4 5] and [5 4 3 2 1]^T give 5 + 8 + 9 + 8 + 5 = 35.

The inner product also has a geometric meaning: a ⋅ b = |a| |b| cos θ, where θ is the angle between the two vectors. Compare this with the correlation formula. For zero-mean sequences, the correlation coefficient is the inner product divided by the two lengths, so it is exactly cos θ. Two sequences with r = 0, like x1 and x2, are orthogonal vectors. The Vector Product page goes further into this geometry.

• The inner product is sum of times with vector notation : a ⋅ b = Σ a_ib_i.
• It measures alignment : it is largest when the vectors point the same way, zero when they are orthogonal, and negative when they point against each other.
• Correlation is a normalized inner product : remove the means, divide by the lengths, and cos θ remains.

Neural Network

A neural network repeats the neuron of Figure 2 many times. Every node of one layer computes a sum of times over all nodes of the previous layer, and a matrix product does all of these sums at once.

Just look at the following diagrams first and think on your own about how this can be related to the "sum of times/sum of multiplication" process. Refer to Neural Network section in Matrix page.

Figure 8. A network with 9 inputs, 5 hidden nodes and 3 outputs. Every line is one weight, and every node adds up the weighted values that arrive on its lines.

Again, refer to Neural Network section in Matrix page to know how you can get the following representdation from the diagram above.

Figure 9. The hidden layer as one matrix product. Row j of the 5 x 9 weight matrix times the input vector is the sum of times that feeds H_j.

Again, refer to Neural Network section in Matrix page to know how you can get the following representdation from the diagram above.

Figure 10. The output layer as one matrix product. The 3 x 5 weight matrix takes the five hidden values h₁ to h₅.

Look at the dimensions in the drawing above. The weight matrix is 3 x 5 and the vector is 5 x 1, so the product is 3 x 1. There are three outputs, o₁ to o₃, as in Figure 8. The right side of the drawing lists five rows, up to o₅, and the last two rows should not be there.

The weight w_ij in these drawings connects node i of one layer to node j of the next. So row j of the matrix collects every weight that ends at node j. In a working network, each total input then gets a bias added and passes through an activation function, such as the sigmoid, before it becomes the value of the node.

• Each node computes one sum of times : H₁ receives w₁₁i₁ + w₂₁i₂ + ... + w₉₁i₉.
• A layer is a matrix times a vector : 5 hidden nodes need 5 sums of 9 products, which is one 5 x 9 by 9 x 1 product.
• The whole network in Figure 8 needs 60 multiplications : 9 x 5 = 45 for the hidden layer and 5 x 3 = 15 for the output layer.
• The nonlinearity sits outside the sum : without an activation function, the two layers would collapse into one 3 x 9 matrix.

Application - Convolution - FIR - Low Pass Filter

In a previous section, you understand how the process called "sum of times" is used for a complicated process called 'Convolution'. This section would give you a example on how the process 'Convolution' is used in a engineering application. This example shows you how covolution is used to implement a digital filter and of course, the fundamental mathemtical process is "sum of times".

Most of digital filter (FIR filter in this case) is represented as follows. Take bi = g[] and x[] = f[] and read the 'convolution' section. then you would understand how this works. The only difference between this mathematical representation and the one used in 'convolution' section is that in this case f[] is shifting at each iteration, but in convolution section g[] was shifting at each iteration. But result is the same.

Figure 11. A 49-tap low pass FIR filter at work. The 3200 Hz component x2 is removed from x = x1 + x2, and the 500 Hz sine x1 comes out in y.

• x1 is 500 Hz and x2 is 3200 Hz : at Fs = 8 kHz, x1 has 16 samples per cycle and x2 has only 2.5, which is why x2 looks like a solid band.
• b holds the 49 filter coefficients : fir1(48,0.35) gives order 48, so N = 48 in the formula above and there are 49 taps. The cutoff is 0.35 x Fs/2 = 1400 Hz, between the two tones.
• y is longer and delayed : conv returns 512 + 49 - 1 = 560 samples. A symmetric 49-tap filter delays the signal by 24 samples, and the ramps at both ends are where the kernel only partly overlaps the data.

Following is the Matlab and Octave code you can try.

Fs=8e3;
Ts=1/Fs;
Ns=512;
t=[0:Ts:Ts*(Ns-1)];
f1=500;
f2=3200;
x1=sin(2*pi*f1*t);
x2=0.3*sin(2*pi*f2*t);
x=x1+x2;
b = fir1(48,0.35,'low');
y=conv(x,b);
subplot(5,1,1); plot(x1);ylabel('x1');axis([0,550,-1.5,1.5]);
subplot(5,1,2); plot(x2);ylabel('x2');axis([0,550,-1.5,1.5]);
subplot(5,1,3); plot(x);ylabel('x=x1+x2');axis([0,550,-1.5,1.5]);
subplot(5,1,4); plot(b);ylabel('b');axis([0,550,-0.5,0.5]);
subplot(5,1,5); plot(y);ylabel('y=conv(x,b)');axis([0,550,-1.5,1.5]);

Each output sample y[n] is one sum of times of 49 coefficients with the latest 49 input samples. So this filter costs 49 multiplications and 48 additions per sample, and at 8 kHz that is 392,000 multiplications per second. This fixed cost per sample is why FIR filters map so well onto DSP hardware, which has a multiply-accumulate instruction for exactly this loop.

• An FIR filter is a convolution : y = conv(x,b), with the coefficients b as the kernel.
• The coefficients define the frequency response : the Fourier Transform of b shows which frequencies pass, and fir1 designs b from the cutoff you ask for.
• More taps give a sharper filter : but they also add delay and cost one more multiplication per tap per sample.

Application - Convolution - Image Processing

I will not say much about the image processing here since this page is not for the theory of image processing even though this is one of my favourite topics. If you are interested in the image process, I strongly recommend to refer to this excellent presentation here.

Very briefly speaking, a lot of image processing techniq is implemented by Convolution and this convolution is done in 2 D data (2 D array) as illustrated below. My point here is that you see the convolution and "sum of times" process in this techniq as well.

Figure 12. Two-dimensional convolution. The kernel g[k,l] slides over the image f[k,l], and each position gives one pixel of y as a double sum of times.

The formula in the drawing is the 1-D convolution with a second index. For a kernel of size (2m + 1) x (2m + 1), each output pixel needs (2m + 1)² products. A 3 x 3 kernel therefore costs 9 multiplications per pixel.

Following is one example of the process that I mentioned here with Octave/Matlab. You would see the source code at the bottom. Enjoy !!!

Figure 13. The kernel [0 0 0; 0 1 0; 0 0 -1] applied to a grayscale photo. Flat areas turn black, and diagonal edges stand out in white.

img=imread('Lena.png');
imgBW = rgb2gray(img);
imshow(imgBW);
b=[0 0 0;0 1 0;0 0 -1]
pimg = conv2(imgBW,b);
figure;
imshow(pimg);

Let's see what this kernel computes. The function conv2 flips the kernel, so the output pixel is y[i,j] = f[i,j] - f[i-1,j-1]. That is the difference between a pixel and its upper-left neighbour. It is near zero on flat areas and large where the brightness changes along the diagonal, so the kernel works as a simple diagonal edge detector.

Two practical notes explain the harsh look of the result. First, pimg holds values from -255 to 255 as doubles, but imshow expects doubles between 0 and 1. Every positive difference shows as white and every other value as black. The call imshow(pimg,[]) scales the full range to gray levels instead. Second, current Matlab requires double input for conv2, so conv2(double(imgBW),b) is the safe form there.

• Image filters are 2-D sums of times : blur, sharpen and edge detection differ only in the kernel values.
• A kernel that sums to 1 keeps the brightness : a 3 x 3 kernel of 1/9 each is a blur, the 2-D moving average.
• A kernel that sums to 0 finds changes : this example, 1 - 1 = 0, returns 0 on every flat area.