I think Hypothesis Test has been one of the most confusing concept in Statistics. Probably I am still getting confused a little :). However, I decided to write down my current understandings as it is and keep updating it until this will be very clear to me and hopefully to you as well.

This is just beginning of the writing and will keep updating for pretty long time.

The later sections complete the procedure. They cover the significance level, the test statistic, the p-value and the decision, with one worked example, a coin tossed 100 times. Where the standard textbook practice differs from the earlier notes, the new paragraphs say so and explain why.

• What is Hypothesis ?
• Is Any Hypothesis be testable ?
• Common Strategy in Hypothesis Testing
• Steps to Hypothesis Testing
• How is the decision criterion set ? - Step 2
• How is the test statistic computed ? - Step 3
• How is the decision made ? - Step 4

What is Hypothesis ?

Before you try anything for testing, you first have clear understanding on what you will test. Hypothesis Test is a method of tesing a Hypothesis. So first thing you need to do is to understand exactly what it mean by "Hypothesis".

I just copied two defintions by Googling as below.

1) A supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.

2) In science, a hypothesis is an idea or explanation that you then test through study and experimentation. Outside science, a theory or guess can also be called a hypothesis

Do you agree with these definitions ? Is this same as what you thought ?

My feeling is "May be.. I agree.. but not exactly sure" because I've never thought of this definition myself.. I just thought I know the meaning of "Hypothesis".

Is this definition the one used in "Hypothesis Testing" ?  Maybe yes in broad sense. But let's make it in simpler/clearer form that is being used in "Hypothesis Testing" . It can be described as follows :

A Statement (a short sentense) that "you think" is true. i.e,a statement that you claim to be true.

As you see, an Hypothesis just a statement that you think is true, but may not be true in reality because it is not proved yet.

If you toss a coin 100 times, how many heads you would get ?

I THINK it is 50 times. Are you sure ?  Can you prove this ? or you just think like that because you have learned as such at school ?

It doesn't matter... If you think (claim) as such, it can be a Hypothesis.

In our class at school, we have 20 girls and 20 boys. What do you expect if they do math test ?

I THINK (Claim) boys would be better ? Are you sure ? Isn't it a kind of biased opinion (prejudice) ?

It doesn't matter... If you think (claim) as such, it can be a Hypothesis.

Both examples share one feature that makes them testable later. Each claim is about a number that describes a whole population, not about a single toss or a single student. For the coin, that number is the probability p of heads. For the class, it is the difference between the mean scores of boys and girls. A statistical hypothesis is a statement about such a population parameter, and the data is a sample that gives evidence about it.

• A hypothesis is a claim that is not yet proven : it can come from theory, experience or simply a guess.
• A testable hypothesis names a population parameter : for example p = 0.5 for the coin.
• The data is a sample : a sample can support or contradict a claim, but it cannot prove it with certainty.

Is Any Hypothesis be testable ?

This is the most important thing and at the same time the most confusing thing to me.  Is Any Hypothesis that pops  up in my mind can be testable with "Hypothesis Test" method ?

The Answer is NO.

Hypothesis Test requires you to set Hypothesis in a specific way (format). What is the format ?

It requires to convert your claim into two Hypothesis in such a way that if one hypothesis become true, the other hypothesis automatically become false. The Hypothesis that is assumed to be true unless the data gives strong evidence against it is called 'Ho' and the other one is called H₁. It can be described as below.

Ho : A statement assumed to be true at the start, usually 'no difference' or 'no effect'

H₁ : A statement that is accepted when Ho is rejected. It is usually the claim you want to show.

Ho Hypothesis is called "Null Hypothesis" and H1 Hypothesis is called "Alternative Hypothesis".

Let's do some practice. How about trying with "I think Boys' average math score is better than girls' in my class". How do we convert this into proper format for testing.

Case 1 :

Ho : Boys' average math score is better than girls'

H₁ : Boys' average math score is Not better than girls' .

Case 2 :

Ho : Boys' average math score is same as girls'

H₁ : Boys' average math score is Not same as girls' .

Which one is better form of Hypothesis ? In principle, both one are allowed but I think 'Case 2' is better.

I know you would ask "Why Case 2 is better ?". You will see later. This is very important to clearly understand why Case 2 is better.

Here is the reason, in the standard form of the test. The null hypothesis must contain the equality, because the test computes probabilities as if Ho were true. In Case 2, Ho says the means are the same. That fixes the difference at exactly 0, so the distribution of the test statistic is known. In Case 1, Ho says boys are better. That allows any positive difference, so no single distribution follows from it. So Case 1 is not a valid null hypothesis in the usual framework.

If your real claim is directional, put it into H₁. The pair is then Ho : mean of boys = mean of girls, and H₁ : mean of boys > mean of girls. This is called a one-sided test. Case 2 with 'not same' in H₁ is a two-sided test, which looks for a difference in either direction.

• Ho always holds the equality : it states 'no difference' or a specific value, such as p = 0.5.
• H₁ holds the claim you want to show : it is 'not equal' for a two-sided test, or 'greater' or 'less' for a one-sided test.
• The two hypotheses do not overlap : exactly one of them is true for the population.

Before we don't even start trying "Test", just try to make various kind of Hypothesis on your own. You may find a lot of examples from internet.

Common Strategy in Hypothesis Testing

Once you become familiar with setting Hyptotheis and now you have a set of Hypothesis (Ho, H₁), the next step is to test it. I think there can be roughly two types of strategies that you case think of.

Case 1 : Try to prove Ho is true. If you are successful here, it would prove Ho is true and automatically H₁ become false.

Case 2 : Try to prove Ho is false. If you are successful here, it would prove Ho come false and automatically H₁ become true.

You can take whatever approaches depending on situation, but what you need to notice is "We always try with Ho (Null Hypothesis)".

In standard practice only Case 2 is used. A test can collect evidence against Ho, but it can never prove Ho true. Suppose you toss a coin 100 times and get 52 heads. This result fits a fair coin well. But a coin with p = 0.55 would give 52 or fewer heads in about 31 % of such experiments, so the same data also fits a slightly biased coin. The data cannot tell the two apart.

So the two possible results of a test are 'reject Ho' and 'fail to reject Ho'. Failing to reject means that the data is not strong enough, not that Ho has been proven. Note also that a very large sample will reject almost any Ho that is only approximately true.

• A test only tries to reject Ho : this is Case 2 above, and it is the only strategy that statistics supports.
• Not rejecting is not proving : 52 heads fits a fair coin and also a coin with p = 0.55.
• The conclusion is stated about Ho : write 'reject Ho' or 'fail to reject Ho', never 'accept Ho as proven'.

Steps to Hypothesis Testing

There are roughly 4 steps to Hypothesis Testing as decribed below. Step 1 is the setup from the sections above, and steps 2 to 4 are where the statistics comes in. The next three sections go through them in order with one example, a coin tossed 100 times.

Step 1 : Make the Hypothesises

Step 2 : Deterime the criterial for decision. that is, set the criteria to prove or disprove Ho (Null) Hypothesis.

Step 3 : Compute the test statistics

Step 4 : Make a decision

Step 1 is explained in previous section and all other steps needs long explanation which will come later.

For the coin example, the four steps read as follows. Step 1 sets Ho : p = 0.5 and H₁ : p ≠ 0.5. Step 2 chooses α = 0.05 and fixes the rejection region, 39 heads or fewer or 61 heads or more. Step 3 counts 60 heads and computes the p-value, 0.057. Step 4 compares 0.057 with 0.05 and fails to reject Ho.

Keep the steps in this order. If α or the rejection region were chosen after seeing 60 heads, it would be easy to pick a rule that gives the answer you hoped for. Steps 1 and 2 use no data at all, and that is what makes the result of Step 4 meaningful.

• Steps 1 and 2 come before the data : the hypotheses, α and the rejection region are fixed first.
• Step 3 reduces the data to one number : the test statistic or its p-value.
• Step 4 applies the rule from Step 2 : there is no judgement left at this point.

How is the decision criterion set ? - Step 2

Before looking at the data, you decide how much evidence is enough to reject Ho. This order matters. A rule chosen after seeing the data can be bent to fit any result. The rule is set by the significance level α, the probability of rejecting Ho when Ho is actually true. The usual choice is α = 0.05, and 0.01 is used when a false alarm is expensive.

Two kinds of error are possible, and the table below shows where each one comes from. The rows give the truth, which we never know, and the columns give the decision.

For the coin, Ho : p = 0.5 and H₁ : p ≠ 0.5, with 100 tosses and α = 0.05. Under Ho the number of heads X follows a binomial distribution with mean 50 and standard deviation √(100 x 0.5 x 0.5) = 5. The rejection region is the set of results that are too far from 50. With the exact binomial probabilities, the region is X ≤ 39 or X ≥ 61. Its total probability under Ho is 0.035, the largest value that does not exceed 0.05. Including 40 and 60 would raise it to 0.057.

The plot below shows the probability of each number of heads when Ho is true, from 30 to 70 heads. The red bars form the rejection region, and the blue bars form the region where Ho is not rejected. The dashed line marks the result used in the next two sections, 60 heads.

Figure 1. Distribution of heads in 100 tosses of a fair coin. Results in the red tails are rare enough under Ho to reject it at α = 0.05.

• The peak is at 50 : P(X = 50) = 0.080. Even the most likely result has a probability below 10 %.
• The tails are small : P(X ≤ 39) + P(X ≥ 61) = 0.035. This is the actual Type I error rate of this rule.
• 60 heads is inside the blue region : P(X = 60) = 0.011 by itself, but the decision uses the whole tail, as Step 3 shows.

The price of a small α is a larger β. With this rule, a coin with p = 0.55 falls in the blue region 86 % of the time, so the power is only 0.14. For p = 0.6 the power is 0.46. More tosses raise the power, and this is how a sample size is chosen.

• α is chosen before the data : it is the accepted risk of a false alarm.
• α and β pull against each other : for a fixed sample size, lowering one raises the other.
• A larger sample raises the power : it narrows the distribution, so a real effect is easier to see.

How is the test statistic computed ? - Step 3

The test statistic turns the data into one number, and the distribution of that number under Ho must be known. Then you can say how unusual the observed value would be if Ho were true. Let's say the coin gave 60 heads in 100 tosses.

The simplest statistic here is the count itself. A standardized form is more common, because it works for any sample size. It is z = (X - 50)/5, the distance from the mean in units of the standard deviation. For 60 heads, z = (60 - 50)/5 = 2.0. By the central limit theorem, z follows the standard normal distribution approximately when Ho is true.

The p-value is the probability, computed under Ho, of a result at least as extreme as the one observed. For a two-sided test, extreme means far from 50 in either direction. The table below gives it three ways for a few results.

The plain normal approximation treats the count as continuous. The continuity correction subtracts 0.5 from the distance, z = (60 - 0.5 - 50)/5 = 1.9, and it brings the approximation close to the exact value. For 60 heads the three methods give 0.046, 0.057 and 0.057.

The same idea covers the class example. To compare two means, the usual statistic is Welch's t = (mean_B - mean_G) / √(s_B²/n_B + s_G²/n_G), where s is the sample standard deviation. For example, suppose 20 boys average 72 and 20 girls average 68, both with s = 10. Then t = 4/√(5 + 5) = 1.26. With 38 degrees of freedom, the two-sided p-value is 0.21.

• A test statistic measures distance from Ho : in units of the spread that Ho predicts.
• The p-value is a tail probability under Ho : it is not the probability that Ho is true.
• The approximation can change a borderline result : 60 heads gives 0.046 with the plain normal approximation and 0.057 exactly.

How is the decision made ? - Step 4

The last step compares the evidence with the rule from Step 2. If the p-value is at or below α, reject Ho. Otherwise, fail to reject it. The equivalent rule compares the statistic with a critical value. For a two-sided z test at α = 0.05 that value is 1.96, so you reject Ho when |z| > 1.96.

For 60 heads, the exact p-value is 0.057, which is above 0.05. So we fail to reject Ho, and the data does not show that the coin is biased. The plain normal approximation gives 0.046 and would reject. This is a borderline case, and it shows why the method must be fixed in Step 2 together with α. With 61 heads, all three methods give a p-value below 0.05, and Ho is rejected. For the class example, p = 0.21 is well above 0.05, so the data does not show a difference between boys and girls.

Report the p-value together with the decision. 'p = 0.057, not rejected at α = 0.05' tells the reader much more than 'not significant'. And a rejected Ho says only that the effect is unlikely to be zero. It says nothing about whether the effect is large enough to matter.

• Reject Ho when p ≤ α : or, equivalently, when the statistic falls in the rejection region.
• 60 heads in 100 tosses does not reject a fair coin : the exact p-value is 0.057, just above 0.05.
• Statistical significance is not practical importance : check the size of the effect as well as the p-value.