Errors in Hypothesis Testing

Rajesh

4 years ago

What are the errors in Hypothesis Testing?

A hypothesis test always leads to one of two decisions.

The sample data provide sufficient evidence to reject the null hypothesis and conclude that the treatment has an effect.
The sample data do not provide enough evidence to reject the null hypothesis. In this case, fail to reject H_0. It concludes that the treatment does not appear to have an effect.

Hypothesis testing is an inferential process,
The hypothesis test uses a sample to draw a conclusion about the population.
A sample provides only limited or incomplete information about the whole population,
There is always the possibility that an incorrect conclusion will be made.

In a hypothesis test, there are two different kinds of errors that can be made.

Type I Error

A Type I error occurs when the Null hypothesis is rejected when it is actually true. This error analysis concludes that treatment does have an effect when in fact it has no effect.

The Probability of a Type I Error

The alpha level determines the probability of obtaining sample data in the critical region even though the null hypothesis is true.
The alpha level for a hypothesis test is the probability that the test will lead to a Type I error.
If sample data are in the critical region just by chance, without any treatment effect. When this occurs, it will be a Type I error. It will conclude that a treatment effect exists when in fact it does not.
With an alpha level of α = .05, only 5% of the samples have means in the critical region. Therefore, there is only a 5% probability (p = .05) that one of these samples will be obtained.
The probability of a Type I error is equal to the alpha level

Type II Errors

A Type II error occurs when it fails to reject a null hypothesis that is really false. In this error, the study concludes that treatment does have no effect when in fact it has an effect. The hypothesis test has failed to detect a real treatment effect

A Type II error occurs when the sample mean is not in the critical region even though the treatment has an effect on the sample.
Often this happens when the effect of the treatment is relatively small. In this case, the treatment does influence the sample, but the magnitude of the effect is not big enough to move the sample mean into the critical region.
Because the sample is not substantially different from the original population (it is not in the critical region), the statistical decision fails to reject the null hypothesis and to conclude that there is not enough evidence to say there is a treatment effect
Unlike a Type I error, it is impossible to determine a single, exact probability for a Type II error. Instead, the probability of a Type II error depends on a variety of factors and therefore is a function, rather than a specific number.
The probability of a Type II error is represented by the symbol β, the Greek letter beta.

Selecting an Alpha Level

Alpha helps determine the boundaries for the critical region by defining the concept of “very unlikely” outcomes.
At the same time, alpha determines the probability of a Type I error.
When you select a value for alpha at the beginning of a hypothesis test, your decision influences both of these functions
The primary concern when selecting an alpha level is to minimize the risk of a Type I error.
Thus, alpha levels tend to be very small probability values.
As the alpha level is lowered, the boundaries for the critical region move farther out and become more difficult to reach.
Notice that z = 0, in the center of the distribution, corresponds to the value of μ specified in the null hypothesis. The boundaries for the critical region determine how much distance between the sample mean and μ is needed to reject the null hypothesis. As the alpha level gets smaller, this distance gets larger.
By convention, the largest permissible value is α = .05. When there is no treatment effect, an alpha level of .05 means that there is still a 5% risk, or a 1-in-20 probability, of rejecting the null hypothesis.

Results of the Statistical Test

A result is said to be significant or statistically significant if it is very unlikely to occur when the null hypothesis is true.
Treatment has a significant effect if the decision from the hypothesis test is to reject H₀ .

Factors that Influence a Hypothesis Test

The most obvious factor influencing the size of the z-score is the difference between the sample mean and the hypothesized population mean
A big mean difference indicates that the treated sample is noticeably different from the untreated population and usually supports the conclusion that the treatment effect is significant.
The size of the z-score is also influenced by the standard error, which is determined by the variability of the scores (standard deviation or variance) and the number of scores in the sample (n).

The Variability of the Scores

The increased variability means that the sample data are no longer sufficient to conclude that the treatment has a significant effect.
In general, increasing the variability of the scores produces a larger standard error and a smaller value (closer to zero) for the z-score. If other factors are held constant, the larger the variability, the lower the likelihood of finding a significant treatment effect.
In a hypothesis test, higher variability can reduce the chances of finding a significant treatment effect.

The Number of data points in the Sample

The second factor that influences the outcome of a hypothesis test is the number of data in the sample.
If all other factors are held constant, the larger the sample size is, the greater the likelihood of finding a significant treatment effect. In simple terms, finding a 0.9-point treatment effect with a large sample is more convincing than finding a 0.9-point effect with a small sample.

Assumptions for Hypothesis Tests with z-Scores

The mathematics used for a hypothesis test is based on a set of assumptions.
When these assumptions are satisfied, you can be confident that the test produces a justified conclusion.
If the assumptions are not satisfied, the hypothesis test may be compromised. The assumptions for hypothesis tests with z-scores are summarized as follows.

a) Random Sampling

- It is assumed that the participants used in the study were selected randomly. The sample must be representative of the population from which it has been drawn. Random sampling helps to ensure that it is representative.

b) Independent Observations

- The values in the sample must consist of independent observations.
- Two observations are independent if there is no consistent, predictable relationship between the first observation and the second.
- More precisely, two events (or observations) are independent if the occurrence of the first event has no effect on the probability of the second event.
- This assumption is satisfied by using a random sample, which also helps ensure that the sample is representative of the population and that the results can be generalized to the population

c) The value of σ is unchanged by the Treatment

- A critical part of the z-score formula in a hypothesis test is the standard error, σ_M. To compute the value for the standard error, we must know the sample size (n) and the population standard deviation (σ).
- In a hypothesis test, however, the sample comes from an unknown population
- If the population is really unknown, we do not know the standard deviation and, therefore, we cannot calculate the standard error.
- To solve this dilemma, we have made an assumption. Specifically, we assume that the standard deviation for the unknown population (after treatment) is the same as it was for the population before treatment.
- The effect of the treatment is to add a constant amount to (or subtract a constant amount from) every score in the population. A constant changes the mean but has no effect on standard-deviation.

d) Normal Sampling Distribution

- To evaluate hypotheses with z-scores, we have used the unit normal table to identify the critical region. This table can be used only if the distribution of sample means is normal.

e) Independent Observations

- Independent observations are a basic requirement for nearly all hypothesis tests. The critical concern is that each observation or measurement is not influenced by any other observation or measurement.
- An example of independent observations is the set of outcomes obtained in a series of coin tosses. Assuming that the coin is balanced, each toss has a 50–50 chance of coming up with either heads or tails. More important, each toss is independent of the tosses that came before. On the fifth toss, for example, there is a 50% chance of heads no matter what happened on the previous four tosses; the coin does not remember what happened earlier and is not influenced by the past.
- Many people fail to believe in the independence of events. For example, after a series of four tails in a row, it is tempting to think that the probability of heads must increase because the coin is overdue to come up with heads. This is a mistake, called the “gambler’s fallacy.”
- Remember that the coin does not know what happened on the preceding tosses and cannot be influenced by previous outcomes.
- In most research situations, the requirement for independent observations is typically satisfied by using a random sample of separate, unrelated individuals. Thus, the measurement obtained for each individual is not influenced by other participants in the study.