What does a p-value actually mean?
A p-value is the probability of getting a result at least as extreme as yours, assuming the null hypothesis is true. A small p-value means your data would be unusual under the null, which counts as evidence against it; it is not the probability that the null hypothesis is true.
Start by pretending the null is true
Every hypothesis test begins with a skeptical assumption, the null hypothesis H₀, usually "no effect" or "no difference." You then ask how surprising your data would be in a world where H₀ holds. The p-value is that surprise, measured as a probability. It is computed entirely inside the null world, which is why it cannot tell you the chance that the null world is the real one.
A worked test
You flip a coin 100 times and get 61 heads. Is it fair? Take H₀: p = 0.5 and Hₐ: p ≠ 0.5. Under H₀ the number of heads has mean 50 and standard deviation √(100 × 0.5 × 0.5) = 5, and the normal approximation applies because both expected counts are 50.
The test statistic is z = (61 − 50) / 5 = 2.2. The probability of z ≥ 2.2 is about 0.014, and because the alternative is two-sided you count both tails: p ≈ 0.028. With a significance level α = 0.05 chosen in advance, 0.028 < 0.05, so you reject H₀.
On the AP Statistics exam you would interpret it in context: assuming the coin is fair, there is about a 0.028 probability of getting a result at least as far from 50 heads as 61. Then conclude: there is convincing evidence that the coin's true probability of heads differs from 0.5.
What a p-value does not mean
- It is not the probability that H₀ is true, and 1 − p is not the probability that Hₐ is true.
- It is not the probability that the result happened by chance.
- It does not measure how big or important an effect is. With a large enough sample, a tiny effect can give a very small p-value.
- A p-value above α does not prove H₀. You "fail to reject" it; the data were just not strong enough evidence against it.
A confidence interval is often more informative, because it shows how large the effect plausibly is.
Seeing the tail
Ask Isotoon "What does a p-value actually mean?" and you get a short animated video. The version that makes it click runs thousands of fair-coin trials, stacks their head counts into a distribution, then shades the tails beyond 39 and 61 so you see the p-value as a share of the null world. That distribution's bell shape comes from the central limit theorem. See also AP Statistics and statistics.
Questions
- Is a p-value of 0.05 the cutoff for significance?
- Only by convention. The significance level α is chosen before the study, and 0.05 is the most common choice. Some fields use 0.01 or stricter. A p-value of 0.049 and one of 0.051 carry nearly the same evidence.
- Does a small p-value mean the effect is large?
- No. A p-value depends on both the size of the effect and the sample size. A huge sample can make a trivial difference statistically significant, so report an effect size or confidence interval too.
- What should I write when the p-value is larger than alpha?
- Say you fail to reject the null hypothesis and that there is not convincing evidence for the alternative, in the context of the problem. Do not say you accept the null or that it has been proven true.