Statistics

Hypothesis testing

In French: Tests d'hypothèse

A hypothesis test answers a single question: is the observed gap between two situations larger than what sampling chance produces on its own?

The logic of the test

One starts from a null hypothesis: there is no difference. One then calculates the probability of observing a gap at least as large as the one measured, if that hypothesis were true. That is the p-value.

If that probability is low — below a threshold set in advance, very often 5 % — one concludes that the null hypothesis is not very credible and that a difference exists. Otherwise, nothing is concluded: absence of evidence is not evidence of absence.

That asymmetry is the point everyone glosses over. A test that detects nothing does not say "the two are identical"; it says "with this data, I cannot assert it". The difference may be real and the sample too small to see it.

Choosing the right test

The choice depends on three things: the nature of the data, the number of groups compared, and the question asked.

Comparing means — Student’s t-test for two groups, analysis of variance beyond. Comparing spreads — tests on variances, often forgotten even though a difference in consistency is sometimes more important than a difference in mean. Comparing proportions or counts — the chi-square test and related tests.

These tests assume conditions: independence of the observations, often a distribution close to normal, sometimes comparable spreads. When they do not hold, nonparametric tests take over, at the cost of a little power.

The two risks, and the one that gets forgotten

The alpha risk is that of concluding on a difference that does not exist — it is the threshold one sets, 5 % in general. The beta risk is that of missing a real difference, and nobody ever sets it.

Yet it is the one that decides the sample size. A test run on too little data has a high probability of seeing nothing, whatever the reality — and one then walks away with the reassuring conviction that there is nothing to see. Calculating the necessary sample size BEFORE collecting is the only defence.

The common mistakes

Confusing statistical significance with practical importance. On ten thousand measurements, a two-minute gap in lead time comes out as highly significant. The next question — does that gap change anything for the customer? — is not statistical, and it is the one that decides.

Deciding the threshold after seeing the result, or multiplying the tests until one passes. Out of twenty tests at the 5 % threshold, one comes out significant by pure chance. It is mechanical, and it is what produces irreproducible conclusions.

On the shop floor — Logistics

Two order-picking methods showed a 4 % productivity gap over a month. The test concluded nothing: the variability between operators far exceeded the gap between methods. Collection resumed on the same operators, each testing both methods, showed the real gap — and it went the other way.

The other tools illustrated by this trade: logistics and distribution.

Frequently asked questions

What does a p-value of 0.03 mean?

That if no difference existed, there would be a 3 % chance of observing a gap at least as large as the one measured. It is not the probability that the null hypothesis is true, nor the probability of being wrong in concluding — two false readings that are extremely widespread.

Why is the threshold at 5 %?

By convention, not by necessity. It is a compromise inherited from agronomic practice at the start of the twentieth century. A context where being wrong is costly — safety, health — justifies a more demanding threshold; a reversible decision can live with a looser one. What counts is setting it before looking at the data.

How much data is needed for a test?

It can be calculated, from three elements: the gap one wants to be able to detect, the variability of the process, and the risks accepted. A small gap on a dispersed process requires a lot of data. Doing that calculation before collection avoids the most frequent scenario: a test that concludes nothing because it could not conclude.

Does a test replace a chart?

Never. The chart shows the shape, the extreme values, the hidden groups, the chronological order — everything a test summarises in a single number and loses. The useful sequence is always: plot first, test next to confirm what the eye suggested.