Statistics

ANOVA (analysis of variance)

In French: ANOVA — analyse de la variance

Analysis of variance compares the means of several groups at once, judging whether the gap between the groups is large compared with the spread observed within each of them.

The reasoning

The method sets two sources of variation against each other. The first is the gap between the groups: the means of each are not identical. The second is the internal spread: within the same group, the values already vary among themselves.

The conclusion hinges on the ratio of the two. If the groups differ from one another much more than their values differ within, the difference is probably not due to chance. If the two variations are of the same order, nothing allows the groups to be told apart.

That is what explains a result often poorly received: two very different means may prove nothing if the measurements are widely spread, whereas a modest gap becomes significant on consistent data.

Why not compare pairwise

The temptation with four machines is to make six pairwise comparisons. The problem is cumulative: each comparison carries its own risk of concluding wrongly, and six comparisons in a row make a false alarm probable even when all the machines are identical.

Analysis of variance asks a single overall question — "do these groups differ somewhere?" — and therefore controls that risk. That is its reason for being, even more than convenience.

It does not, however, say which ones differ. When the overall answer is positive, the groups concerned are identified by comparisons designed for that purpose, which take into account the number of comparisons made.

What it assumes

It assumes groups made up of independent observations, internal spreads comparable from one group to another, and departures from the model that are distributed with no particular shape. Those conditions are checked, in particular by examining the residuals, before commenting on a result.

The condition most often violated in practice is independence: measurements taken one after another on the same machine resemble each other for reasons that have nothing to do with the group. That dependence brings out differences that do not exist.

A marked imbalance between the group sizes also weakens the conclusion, without invalidating it. A group of three values against a group of a hundred weighs little, and the result mainly says something about the larger of the two.

The common mistakes

Concluding "the groups are different" without saying which. The overall answer is a green light to search, not a conclusion presentable as is.

Chaining pairwise comparisons once a group has proved interesting, without taking their number into account. It is exactly the risk the method served to control, reintroduced at the next step.

Taking the absence of a detected difference for proof of equivalence. With few measurements or a lot of spread, the method detects nothing even when a real gap exists.

On the shop floor — Surface treatment

Four baths were suspected of giving different thicknesses. The analysis concluded that there was a difference, and examination of the residuals showed that it came down to a single bath, the one whose temperature controller had been replaced. The other three were indistinguishable, which the overall mean would never have shown.

The other tools illustrated by this trade: machining, mechanical engineering and finishing.

Frequently asked questions

When should ANOVA be used rather than a two-mean comparison test?

As soon as there are more than two groups to compare on the same quantity. For two groups only, both approaches give the same conclusion and the two-sample test is more direct. The value of analysis of variance appears from three, because it avoids piling up comparisons and the risk that goes with them.

What should be done if the spreads are very different from one group to another?

Do not ignore the finding, because it is often more interesting than the comparison of means itself: a group that is clearly more dispersed signals an instability that deserves to be treated first. On the calculation side, there are variants that do not assume equal spreads, and a transformation of the data sometimes solves the problem.

How many measurements are needed per group?

Enough for the internal spread to be estimated correctly, which means at least about ten values per group under ordinary conditions, more if the measurements are noisy. The right reflex is to determine that sample size before collecting, from the gap one wants to be able to detect, rather than to find out afterwards that nothing was found.

Is a significant difference an important difference?

Not necessarily, and it is the most widespread confusion. Significance says that a gap is probably not due to chance; it says nothing about its size or its practical interest. On a lot of data, an inconsequential gap comes out as significant — the next question is therefore always "by how much", asked in the unit of the process.