Statistics

Regression analysis

In French: Régression

Regression models the link between a quantity one wants to explain and one or more quantities that influence it. It quantifies what the scatter plot lets one guess.

What regression provides

It gives three things. An equation, which makes it possible to predict the output from the inputs. Coefficients, which say by how much the output moves when an input varies by one unit. A goodness-of-fit measure, which says what share of the observed variation the model explains.

In Lean Six Sigma, the main use is the analysis step: one suspects that a process parameter influences a characteristic, and regression decides — does the effect exist, how large is it, and does the model explain enough for it to be used.

Multiple regression extends the reasoning to several simultaneous inputs, which is almost always the real situation. It then makes it possible to separate the effect of each factor from the effects of the others — something no pairwise comparison can do.

Correlation is not causation

Two quantities can vary together without either acting on the other: a third influences both, or the coincidence is fortuitous. Regression measures an association, it does not demonstrate a mechanism.

Three elements make it possible to go from association to causal link, and none is statistical: a plausible physical or logical mechanism, the cause preceding the effect, and above all an experimental check — the factor is changed and the result observed. That is exactly what a design of experiments does.

Reading the results without going wrong

The coefficient of determination indicates the share of variation explained. A model at 0.9 explains a lot; a model at 0.2 leaves most of it out. But a high coefficient guarantees neither that the model is right, nor that it will predict correctly outside the observed range.

The residuals — the gaps between observed and predicted values — often say more than the model itself. Plotted, they should look like a cloud with no structure. A curvature signals that a non-linear relationship has been forced into a straight line; a progressive widening signals that variability depends on the level.

Extrapolating is the classic trap. A model established between 20 and 60 degrees says nothing about what happens at 90. The relationship may reverse, saturate, or the process may change regime.

The common mistakes

Adding variables until the fit looks good. Each variable added mechanically improves the fit, even if it has no bearing. A twelve-factor model on thirty observations describes the noise of those thirty and will predict nothing.

Confusing a significant coefficient with a large effect. On a lot of data, a tiny effect comes out as significant. The coefficient says the size, significance says only that it is probably not zero — those are two different readings.

On the shop floor — Surface treatment

A regression linked coating thickness to bath temperature, with a convincing fit. The residual plot showed a clear curvature: the relationship plateaued beyond a threshold. The linear model recommended raising the temperature; the process had long stopped gaining anything from it.

The other tools illustrated by this trade: machining, mechanical engineering and finishing.

Frequently asked questions

What is the difference between correlation and regression?

Correlation measures the strength of a link between two quantities, without distinguishing which explains the other — the coefficient is the same in both directions. Regression assigns a role: one quantity is explained, the others explain it, and the result is a model usable for prediction.

What coefficient of determination should be aimed for?

There is no universal threshold. On a well-controlled physical process, 0.8 is common; on a phenomenon involving human behaviour, 0.3 can be an excellent result. The right question is not "is it high enough" but "is the model good enough for the decision I have to take".

How much data is needed?

A widespread rule of thumb asks for at least about ten observations per factor studied, and more if the effects are weak or the data noisy. Below that, the model fits the chance of the sample and changes completely with the next one.

Can regression be used on historical data?

Yes, and it is often the least expensive starting point. But historical data covers only the conditions already encountered, and the factors in it rarely vary independently — which makes their effects hard to separate. That is precisely what a design of experiments corrects by organising the variation.