The real beauty of correlation and regression missed out by Maths

Maths has given us the concept without relatability and anything which is not relatable requires more effort to understand.

All of us have come across strict disciplinarians in our lives who believe punishment is more effective than praises. If prodded they would even tell you how a worst performing candidate improved after being shouted at, while a great performing candidate’s next performance dint improve after being showered with accolades. The results which they talk about are true but the cause which they try to associate with it could not be more wrong. The fact of the matter is that it is plain regression to the mean. If the outcome of a random event is extreme then a future outcome may be less extreme (or closer to the average in other words). There is an unpopular luck factor involved which could yield in extreme results again but applying the law of large numbers, it would move towards the average, with all other factors staying constant. Success is a result of skill plus luck. This is true for anything irrespective of whether you praise or punish. An unusually bad performance will improve the next time because there is regression to the mean. An exceptionally good performance will be followed by a slightly mediocre performance, if the skill level does not change drastically. You just tend to attach a causal explanation to the inevitable fluctuations of a random process because human beings like stories. They have this need to believe that there is a cause attached to it so as to feel more in control. We don’t want to consider that there sometimes is a luck factor involved because it makes us feel powerless. It doesn’t take long after that to cross the bridge and walk towards nihilism but I guess that is why human brains are wired in such a way that we can’t really spot it even though it is in our plain sight.

Same holds true even in sporting events. We know that an exceptional performance is followed by an average performance but the fans and commentators have very convincing and beautiful justifications which blows your mind. Sure, there are many factors to a player’s or team’s performance and all this makes the analysis very interesting but sometimes, at the very base of it, there is nothing more to it than plain regression at play.

You must have read the motivational phrase that if you have hit the rock bottom then you have nothing to worry about as you can go only up from here! Not to take the charm out of it but it is just stating a statistical fact which will more or less hold true always.

Regression and correlation go hand in hand.

We all have read articles or news snippets with findings such as the below, which catch our attention-

Studies show consumption of regular breakfast may help reduce teen obesity.

It is very important to understand that there is a high probability a group of obese people will perform somewhat better as it is an extreme-weight group. Extremes regress to mean over time when there is no perfect correlation between them. But we tend to look for causal explanations as mere statistics bores us. It took even scientists a long time to carefully navigate through these pits and have proper controls or placebo experiments to have better results.

We should consciously train our mind to look at everything objectively and look for facts rather than falling for the stories our mind weaves around everything, as this does not come naturally to us and our mind plays games. Same is true even with people- If we like a person A, we are more inclined to believing that he is intelligent too. This cognitive bias is the Halo Effect where one trait of a person influences our overall judgement leading us into making biased decisions. This is because it is easy and our brain loves taking shortcuts.

We all calculated correlation coefficients, solved huge equations in schools and colleges but we would still take the easy way of overlooking facts and associating causes to events which are completely random. Next time try to take a second look because the obvious need not be true. Wish they taught us this in school.

Reference: Thinking Fast and Slow- Daniel Kahneman