An example: say we want to know how a drug affects cognition. We give a simple test to a bunch of people on and off it, blinded, etc. The control group's average score is 74, and the test group's average score is 72. We can use a t test to see if there's a statistical difference, and find there isn't. We can't conclude anything about the drug.

Now imagine we have exactly that same data, but we were careful to give two tests to each person (in a random order, and different tests, of course). We take another look at the data and find out that every single participant scored lower when they were on the drug. With even a fairly small sample size this provides strong evidence that the drug impairs cognition, and probably tells us quite a bit about how much it does.

The article is probably talking about multivariate regression; the more important number comes a few sentences later---"retention was related more strongly to manager quality than to seniority, performance, tenure, or promotions". So presumably, they did the same sort of analysis, carefully pairing people who were similar in as many ways as possible, and found out that good managers are more important than seniority in terms of employee retention. The more variables you have, the more even large differences can hide in raw group averages.

'For example, in 2008, the high-scoring managers saw less turnover on their teams than the others did—and retention was related more strongly to manager quality than to seniority, performance, tenure, or promotions. The data also showed a tight connection between managers’ quality and workers’ happiness: Employees with high-scoring bosses consistently reported greater satisfaction in multiple areas, including innovation, work-life balance, and career development.'

If their scores predicted those things, then they were measuring something real, regardless of whether they went looking for what they wanted or not. The question then becomes one of whether altering those scores alters the dependent variable or whether you've just created a correlation by doing evil to your numbers.

Which... they did look at their results down the line and I'd imagine they'd have looked at turnover, it'd seem really odd not to considering the other things they looked at and the obvious business case for doing so.

One of the big things I retained from my stats classes is the idea that, once you deviate from a pre-specified analysis technique, the strength of your conclusion is strongly diminished. Also, sophisticated statistical techniques are often less robust than simple ones. Maybe some other ideas apply that I can't think of off the top of my head.

On the other hand, the author may not have appreciated the statistical iffyness of that phrasing, and perhaps misrepresented the rigor of the actual analysis.

That's actually how one performs experiments and develops theories in the social (and managerial) sciences, such as economics: given data, and a hypothesis, and try to develop a model that fits both and offers demonstrable predictive power for future circumstances and datasets.

Of course, one's math might be wrong, and the model may still be falsified by future data. But that's what makes social science interesting.