I think you got downvotes because you misunderstood what people were taking umbrage with in the first example.
It’s not that birth time can never have real effects. It’s that if you keep rolling a die long enough, eventually you’ll hit a “statically unlikely” event like rolling 4 fives in a row or hitting 1 2 3 4 in order.
Extraneous sub group analysis are like rolling the die again. Say you’re searching for a p-value of .05 with a confidence interval of 95%. That means 19 out of 20 times it’s indicative of a real relationship and 1 out of 20 times it was due to random chance.
If you do a bunch of extraneous sub group analyses like the reviewer wanted, you’re banking on the statistical likelihood that eventually you’ll get the result you want even if it’s not a real relationship.
This is what follow up studies are for. To separate the wheat from the chaff. Don't separate before. See my rant below as to why I think pre-hoc decisions on analysis is a bad idea: https://news.ycombinator.com/item?id=35883276
At the very least, I'd like to see people say in advance which parameters they are interested in. That sort of thing is fine and important to avoid un-backed p-hacking. But for researchers who come after, for their sake, if you are not publishing the entire data so that they can reanalyze it de novo, please do as much analysis as possible (and record as many parameters as possible), if only in the supplementary data.
Science can only build on previous science if the authors of that previous science allow it to happen.
It’s not that birth time can never have real effects. It’s that if you keep rolling a die long enough, eventually you’ll hit a “statically unlikely” event like rolling 4 fives in a row or hitting 1 2 3 4 in order.
Extraneous sub group analysis are like rolling the die again. Say you’re searching for a p-value of .05 with a confidence interval of 95%. That means 19 out of 20 times it’s indicative of a real relationship and 1 out of 20 times it was due to random chance.
If you do a bunch of extraneous sub group analyses like the reviewer wanted, you’re banking on the statistical likelihood that eventually you’ll get the result you want even if it’s not a real relationship.