Hi all. Jimmy, statistician from Optimizely chiming in.
We were excited to collaborate with the authors on this study. Keep in mind the data used in this analysis is from 2014, before we introduced sequential testing and FDR correction as to specifically address this p-hacking issue. I expect these results are in line with any platform using fixed-horizon frequentist methods.
We were excited to collaborate with the authors on this study. Keep in mind the data used in this analysis is from 2014, before we introduced sequential testing and FDR correction as to specifically address this p-hacking issue. I expect these results are in line with any platform using fixed-horizon frequentist methods.
Check out this paper for more details: http://www.kdd.org/kdd2017/papers/view/peeking-at-ab-tests-w...