Hi Rohit, I'm a product manager on Target and let me try to address your concern. First and most importantly, the sample size calculator does not provide an estimate. It stipulates the minimum sample size required in order to guarantee that your false-positive rate (ie inverse of Confidence) is bounded. Which means that if you desire a 95% confidence (or 5% false-positive rate), you MUST wait until this sample size has transpired in order to guarantee that only 1 out of 20 times (ie 5%) will a test yield a false-positive. Only after the test has crossed the sample size, a user should look at the Confidence-value and ascertain that it is indeed above 95%. If the confidence-value after the sample side has been acquired is below 95%, this means that at a 95% threshold for significance, your test in inconclusive. If all of this didnt make sense, here is a simple 3-step workflow to do AB-testing correctly:
1. Compute the sample size with desired significance (say 95%) and most accurate guesses for "Baseline CR", "Minimum detectable lift". If you have more than 2 experiences, dont forget to apply Bonneferroni correction.
2. Wait until each experience has acquired this sample size.
3. Evaluate only at this point, whether the Confidence value shown in the Reports is above 95%. If its not, your test is inconclusive and you do not have a winner for this test.
I understand this is something you may not have done before, but our years of analysis have shown that if users dont wait until the sample size, their tests are 56% likely to find a false-positive (ie a 'winner' that actually performs worse than control in reality).
Hope that helps!
