Investigating
A/B testing winner based on
A/B testing winner based on % not raw numbers. Using the current method with raw numbers can lead to a false positive.
For example,
We ran an A/B test with three different subject lines. We use the open rate to declare a winner. Unfortunately the 'winner' was not the true winner. Due to bounces the deliverability is not equal between all variations.
Our test group was 10684.
8504 were delivered within our time frame.
A. 3232 were delivered, 798 were opened.
B. 2806 were delivered, 674 were opened.
C. 2466 were delivered, 644 were opened.
Looking at the raw opens the winner was A. However, based on open rates, C was the true winner followed by A.
Open Rate
A. 24.69%
B. 24.02%
C. 25.70%
For example,
We ran an A/B test with three different subject lines. We use the open rate to declare a winner. Unfortunately the 'winner' was not the true winner. Due to bounces the deliverability is not equal between all variations.
Our test group was 10684.
8504 were delivered within our time frame.
A. 3232 were delivered, 798 were opened.
B. 2806 were delivered, 674 were opened.
C. 2466 were delivered, 644 were opened.
Looking at the raw opens the winner was A. However, based on open rates, C was the true winner followed by A.
Open Rate
A. 24.69%
B. 24.02%
C. 25.70%