How long should the test duration for an a/b test be? | Community
Skip to main content
A
Anonymous
January 21, 2015
Solved

How long should the test duration for an a/b test be?

  • January 21, 2015
  • 5 replies
  • 4718 views
I have a email going out tomo to 20K ppl with an a/b test on the subject line.

I really don't want to wait too long to send out the winner bc I want it to go out tomorrow, not another day (it's related to a launch).

How long does Marketo wait before it declares an email has not been opened? I suppose that 1 hour might not be enough?

Thus, is 1 hour enough to reach the sample size of 10%? 

Thanks!
 
This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Dory_Viscoglio
It might have changed, but I believe the minimum time for the automatic winner email program sends was 4 hours at one point. If all of your recipients are in the US then Kim's suggestion of waiting about 6 hours should be good. Otherwise, you're not necessarily going to have a very representative sample of opens, and your final send might not go at a very desirable time which could impact the overall outcome. At that point, I'd probably wait 24 hours. 

5 replies

A
Anonymous
January 25, 2016

Hi, so there is no possible way to declare a winner after 30 minutes?

Dory_Viscoglio
Dory_Viscoglio
New Participant
January 25, 2016

Not if you're doing it through an email program. You can set it up manually and do it after 30 minutes, but I would advise against it. Just as food for thought, on Friday, 4 hours after an email send I saw a ~20% open rate (11am ET send). As of this morning it was 39%. Obviously this could have had some regionality associated with it, but 30 minutes won't always be very indicative of actual results.

A
Anonymous
January 21, 2015
Thanks everyone, helpful stuff!
Dory_Viscoglio
Dory_ViscoglioAccepted solution
New Participant
January 21, 2015
It might have changed, but I believe the minimum time for the automatic winner email program sends was 4 hours at one point. If all of your recipients are in the US then Kim's suggestion of waiting about 6 hours should be good. Otherwise, you're not necessarily going to have a very representative sample of opens, and your final send might not go at a very desirable time which could impact the overall outcome. At that point, I'd probably wait 24 hours. 
K
kpara3-2
New Participant
January 21, 2015
I agree with Josh, 1 hour isn't enough time. However, I know I get a lot of pushback from the team if I don't have the sends all in one day. So I usually have the test send early in the morning and give it a good 6 hours or so before the automatic winner is sent.
Josh_Hill13
Josh_Hill13
New Participant
January 21, 2015
I doubt 1 hour is enough. A good test would be 24-48 hours. I think marketo just waits the allotted time to declare a winner when you set it automatically. Whoever opens, opens.
Terms & ConditionsAccessibility statement

Loading footer...