A/B Testing & Statistical Significance | Community
Skip to main content
A
Anonymous
June 24, 2014
Solved

A/B Testing & Statistical Significance

  • June 24, 2014
  • 6 replies
  • 3673 views
I'm a relatively new user, so be gentle!

I'm wondering if there is a way to do A/B testing in Marketo such that the system automatically sends the winning email the moment testing identifies a winner with a predetermined level of statistical significance.  In other words, the standard method has us identify a pre-set % of our target list to send the email.  Say we have a list of 10,000...we can select a winner by sending the variations randomly to 20%, or 2,000.  However, this could be a wasted opportunity.

Imagine an extreme (and extremely simplified) scenario where 100% open version A and 0% open version B.  In that case (without doing any math), let's imagine that we can be 95% confident after seeing the results from only 500 emails (250 for A and 250 for B), that A is superior.  At that point, if we continue sending out B for the purposes of confirming what we already know, we're effectively wasting 750 opportunities(half the remaining 1,500 sample).

So my question is this: is there a way for marketo to choose a winner based on statistical significance if that can be established prior to the pre-determined sample size?

Thanks you in advance,
Dave Gimpel
    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by
    I'll be gentle: No, there is not.

    But you should suggest it as an idea in the community.

    6 replies

    P
    PhillipWi1
    New Participant
    February 24, 2016

    Hi David

    As others have mentioned, there is no way to do this within Marketo - and I haven't seen it in other marketing automation software either. I love that you are into the subject since A/B testing is one thing, A/B testing with proper science behind it is completely another.

    One thing Marketo lacks that is much simpler to implement is a p value calculator for the results. We always see "A is the winner since it had 1100 opens, while B only had 1050 opens". But in reality, as you probably know, that result might be completely meaningless if the sample size wasn't large enough, and might be purely random variation.

    Here's a calculator I use to determine whether my results are statistically significant after the test is run: A/B Test Calculator: MeasuringU

    Estimating the sample size, as Rhianne Makovec​ mentioned, can be done with online calculators. But - this requires estimating the proportions ahead of time, which is a bit of educated guesswork. So it's always worthwhile analysing the results for statistical significance after the fact to make sure the effect you see is likely the result of the test.

    Phil

    A
    Anonymous
    March 3, 2015
    Hi David / All,

    Athough you cannot detersome a statistically significant sample size in Marketo, if it helps anyone, here are some online tools and calculations I am using to determine sample size (audience %) to put into Marketo.

    I am going to use an external sample size calculator. Here is one: http://www.surveysystem.com/sscalc.htm#one. 
    An email A/B testing article suggested using a confidence interval (margin of error) of 5%, and the 95% confidence level. After you get the sample size by the total number of emailable contacts in your list. If you use these numbers, and 60% of your sample opens Email A, you can be 95% sure that between 55% (60 minus 5) and 65% (60 plus 5) would have also opened Email A.

    Hope this helps someone! 

     
    A
    Anonymous
    June 25, 2014
    That is a bummer, and I think I will.

    Thanks!
    A
    Anonymous
    Accepted solution
    June 24, 2014
    I'll be gentle: No, there is not.

    But you should suggest it as an idea in the community.
    A
    Anonymous
    June 24, 2014
    Respectfully, that didn't answer my question at all.  I do know how to set up a simple A/B test (including scheduling and defining the winner criteria), but that wasn't the point of my original question.  At your convenience, please take a second look at my previous email and if you require clarification, please let me know where I'm being unclear.

    Thanks,
    Dave
    A
    Anonymous
    June 24, 2014
    Email Programs.

    Add one: https://community.marketo.com/MarketoArticle?id=kA050000000LC8FCAW
    Schedule one: https://community.marketo.com/MarketoArticle?id=kA050000000LCSVCA4
    Define one: https://community.marketo.com/MarketoArticle?id=kA050000000LCMXCA4

    If you do a search for Email Programs A/B Testing, you should be able to pull up all of the other references as there are many that you'll want to go through in order to gain an understanding of how it all comes together.  
    Terms & ConditionsAccessibility statement

    Loading footer...