100% Test Sample Size in Email A/B Testing | Community
Skip to main content
Chelsey_Davis
Chelsey_Davis
New Participant
February 21, 2018
Solved

100% Test Sample Size in Email A/B Testing

  • February 21, 2018
  • 1 reply
  • 4174 views

Howdy!

I have a quick question...

I have an email test with 2 subject lines and 2 tweaks to copy ( a 2 x 2 test) with a total population of 160,000 emails. I want to set the test up like this (whole email test):

Email 1 (Subject Line A, Copy A)

Email 2 (Subject Line B, Copy A)

Email 3 (Subject Line B, Copy A)

Email 4 (Subject Line B, Copy B)

If I'm just looking for data and not necessarily a 'winner' and will be uploading a static list for this test, can I set the test sample size to 100% without any issues?

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Devraj_Grewal

Chelsey,

The easiest way to ensure this is done correctly is to just duplicate the emails so there are 4 in total, each with their own subject line/copy combination. Then create a smart campaign and have a flow step like this:

1. Send Email

     Choice 1: if random sample is 25%: send Email 1

     Choice 2: if random sample is 25%: send Email 2

     Choice 3: if random sample is 25%: send Email 3

     Choice 4: if random sample is 25%: send Email 4

     Default Choice: do nothing

This ensures every person in the list is only sent one variation of the email and each email is sent to about the same size of the list (25% of 160,000).

1 reply

Devraj_Grewal
Devraj_GrewalAccepted solution
New Participant
February 21, 2018

Chelsey,

The easiest way to ensure this is done correctly is to just duplicate the emails so there are 4 in total, each with their own subject line/copy combination. Then create a smart campaign and have a flow step like this:

1. Send Email

     Choice 1: if random sample is 25%: send Email 1

     Choice 2: if random sample is 25%: send Email 2

     Choice 3: if random sample is 25%: send Email 3

     Choice 4: if random sample is 25%: send Email 4

     Default Choice: do nothing

This ensures every person in the list is only sent one variation of the email and each email is sent to about the same size of the list (25% of 160,000).

Chelsey_Davis
Chelsey_DavisAuthor
New Participant
February 21, 2018

Thanks, Devraj. I was trying to keep it in an email program, but your route seems like a cleaner option.

Devraj_Grewal
Devraj_Grewal
New Participant
February 21, 2018

It's much easier for reporting purposes as well just to have each email variation created as a separate email then you don't need to mess with the test variant constraints.

Terms & ConditionsAccessibility statement

Loading footer...