A/B test reports are not making sense | Community
Skip to main content
T
ThaisMacedo
New Participant
July 12, 2022
Question

A/B test reports are not making sense

  • July 12, 2022
  • 1 reply
  • 2443 views

Hello, Marketo Community!

 

I'm having trouble figuring out how many people received an email that includes an A/B test. 

 

To give context, this triggered email went out for a few weeks without a challenger, then I added an AB test 50/50 around 1 month ago. 

 

If I pull out an email performance report, I get (sent):

Total: 6,523 

Version A: 14
Version B: 24

 

Now, that's when things start to get weird: if I create a smart list with the filter "was sent email", I get:

Total: 3,943

Version A: 981

Version B: 97

 

Do you have any idea why these numbers are so different? Is this a bug or am I doing something wrong?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

C
ChristianeRo1
Community Manager
July 12, 2022

How long has your email been going out? The Send Email activity in a person's record is only retained for 90 days, so, depending on how long your email has been going out, you may see a discrepancy between the report and your smart list. You'll still see data in the email performance report, but "Was sent email" data over 90 days can't be pulled with a smart list.

T
ThaisMacedoAuthor
New Participant
July 13, 2022

 Thank you for the answer. I think it solves part of the mystery (why there are more people on the smart list than on the report when we look at total numbers).

 

But still, there are more people in the smart list than on the report for each variant, which goes against this logic.

 

Darshil_Shah1
Darshil_Shah1
Community Manager
July 13, 2022

Hi @thaismacedo, 

 

Apologies if I'm missing out on anything here, but how are you splitting the email metrics by the test variants in a single email performance report? AFAIK the email metrics for all the test variants roll-up to the email asset itself (and are not broken by the test variants). There's also an idea on the community to add the ability to split the email performance data by the test variants.

 

Terms & ConditionsAccessibility statement

Loading footer...