Does anyone run A/B tests without relying on Marketo A/B test native randomization capabilities? | Community
Skip to main content
A
Andy2023
New Participant
January 7, 2021
Question

Does anyone run A/B tests without relying on Marketo A/B test native randomization capabilities?

  • January 7, 2021
  • 1 reply
  • 3001 views

Hey everyone,

 

Working now with our data team on the best way for us to split the audience for the email A/B test campaigns. Until now we’ve been randomizing people for that purpose in our CRM using signup time and/or native A/B testing capabilities of Marketo when sending emails.

 

We’ve noticed though that it might be not ideal as some segments of our clients were overrepresented in one group vs another and this skewed success metrics since different segments are treated differently by sales. In general, different segments use our product differently so if they aren't represented roughly equally in A and B then we'll have an issue.

 

I there anyone here who has experience spliting their audiences into A and B themselves and how do you do that? Which formulas, algorithms do you use?

 

Any insights are appreciated.

 

Thank you!

Andy

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

SanfordWhiteman
SanfordWhiteman
New Participant
January 8, 2021

Native A/B is a uniform distribution, so not sure what kind of skew you're concerned about.

 

However if you want to shuffle leads in a specific way, so there is a skew based on certain characteristics, you could use a webhook-powered service. You can also use a  webhook for strict round-robin distribution, which of course will be non-random and just based on order of execution.

A
Andy2023Author
New Participant
January 12, 2021

The concern with our previous experiments was the skews to our KPIs because different segments of our clients weren't represented equally in both groups.

 

What do you mean by webhook-powered services that can randomize on certain characteristics?

 

Can you give an example of a webhook that's used for strict round-robin distribution?

SanfordWhiteman
SanfordWhiteman
New Participant
January 13, 2021
The concern with our previous experiments was the skews to our KPIs because different segments of our clients weren't represented equally in both groups.

To be clear: true randomization could never ensure that any particular characteristic is always equally distributed across 2 groups, regardless of the number of items. It's a clear sign of a broken/deceptive (P)RNG is if some arbitrary value appeared to be exactly round-robined across the groups!

 

What do you mean by webhook-powered services that can randomize on certain characteristics? Can you give an example of a webhook that's used for strict round-robin distribution?

Any service that uses a shared counter can distribute people into a fixed number of buckets, A-B-A-B-A-B. If you want to sample across the entire database then you'd need to run people through the webhook twice (once to preload or prime the database, then again to sample from it).

Terms & ConditionsAccessibility statement

Loading footer...