Interpreting AT test result when the control is winner in one version, and the test is successful in the other version. | Community
Skip to main content
B
bilalhowdoiuseA
New Participant
May 5, 2020
Solved

Interpreting AT test result when the control is winner in one version, and the test is successful in the other version.

  • May 5, 2020
  • 2 replies
  • 8823 views

Hi, 

 

I recently ran an A/B test on two versions of the same page - an english version and a chinese version - the results were pushed to Adobe Analytics. The results are the opposite i.e. the chinese version shows that the experience B (experiment) was the winner, whereas in the english version of the test, experience A (control) was the winner. The winner is based on a conversion metric..Point to note: the chinese version of the site receives significantly more traffic and conversion.

 

My question: 

- In such cases, what are some of the recommended decision rules to consider for determining if the test was successful or not?

- The answer may be that it depends, but what is the thought process or better approach to deciding if the test was successful or not? Thanks!

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Tanvi_Agarwal_

    @bilalhowdoiusea 

     

    Is conversion metric same for both the Chinese and English version?

     

    If yes , it simply says that visitors of both the versions have different liking.

    It may happen that Exp A is a winner in one region and Exp B is a winner in another region, so future decisions should be taken based on the region conversions.

     

    I am referring to regions as I understand that Chinese version is targeting only Chinese audience or region.

     

    Hope that helps to understand!

     

     

    2 replies

    Tanvi_Agarwal_
    Tanvi_Agarwal_Accepted solution
    Employee
    May 21, 2020

    @bilalhowdoiusea 

     

    Is conversion metric same for both the Chinese and English version?

     

    If yes , it simply says that visitors of both the versions have different liking.

    It may happen that Exp A is a winner in one region and Exp B is a winner in another region, so future decisions should be taken based on the region conversions.

     

    I am referring to regions as I understand that Chinese version is targeting only Chinese audience or region.

     

    Hope that helps to understand!

     

     

      B
      bilalhowdoiuseAAuthor
      New Participant
      May 21, 2020
      Thank you for the answer - i should have been more clear, my mistake.
      K
      karandhawan
      Employee
      May 20, 2020

      @bilalhowdoiusea Can you please confirm if the activity created on Target was an A4T(Analytics for Target) which uses Analytics as a reporting source?

      Also, was the metric used in the activity an Analytics metric?

      For comparing the results between two solutions, it is best to have integrations(A4T) in place.

       

      Regards,

      Karan

        B
        bilalhowdoiuseAAuthor
        New Participant
        May 21, 2020
        Yes, the activity was created on A4T and we used a custom analytics metric (form submissions) to compare.
        Terms & ConditionsAccessibility statement

        Loading footer...