Is there any way to check whether AB test is having flicker effect? | Community
Skip to main content
Ramya_SR
Level 2
August 11, 2023
Solved

Is there any way to check whether AB test is having flicker effect?

Traffic is split 50/50% between control and variation. The data analysis shows sample size mismatch. 

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    PratheepArunRaj
    PratheepArunRajAccepted solution
    New Participant
    August 14, 2023

    Dear Ramya,

    When you say flicker, are you saying that Control is loading more due to the flicker creating the size mismatch?

    If yes, it should not be the case since the traffic is counted only when the offer is delivered (Either it is control or variation).

    The unequal split is common during the initial days and settles over a week. Just continue with the test and see whether the numbers are normalizing in a week or so. If not, reach us back.

    Thank You, Pratheep Arun Raj B (Arun) | NextRow Digital | Terryn Winter Analytics

    View reply

    1 reply

    PratheepArunRaj
    PratheepArunRajAccepted solution
    New Participant
    August 14, 2023

    Dear Ramya,

    When you say flicker, are you saying that Control is loading more due to the flicker creating the size mismatch?

    If yes, it should not be the case since the traffic is counted only when the offer is delivered (Either it is control or variation).

    The unequal split is common during the initial days and settles over a week. Just continue with the test and see whether the numbers are normalizing in a week or so. If not, reach us back.

    Thank You, Pratheep Arun Raj B (Arun) | NextRow Digital | Terryn Winter Analytics

    Translate
      Terms & ConditionsAccessibility statement

      Loading footer...