Bot Traffic Through Raw Full URL | Community
Skip to main content
skatofiabah
skatofiabah
Level 5
Oct 23, 2024 3:14 PM UTC
Question

Bot Traffic Through Raw Full URL

Hi All,

 

We have a system where we look at the Raw Full URL of the page, then the Operating System, then the IP Address in Adobe Analytics to identify bot traffic. We are getting a full URL with "?test=1" in the query string parameter that seems to be bot traffic. Are there other ways or query strings in the full raw URL that would indicate Bot Traffic too?

 

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

12 replies

Vernon_H
Level 2
Oct 24, 2024 12:53 AM UTC

Hi @skatofiabah,

 

Agreed that the query string parameter would not be indicative of a bot. Personally, what we use is a mixture of User Agent, IP address, Countries, Click behavioural data, Timestamp and page views events to help identify the larger bots. I would also check the IAB filter to ensure that it is turned on (though it doesn't capture new/smaller bots).

Cheers,

Vernon

Translate
    skatofiabah
    skatofiabahAuthor
    Level 5
    Oct 24, 2024 2:18 PM UTC

    Hi @vernon_h,

    What is User Agent? I don't see that dimension in Adobe. Are there any other ways to slice besides those dimensions above?

    Translate
    Vernon_H
    Level 2
    Oct 25, 2024 1:25 AM UTC

    Hi @skatofiabah,

    For more information on user agent, here's a documentation of the string format.

    We capture user agent as a seperate eVar via the JS variable navigator.userAgent. I would personally highly recommend having this captured (especially with the rise of crawlers to power Gen AI tools). 
    An example of an infamous bot from ByteDance that appeared on our radar earlier this year:  
    mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; spider-feedback@bytedance.com) 

    From the user agent alone, you can easily do a Google search or immediately tell its some sort of a spider from bytedance.com. So you can then do an exclusion in your reports for this. 
    Other smaller scale bots might use legit user agents, hence, you will need other dimensions and metrics to support your exclusion (e.g. link clicks actions, bounce rates, time spent on page, etc.).

    Hope this info helps! Goodluck!

     

    Cheers,

    Vernon

    Translate
      PratheepArunRaj
      PratheepArunRaj
      New Participant
      Oct 23, 2024 6:58 PM UTC

      Dear @skatofiabah ,

      '?test=1' need not be a bot ideally; rather, somebody is testing the page load in production (it might be your QC team) or trying to clear Akamai cache (usually, we use the query string parameter). 

      Pretty not sure that using query string parameter is a reliable method to identify BOT.

      Thank You, Pratheep Arun Raj B (Arun) | NextRow Digital | Terryn Winter Analytics

      Translate
        Terms & ConditionsAccessibility statement

        Loading footer...