AEM Dispatcher Logs: How to add extra data to dispatcher logs? | Community
Skip to main content
Anonymous
Aug 9, 2017 9:09 PM UTC
Solved

AEM Dispatcher Logs: How to add extra data to dispatcher logs?

Background/Use-case: We are using AEM6.1; have configured AEM to inject log to Splunk tool.

For creating SPLUNK dashboards, we need  custom/additional data from dispatcher logs.The  Splunk expects data in a specific format, to filter data from AEM logs- KEY:VALUE pair format(path:/content/us/en_us/testPage.html)

ASK:

Can we modify Dispatcher config/service to add new KEY:VALUE pair?  appreciate any suggestions/ideas to support the above use case.

Thanks

Guru

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    joerghoh
    joerghohAccepted solution
    Employee
    Aug 9, 2017 9:17 PM UTC

    This is not possible.

    But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

    In the httpd.conf you can use something like this:

    DispatcherLog "|/path/to/my/script"

    Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

    Jörg

    View reply

    9 replies

    MaxBarrass
    MaxBarrass
    Level 3
    Jun 25, 2019 7:00 AM UTC

    I have configured my dispatcher container to return logs like this, which should suit any current log management tools

    https://hub.docker.com/r/aemdesign/dispatcher

    ### Container Logs

    All apache and dispatcher logs are piped to container stdout, following is an example of ```docker logs -f publish-dispatcher```

    ```
    {"@timestamp":"2019-06-25 13:09:31.317603", "level":"notice", "module":"suexec", "process":"6", "message":"AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.317775", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.317797", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.344707", "level":"notice", "module":"auth_digest", "process":"6", "message":"AH01757: generating secret for digest authentication ...", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.345385", "level":"notice", "module":"lbmethod_heartbeat", "process":"6", "message":"AH02282: No slotmem from mod_heartmonitor", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.345490", "level":"warn", "module":"ssl", "process":"6", "message":"AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.345497", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.345510", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.349115", "level":"notice", "module":"mpm_prefork", "process":"6", "message":"AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Communique/4.3.2 configured -- resuming normal operations", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.349154", "level":"info", "module":"mpm_prefork", "process":"6", "message":"AH00164: Server built: Apr 24 2019 13:45:48", "host":"dispatcher",
    {"@timestamp":"2019-06-25 13:09:31.349174", "level":"notice", "module":"core", "process":"6", "message":"AH00094: Command line: 'httpd -f /dispatcher/httpd/httpd.conf -D FOREGROUND'", "host":"dispatcher",
    {"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"IP list for backend rend01 will be resolved on every request."}
    {"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"Dispatcher initialized (build 4.3.2/apache2.4-linux-x86_64)"}
    {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "query_string":"", "referer":"-", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2375", "bytes_out":"8486", "keepalive":"+", "duration_micros":"4810243", "duration_sec": 4, "status":"200", "short_message": "GET /content/aemdesign-showcase/au/en/component/lists/page-list.html HTTP/1.1" },
    {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "status":"200", "size":"158170", "duration":"4809ms", "renderer":"publish/rend01"}
    {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"22320", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
    {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"22ms", "renderer":"publish/rend01"}
    {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/libs/granite/csrf/token.json", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"application/json", "bytes_in":"2260", "bytes_out":"348", "keepalive":"+", "duration_micros":"17257", "duration_sec": 0, "status":"200", "short_message": "GET /libs/granite/csrf/token.json HTTP/1.1" },
    {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/libs/granite/csrf/token.json", "status":"200", "size":"2", "duration":"17ms", "renderer":"publish/rend01"}
    {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"10906", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
    {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"11ms", "renderer":"publish/rend01"}
    ```

    #### Log Format
    - All records with ```@timestamp``` are from apache generated using CustomLog and ErrorLog

    - All records with ```@date``` are from apache Dispatcher module generated using DispatcherLog

    - Apache can spit out non JSON formatted logs into STDOUT

    Translate
    joerghoh
    joerghoh
    Employee
    Jan 16, 2018 8:57 PM UTC

    You can feed logfiles into Splunk.

    Getting data in | Splunk

    Jörg

    Translate
    paull10780674
    New Participant
    Jan 16, 2018 10:13 AM UTC

    May I know how to configure AEM to inject log to Splunk tool?

    Translate
    Anonymous
    Aug 18, 2017 4:54 PM UTC

    All

    I was able to solve the issue by using customLog.

    Solution:

    LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"urlpath\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"True-Client-IP\":\"%{True-Client-IP}i\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" splunk

    CustomLog /cq/log/exp-splunk.log splunk

    Translate
      ajayk94129834
      ajayk94129834
      New Participant
      May 7, 2020 8:29 PM UTC

      Hi,

       

      I am trying to implement custom log to capture details in to splunk and this article completely aligned with my requirement.

      Could you please share the custom log script it will be helpful to me.

       

      Thank you.

      Translate
      Anonymous
      Aug 9, 2017 11:00 PM UTC

      great, thanks Jorg.

      Translate
      joerghoh
      joerghoh
      Employee
      Aug 9, 2017 9:39 PM UTC

      DispatcherLog "| tee -a myLogfile | /path/to/my/script"

      Will still write to myLogfile and also forward it to your script. "tee" is a standard unix command, please see its manpage for details.

      Jörg

      Translate
        Anonymous
        Aug 9, 2017 9:36 PM UTC

        Thanks Jorg, for the quick response.

        Is it possible to log to log file & to  a program? so I can keep the current flow & in my program I'll just extract specific dataset.

        Translate
        joerghoh
        joerghohAccepted solution
        Employee
        Aug 9, 2017 9:17 PM UTC

        This is not possible.

        But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

        In the httpd.conf you can use something like this:

        DispatcherLog "|/path/to/my/script"

        Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

        Jörg

        Translate
          Terms & ConditionsAccessibility statement

          Loading footer...