AEM - ROBOTS.Txt file not showing content instead it's downloading | Community
Skip to main content
swamulu143
New Participant
Dec 8, 2021 9:44 AM UTC
Solved

AEM - ROBOTS.Txt file not showing content instead it's downloading

Hi Team,

Can you please help me with below issue.

We have created robots.txt file to deny access to crawlers on some paths but we are facing below issue.

 Steps we followed:

   1. Created robots.txt file in DAM path(/content/dam/folder)

   2. Added relevant content as per business rules

   3. Allowed robots.txt path by adding rules in filters& added rewrite rules in dispatcher and restarted apache server

   4. When we hit url(https:host:port/robots.txt) it's downloaded instead of opening.

 

Thanks,

Swamulu

   

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Kishore_Kumar_
Kishore_Kumar_Accepted solution
Level 9
Dec 8, 2021 11:31 AM UTC

Hi @swamulu143 ,

 

It might be due to content disposition, Can you please try adding a path in Apache Sling Content Disposition Filter as mentioned in below link.

 

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/how-to-allow-a-robots-txt-file-to-be-visible-on-browser-when/td-p/430829 

 

View reply

4 replies

arunpatidar
arunpatidar
New Participant
Dec 8, 2021 1:03 PM UTC

Hi,

Can you check if the response header, if it has this header then you can remove this header from httpd server for this file

 

Content-Disposition: attachment; filename="robots.txt"

 

Translate
Arun Patidar
    Raja-Karuppsamy
    Raja-Karuppsamy
    New Participant
    Dec 8, 2021 10:08 AM UTC

    Can you tell me what's the issue here ?

    Translate
      swamulu143Author
      New Participant
      Dec 8, 2021 11:17 AM UTC

      Hi Raja,

      Robots.txt file is downloading instead of opening file when we hit Url.

       

       

       

      Translate
        Kishore_Kumar_
        Kishore_Kumar_Accepted solution
        Level 9
        Dec 8, 2021 11:31 AM UTC

        Hi @swamulu143 ,

         

        It might be due to content disposition, Can you please try adding a path in Apache Sling Content Disposition Filter as mentioned in below link.

         

        https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/how-to-allow-a-robots-txt-file-to-be-visible-on-browser-when/td-p/430829 

         

        Translate
          Terms & ConditionsAccessibility statement

          Loading footer...