Robots.txt in AEM codebase

Forum|Forum|8 years ago
May 3, 2017
3 replies
2481 views

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Best answer by Jitendra_S_Toma

Hi CQUser1,

The purpose of the Robots.txt is to provide details about the application to the search engine. One of the thing is to instruct web crawlers that a particular path is not index-able (Disallow: /).

i hope this helps.

cquser1 wrote...

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: * Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

cquser1Author

New Participant

Hi Amit/Jitendra,

Thank you for your replies.

Jitendra_S_TomaAccepted solution

New Participant

Hi CQUser1,

The purpose of the Robots.txt is to provide details about the application to the search engine. One of the thing is to instruct web crawlers that a particular path is not index-able (Disallow: /).

i hope this helps.

cquser1 wrote...

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: * Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

Amit_Kumar

New Participant

Hi,

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Red more here: https://moz.com/learn/seo/robotstxt

Regards,

Amit

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded