Block search engine crawlers to access downloadable assets (pdfs, etc) | Community
Skip to main content
sahil_1512
sahil_1512
New Participant
March 15, 2024
Solved

Block search engine crawlers to access downloadable assets (pdfs, etc)

  • March 15, 2024
  • 2 replies
  • 1999 views

Hi Everyone,

Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Since the assets are gated via Marketo forms...so I don't want anyone to get the direct access to them.

Best answer by SanfordWhiteman
Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Yes, set up your robots.txt accordingly, upload it to Design Studio, and create a Redirect Rule so it appears as pages.example.com/robots.txt.

2 replies

SanfordWhiteman
SanfordWhitemanAccepted solution
New Participant
March 15, 2024
Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Yes, set up your robots.txt accordingly, upload it to Design Studio, and create a Redirect Rule so it appears as pages.example.com/robots.txt.

Darshil_Shah1
Darshil_Shah1
Community Manager
March 15, 2024

Well, people will be able to access the files if they have direct URLs to your assets (e.g., someone who filled out the form sent them); however, since your assets are gated behind Marketo forms, crawlers and search engines won't be able to access the content itself. The crawlers follow links to discover and index web pages. Crawlers won’t submit the form, so they won't be able to access the gated content behind the form.

Terms & ConditionsAccessibility statement

Loading footer...