AEM 6.5 - Binary Less replication in Shared S3 datastore | Community
Skip to main content
V
Vikashyadav
New Participant
July 9, 2020
Solved

AEM 6.5 - Binary Less replication in Shared S3 datastore

  • July 9, 2020
  • 1 reply
  • 2489 views

Hi,

We do have customized AEM asset portal that uses shared S3 datastore. While replicating assets in bulk amount(600 assets) from author to publish via binary-less replication we have observed that publisher becomes unresponsive and start giving oops page(Error : 500). Publisher automatically recovered when replication queue items reduced. For instance, within 30 sec. author makes around 300-400 POST request to publisher to publish the assets. Also publisher makes request to dispatcher flush to clear cache from dispatcher. Sometimes we observed around 1500+ items in replication queue which makes publisher unresponsive. 

 

How we can  increase throughput of binary-less replication?

Is there any way to make replication in chunks or batches?

How we can tune-up our publisher to handle bulk request?

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by joerghoh

Hi,

your scenario is not really clear to me. I understand that you are replicating hundreds of assets in bulk and that during that time the publish instance is unable to respond to other incoming requests.

 

Let me comment some of your statements:

  • within 30 seconds 400-500 POST requests: That makes me think that the replication itself is quite quick. How do you trigger the replication on author? Using the standard "replicator.replicate()" method or do you use synchronous replication with many threads in parallel? The first one will only replicate one asset at a time, while in the second case you will have multiple replication requests at once.
  • You should validate the dispatcher logs and see what is happening. Do a few threaddumps and check what's happening. And what is the exception message of these "internal server error" requests you mentioned?

If you can answer these questions we can help you further.

 

Jörg

 

1 reply

joerghoh
joerghohAccepted solution
Employee
July 9, 2020

Hi,

your scenario is not really clear to me. I understand that you are replicating hundreds of assets in bulk and that during that time the publish instance is unable to respond to other incoming requests.

 

Let me comment some of your statements:

  • within 30 seconds 400-500 POST requests: That makes me think that the replication itself is quite quick. How do you trigger the replication on author? Using the standard "replicator.replicate()" method or do you use synchronous replication with many threads in parallel? The first one will only replicate one asset at a time, while in the second case you will have multiple replication requests at once.
  • You should validate the dispatcher logs and see what is happening. Do a few threaddumps and check what's happening. And what is the exception message of these "internal server error" requests you mentioned?

If you can answer these questions we can help you further.

 

Jörg

 

    V
    VikashyadavAuthor
    New Participant
    July 10, 2020

    Hi Jorg,

     

    Your understanding is correct. We have one Author and one Publish instance sharing S3 datastore.

    • Author instance has scheduler which is running every hour and pull assets from S3 bucket, process it and replicate using "replicator.replicate()" method to publish. This replication is one at a time.
    • We'll analyse the dispatcher logs and share our findings.
    • This bulk replication was working fine when we were running on AEM 6.3. Post up-gradation to AEM 6.5 we started facing this issue.

     

    Terms & ConditionsAccessibility statement

    Loading footer...