Publishing Huge data to scene7 | Community
Skip to main content
A
asn11
New Participant
January 13, 2016
Solved

Publishing Huge data to scene7

  • January 13, 2016
  • 7 replies
  • 1473 views

Hi Team,

We are using CQ5.6.1

Our prod instance has got close to 30 GB of assets, and we are looking for ways to publish this huge data to scene7.

For starters we tried with close to 2 GB of assets in one of the lower environments.

 -It took 5 hours for data to get published to scene7.

- Though the data was around 2 GB, the CQ server went on occupying space exponentially.

- For publishing 2 GB of data the CQ instance space utilization grew by alarmingly atleast 18-20 GBs.

Why is it so?

We do not have importer (scene7 to DAM) enabled.

All we did was selecting a high level folder and click on publish to scene7.

 

It would be very helpful if I am guided how to deal with such huge data.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by mvanceadobe

The Scene7 uploader in 5.6.1 sends files one a time, so this explains that it takes a long time to go through the content to the Scene7 servers.

You may check the job logs in your SPS instance to see if there are any errors or warnings logged.

I don't have an answer about the space utilization topic

7 replies

M
mvanceadobeAccepted solution
Employee
January 15, 2016

The Scene7 uploader in 5.6.1 sends files one a time, so this explains that it takes a long time to go through the content to the Scene7 servers.

You may check the job logs in your SPS instance to see if there are any errors or warnings logged.

I don't have an answer about the space utilization topic

Kunal_Gaba_
Kunal_Gaba_
New Participant
January 14, 2016

My hunch is that when the scene7 publish job runs on DAM assets it writes some status metadata property on each asset to track the status of the publish operation. And if there are huge number of assets then the number of writes to the repository increases proportionately which leads to creation of new tarpm tar files on the file system. (As tar file is never overwritten, new data is always appended)

I guess if you run the tarpm optimization job after the scene7 publish is complete then you should be able to reclaim the extra redundant disk space which was consumed during the publishing. This is my guess but there could be other reasons for the growth during the publishing. 

https://helpx.adobe.com/experience-manager/kb/analyze-unusual-repository-growth.html

https://helpx.adobe.com/experience-manager/kb/TarPMOptimization.html

Jitendra_S_Toma
Jitendra_S_Toma
New Participant
January 14, 2016

Sorry man. If cloud storage isn't there, I don't think it would be right to proceed with this at this point of time. It was just a thought that in these situations, cloud storage could have a better option.

Here is one doc which talks about configuring Amazon S3 storage with CQ5.6.1

http://cq-ops.tumblr.com/post/69692603146/how-to-host-cq-data-store-on-amazon-s3

Jitendra

A
asn11Author
New Participant
January 14, 2016

Thank You Scott and Daniel for passing to right direction.

@Jitendra,

We do not have any cloud storage available in current setup.

Would want to know if its possible at all, with just CQ 5.6.1 and Scene7 account for company.

If yes,could you please point us to right direction.

Regards,

Jitendra_S_Toma
Jitendra_S_Toma
New Participant
January 14, 2016

@asn_177,

This is the one reason, we should look for data storage/Cloud for large amount of data. Keeping everything in our repository is always problematic in such cases.

Jitendra

smacdonald2008
smacdonald2008
New Participant
January 14, 2016

I passed this question to the S7 team. We should hear back soon. 

Daniel_Hollerun
Daniel_Hollerun
New Participant
January 13, 2016

I think this is a great question; one that I'd like to hear a response. However, I don't think it is a question of Target but rather Experience Manager.

Would it make sense to ask it here instead?

http://help-forums.adobe.com/content/adobeforums/en/experience-manager-forum/adobe-experience-manager.html

Terms & ConditionsAccessibility statement

Loading footer...