Bulk Data extract | Community
Skip to main content
J
jay2
New Participant
March 26, 2024
Solved

Bulk Data extract

  • March 26, 2024
  • 2 replies
  • 2385 views

I have raised a support ticket to extract bulk records of approx 1M+ data from Marketo and Adobe support suggested to extract data via the bulk API as per the product limitation issue can not be extracted bulk data. (considering the schema file format contains 500+ fields names in the header)  

https://developers.marketo.com/rest-api/bulk-extract/bulk-lead-extract/ 

https://developers.marketo.com/rest-api/bulk-extract/ 

When I tried extracting bulk data via API the job status endpoints return a checksum in the fileChecksum attribute when status is “Completed”.

The checksum is a SHA-256 hash of the exported file. https://sha256sum.com/ 

I was wondering how to extract the data in such a case. I have not used any SHA-256 coding in the past. is there any workaround to download this checksum file?

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by SanfordWhiteman
I was wondering how to extract the data in such a case. I have not used any SHA-256 coding in the past. is there any workaround to download this checksum file?

The checksum is so you can be sure you downloaded the entire file.

 

This is usually not worth worrying about, but with very large files the chance of a network error (on either side) rises. And some HTTP clients can’t/won’t tell you if the connection was dropped prematurely, or they may not care if the length of the downloaded content matches the Content-Length header, etc.

 

If you run a checksum over the saved file, it must match the checksum on the server side exactly or else it’s a different file.

Note your app might be changing line breaks or perhaps adding/removing a BOM, in which case the files and thus checksums will be different, but at the business level the file is OK.

2 replies

SanfordWhiteman
SanfordWhitemanAccepted solution
New Participant
March 26, 2024
I was wondering how to extract the data in such a case. I have not used any SHA-256 coding in the past. is there any workaround to download this checksum file?

The checksum is so you can be sure you downloaded the entire file.

 

This is usually not worth worrying about, but with very large files the chance of a network error (on either side) rises. And some HTTP clients can’t/won’t tell you if the connection was dropped prematurely, or they may not care if the length of the downloaded content matches the Content-Length header, etc.

 

If you run a checksum over the saved file, it must match the checksum on the server side exactly or else it’s a different file.

Note your app might be changing line breaks or perhaps adding/removing a BOM, in which case the files and thus checksums will be different, but at the business level the file is OK.

Jo_Pitts1
Jo_Pitts1
Community Manager
March 26, 2024

@jay2 ,

which language are you developing in?

You shouldn't really need to worry about the checksum, but regardless knowing your dev stack will help immensely.

Regards

Jo

Terms & ConditionsAccessibility statement

Loading footer...