Activity Extract Options. Bulk vs Batch
Option 1 – “Batch API”
The Batch API solution allows you to extract multiple activity types (listed here ) over a long interval at your discretion. Each API call returns one page of data and the following. The first call requires you to create a paging token. https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#/Activities/getActivitiesPagingTokenUsingGET For subsequent calls to this endpoint, use the nextPageToken returned from the response. This endpoint will always return the nextPageToken.
- A “next page token” which represents the subsequent page of data.
- A “moreResult” Boolean. You’ll loop through until a “false” is returned, indicating that there is no more data to be returned.
More information is here https://developers.marketo.com/rest-api/lead-database/activities/ and the API specifics are here https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#/Activities
Restrictions
- See below
Option 2 – “Bulk API”
The Bulk Activity API may be used to extract up to 500mb of data per day, and up to a date range of 31 days in each call. the Bulk API has a very different process than the batch. At a high level it is as follows
- Create the Bulk Extract job (API call #1, returns the job ID)
- Enqueue the Bulk API Job using the job ID
- Poll the API using the Job ID until a status of “completed” is returned. This indicates the full CSV activity extract is ready to be streamed from the Marketo server. It will remain there for 7 days.
- Stream the file down (API call #4)
You can have up to 10 jobs enqueued, two will run concurrently. A good practice is to extract them by calendar month. One call for January, One call for February, etc.
More detail is available here https://developers.marketo.com/rest-api/bulk-extract/bulk-activity-extract/ and the API endpoints are here https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#/Bulk_Export_Activities
Restrictions
- Limited to 500mb of data per day
- Limited to extracting a range of 31 days per call
- Limit resets at 12 midnight US Central Time
Helpful links
Blog: https://nation.marketo.com/t5/Product-Blogs/Activity-Types-Attributes-and-Exporting/ba-p/244377
Activity types: https://docs.google.com/spreadsheets/d/1i-3GVbpUK7oc8YanOokY_1yd_Ly124urK2HHydrN0d8/edit#gid=1253153029
Retention Policy: https://nation.marketo.com/t5/Knowledgebase/Marketo-Activities-Data-Retention-Policy-Overview-amp-FAQ/ta-p/250750