Asynchronous Data Import | Community
Skip to main content
New Participant
October 8, 2020
Solved

Asynchronous Data Import

  • October 8, 2020
  • 1 reply
  • 2045 views

In Marketo, 10 API call can be processed simultaneously (2 parallel and 8 queue) but business need is to do bulk import (more than 10 API). Does Marketo support team allows Asynchronous process to archive this function.  The current Synchronous process is taking 2 to 3 min for 10 MB file but business requirement is to speed up the process and looking for suggestion. Asynchronous process can be helpful but any other suggestion will be welcome.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by SanfordWhiteman

You're using "asynchronous" in a sort of confusing way, because bulk import jobs are already asynchronous.

 

So "the current Synchronous process" doesn't exist.

 

That's why you can have a queue at all. Jobs are pulled off the queue asynchronously — otherwise, you wouldn't even get an HTTP response from a submission until each entire file is processed. The fact that you submit, then the back end does the jobs later is kind of the definition of asynchronous!

 

Now, if what you're asking is whether there can be some sort of other hierarchical queue, like a second-level queue, the answer is: Not within Marketo itself. You could certainly build your own queue that accepts an unlimited number of simultaneous jobs, and then pulls jobs off the queue only when Marketo signals that it has an open slot. As long as you submit to this second queue, it would appear to the client app as if there were no queue limit.

1 reply

SanfordWhiteman
SanfordWhitemanAccepted solution
New Participant
October 8, 2020

You're using "asynchronous" in a sort of confusing way, because bulk import jobs are already asynchronous.

 

So "the current Synchronous process" doesn't exist.

 

That's why you can have a queue at all. Jobs are pulled off the queue asynchronously — otherwise, you wouldn't even get an HTTP response from a submission until each entire file is processed. The fact that you submit, then the back end does the jobs later is kind of the definition of asynchronous!

 

Now, if what you're asking is whether there can be some sort of other hierarchical queue, like a second-level queue, the answer is: Not within Marketo itself. You could certainly build your own queue that accepts an unlimited number of simultaneous jobs, and then pulls jobs off the queue only when Marketo signals that it has an open slot. As long as you submit to this second queue, it would appear to the client app as if there were no queue limit.