Is there anything I can do to speed the getMultipleLeads API call?

Forum|Forum|13 years ago
January 17, 2013
16 replies
2534 views

I'd like to stay current on lead changes, so I've set up an app to call getMultipleLeads on an hourly basis, with a last modified threshold of an hour. Or half an hour. Or even ten minutes - it still usually takes over an hour to get any results. If I pull as little as the last three minutes' updates, this may complete in just five minutes. Obviously at that rate I won't be able to keep current.

We do have a fairly high volume of touches, as our data is synched with Salesforce on a near-real time basis. Still, a report in the web GUI for all leads updated in the last hour returned "about 377" records (though it took several minutes to return the first page of results). Surely 377 records shouldn't take minutes to process!

What can I do to keep current on lead changes? Is there a way to speed getMultipleLeads, or is there a better approach I should be using?

To be clear, the span of time I'm measuring is literally the length of time for the getMultipleLeads method to return. I'm using very verbose logging to pin this down.

Marketo

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Best answer by

You can't do UI Smart Lists, but you can use recurring batch campaigns to populate the static lists.

Are you pulling all the activities for each lead? That is probably the core area where we can speed this up. There is a getLeadChanges API call which gives you new activities. Many folks use that for capturing what is needed. Even better, filering out some activity types will help. This is probably the most important thing to look at.

You can have multiple connections to the Marketo API at once, although keep it in the lower numbers (less than 5).

-Erik

Show previous replies

A

Anonymous

Also, if you're making a call to fetch activity every time a lead changes (and 100k changes per hour) you're probably making a ton of API calls. You know the limit is at 10K/day right? It is a soft limit, but with what you described you may be going waaay over.

Hopefully getLeadChanges will help with all of that.

A

Anonymous

Accepted solution

You can't do UI Smart Lists, but you can use recurring batch campaigns to populate the static lists.

Are you pulling all the activities for each lead? That is probably the core area where we can speed this up. There is a getLeadChanges API call which gives you new activities. Many folks use that for capturing what is needed. Even better, filering out some activity types will help. This is probably the most important thing to look at.

You can have multiple connections to the Marketo API at once, although keep it in the lower numbers (less than 5).

-Erik

A

Anonymous

For now, I'd like to capture changes to all leads. If we were to reduce this to a subset (maybe only high-score leads?), could we use a smart list generated in the Marketo web GUI? I see the LeadSelector parameter, but it seems to be limited to filtering by date or "static list", which based on the name I assume is distinct from a smart list.

Another option might be to update leads only daily, but activities hourly. Unfortunately activities are taking more time than leads, even with an activityFilter in place. There doesn't seem to be an equivalent to includeAttributes for the getLeadActivity method. I am calling getLeadActivity once each for the tens-of-thousands of leads identified as having changed, which entails a lot of overhead; is there a method I've missed which would let me query activities posted in the last hour, for all leads?

I like your multi-threaded suggestion. I've generally avoided doing so with APIs, as most use some kind of authentication token which cannot be used for multiple simultaneous queries, but if the Marketo API is cool with that, that opens up some posibilities! I'll check that out next week.

Thank you for your suggestions, I'm hopeful that this will get me where I need to be.

A

Anonymous

Well, 100k an hour is pretty high volume, so getting some kind of backlog in those cases seems possible. Is the goal to sync all leads, or do you only really need a subset?

Also, you could possibly try a multi-threaded approach where you use the timeframe constraints on getMultipleLeads to have different threads taking different time slices.

A

Anonymous

That (setting parameters.includeAttributes and parameters.activityFilter.includeTypes) has helped, but it's still extremely slow. Examining leads updated in the last hour still often takes longer than 58 minutes, considering both leads and their activities. Just 10,000 leads takes around ten minutes, and a typical hour sees 10 to 100K updated leads.

Is there a more scalable approach I should be using? Or are daily updates the only realistic option?

A

Anonymous

One piece that can really help is reducing the number of fields you need. That can make things quite a bit quicker.

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded