Lead enrichment - how many webhook calls is too many? | Community
Skip to main content
Marc_Asmus
Marc_Asmus
New Participant
September 3, 2021
Solved

Lead enrichment - how many webhook calls is too many?

  • September 3, 2021
  • 1 reply
  • 2905 views

I'm interested in enriching every lead in my Marketo database daily with data from a different tool within our tech stack. 

 

I'm running into an issue with the native integration between this tool and Marketo that's causing duplicates to be created within Marketo. So, I'm exploring alternatives. 

 

One idea I have is to configure Marketo to "ask" the tool for the data about each lead. Because Marketo would only ask for data about leads who already exist in Marketo, I believe I'd eliminate the risk of duplicates. I'm looking at leveraging Marketo webhooks to make this happen. Going this route would mean sending tens of thousands of webhooks from Marketo to the tool every day. I'd schedule so that this process ran overnight or on the weekends when there's less happening within our Marketo instance. 

 

  • How many webhooks are too many? What are the limitations?
  • Is this a bad idea? Why?
  • Ideas for a better way to solve?
This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by SanfordWhiteman
There's no hard limit. It depends on how fast each call returns (up to the max of 30s) because only 20 connections can be open simultaneously.

If your calls respond in under 5s I wouldn't worry about making tens of thousands per day. Gets a little hairy above 50,000.

1 reply

SanfordWhiteman
SanfordWhitemanAccepted solution
New Participant
September 3, 2021
There's no hard limit. It depends on how fast each call returns (up to the max of 30s) because only 20 connections can be open simultaneously.

If your calls respond in under 5s I wouldn't worry about making tens of thousands per day. Gets a little hairy above 50,000.
    Marc_Asmus
    Marc_AsmusAuthor
    New Participant
    September 3, 2021

    Thanks for the reply @sanfordwhiteman!

     

    Does Marketo provide visibility into how fast calls return?

     

    I ran a few tests through Postman that suggest these calls take <1 second.

     

    Can you describe what gets hairy above 50k calls? Is it that Marketo needs time to process all the calls and could delay subsequent flow steps or other processes?

     

    Assuming 1 second/call...

    100k calls * 1 sec/call = 100k seconds = ~1,667 minutes = ~28 hours

     

    Does 20 simultaneous connections mean I could divide that ~28 hours by 20 to estimate those 100k calls might take ~1.4 hours?

     

    Related:

    https://developers.marketo.com/webhooks/

    Webhook execution time is at the mercy of the response time of the service being used and can result in long campaign execution delays.  Even if a service only takes 50ms to execute, that is 1.5 hours when executed 100,000 times.

     

     

    SanfordWhiteman
    SanfordWhiteman
    New Participant
    September 4, 2021

    Yes, divide by 20!

     

    The problems arise when you're running people through a daily campaign and the campaign takes nearly a day (let alone more) to finish! You want to keep your runs under ~4h, whatever they do, to make sure you can deliver usable results every day.

      Terms & ConditionsAccessibility statement

      Loading footer...