none
Can I make interval of Web.Contens() ?

    Question

  • Can I make interval of Web.Contens()  ?

    I want to wait seconds for each Web.Contents() functon,
    Some Web sites block client ip address which make a lot of request per minutes.
    also some open API services,

    Example:
    I want to get 100 XML contents at same site,
    but the site block at over 30 request / second.

    Regards.
    Yoshihiro Kawabata.

    Friday, September 27, 2013 1:22 PM

Answers

  • This function will act like a wait:

    = (seconds as number, action as function) => if (List.Count(List.Generate(() => DateTimeZone.LocalNow() +
    #duration(0,0,0,seconds), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()

    Let's call this query "Wait." If I want to make a call to Web.Contents every so often, I can now do:

    Wait(time, () => Web.Contents(myPage))

    For example, to get the contents from Bing every 0.5 seconds, we would do

    Wait(0.5, () => Web.Contents(www.bing.com))

    If you are curious about how the wait function works:

    • List.Generate creates a list with the values which satisfy the condition (DateTimeZone.LocalNow() < x).
    • The initial value is the current time plus the amount of time we want to wait. This is the time when we want to perform the action.
    • The condition checks if we have passed the time when we want to perform the action. If we haven't, we continue generating the list.
    • This list will be completed once the desired amount of time has passed. There is at least one value in the list, so the if statement will be false (as long as we wait more than 0 seconds).
    • Because the if statement is false, we will evaluate the function.

    You could also use a recursive function to perform the wait.

    -Alejandro Lopez-Lago (MSFT)

    Thursday, October 03, 2013 8:35 PM

All replies

  • This function will act like a wait:

    = (seconds as number, action as function) => if (List.Count(List.Generate(() => DateTimeZone.LocalNow() +
    #duration(0,0,0,seconds), (x) => DateTimeZone.LocalNow() < x, (x) => x)) = 0) then null else action()

    Let's call this query "Wait." If I want to make a call to Web.Contents every so often, I can now do:

    Wait(time, () => Web.Contents(myPage))

    For example, to get the contents from Bing every 0.5 seconds, we would do

    Wait(0.5, () => Web.Contents(www.bing.com))

    If you are curious about how the wait function works:

    • List.Generate creates a list with the values which satisfy the condition (DateTimeZone.LocalNow() < x).
    • The initial value is the current time plus the amount of time we want to wait. This is the time when we want to perform the action.
    • The condition checks if we have passed the time when we want to perform the action. If we haven't, we continue generating the list.
    • This list will be completed once the desired amount of time has passed. There is at least one value in the list, so the if statement will be false (as long as we wait more than 0 seconds).
    • Because the if statement is false, we will evaluate the function.

    You could also use a recursive function to perform the wait.

    -Alejandro Lopez-Lago (MSFT)

    Thursday, October 03, 2013 8:35 PM
  • Thank you, Alejandro .

    Wait function work fine.
    I can call Web API 1,000 times without block by web site.

    DEFINITOON : Wait fuction

        Wait = ( seconds as number, action as function) =>
          if List.Count(
              List.Generate (
                  () => DateTime.LocalNow() + #duration(0,0,0, seconds),
                  each DateTime.LocalNow() < _ ,
                   each _)
              ) = 0
          then null
          else action(),

    USAGE : Wait function

        s0 = Wait ( 0.5, () => Web.Contents( xxx )),

    Web API report on my Facebook,
    use Yahoo! JAPAN Web API,
    https://www.facebook.com/photo.php?fbid=10151888379514394&l=9b751e1788

    // I wish Wait function without CPU/Memory usage.

    Regards,
    Yoshihiro Kawabata



    Friday, October 04, 2013 3:00 AM