Data Factory - Minimum AcquiringCompute Time RRS feed

  • Question

  • At the moment we are evaluating the feasibility to move our on-premises data infrastructure into the cloud. For this purpose we are testing the ADLS Gen2 and the ADF v2. Our understanding is that it will generally take 5-7 min to spin up a spark cluster to do data transformations. As this is to long our work around in the testing environment is to wake up our IR that is configured with some TTL with a dummy job. Subsequently, the actual job is started within that TTL. For the real job the AcquiringCompute time will still be around 1:20 min.    

    Now the question is whether this is the lower limit for the AcquiringCompute time or if there are strategies to cut this time down even further. In certain videos (links below) I got the impression that the time can be reduced to 10s. It would be nice to know how this can be achieved as it is critical to our decision wheter or not to move our infrastructure to azure.

    https://www.youtube.com/watch?v=5KUek4JfSYs (min 30:03)

    https://www.youtube.com/watch?v=iyZT5CY3V_4 (min 6:11)

    Any help is appreciated - Thank you very much in advance.

    Best Regards


    • Edited by TechBerg Thursday, August 27, 2020 1:57 PM
    Thursday, August 27, 2020 1:48 PM