locked
size of data processed by the engine RRS feed

  • Question

  • The Power Query specification here describes the "Size of data processed by the Engine" as:

    "Limited to available virtual memory (for 64-bit version) or about 1GB for 32-bit version, if data cannot be fully streamed, such as when sorting the data set locally before filling it"

    What is meant by the part where it says "if data cannot be fully streamed, such as when sorting the data set locally before filling it"?

    Is it saying that the limits don't apply when sorting? Is it applicable to 32-bit only?

    Thanks

    Friday, November 7, 2014 5:47 PM

Answers

  • Power Query will try to tell the server to do the transformations on their side if they can which means that Power Query does not need to store all of the data locally. In  certain cases this is not possible, so Power Query has to download the data locally before doing the transformation, which takes up memory.

    In this case, if we have to do the sort locally (because the user has buffered the result or because the data source we are using cannot be streamed), then the 32-bit version of Power Query can use up to around 1 GB of memory while the 64-bit version of Power Query can use as much virtual memory that's available on the system.

    I hope that helps.

    Thanks.

    Monday, November 10, 2014 10:32 PM

All replies

  • Power Query will try to tell the server to do the transformations on their side if they can which means that Power Query does not need to store all of the data locally. In  certain cases this is not possible, so Power Query has to download the data locally before doing the transformation, which takes up memory.

    In this case, if we have to do the sort locally (because the user has buffered the result or because the data source we are using cannot be streamed), then the 32-bit version of Power Query can use up to around 1 GB of memory while the 64-bit version of Power Query can use as much virtual memory that's available on the system.

    I hope that helps.

    Thanks.

    Monday, November 10, 2014 10:32 PM
  • Thank you - that helps. So if you have to sort locally then everything must be read into memory, I understand.

    Can you tell me what defines a data source that can or cannot be streamed? What about .csv files?

    And what do you mean by buffering the result in the context of Power Query?

    Thursday, November 20, 2014 9:56 AM