none
Data input from folder metadata - add in Evaluation ran out of memory and can't continue.

    Question

  • I am trying to import file data from folders on my desktop.

    I have a 64 bit Win10, ca 32 GB RAM computer using Excel13 32 bit

    1. The folder has 38 items: All 38 file data is loaded into an Excel sheet.  However the Query side panel says that the Download failed.

    2. The folder has 230 items: Only 101 file data is loaded. The Query side panel says that the Download failed.

    The columns created by both queries are

    Name: Extension: Date accessed: Date modified: date created: Attributes: Folder Path

    Previous error messages have been: add in Evaluation ran out of memory and can't continue.

    The files themselves are of large images, up to 1.7 GB, but there is no sign that the file content is being loaded, not even the file size data (though that would be useful). I want only the file metadata.  In various folders I have ca 1,800 files whose file data I wish put into an Excel database.

    Can PQ do this fairly simple job?


    CVH Manchester

    Wednesday, January 03, 2018 5:37 PM

Answers

  • Hey,

    In short, yes. PQ can do the job, but it's a tough situation to determine the root cause of this issue without being able to replicate it. I'd recommend that you click on the frown icon to submit all the telemetry need to the PQ team so they further help you with it.

    From what I'm reading, you deleted the Content column which is the one that reads the binaries of each of the files in that folder. I'm guessing that you have the latest version of PQ installed, but I might as well point out that you should test this situation with the latest version of Power Query.

    PQ can be REALLY slow when it tries to discover a lot of complex and large files from a folder. I'd also suggest that you place those same files in a different folder (like a specific folder inside your Pictures folder for instance).

    Best of luck!

    • Marked as answer by Manchester05 Thursday, January 04, 2018 7:32 AM
    Wednesday, January 03, 2018 11:23 PM

All replies

  • Hey,

    In short, yes. PQ can do the job, but it's a tough situation to determine the root cause of this issue without being able to replicate it. I'd recommend that you click on the frown icon to submit all the telemetry need to the PQ team so they further help you with it.

    From what I'm reading, you deleted the Content column which is the one that reads the binaries of each of the files in that folder. I'm guessing that you have the latest version of PQ installed, but I might as well point out that you should test this situation with the latest version of Power Query.

    PQ can be REALLY slow when it tries to discover a lot of complex and large files from a folder. I'd also suggest that you place those same files in a different folder (like a specific folder inside your Pictures folder for instance).

    Best of luck!

    • Marked as answer by Manchester05 Thursday, January 04, 2018 7:32 AM
    Wednesday, January 03, 2018 11:23 PM
  • Miguel

    Thank you for the expertise.  This was my first try with PQ, using the latest version.

    I used the tool without changing any defaults, and certainly did not delete the Content column. From your comments and the memory restrictions on 32 bit Excel, I guess that the tool tried to import the content.  However, the file content is far greater than the ca 2 GB limit on 32 bit, so PQ fell over.  It seems that error trapping and reporting in PQ needs improvement.

    I have found a simple tool, Kutools, that does my job quickly and easily. 

    Discovering file metadata, and loading the underlying data, are different tasks. Increasingly, we do not need the data to hand (except for financial services), we need it available. Which was what I was trying to do.


    CVH Manchester

    Thursday, January 04, 2018 8:09 AM