XML Task: System Out of Memory
-
Friday, September 01, 2006 12:11 PM
I have an XML Task to validate an xml document on disk (size: 78,464K). When I try to validate it I get error:
[XML Task] Error: An error occurred with the following error message: "Exception of type 'System.OutOfMemoryException' was thrown.".Task Manager shows 904,492K Physical Memory Available which drops to about 586,000K when the task fails. Any ideas?
All Replies
-
Thursday, May 17, 2012 7:29 PM
I have an XML Task to validate an xml document on disk (size: 78,464K). When I try to validate it I get error:
[XML Task] Error: An error occurred with the following error message: "Exception of type 'System.OutOfMemoryException' was thrown.".Task Manager shows 904,492K Physical Memory Available which drops to about 586,000K when the task fails. Any ideas?
I have this same problem. I have searched this forum and googled the topic. No luck on a solution.
I would welcome a solution. If I cannot solve this using the XML Task, then perhaps a command line XML validator that I can call via the Execute Process Task. Any suggestions?
Thanks.
-
Thursday, May 17, 2012 7:47 PMModerator
-
Thursday, May 17, 2012 8:01 PM
This is not a practical approach. I will have files, 10, 20, 100 times larger or even more than the test file that is resulting in this error. I cannot add memory ad infinitum. I cannot even imagine what the reaction would be if I kept making such upgrade requests.This is a simple lack of memory, just increase it.
Arthur My Blog
-
Thursday, May 17, 2012 8:21 PMModerator
I disagree, 1 GB of memory is no good for a prod system even a dev machine. Memory is not too much money.
If you want to process complex, large files with no memory what do you expect? How large/complex is your file? What is planned to happen after you validated it?
Besides, please tell how did you set all up. I think there is a chance for some misconfig.
Arthur My Blog
-
Thursday, May 17, 2012 11:15 PMModeratorPerhaps you can use this script code instead: XML Validation in SQL Server Intergration Services
Talk to me now on
-
Friday, May 18, 2012 11:23 AM
I disagree, 1 GB of memory is no good for a prod system even a dev machine. Memory is not too much money.
If you want to process complex, large files with no memory what do you expect? How large/complex is your file? What is planned to happen after you validated it?
Besides, please tell how did you set all up. I think there is a chance for some misconfig.
Arthur My Blog
The XML file is not particularly complex -- just large. The single XML data source flows to about 40 destination tables. The destination tables have just a handful of columns. I had to use a derived column transformation for each destination table. For two or three destination tables I had to do some simple data conversion transformations.
I created an XML test file that generates 1.1 million rows in the table associate with the primary element. It's a bit over 1GB in size. That might be the upper end of actual files. I tested files 1/10th that size that also failed with the memory error. Very small files do not fail -- well, except for XML files that I screw up just to make sure the validation fails as expected. I did not test anything between the 1/10th size file and the very small files because it's not really relevant as I need to be able to process files as large as the 1GB file.
-
Friday, May 18, 2012 2:51 PMModerator
Why a Derived Column Transfrom and not a Split and/or Multicast ?
Well, I would really then advocate looking into Todd's suggestion: http://sqlblogcasts.com/blogs/simons/archive/2010/01/20/XML-Validation-in-SQL-Server-Intergration-Services.aspx
Arthur My Blog
.png)
