Answered by:
Biztalk receive location

Question
-
I'm trying to process a file with 70 MB but the receive location "IN folder" is not picking it up.
To make sure that the receive location is working, I dropped a 10 KB size file and it picked it up quickly. So the receive location is working properly.
So why my receive location is not able to process a 70 MB ?
Christiane
Friday, January 17, 2014 9:19 PM
Answers
-
Hi Christiane,
I tried with 94.3037 MB EDI file using EDIReceive pipeline and it was processed, but it took 28 mins to process the file.
And then after individual files (transactions) are getting send to destination... so far 5566 files are done (in one hour).... still counting
Offcourse the performance is not great but that was expected as the encounter is with EDI file that too large in size. And it is because while transformation from EDI to XML in pipeline annotations are added thus leading to increase of size.
This is certainly a limitation, adding a dedicated host should help upto some extent, and I guess there is a EDI Accelerator from covast which does overcome this limitation by removing the annotations (I haven't used it)
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Edited by Maheshkumar S Tiwari Wednesday, January 22, 2014 6:45 PM
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 6:30 PM -
Hi Christiane,
This error is due to the size of the Message you are trying to process, as the message size is greater than that of resources available for processing it.
I guess you tried with all the suggestions, but did you tried with EDI Fallback settings and dedicated Host (There should be no harm in trying that as well :) )
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Edited by Maheshkumar S Tiwari Wednesday, January 22, 2014 4:25 PM
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 4:25 PM -
One thing at a time.
Change the Send Port to the File Adapter and run the batch to confirm your system is tuned for that receive volume.
Does the process work otherwise with smaller files of ~100 transactions?
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 4:03 PMModerator -
If your intent is to separate the Receive operation then:
1. Create a new Host. You can choose any appropriate name but it must be different than any other Host.
2. Create an Instance of the new Host. It can use the same credentials as any other Host Instance.
3. Add Handlers for any Adapter you will be running in that Host. In your case, that would be the FILE Adapte at least, and any other Adapters you expect to run in the new Host.
4. Change the Receive Handler to your new Host on the Receive Location where you're receiving the large file.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 3:25 PMModerator -
It's because there are suspended messages still tagged for previous host.
You only option is to Terminate them and start over.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 9:12 PMModerator -
Just to clarify, the Receive Location and the Send Port should be in different Hosts/Instances. So, one Host/Instance for Receive and one Host/Instance for Send.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 7:13 PMModerator -
Well, I only have 1 host instance on the Biztalk Server!
And that is a likely contributor to your performance situation which is why you should first create a new Host and Host Instance for the Send Port.
Here are the steps to create Hosts and Host Instances: http://msdn.microsoft.com/en-us/library/aa561042.aspx
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Thursday, January 23, 2014 2:36 AMModerator -
Well, consider, the goal of this process is to run the Receive and the Send in different Host Instances so while you technically could change the Send Handler as well, you'd be in the same place you are now.
To clarify, I'm recommending you run the Receive Location and the Send Port in different Hosts/Instances.
Another option is to create two new Host/Instances, one dedicated to Receive, onde dedicated to Send while your existing Host, BizTalkServerApplication presumably, continues to run any other operations.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 4:02 PMModerator
All replies
-
Most likely it is working but is taking a very long time to process all 70MB.
The FILE Adapter will not delete the file until it has been accepted and persisted to the MessageBox. That process though includes all Pipeline processing, including decoding, debatching, mapping, etc.
What type of file is it and what do you do in the Pipeline? Perhaps we can recommend some alternate approaches.
- Proposed as answer by Sachin142 Tuesday, January 21, 2014 7:17 AM
Saturday, January 18, 2014 3:43 AMModerator -
Hi Christiance,
Can you check if your receive location is subscribing to the message type (or file type) ? if yes than you can do the following
1)Increase the batch size at your receive location.It is available under Batching section of your file receive location.
2) Change resource based throttling setting for your host instance.
Do stop/start your host instance .
Thanks
Abhishek
- Proposed as answer by Abhishek0127[Abhishek kumar]MVP Saturday, January 18, 2014 3:53 AM
- Edited by Abhishek0127[Abhishek kumar]MVP Saturday, January 18, 2014 3:55 AM ok
Saturday, January 18, 2014 3:53 AM -
Make sure the file is not read only.
Søren Andersen
- Proposed as answer by Sachin142 Tuesday, January 21, 2014 7:12 AM
Saturday, January 18, 2014 7:37 AM -
Hi Christiane,
As the size of message increases, the resource usage for processing also increases and depending on various factors the time taken to process increases. Can I ask you to go through How BizTalk Server Processes Large Messages , this will help you to detect the problem area with your solution.
Debatching messages at receive side does help in such scenario.
And can you provide more details like:
1. Type of file you are receiving
2. What processing is done on the message
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Proposed as answer by Sachin142 Tuesday, January 21, 2014 7:18 AM
Saturday, January 18, 2014 1:31 PM -
Boatseller,
The file is an EDI file but I put in the file mask *.*
I have on my receive pipeline: EDI receive
The receive location adapter is file and it is trying to pickup the 70MB file and process it to my sql database by a send port as WCF custom adapter using SQL binding.
Christiane
Saturday, January 18, 2014 6:44 PM -
Maheshkumar
The file is an EDI file but I put in the file mask *.*
I have on my receive pipeline: EDI receive
The receive location adapter is file and it is trying to pickup the 70MB file and process it to my sql database by a send port as WCF custom adapter using SQL binding.
Christiane
Saturday, January 18, 2014 6:45 PM -
Christiane
Should I increase any? or that is good enough! because still with these settings, not able to process 70MB- Edited by Christiane0696 Saturday, January 18, 2014 7:04 PM edit
Saturday, January 18, 2014 7:01 PM -
Hi Christiane,
I think your File processing is taking more time as your file size is large . Try to increase Batch size at your receive location and restart the application .
Thanks
Abhishek
- Proposed as answer by Abhishek0127[Abhishek kumar]MVP Saturday, January 18, 2014 7:05 PM
Saturday, January 18, 2014 7:05 PM -
Abhishek,
these are the batching settings on my receive location.
Christiane
Saturday, January 18, 2014 7:09 PM -
Hi Christiane,
Can you increase Maximum Batch Size (in Bytes) and change your threshold settings and than you can restart the application and your respective host .
Thanks
Abhishek
- Proposed as answer by Abhishek0127[Abhishek kumar]MVP Saturday, January 18, 2014 7:28 PM
- Edited by Abhishek0127[Abhishek kumar]MVP Saturday, January 18, 2014 7:31 PM
Saturday, January 18, 2014 7:15 PM -
Abhishek,
Can you be specific with your numbers?
Christiane
Saturday, January 18, 2014 8:21 PM -
Hi Christiane,
EDI files when transformed into XML increases the size as annotations are added while transforming, thus it is recommended to split the interchange in the pipeline. Also, having a dedicated host can help.
Have you set the Inbound batch processing option, agreement property to Split Interchange as Transaction Sets.
Setting this will make EDI pipeline to split each transaction set .
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
Saturday, January 18, 2014 8:37 PM -
Maheshkumar,
I don't need agreement for this process.
I only have a receive location and a send port to store data in SQL.
Christiane
Saturday, January 18, 2014 9:20 PM -
Hi Christiane,
In that case you can use Fallback settings. BizTalk Server uses the fallback agreement settings if no agreement is resolved for incoming or outgoing messages
1.Administration Console --> right-click the Parties node, and then click X12 Fallback Settings or EDIFACT Fallback Settings
2.On Fallback Settings General Pages tab --> General page --> Click Enable Fallback Settings
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Edited by Maheshkumar S Tiwari Saturday, January 18, 2014 9:38 PM
Saturday, January 18, 2014 9:35 PM -
So, let's start from the beginning.
1. The "Maximum batch size" is pretty much irrelevant in this case since the file is larger than the default (unless you changed the default). So all that means is the file will be delivered by the Adapter in a Batch of 1.
2. Test the Receive Location with the PassThrough Pipeline. Don't change anything else. You should get an error somewhere, Routing, Send Port, doesn't really matter. A 70MB file could take 1-3 min to get to the database depending on your system.
3. Try again and be patient. Check either Performance Monitor or Task Manager for BTSNTSvc[64].exe, if it's taking CPU, it's doing something.
Just keep in mind, the file will not be deleted from the receive folder until all pipeline processing is complete and all resulting messages are committed to the MessageBox.
If a 10KB file in the same format, EDI in this case, processes, the 70MB file will also process but it will take a lot longer.
As an aside, I wouldn't recommend using the Fallback Settings unless you will never, ever use Agreements.
Saturday, January 18, 2014 10:14 PMModerator -
I'm just trying to test this biztalk application in the dev server.
so if I want to change the batch size, what exact number should I enter.
Christiane
Sunday, January 19, 2014 7:01 PM -
What are you trying to test? Any value < 70MB (for purposes of this discussion) will yield the same result. The batch size setting affects only the number of Messages, files in the case of the FILE Adapter, submitted in the same batch. It is not a limit on message size.
Any number < 70MB simply means the 70MB file will be submitted by itself regardless of the "Number of messages in a batch" setting. It will always be the lower of the two settings.
If all you want to do is test the Receive Location/Adapter, you have to do that with a PassThrough Pipeline as described in #2 above.
Sunday, January 19, 2014 8:16 PMModerator -
Now that I changed the receive pipeline to PassThrough, the file is picking up fast but I don't see any data in SQL.
That's the error I'm receiving:
The adapter "SQL" raised an error message. Details "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.".
Christiane
Monday, January 20, 2014 4:01 PM -
Hi Christiane,
Can you set UseAmbientConnection to false and use transaction = false in your custom configuration and stop/Start your host instance
Thanks
Abhishek
Monday, January 20, 2014 5:25 PM -
The transaction on the Send Port has nothing to do with it. Also, at the point any SQL errors are also irrelevant since the native EDI is being passed to the SQL Adapter.
Here is your situation. By using the PassThrough Pipeline, you have proven that the Receive Location is working and the Adapter is in fact picking up the file/message.
You have also isolated the lengthy duration to the Pipeline execution. 99.9% probability that the EDI Disassembler is just taking a long time to process the 70MB file. Did you look at Performance Monitor or Task Manager?
Set the Pipeline back to your EDI Pipeline, drop the file and let it run.
Reminder, the file will not be deleted from the file system until all pipeline processing, debatching, serializing, validation, mapping, etc. is compete. With 70MB, this can take a while.
Monday, January 20, 2014 5:35 PMModerator -
Boatseller,
I will give it time and let you know what happens. I think it takes about 40 minutes to see some row counts in SQL server. I dropped the file and it quickly got picked up as PassThroughreceive, the send port pipeline configuration is set as passthrutransmit.
Christiane
Monday, January 20, 2014 5:46 PM -
Boatseller,
I keep getting this warning not error from sql:
The adapter "SQL" raised an error message. Details "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied."
I changed the receive location > receive pipeline to EDI receive and the file is taking forever to get picked up!!!
is this performance issue? if yes how can I fix this old Biztalk 2009! maybe run messageboxviewer tool?
Christiane
Monday, January 20, 2014 7:12 PM -
The adapter "SQL" raised an error message. Details "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied."
Do you get this with the 10k file? If so, it is unrelated to the large file size.
If you are splitting the 70MB file, into individual 837's for instance, you should check Ordered Delivery on the Send Port.
Again, the file in not taking forever to get picked up. It is being processed but will not be removed from the file system until all Pipeline processing is complete.
The reason the PassThrough is quick is because the 70MB is being streamed directly to the MessageBox as is, not processing whatsoever.
- Edited by Johns-305MVP, Moderator Monday, January 20, 2014 8:49 PM
Monday, January 20, 2014 8:47 PMModerator -
If I choose PassThrough, will it process the file to SQL ? or only it will sit at the MessageBox?
Christiane
Tuesday, January 21, 2014 8:09 PM -
If I choose PassThrough, will it process the file to SQL ? No.
or only it will sit at the MessageBox? Yes, suspended.
The PassThrough Pipeline does no processing on the message at all so the output of the Pipeline is still EDI which the SQL Adapter cannot process.
The EDI Disassembler is what converts the EDI to Xml. That process takes a while to finish. After that, I assume you have a Map to the SQL Insert message/Stored Procedure.
Have you let it run? How long did you wait? Did it eventually complete?
- Edited by Johns-305MVP, Moderator Tuesday, January 21, 2014 8:24 PM clarification
Tuesday, January 21, 2014 8:23 PMModerator -
Running services instance was in the 90,000 now it is decreasing to 40,000.
What does that means?
At this time, I still don't see my data processed to SQL tables.
Christiane
Tuesday, January 21, 2014 8:32 PM -
The EDI interchange will be completely processed and persisted to the MsgBox DB before the outbound subscriptions are fulfilled. The count should drop to almost 0 and then it should increase rapidly as the outbound processing begins.
David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.
Tuesday, January 21, 2014 8:54 PM -
It depends on what you're counting. What's the Service Class or Name?
Is is possible your EDI Interchange had ~90,000 individual transactions? They were split by the EDI Disassembler resulting in the 90k.
Depending on your query in BT Admin, that number might also include Send Pipeline instances.
Tuesday, January 21, 2014 9:15 PMModerator -
Boatseller,
this is the warning I'm receiving in my application log:
The adapter "SQL" raised an error message. Details "Insufficient memory to continue the execution of the program.".
this is the error I received
The description for Event ID ( 4838 ) in Source ( COM+ ) cannot be found. Either the component that raises this event is not installed on your local computer, or the installation is corrupted. You can install or repair the component on the local computer, or contact the component manufacturer for a newer version.
If the event was saved from another computer or forwarded from a remote computer, you might have to include display information with the events when saving them or when setting up the forwarding s
Server Application ID: {CA8B635A-933F-41F8-81EE-CA088DE1B133}
Server Application Instance ID:
{789945DC-CDA3-4006-9C90-88F2947D6AE2}
Server Application Name: XLANG Internal Utilities
Error Code = 0x8007000e : Not enough storage is available to complete this operation.
COM+ Services Internals Information:
File: d:\w7rtm\com\complus\src\comsvcs\byot\byotex.cpp, Line: 450
Comsvcs.dll file version: ENU 2001.12.8530.16385 shp.another warning:
The adapter "SQL" raised an error message. Details "[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.".
Christiane
Wednesday, January 22, 2014 3:48 PM -
The adapter "SQL" raised an error message. Details "The adapter recovered from recent failures. The Event Log Error Threshold count reset. ".
Christiane
Wednesday, January 22, 2014 3:50 PM -
There is 300,000 "count" running services with status "Ready to run"
Christiane
Wednesday, January 22, 2014 3:52 PM -
One thing at a time.
Change the Send Port to the File Adapter and run the batch to confirm your system is tuned for that receive volume.
Does the process work otherwise with smaller files of ~100 transactions?
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 4:03 PMModerator -
my system used to process half the size of what I'm trying to process now.
Before it was processing 30 mb. now that the file size is 70 mb then it doesn't process it. Do you know which settings I have to change? and increase the volume of memory:?
Christiane
Wednesday, January 22, 2014 4:09 PM -
Christiane
Wednesday, January 22, 2014 4:17 PM -
Hi Christiane,
This error is due to the size of the Message you are trying to process, as the message size is greater than that of resources available for processing it.
I guess you tried with all the suggestions, but did you tried with EDI Fallback settings and dedicated Host (There should be no harm in trying that as well :) )
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Edited by Maheshkumar S Tiwari Wednesday, January 22, 2014 4:25 PM
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 4:25 PM -
Post back the results of using the File Adapter on the Send Port.
Did you set Ordered Delivery on the Send Port?
Fallback Settings vs a full Agreement shouldn't have an effect, the Fallback Settings are just a built-in Agreement.
Either way, your next configuration step is to create new Host and Host Instance dedicated to running the Send Port.
Wednesday, January 22, 2014 4:42 PMModerator -
Hi Christiane,
I tried with 94.3037 MB EDI file using EDIReceive pipeline and it was processed, but it took 28 mins to process the file.
And then after individual files (transactions) are getting send to destination... so far 5566 files are done (in one hour).... still counting
Offcourse the performance is not great but that was expected as the encounter is with EDI file that too large in size. And it is because while transformation from EDI to XML in pipeline annotations are added thus leading to increase of size.
This is certainly a limitation, adding a dedicated host should help upto some extent, and I guess there is a EDI Accelerator from covast which does overcome this limitation by removing the annotations (I haven't used it)
Maheshkumar S Tiwari|User Page | http://tech-findings.blogspot.com/
- Edited by Maheshkumar S Tiwari Wednesday, January 22, 2014 6:45 PM
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 6:30 PM -
Boatseller,
I will create a host and host instance for this particular Biztalk application and will see what happens.
Christiane
Wednesday, January 22, 2014 7:00 PM -
Just to clarify, the Receive Location and the Send Port should be in different Hosts/Instances. So, one Host/Instance for Receive and one Host/Instance for Send.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Wednesday, January 22, 2014 7:13 PMModerator -
Well, I only have 1 host instance on the Biztalk Server!
Christiane
Thursday, January 23, 2014 1:00 AM -
Well, I only have 1 host instance on the Biztalk Server!
And that is a likely contributor to your performance situation which is why you should first create a new Host and Host Instance for the Send Port.
Here are the steps to create Hosts and Host Instances: http://msdn.microsoft.com/en-us/library/aa561042.aspx
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Thursday, January 23, 2014 2:36 AMModerator -
Boatseller,
Do I also need to add new adapters? such as file and WCF custom ?
Can I have the same host instance name which is for example: BizTalkServerApplication
I just need a receive location with file adapter and a send port with WCF custom.
Christiane
- Edited by Christiane0696 Tuesday, January 28, 2014 2:40 PM
Tuesday, January 28, 2014 2:38 PM -
If your intent is to separate the Receive operation then:
1. Create a new Host. You can choose any appropriate name but it must be different than any other Host.
2. Create an Instance of the new Host. It can use the same credentials as any other Host Instance.
3. Add Handlers for any Adapter you will be running in that Host. In your case, that would be the FILE Adapte at least, and any other Adapters you expect to run in the new Host.
4. Change the Receive Handler to your new Host on the Receive Location where you're receiving the large file.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 3:25 PMModerator -
What about the send handler ? do I also change that to the new Host ? on the send port that is sending to my sql database using WCF custom adapter!
If I have to change the send handler to the new Host, Do I have to add a new WCF custom adapter?
Christiane
Tuesday, January 28, 2014 3:36 PM -
Well, consider, the goal of this process is to run the Receive and the Send in different Host Instances so while you technically could change the Send Handler as well, you'd be in the same place you are now.
To clarify, I'm recommending you run the Receive Location and the Send Port in different Hosts/Instances.
Another option is to create two new Host/Instances, one dedicated to Receive, onde dedicated to Send while your existing Host, BizTalkServerApplication presumably, continues to run any other operations.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 4:02 PMModerator -
As I changed the receive handler in the receive location to the new host/instance.
This is the error I'm getting in application log:
The InboundTransportLocation is disallowed. This could be because the receive location is disabled or its service window is currently inactive.
Christiane
Tuesday, January 28, 2014 8:43 PM -
It's because there are suspended messages still tagged for previous host.
You only option is to Terminate them and start over.
- Marked as answer by Christiane0696 Thursday, January 30, 2014 2:31 AM
Tuesday, January 28, 2014 9:12 PMModerator -
I'm half way, my send port that has the WCF custom that is storing data to my sql server displayed a warning which is the following:
The adapter failed to transmit message going to send port "LA_send_port_5010" with URL "mssql://TPA*****/OP*****/HiCo******?". It will be retransmitted after the retry interval specified for this Send Port. Details:"Microsoft.ServiceModel.Channels.Common.ConnectionException: A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 0 - The specified network name is no longer available.) ---> System.Data.SqlClient.SqlException: A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 0 - The specified network name is no longer available.)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at Microsoft.Adapters.Sql.SqlAdapterConnection.OpenConnection()
--- End of inner exception stack trace ---
Server stack trace:
at Microsoft.Adapters.Sql.SqlAdapterConnection.OpenConnection()
at Microsoft.Adapters.Sql.ASDKConnection.Open(TimeSpan timeout)
at Microsoft.ServiceModel.Channels.Common.Design.ConnectionPool.GetConnection(Guid clientId, TimeSpan timeout)
at Microsoft.ServiceModel.Channels.Common.Design.ConnectionPool.GetConnectionHandler[TConnectionHandler](Guid clientId, TimeSpan timeout, MetadataLookup metadataLookup, String& connectionId)
at Microsoft.ServiceModel.Channels.Common.Channels.AdapterRequestChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open()
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open()
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.GetChannel[TChannel](IBaseMessage bizTalkMessage, ChannelFactory`1& cachedFactory)
at Microsoft.BizTalk.Adapter.Wcf.Runtime.WcfClient`2.SendMessage(IBaseMessage bizTalkMessage)".
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.Christiane
Tuesday, January 28, 2014 10:02 PM