none
Orchestration slow under load.

    Question

  • Hello,

    I have an orchestration that connects to a ftp Server. This Orchestration is configured with a Host Instance that is only assigned to it. There are many intances of this Orchestration running at the same time. Sometimes it processes very slow. When I dig further I got to know that while making connection to FTP Server the process' are taking a lot of time. What could be the issue? Do I need to configure my Host Instance or Host for some settings? Is there issue while connecting to FTP Server.

    Fyi, To make connection to FTP, we are using Ftp Class of nsoftware (as we are using SFTP).

    Thanks.


    Kunal G

    Thursday, March 28, 2013 10:23 AM

All replies

  • Hi Kunal,

    If you’re sure that error is while connecting the SFTP server, then this could be due to data connection issues.

    The FTP protocol uses two connections for its operations. The first connection is the "Command Connection". This connection is the initial connection that is made using the port number you would have given during the FTP port configuration and is where all of the FTP commands are issued. The other connection that is used is the "Data Connection" and is where the actual data being uploaded/downloaded is communicated. There are two methods for opening the Data Channel with an FTP server. The most common (and easiest) is to use Passive Mode FTP. Passive Mode is where the FTP client (the FTPS Adapter) tells the FTP Server that it wants to open a Data Connection in Passive Mode. This is done by sending the PASV command to the server. When this command is received by the server, the server opens another port to listen for the incoming data connection from the client (the FTPS Adapter) and responds to the client (the FTPS Adapter) with a message instructing which port to use. By enabling logging in SFTP adapter you can see how long it takes to achive this connection.

    Contact nSoftware support for more help on this context.

     

    Anyway for the above issue, your port (adapter) should process message slowly. If you have configured SFTP on the send side, then is your orchestration after sending the message to SFTP do you continue processing any message ? or are you waiting for any acknowledgment from SFTP adapter ? Why your orchestration is slow when the problem is in the SFTP adapter. Use performance monitors to identify the actual bottleneck.

    http://apmblog.compuware.com/2010/02/25/identify-performance-bottlenecks-in-your-biztalk-environment-part-i/

    http://social.technet.microsoft.com/wiki/contents/articles/6819.biztalk-orchestration-performance-thoughts-from-biztalk-support.aspx

    Regards,

    M.R.ASHWINPRABHU


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful.


    Thursday, March 28, 2013 10:38 AM
  • Anyway for the above issue, your port (adapter) should process message slowly. If you have configured SFTP on the send side, then is your orchestration after sending the message to SFTP do you continue processing any message ? or are you waiting for any acknowledgment from SFTP adapter ? Why your orchestration is slow when the problem is in the SFTP adapter. Use performance monitors to identify the actual bottleneck.

    I'm connecting to Ftp Server by calling a helper library in C#. I have added logging to it. Between the two logs it takes approx 3-4 minutes.

    private void ConnectToFTPSvr()
            {
                 this.WriteAuditFile(Logger.AuditLogLevel.Low, "FTPInbound-ConnectToFTPSvr", String.Format("Connecting to FTP Server{0}:{1}", objOperationTrigger.ServerName, objOperationTrigger.Port));
                    objFtpClient = new Ftp();
                    objFtpClient.RemoteHost = objOperationTrigger.ServerName;
                    objFtpClient.User = objOperationTrigger.UserId;
                    objFtpClient.Password = objOperationTrigger.UserPassword;
                    objFtpClient.RemotePort = int.Parse(objOperationTrigger.Port);
                    objFtpClient.Timeout = 0;
                    objFtpClient.Logon();
    
                    this.WriteAuditFile(Logger.AuditLogLevel.High, "FTPInbound-ConnectToFTPSvr", "FTP Connection-Completed");
                }
    			
    
    
    
    

    Here Ftp is a class in namespace nsoftware.IPWorks

    Is the issue due to it or Host Instance being under load?


    Kunal G

    Thursday, March 28, 2013 11:25 AM
  • Have you checked on the FTP Server against how many sessions are open or permitted? There is nothing wrong with the client/code. If on the FTP server there is a restriction on the number of sessions that can be opened per client, etc then there are properties such as connection timeout, sessions timeout which would control why BizTalk is putting your session connects on hold and thus slowing down the processing.

    Regards.

    Thursday, March 28, 2013 11:33 AM
  • ok, I will see to that and let you know. Thanks.

    Kunal G

    Friday, March 29, 2013 6:13 AM
  • There don't seem to be issue with FTP. It throws error if we try to connect to it when it has already reached it's max limit. So, could it be something related to Host or Host Instance?

    Kunal G


    • Edited by Kunal G Monday, April 01, 2013 6:48 AM
    Monday, April 01, 2013 6:47 AM
  • I would use the above code to create a test client and test the FTP with similar payload.

    It would be great to monitor the number of calls/sessions from the FTP site side. If many service instances trying to get sessions it is possible a race condition.


    Leonid Ganeline [BizTalk MVP] BizTalk: the Naming Conventions in Examples

    Tuesday, April 02, 2013 4:22 PM
  • Thanks, I will try that and let you know the results. ( Sorry for late reply. Couldn't get time to try that due to other issues. )

    Kunal G

    Thursday, April 04, 2013 12:06 PM
  • Have you checked on the FTP Server against how many sessions are open or permitted?

    How to check that?

    After further debugging I found that the statement "objFtpClient.Logon();" is sometimes taking more than 5 minutes to execute. I don't know why it is happening. Ant ideas/suggestions?


    Kunal G

    Thursday, May 30, 2013 4:57 PM

  • It would be great to monitor the number of calls/sessions from the FTP site side. If many service instances trying to get sessions it is possible a race condition.

    How to monitor that?

    Kunal G

    Thursday, May 30, 2013 4:58 PM
  • If the FTP is hosted on a U**X Box, the number of login sessions permitted per user restricts the number of logon sessions that the FTP server will permit per user. Since you;re connecting from BizTalk and using the server credentials of the host, once the number of connections has been reached, the next logon attempt would wait or fail. In the FTP client there is a property that controls the connection idle timeout, if this is set to 5 mniutes or 300 seconds, would explain the behaviour. Your client logon request waits till one of the existing sessions has timeout post the transmit which reduces the number of sessions and permits a fresh one.

    You could reduce the idle timeout configured for the client OR increase the BATCH size. The batch size controls the number of messages that would be transmitted under one session. Otherwise you'll need to get in touch with the system administrator of the FTP Server and have him increase the number of sessions permitted for the U**X user on the system.

    Regards.

    Friday, May 31, 2013 4:30 AM
  • Try turning off all your tracking unless you are using them ( which will increase a bit of performance)

    Monday, June 03, 2013 8:09 PM
  • Thanks, but I don't have access to the FTP. I will try to raise a request and check that.

    I have noticed one more behavior:
    Let say there are two Host Instance, HI1 and HI2. HI1 performs 'n' requests and HI2 performs 'm' requests, where n > m. If we use one host instance at a time, then the amount of time it waits at "objFtpClient.Logon()" for HI1 is greater than that of HI2. So, in some way it is depending on the number of request that are being handled. Also, if it was FTP server issue then wait should have been same as it will never know how many more requests are about to come.

    Any comments?


    Kunal G

    Tuesday, June 04, 2013 6:27 AM
  • if HI host is reliably processing connection because it has established sessions while the H2 (which BizTalk tried to load balance onto) starts facing connection issues, the internal BizTalk logic would mark H2 as unreliable and continue pushing the bulk through H1. This would mean that H1 also does not reach an idle timeout and thus never looses connection and keeps sending message batches over the existing sessions?

    Have you tried reducing the Batch count on the FTP to 1 (implying 1 message/batch) and then see what happens?

    Regards.

    Tuesday, June 04, 2013 6:47 AM
  • I have started H1 and H2 at different time so that they won't interfere in each others processing. Eg: when H1 was started then H2 was stopped and vice versa.

    I don't have access to FTP, so changing anything on that would be tough as unfortunately due to some of our internal policies I could ask for FTP access only If I'm very sure that the issue is due to FTP. :(


    Kunal G

    Tuesday, June 04, 2013 6:59 AM
  • Use the FTP login creadentials and try and open a session from multiple PC's or multiple command prompt windows and see if you hit a limit. Then use that to prove that there is a session restriction and get FTP/Server access.

    Regards.

    Tuesday, June 04, 2013 7:20 AM
  • I have created a console application with the same code and credentials as my helper library. When the BizTalk -> Helper code was waiting at objFtpClient.Logon() I started the console application to check if it FTP will respond or not but the FTP responded in a timely fashion. I believe that it should have waited if it was sessions issue.

    Kunal G

    Thursday, June 06, 2013 11:51 AM
  • Look under the BizTalk Performance Optimization Guide for the registry key associated with TCP/IP port exhaustion and verify that it is set on H2.

    Regards.


    Thursday, June 06, 2013 1:07 PM
  • I have followed "Avoiding TCP/IP Port Exhaustion" and tried increasing the upper range of ephemeral ports that are dynamically allocated to client TCP/IP socket connections to 65533, even then the same issue. Also, tried reducing the client TCP/IP socket connection timeout value to 40 seconds but still the same :(

    Kunal G

    Tuesday, June 11, 2013 3:37 PM
  • After the registry key change, you need to fix the MSDTC and then rerun the analyser to see if it reports or does not report the issue.

    IMHO it is time you logged a cal with Microsoft Support to get a resolution of your problem.

    Regards.

    Wednesday, June 12, 2013 5:44 AM
  • Again, create the console app with your code and test it. Looks like it is not related to the BizTalk but only to the FTP server.

    Leonid Ganeline [BizTalk MVP] BizTalk: the Naming Conventions in Examples

    Thursday, July 18, 2013 8:52 PM
  • I have tested it with a console application also.When BizTalk  was waiting at objFtpClient.Logon(), I started the console application with the same code to check if FTP will respond or not because BizTalk was waiting on that but in console app FTP responded in a timely fashion.

    Kunal G

    Friday, July 19, 2013 4:29 AM
  • Kunal,

    is this still a current problem?

    It is not clear from the above whether you are using FTP or SFTP - they are quite different. 

    If you are using SFTP, it is important to consider SFTP sessions at the receiver end. By default you will get new sftp server process on the remote side. THere is a setting somewhere on NSOFTWARE that allows you to specify whether to keep sessions alive. If you set that you will get much better performance.

    You can tell if you have this problem if a reboot of the remote server gives good performance initially  but then degrades after a few thousand requests. On the Server they will see lots of SFTP server processes not doing much but consuming all the memory.

    BizTalk will back up because orchestrations are being kept alive that wold normally have completed their work.

    mark 


    mark

    Friday, July 19, 2013 11:00 AM
  • Hi,

    Your problem could be due to data connection. Command Connection and Data Connections are the two types of connections. First Command Connection runs than Data Connection runs. Within two ways to open data connection, it is common to use passive form by giving PASV command to open Data Connection. After receiving the command, server opens other port to take note on received data connection from BizTalk adapters (client) and reply to client with the instruction to use which port.

    For more help on this issue visit:  http://www.athenainfotech.co.uk

    They provide free consultancy service.

    Thanks

    Saturday, July 20, 2013 6:14 AM
  • @Mark: Thanks. Yes, this is the current problem. I will check that and let you know. (Sorry for the late response, for some reason I didn't get any update that something is posted to this thread. )

    Kunal G

    Tuesday, August 13, 2013 6:24 AM
  • THere is a setting somewhere on NSOFTWARE that allows you to specify whether to keep sessions alive. If you set that you will get much better performance.

    I can't figure out how to do that. Could you please give me some pointers in that direction? I also tried searching the properties of the FTP class but there is no property for that.


    Kunal G

    Thursday, August 22, 2013 12:24 PM
  • Could it be something related to throttling? How can I figure out that? I have read this link and it feels that it could be the case. Are there any tools with which I can know that or it's just hit and trial with the settings?

    Kunal G

    Monday, October 21, 2013 12:38 PM
  • Could it be something related to throttling? How can I figure out that? I have read this link and it feels that it could be the case. Are there any tools with which I can know that or it's just hit and trial with the settings?

    Kunal G


    Performance Monitor:  http://msdn.microsoft.com/en-us/library/aa578302.aspx
    Monday, October 21, 2013 1:56 PM
  • Try following:

    - If possible have 64 bit Orchestration host

    - Change Resource-Based Throttling of orchestration host Memory Usuage->Process Virtual to adjust based on available memory

    - Change Rate-Based Throttling , Publising and Delivery->Maximum Throtting Delay to 0 and Throttling override to "Do not throttle"

    Please note, after these changes host not will not throttle and your memory usage can go beyond and in those condition your host will restart.

    The actual setting depends on your volume and other load on the servers

    Monday, October 21, 2013 10:07 PM