none
Biztalk 2010 throttling / scaling Biztalk Application -> Limit CPU load

    Question

  • Hi, I am looking for help in throttling / scaling a Biztalk Application.

    We are using a BiztalkServer 2010. On MS Server 2008 R1 in a 2 CPU configuration.

    We need to meet 2 very different requirements with our application:

    The application has to be scaled for:

    1. acceptable throughput, as well as
    2. latency.

    Messages "entering" the application need to run through a series of "suborchestrations" which represent a series of "business cases" or just enriches the message with some information from a database. These sub orchestrations are subscribing to a certain message type and an action parameter.There is a dispatcher orchestration which hosts a loop, creating correlation sets and setting an action parameter to the message and puts the message to the message box. The corresponding sub orchestration grabs the message, applies some "rules", changes the action and puts the message back into the message box, the corresponding dispatcher orchestration grabs the message from the correlation set, rinse repeat. The last Action is to save the "enriched" message to a database. All this is working fine. The Problem is: In our scenario the application has to be reached by a flat file receive port as well as a WCF-service which can be called from a web-application(GUI). The call from the web application is a synchronous call where the user is waiting for the processed and enriched message. The GUI process must not timeout. There is a "dispatcher"(as described above) for each of the receive ports using the same set of sub orchestrations. Low Latency for the WCF Call must be met during simultaneous processing of messages that enter the application by the flat file receive port. I have to scale the BiztalkServer, so that even during processing of a large flatfile batch like 5000 - 10000 Messages, both throughput as well as latency are acceptable. Our approach was to create different hosts in order to have a possibility to scale the differnt components.

    So I have a:

    1. FileReceiveHost (In-Process)
    2. WCFReceiveHost (In-Process)
    3. BiztalkServerApplication (In-Process)
    4. BiztalkServerIsolatedHost (Isolated)

    I organized the FlatFileReceiveLocations Receive Handler and the FlatFile_Dispatcher Orchestration in the FileReceiveHost.

    All the sub orchestrations are running in the BiztalkServerApplication Host.

    I introduced a new orchestration WCF_CallDispatcherOrchestration.odx receiving the calls from the WCF Port and calling the WCF_Dispatcher Orchestration by Call_Shape (the idea here was to ensure that the orchestrations are running in the same instance/process).

    The WCF_CallDispatcherOrchestration as well as the WCF_Dispatcher itself are running in the WCFReceiveHost.

    The WCReceive_Port ReceiveHandler is running in the BiztalkServerIsolatedHost.

    Here are my settings for the different Hosts (just wrote down the vales I changed from default):

    FileReceiveHost:

    General

    Polling Intervals

    Messages: 750 ms Orchestration: 750 ms

    Per CPU Settings

    Threads: 55

    Database Connections: 50

    Orchestration Throttling:

    Pause at: 40 Resume at: 0

    WCFReceiveHost:

    General

    Polling Intervals

    Messages: 50 ms Orchestration: 50 ms

    Ressource based throttling

    Internal Message queue size: 10000

    BiztalkServerIsolatedHost:

    General

    Polling Intervals

    Messages: 50 ms Orchestration: 50 ms

    Ressource based throttling

    Internal Message queue size: 10000

    FileReceiveHost: General

    Polling Intervals

    Messages: 500 ms

    Orchestration: 500 ms

    Per CPU Settings

    Threads: 55

    Database Connections: 50

    Orchestration Throtteling:

    Pause at: 200 Resume at: 100

    With these connections, I get a decent latency for the web application. BUT after a few minutes the BizTalk server stabilizes at 3-10 % CPU (2 CPU) and the processing of flat files takes ages. It takes over 2 hours now for a flatfile with 103 records to be processed. If I am not scaling for latency and let the BizTalk server run free, it takes like 8 seconds per record to process (which would ?? like 15min max for the whole batch). Iin this case, I have a very bad latency, the web application timeouts, CPU load almost all the time at 100%. I think, if I would be able to scale the server so that the flatfile processing would consume like 25-30% CPU, latency will be acceptable well in this configuration. However any tweaks I made with respect to threads per CPU or Orchestration throttling for the FileReceivePort it seems like the BizTalk will either run at 3-5 % or at 100%. So either I have a good throughput for batch file processing or a low latency but I can't find a setup giving proper results for both scenarios.

    Any ideas?

    Wednesday, February 22, 2012 1:00 PM

Answers

  • Hi Martin,

    Thanks for providing the details!

    A suggestion from my side about the settings of your Isolated and WCF receive host which is 50 MS. I think keeping this value too low might create aload on SQL server , imagine in every 50Ms your host (BizTalk Service) is reading your database for more work in the Queues (WCF Receive and ISolatedQueue).

    Also you would need to make sure that your database infrastcture is properly tuned up to achieve required latency. There are guidelines from Microsoft to tune various parameters to achieve that. Please see following guides

    Performance Optimization Guide-BizTalk 

    Operations readiness Guide-BizTalk

    I would strongly recommend you to Run PAL ( get it from here Please) to your BizTalk farm while you are facing the latency issues. You would need to run the PAL on Biztalk and SQL box, by choosing the templates file in your perfmon. Please see here about how to use PAL.

    PAL might give you lots of infromation if there are any suspected bottleneck in your Farm.

    Also you run the MessageBox Viewer tool , It should be your starting point I think. Please see about Message Box Viewer tool here.

    You can refer this PPT , Very good by Paolo about perfromance tuning.

    All The very Best!

    HTH

    Naushad

    Wednesday, February 22, 2012 1:37 PM
    Moderator
  • Hi Martin,

    May be you would have resolved this issue, However I would like to suggest you that there is a new Technet Wiki Article "BizTalk Server: Performance Tuning & Optimization" , Please refer , Might give you some more details at one place!

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com,My New Technet Wiki Article “BizTalk Server: Performance Tuning & Optimization"

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!


    Monday, February 27, 2012 11:33 PM
    Moderator

All replies

  • Hi Martin,

    Thanks for providing the details!

    A suggestion from my side about the settings of your Isolated and WCF receive host which is 50 MS. I think keeping this value too low might create aload on SQL server , imagine in every 50Ms your host (BizTalk Service) is reading your database for more work in the Queues (WCF Receive and ISolatedQueue).

    Also you would need to make sure that your database infrastcture is properly tuned up to achieve required latency. There are guidelines from Microsoft to tune various parameters to achieve that. Please see following guides

    Performance Optimization Guide-BizTalk 

    Operations readiness Guide-BizTalk

    I would strongly recommend you to Run PAL ( get it from here Please) to your BizTalk farm while you are facing the latency issues. You would need to run the PAL on Biztalk and SQL box, by choosing the templates file in your perfmon. Please see here about how to use PAL.

    PAL might give you lots of infromation if there are any suspected bottleneck in your Farm.

    Also you run the MessageBox Viewer tool , It should be your starting point I think. Please see about Message Box Viewer tool here.

    You can refer this PPT , Very good by Paolo about perfromance tuning.

    All The very Best!

    HTH

    Naushad

    Wednesday, February 22, 2012 1:37 PM
    Moderator
  • Hi Martin,

    In addition to what Naushad suggested,  i would also suggest to review your design,

    If your front end user is waiting for the response and backend processing is time consuming process then its better to implement the Asynchrounous call where you need to implement Callback mechanism. Please check if its possible and useful in your case.


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/

    Wednesday, February 22, 2012 2:08 PM
  • Hi Naushad,

    thank you for reply and the suggestions.

    I will try running PAL to identify the "bottleneck" / reason for the latency problems. Do you know some best practice to limit CPU load for a biztalk host, meaning what would be the best parameter to tweak? Limiting Threads per CPU ? Orchestration Throttling, setting "pause at" and "resume at" values?

    I allready increased the polling intervals for the isolated and the wcf receive host, unfortunatly it didn't help me in scaling the CPU Load.


    Wednesday, February 22, 2012 5:37 PM
  • Hi Martin,

    I would say (based on my exp) there is nothing fixed which you can do to achieve required latency.  Tuning is not fixed set of things to perform, I guess It's an ongoing exercise (to change/record/evaluate) till you reach to the point of your required performance.

    However i will dig out some more information about your above question for Threads per CPU and Throttling settings. Also it depends upon the right choice of tools to perform your load testing/recording the results.

    Thanks

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!

    Wednesday, February 22, 2012 5:51 PM
    Moderator
  • Hi Martin,

    Just to add few more links, about performance tuning, I have compiled few links of Videos related with BizTalk, Please see those related with performance tuning.

    You can find them at http://alamnaushad.wordpress.com/videos/  

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com |@naushadalam

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!

    Wednesday, February 22, 2012 11:21 PM
    Moderator
  • Hello!

    Fun to see people using the threshold to have fun!

    The settings you have set up are decent. Where are you splitting the batch message? In the receive pipeline?

    When you encounter the issue when it goes slow what throttling state are you in? Can you also check the row count in the spool table (messagebox) when you encounter these results to provide some more information?

    Have you tried to turn global tracking off? and is all orchestration tracking turned off?

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Thursday, February 23, 2012 8:50 AM
  • Hi Martin,

    May be you would have resolved this issue, However I would like to suggest you that there is a new Technet Wiki Article "BizTalk Server: Performance Tuning & Optimization" , Please refer , Might give you some more details at one place!

    HTH,Thanks, Naushad (MCC/MCTS) http://alamnaushad.wordpress.com,My New Technet Wiki Article “BizTalk Server: Performance Tuning & Optimization"

    If this is helpful or answers your question - please mark accordingly! Please "Vote As Helpful" if this was useful while resolving your question!


    Monday, February 27, 2012 11:33 PM
    Moderator