Fine tuning performance is difficult and time consuming. It will be even harder after the system is put into production. So our first advice is to make sure you perform performance testing before going for production. The performance testing should cover the foreseen peak load as well as the average load.

Here are general guidelines in tuning and troubleshooting Orchestration performance:

Isolate. Isolate. Isolate.

Isolate your Orchestrations into a separate host. The separate host instance uses dedicated process-level resources and allows specific tuning of host-level throttling. If your Orchestration host is experiencing high memory, further separate your Orchestrations into more hosts. This will help narrow down the guilty orchestration(s). Depending on the orchestration design, it may be normal/expected for a process to use high memory.

Throttling

If Orchestration processing is slowing down, open Performance Monitor and monitor the Message Delivery Throttling State and Message Publishing Throttling State counters under the BizTalk:Message Agent object. If BizTalk is throttling, these counters will show a non-zero value. If a non-zero value is returned, look up the status code at the link below:

• How BizTalk Server Implements Host Throttling

This link also explains why throttling occurs and what you can do to fix it. The Process Memory Usage throttling option is usually changed from its default value of 25%. In BizTalk 2010, this setting is the Process Virtual memory threshold in Settings Dashboard.

When the address space of a host process is 2GB (32-bit operation system), throttling may occur after process memory usage crosses over 500MB, which is not much. On a 32-bit server, you can try increasing Process Memory Usage to 50%. Do not configure your host to use more than 1.54GB of memory (75%). On a 64-bit server, set this value to 100%, even if it's a 32-bit host. BizTalk is large address aware. The BizTalk host should not throttle until it uses more than 2GB of memory.

Also, your Orchestration host may not be throttling but Orchestration performance can still be affected if another host is throttling. For example, your Orchestration is bound to a SQL send port. If the SQL send host is throttling, messages cannot be published and your Orchestration will be impacted. As a general rule, if you have throttling, investigate the cause and determine if you can do something about it.

Messages/Variables scope

We often see cases where Orchestration instances use a large amount of memory. During development, take note of the scope of messages and variables. A message or variable can be defined at the Orchestration level or bound to a scope; similar to a global variable vs. local variable in any programming language. The Orchestration engine has logic to invoke garbage collection on messages that are not used later in the Orchestration. But, you don’t have absolute control over when GC kicks in. With large messages or object types, it makes sense to define the message or variable in the scope they will be used. Furthermore, if you load large objects into variables, such as a large XML into XMLDocument, it is good practice to clean up the object as soon as you are done with it.

Persistence Points

For long-running Orchestrations to maintain state information, the Orchestration engine saves instance state to the BizTalkMsgBoxDB at various persistence points. Guaranteed persistence points are:  

• Send Shape (after a message is sent)
• Start Orchestration Shape
• Suspend Shape
• End of a Transactional Scope (atomic or long-running)
• An Orchestration Debugger breakpoint is hit
• Orchestration Engine determines that the instance needs to be dehydrated
• When the Orchestration Engine is shutdown; through controlled shutdown of the host or abnormal circumstances. The engine tries to persist but if that fails, the Orchestration instance will resume from the last successful persistence point.

The Receive Shape, Listen Branch, and Delay Shape are conditional persistence points. They will dehydrate if there is a wait for a subscription that takes longer than 2*MaxThreshold. In BizTalk 2010, the Maximum Threshold value is configured in Settings Dashboard.  In BizTalk 2009 and earlier, these values are configured in BTSNTSvc.exe.config or BTSNTSvc64.exe.config file. 

• How to Calculate Dehydration (http://go.microsoft.com/fwlink/?LinkId=239975) provides more information.

Each persistence point requires your Orchestration instance and the associated data to be serialized to the database. This can impact performance if you have a large number of persistence points or if there is large amount of data to be serialized. During design time, you can use the persistence points listed above as a guide to adjust your workflow. During runtime, the dehydration and persistence counters in Performance Monitor under the XLANG/s Orchestration performance object can be used to determine if excessive persistence may be a factor in your performance issue.

To prevent persistence, enclose a potential persistence point within an Atomic Scope. The Orchestration Engine won’t persist until it reaches the end of the transaction. But, be careful when you do this! It does keep the Orchestration instance from persisting but also forces the Orchestration instance to remain in memory instead of dehydrating. It is possible to run out of system resources if you have a large number of active Orchestration instances.

Map

Transformations are more efficient when they are executed from a send or receive port. There are scenarios where it makes sense to have a map in an Orchestration. For example, you can execute a transformation dynamically within an Expression Shape. Maps in Orchestration can also accept multiple inputs. In general, unless there is an absolute requirement to use a map in an Orchestration, don't do it.

Distinguished Field vs. Promoted Property

When you promote a property, consider how you’ll be using it. If you will only access the data within an Orchestration, promote the property as a Distinguished Field only. If you plan to use a field outside of an orchestration, like as a filter for a Send Port subscription, use it as a Promoted Property. Promoted Property Fields can be used for routing but are also more expensive.

External Components

It is normal and expected for Orchestrations to use external assemblies within an Expression Shape. When you call a method from an external assembly or unmanaged code, be aware that the Orchestration Engine has little control over what happens within the external code. If this external code fails, your Orchestration design needs to handle the exception. If the method call does not return in a timely manner, the Orchestration Engine will continue to wait on it. Do not assume or expect that the Orchestration Engine will "timeout" or terminate the external component. If you want a timeout, build it into the method you are calling.

With any custom code, you should implement tracing and/or debugging information. We have seen many cases where an Orchestration instance fails within custom code. Since there was no tracing within the component, a lot of time was spent isolating the failure. Once the problem was isolated to the custom component, the issue still needed to be resolved by the original developer since we didn’t have any knowledge of the component. It is more efficient for you if we can quickly determine what is failing and who should be engaged to fix the issue.

Latency

If a message is received after an orchestration has been idle, there may be a delay with the orchestration. This may be caused by an application domain unloading within the BizTalk host process. If an AppDomain is shut down, the next message must wait for the Orchestration to reload. This can be a noticeable wait time. To prevent this in a low latency scenario, add the SecondsIdleBeforeShutdown property to the BizTalk config file (BTSNTSvc.exe.config or BTSNTSvc64.exe.config) file:

SecondsIdleBeforeShutdown is the number of seconds that an AppDomain is idle (contains only dehydratable orchestration instances) before being unloaded. Setting it to -1 will prevent the AppDomain from ever shutting down due to being idle.

SecondsEmptyBeforeShutdown is the number of seconds that an app domain is empty (does not contain any orchestration instances) before being unloaded. Setting it to -1 to signal that an AppDomain should never unload, even when empty.

Tracking – Orchestration Events

Document tracking can impact performance throughout BizTalk, not just Orchestration. Orchestration Event Tracking is enabled by default. This can be helpful during development, testing and troubleshooting with Orchestration Debugger. In Production, these tracking events should be disabled.

When enabled, these events are written to the TrackingData_x_x tables in the BizTalkMsgBoxDb database and then moved to the dta_DebugTrace table in the BizTalkDTADb database by the Tracking host. If the TrackingData_x_x tables (BizTalkMsgBoxDb database) and/or the dta_DebugTrace table (BizTalkDTADb database) get too large*, performance will be impacted.

• How to maintain and troubleshoot BizTalk Server databases (http://support.microsoft.com/?id=952555) provides information on these tables.

If the Tracking host cannot move data efficiently into the BizTalkDTADB database, data will accumulate in the BizTalkMsgBoxDb database. A large BizTalkMsgBoxDb database can cause all hosts to slow down and eventually lead to throttling.

Thread Pool Size

The Orchestration Engine and many of the BizTalk Adapters use worker threads from the managed thread pool. If there is a large number of orchestration instances that must be active, it's very common to increase the thread pool size. If you suspect thread starvation, monitor the Running Orchestrations Performance Monitor counter under the XLANG/s Orchestrations object. The number of running orchestration instances may increase and then remain constant.

You can usually detect this by monitoring the Running Orchestrations performance counter under the XLANG/s Orchestrations performance object. You may notice that the number of running Orchestrations increases and then remains constant when it reaches the thread pool limit. In BizTalk 2010, the thread settings can be modified in Settings Dashboard (http://go.microsoft.com/fwlink/?LinkId=239982). In BizTalk 2009 and earlier, add the CLR Hosting registry key (http://technet.microsoft.com/library/dd722826(BTS.10).aspx).

Before you suspect thread pool as an issue, confirm there is no throttling and that your system is not already under stress.

Long-Running Singleton Orchestrations

Singleton orchestrations usually involve correlated receives within a loop. This design pattern is often used for batching, maintaining ordered processing or throttling downstream flow. Using a singleton is fine normally doesn’t raise a performance concern. If there is a looping condition that keeps an orchestration alive for days or weeks, you may notice a gradual performance degradation. The reason? As an orchestration loops, it is possible for different objects to accumulate and eventually impact performance.

During development, make sure all objects you create are cleaned up when no longer needed. Also, add logic to limit the lifespan of your singleton orchestration. The mechanism would depend on your business scenario. For example, terminate an orchestration instance only when it is idle. 

a) Add a Listen Branch into the loop.
b) On one branch, you receive the correlated message as before.
c) On the other branch, use a Delay Shape and specify the timer value.
d) Add an Expression shape with logic to exit the loop.

The concept is simple. When messages are flowing in, the execution path will stay on the receiving side of the Listen Branch. When message delivery stops, the timer value in the Delay Shape will exceed. Then, the execution path moves to the delay branch. After the Delay shape, set a Boolean variable in an Expression Shape to exit the loop. The orchestration instance completes and the next message delivered to the orchestration engine starts a new instance of the orchestration.

Using XLANGMessage and XLANGPart

When passing XLANGMessage and XLANGPart to a custom component, the custom component generally needs to take care of its lifetime requirement and dispose it promptly. Messages Represented as XLANGMessage (http://go.microsoft.com/fwlink/?LinkId=239985) has the details.

Acknowledgements

This Wiki post was taken from the following blog posts and updated to reflect BizTalk 2010:

• Thoughts on Orchestration Performance: http://blogs.msdn.com/b/biztalkcpr/archive/2008/05/08/thoughts-on-orchestration-performance.aspx
• Thoughts on Orchestration Performance – part 2 of ?: http://blogs.msdn.com/b/biztalkcpr/archive/2008/05/30/thoughts-on-orchestration-performance-part-2-of.aspx

See Also

Read suggested related topics:

• BizTalk Server 2006 Operations Guide: http://msdn.microsoft.com/en-us/library/cc296643%28v=bts.10%29.aspx
• BizTalk Server 2009 Operations Guide: http://msdn.microsoft.com/library/ee308947.aspx
• BizTalk Server 2010 Operations Guide: http://msdn.microsoft.com/en-us/library/gg634499.aspx
• BizTalk Server Performance Optimization Guide: http://msdn.microsoft.com/library/ee377064.aspx
• How to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process: http://support.microsoft.com/?id=918643
• How to maintain and troubleshoot BizTalk Server databases: http://support.microsoft.com/?id=952555

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.