Microsoft BizTalk Automatic Throttling

Microsoft BizTalk Automatic Throttling


Introduction

To maintain the stability of your environment Microsoft BizTalk provides automatic throttling according to the conditions of your environment. For most companies this throttling is more than enough, performing manual throttling is not used broad, however more and more are doing this. Microsoft BizTalk also creates states to determine the severity of the throttling states, all these can be monitored and designated actions can be taken for all these states.

BizTalk does a lot of different communication towards the BizTalk database these are called "publishing" this is where are a receive port or orchestration publish messages into the Microsoft BizTalk databases, the other one is "delivery" this is done by send ports or orchestrations, it's when a delivery to a third-part is done. An example of this would be a file share or a WCF service. Microsoft have many great articles regarding this issue. All are mentioned in the reference at the bottom of this page.

Scenarios that may cause an automatic throttling state

There are many scenarios that may expose you to an automatic throttling state, a few of this are:

  • Poorly developed applications
  • Wrong configuration in BizTalk
  • Network related issues
  • Hardware issues
  • Exceeding thresholds

The automatic throttling states can be used to stabilize environment, you can set different thresholds to manage the use of resources for a host, by setting thresholds for threads will make sure that the host never will use more threads, and will make sure it won't eat up all your threads. This can be used when performing performance optimizing by limiting the use of resources for some hosts, and granting more to others.

Message Publishing throttling state:

2 – Incoming rate exceeds outgoing rate

This will happen if the incoming message rate exceeds the message outgoing rate, and the database experience problems with coping with the increasing amount of messages being published to the BizTalk databases. In order for BizTalk to deal with this problems it simply blocks the publishing of messages until incoming rate is at par with the message outgoing rate. You can view this scenario in real-time by using the performance monitor and look at the counters described in the table below:
Counter Sub counter
BizTalk:Message Agent Message publishing incoming rate
BizTalk:Message Agent message publishing outgoing rate

4 – Memory exceeds specified threshold

This may happen if you have a steep increment of memory use in for example large batch files. BizTalk will automatically reduce the size of the EPM thread pool, and stops any creation of new threads, if the batch requires a lot of resources it will actually slow down the process in order to maintain the availability for the rest of the environment. Sometimes the message if it is too big and including too many messages may be blocked out completely by BizTalk. In order to monitor this you need to open the following views in Performance Monitor:

Counter Sub counter
BizTalk:Message Agent High process memory
BizTalk:Message Agent process memory usage (MB)
BizTalk:Message Agent process memory usage threshold (MB)

5- System memory exceeds threshold

This state couldn't be described any better, the system memory exceeds the specified threshold and simply stops the EPM to start processing new messages and reduce the current EPM thread count, if the reason for this state is because a steep memory usage of a batch file it will take longer to finish it, and if it is too big it will be dropped by BizTalk. Use the following counters too see this issue in Performance Monitor:

 

Counter Sub counter
BizTalk:Message Agent Physical memory usage threshold (MB)
BizTalk:Message Agent Physical memory usage (MB)

6 – Host message size

The reason why you encounter this state maybe that some of your SQL jobs have stopped, tracking host is not running or the number of suspended messages is high. This state will as the one above create the same scenario as above, it will reduce the threads for EPM, and it also blocks the EPM to process any new messages and creates a publishing delay for all messages. In this scenario there are several counters to monitor:

Counter Sub counter
BizTalk:Message Agent database size
BizTalk:MessageBox : General Counters Spool size

Tracking data size
BizTalk: MessageBox : Host Counters Host queue length

Host queue suspend message length

8 – Database session exceeds threshold

This state simply states that the sessions used by the host towards the database exceeds the specified threshold. The auto throttling here will do the same as in state 4 and 6, reduce the threads for EPM and stop the execution of any the threads, it will also delay all ongoing messages until the database is back to normal. If this happens I recommend you to reduce the database connection threshold for the host. You can monitor this by looking at the following counters in Performance Monitor:

Counter Sub counter
BizTalk:Message Agent Database session

9 – Process count exceeds threshold

What it says is that the process counts exceeds a specified threshold, the reason for this may be that you have to many transactions on that hosts that creates too many processes, try reducing the amount of allowed threads on the host so that the system don’t create too many threads. The self-throttling will do the exact same as in state 8 and basically reduce the EPM thread count, stop it from creating new and messages will be delayed until the host is back to normal state again. You can monitor this by using the following counters in Performance Monitor:

Counter Sub counter
BizTalk:Message Agent Thread count
BizTalk:Message Agent Thread count threshold

Message delivery throttling state

1 – Incoming rate exceeds outgoing rate

This happens if the message incoming rate is greater and exceeds the message outgoing rate, it can be caused by a shortage in resources or by high process complexity. BizTalk perform a self-throttling blocking the delivery thread for a computed time until the message incoming rate is at par with the outgoing rate. To monitor this use the following counters in the Performance Monitor:
Counter Sub counter
BizTalk:Message Agent Message publishing incoming rate
BizTalk:Message Agent message publishing outgoing rate

3 – in-process delivery exceeds threshold

This state informs us that the a number on in-process messages delivered to a service class exceeds the specified threshold and can be caused by processing high complexity messages, slow outbound adapters or a shortage in resources on the server. BizTalk self-throttles and slows down the message delivery to XLANG or adapters, It also reduce the size of the thread pool by the message agent. You can monitor this by looking at the following counters in Performance Monitor:

Counter Sub counter
BizTalk:Message Agent In-process message count
BizTalk:Message Agent In-process message count threshold

4 – Process memory exceeds threshold

This one is also covered well in publishing throttling states, it basically tells us that we exceed the specified value of the process memory threshold. It occurs when you have large and complex messages that are being processed, or a high amount of message simultaneously. This sate will not make the system idle so no immediate action is needed if this is a once in a life-time scenario, however BizTalk will still perform self-throttling and slow down delivery of messages to adapters and XLANG. It may also reduce the memory usage by dehydrating messages and reduce the size of the EPM thread pool sometimes it also force a .NET garbage collect. You can monitor this by using the counters described below:

Counter Sub counter
BizTalk:Message Agent High process memory

Process memory usage (MB)

Process memory usage threshold (MB)

Active instance count

In-process message count

5 -System memory exceeds threshold

Basically the systems memory exceeds the specified threshold and BizTalk will slow down message delivery to adapters and XLANG, it will also try to decrease the memory consumption by dehydrating messages it also reduce the thread pool size of EPM. If you have this issue often you should consider reducing the EPM thread pool to decrease the occurrence of this throttling state as it will slow down BizTalk drastically. You can read the monitor for this in Performance Monitor as well by using the following counter:

Counter Sub counter
BizTalk:Message Agent Physical memory usage

9 -Process thread count exceeds threshold

You will encounter this sate if BizTalk has a process thread count that exceeds the specified threshold. BizTalk will automatic throttle to decrease the thread count used by EPM and you should consider changing the different thread throttling settings for the different hosts. You can monitor this issue too in Performance Monitor by checking the following counters:

Counter Sub counter
BizTalk:Message Agent Thread count

Thread count threshold

References

MSDN: How BizTalk Server Implements Host Throttling

See Also

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

Sort by: Published Date | Most Recent | Most Useful
Comments
  • Compliments on this great article!

  • Thank you Steef-Jan!

  • A very important issue, and a good article.

    From my experience, I dont remember much cases where a system I was working with, was going through a serious throttling state and managed to overcome the stress using the biztalk automatic mechanism. Even though many parameters are configurable, I think that most of the time, steps taken by the biztalk engine dont solve the cause of the throttling, but more help to avoid critical stress on resources that may cause general crash down.

    I think it's not recommended to count on throttling mechanism to save the day in case of serious load on the systems. Better to make sure you dont get to that state at all

  • Hey Shay, I can see why you feel it like that, however remember that you can use this to limit the use of resources, and the throttling states will hit when such a scenario occur. I updated the article with more information regarding this.

    Throttling states are vital to follow if you use them and need them, and whenever you have issues with BizTalk checking the throttling states will often give you a quick idea of what the issue might be. And network related issues can occur anytime, and by viewing the throttling state you can eaiser locate where the bottleneck is. Understanding BizTalk performance and optimizing is hard and time consuming but from my experience checking throttling states can often locate the bottleneck faster then checking all other locations first.

    Anyways, a good tip and contribution to the wiki article! Have a nice day!

  • Nice article. Easy to understand

  • Awesome article Tord.

  • Thank you Nino!

  • Great article Tord, as you always say, BizTalk Throttling is your friend!

  • Its really a wonderful atricle but my bad i didnt see this while i was perfoming perfomance testing for a newly setup production environment. Anyway, this will help me in future!

  • Great article

Page 1 of 2 (14 items) 12