BizTalk Health Check

BizTalk Health Check


Health checking

Maintaining a good and healthy environment for BizTalk is important, this will provide you with a better overview and safety when it comes to security and scalability of your BizTalk installation. You can use a couple of third-party tools but not all elements are checked. I've gathered a list of checkpoints that should be covered when performing a health check. The health check is pretty much the same as the yearly check you have with the doctor and is recommended to be performed at least once a year, if you don't feel comfortable doing one yourself you can invite Microsoft to do one for you. They will give you a good health report back describing what they found and how to solve it.

There are some free tools online to do some of this job for you, you can download and run the two following applications:

Patching:

What are the internal routines to install windows critical updates? These should be reviewed. As long as routines are followed and preferably updates are installed once a month you should be fine. 

You should also run the Microsoft Baseline Security Analyzer (MBSA) on the servers Both on SQL servers and BizTalk servers. This is to identify and inconsistencies between the deployed updates and the current list of recommended updates.

Update COM+ and MSDTC.

Both COM+ and MSDTC are used by BizTalk, patches and hotfixes for these services often provide hot fixing to improve tuning and stability.

General Network

The following points explains how to maintain and keep your general network information up to date, some of these elements should also be used when you think you encounter network related issues.

DTCPing of all the servers

There should be no packets lost during this ping.

Transfer 100MB data

If you have a 1gb network the response should be 5 seconds or faster (average between 3-5 seconds). Do to all the servers. If you are running a 100mb network up to 20 seconds is within the acceptable transfer rate (average is around 8 - 10 second)

Pathping

Do a pathping from the BizTalk server to all the SQL servers. You should have no packet loss during this ping.

TCP/IP port Exhaustion

Troubleshoot to see if you are using less than 3000 ephemeral ports. To do the counting write the following command on the BizTalk Servers "netstat -ano -p tcp" Count the number of unique Local Address TCP ports open above 1024 for each IP address. Using TCPView tool makes this is a lot easier.

DBNETLIB Exceptions

Avoid this, DBnetLib (Database Network Library). The most common error when this occurs is when one of the BizTalk MessageBox becomes extremely busy. Attempts to communicate with the busy MessageBox database results in a timeout. Look for 5410 errors in the EventLog. Example of error message:
Event Type: Warning
Event Source: BizTalk Server 2006 R2
Event Category: BizTalk Server 2006 R2
Event ID: 5410
Computer:
Description:
An error occurred that requires the BizTalk service to terminate. The most common causes are the following: 1) An unexpected out of memory error. OR 2) An inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shut down and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

TCP/IP

This is only a check for the BizTalk servers that have a direct connection to the internet such as those residing in a perimeter network hosting HTTP and SOAP adapters If the BizTalk servers reside in a vulnerable network location then check to ensure any internet/public facing network adapters have NetBIOS over TCP/IP disabled

Server general

 This section will cover general health checking for the servers, and must be used on both BizTalk servers and SQL servers.

Time synchronization

It is vital that the time between the BizTalk server and SQL server is synchronized. Check to see if the time is within the valid synchronization by typing the following command in command prompt "w32tm /stripchart /computer " to resynchronized the clocks
type the following in the command prompt window "w32tm /resync /computer "

BIOS Version

BIOS should be updated, this is because the releases from the manufactures can provide better stability, and network related performance.

Update the Certificate RevocationListUpdates

If it takes a long time to start up a BizTalk server it may be because you don't have access from the BizTalk server to reach the Microsoft.com domain. You might get some start-up issues because the .NET framework will try to download the Certificate Revocation LIST (CRL) from Microsoft.com. You can update your servers manually by following these two links:

http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl and
http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl

BizTalk in general

This part will cover general health checking of the BizTalk servers

The Host separation policy

BizTalk solutions typically consist of executable artifacts - receive locations, pipelines, orchestrations, send ports, and so on. BizTalk hosts provides a way to logically group these artifacts and allow you to instance each host on one or more BizTalk Servers. A Host instance represents a physical run time windows service or process running on a BizTalk. This gives you a very flexible solution to host BizTalk application components, there are many factors which should be taken into account when distributing your applications across hosts.

Performance

  • When you create a new host a set of dedicated databases tables is provisioned in the BizTalk databases. In the BizTalkMsgBoxDB they act as the work queues for that host. This can help to gain low latency situations by creating multiple queues for different workloads, thus presenting messages with low latency requirements getting delayed behind large batches of none "time-critical messages".
  • Each host instance has its own set of system resources such as memory threads and handles in the .NET thread pool. Instead of hosting all executable artifacts in a single host it can be beneficial to spread the load across multiple hosts.
  • BizTalk throttling is implemented on the host level. You may experience that you need to set a specific throttling parameter in order to control individual artifacts. If you need to do this I would recommend you to separate artifacts across different hosts so that you don't apply unnecessary throttling to all artifacts.

Functionality

It is recommended to separate reviving, orchestration, sending and tracking functionality into separate hosts. This will help you to manage and scale each type of processing independently. IF you want to stop all reviving or all orchestration processing. Or adding additional servers to the BizTalk group to handle dedicated orchestration processing only.

You should also have hosts for application that only can run on one server, POP3, FTP, MSMQ receive. These hosts should be grouped and clustered in the BizTalk console. All others can remain in hosts which supports multiple running host instances.

Separation application processing across hosts ultimately leads to a multiple runtime processes hosting you applications. As BizTalk can host custom written components (all custom components you have), host separation can be used to provide process isolation and protection. If you experience instability in any custom components, you should isolate them into their own BizTalk host. This should be done to prevent the instability to affect other applications.

Security

Each BizTalk host is assigned to a windows group which is used to give the host permission to specific tables in the BizTalk database. This is important when dealing with highly sensitive data.

Certificates can be utilized in BizTalk for message signing and encryption. This settings is assigned in the host level and therefore you may need to separate hosts in order to represent different identities.

All host instances are assigned with a service account to run under. Access to external systems is often a requirement for this account in order to process outbound messages through the adapters.

Set up a dedicated tracking host

Any hosts that holds tracking is responsible for moving the DTA and BAM tracking data from the MessageBox database to the BizTalk Tracking (DTA) and BAM Primary Import databases. The moment of tracking data has an impact on the performances of other BizTalk artifacts running on the same host that is hosting tracking. Therefore you should have a dedicated host that does nothing but tracking.

To use a dedicated host for tracking also allows you to stop other hosts without interfering with the BizTalk server tracking. The movement of tracking data out of the MessageBox database is critical for healthy systems. IF the BizTalk host is responsible for moving the tracking data in the BizTalk group is stopped, the Tracking data decode service will not run. Impacts of this is:

HAT tracking data will not be moved from the MessageBox to the BizTalk Tracking database. Same goes for BAM tracking data, this will not be moved to BAM Primary Import database.

  • Because data is not moved, it cannot be deleted from the BizTalkMsgBoxDb.
  • When the tracking data decode service is stopped, tracking interceptors will still write tracking data to the MessageBox database. IF the data is not moved, this will cause the MessageBox database to become enormous. which will impact performance over time.
  • It is also important for security to have a dedicated tracking host, since the tracking host has access to all the messages across the message box database.
  • Turn of tracking on all other hosts, they will still write tracking but they won't have read access.

Isolated Adapter per Application Pool

An application pool which hosts an isolated adapter represents the physical implementation of an isolated host instance. This means that the application pool (w3wp.exe process) becomes an instance of BizTalk because the isolated adapter loads the BizTalk runtime components into the process to perform message processing. BizTalk only supports a single isolated adapter per application pool.

Remove unnecessary BizTalk items, components and settings

It is important to keep a clean environment. Remove old receive locations, send ports and applications that are no longer in use. Artifacts which only were used during testing and development should be removed. The output from the BizTalk Documenter tool can be used for this.

The BizTalk configuration file

BizTalk runtime process is like other .NET applications and have configuration files that can be used to control certain application behaviors. They control run-time validation, orchestration dehydration and provide custom application settings for custom .NET components hosted in BizTalk.

BizTalk Registry Key Validation

BizTalk stores a variety of configuration data in the windows registry. This data can have an impact on the behavior of the BizTalk runtime processes and associated services. There is one specific area which is recommended to tune for the default registry values and these control the .NET thread pool used by BizTalk. You can find the values here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$hostname\CLR Hosting

Adjust the BizTalk cache refresh interval

The BizTalk cache refresh interval is the time (seconds) that all items in the BizTalk server administration cache must wait for the configuration cache to refresh.
The following elements are refreshed when the cache is refreshed:
  • MessageBox databases
  • Server properties
  • Adapters
  • Connections towards the tracking database (BizTalkDTADb)

By default all objects are refreshed every 60 seconds. Except for the database connection and server properties.

Adjust adapter batch settings

A receive adapter can make us of batching in order to reduce the numbers of database "round trips" required to publish messages to the BizTalkMsgBoxDb (message box).
Batch sizes can be configured on certain receive adapters allowing batch processes to be tuned to the desired requirement on the deployed solution.

If you increase the batch job it can reduce the queries towards the BizTalk MessageBox, but it may increase individual message latency. Alternatively decreasing the batch size can improve individual message latency (fast response) but it comes with a cost of more queries towards the database, and transaction overheads.

Adjust the MaxReceiveInterval

The MaxReceiveInterval is the amount of time each host instance will wait between dequeue requests. The messages that flows though BizTalk are queued in the BizTalk MessageBox. Host instances has its own rows of data, they poll their respective queues within the database to retrieve messages. When it exist messages in the queue the MaxRetriveInterval will be ignored, but when queue is empty the host instance will back off and poll the queue at intervals based on the value, until more messages are retrieved.

The MaxReceiveInterval value can be changed in the BizTalkMgmtDb (BizTalk management database) and the adm_ServiceClass table. The table contains one record for each service type:

  • Messaging Isolated
  • Messaging In-Process
  • XLANG/s
  • MSMQT

Best practice is not to adjust this from the default value of 500 milliseconds unless you require a low latency environment.

Adjust the BizTalk tracking

When a message comes into the system, various elements are stored and pass through a series of processes and databases and ends up writing in to the tracking database.

It is vital that you only track information that are necessary for your use. Make sure the context only provides the information you need. Never store the entire message or huge elements in the context of the page.

The highest cost is when you track message body tracking, business rules tracking and orchestration event shape tracking.

Latest BizTalk server service pack and/or critical hotfixes

Updating and installation of hot fixes can be vital to keep your environment safe and up to date. The updates can provide improvements on reliability, security and performance.

BizTalk SQL Servers

This part will cover information for the BizTalk SQL servers, these are dedicated changes that should be performed to a BizTalk SQL server.

SQL service pack and / or critical hotfixes

Do you have routines on patching the SQL database? Updates from Microsoft can provide you with updates to help performance, security and reliability. It is important to follow upon these updates. The latest SQL services packs at the moment is:

SQL server 2000 - Service Pack 4
SQL server 2005 - Service Pack 4
SQL server 2008 - Service Pack 3 CTP
SQL server 2008 R2 - Service Pack 1

Max degree of parallelism

MDOP (Max degree of parallelism) is an instance-level setting on the SQL server. Default is "1" this value is set on the SQL server instance(s) that host a BizTalk Server MessageBox. This value should not be changed, if you change the parallelism setting to "1" for an instance of SQL server will have an adverse effect on other database applications that also is hosting the BizTalk MessageBox database(s).

Auto update/create on SQL

Auto create and auto update are turned off by default in the BizTalk Server MessageBox database. If you choose to have this on it can cause undesirable query execution delays.

SQL server agent

The SQL agent runs many important tasks for BizTalk in order to maintain and take backup of the databases. On a clustered environment the SQL agent bust follow the cluster and the service should start automatically. If it is running on a single server you should have the service on automatic start up.

SQL jobs for BizTalk

There are a set of jobs defined by BizTalk and used to maintain stability and backup.

  • MessageBox_Message_Cleanup_BizTalkMsgBoxDb - The job removes all messages that are no longer being referenced by any subscribers in the the BizTalk MessageBox.
  • MessageBox_Message_MessageRefCountLog - This job manages the reference count logs for messages and determines when a message is no longer referenced by any subscribers. NOTE: Although this agent job is scheduled to run every minute the stored procedure that is called by this job contains logic to ensure it runs continually. Do not edit this job.
  • MessageBox_Parts_Cleanup_BizTalkMessageBoxDb - This job removed message parts that are no longer in use or referenced by and messages in the MessageBox (BizTalkMsgBoxDb).
  • CleanUpBTFExpiredEntriesJob_BizTalkMgmtDb - This job cleans up expired BizTalk framework.
  • MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb - If a host is instance has stopped, this job releases all the work so another host instance can take over the work.
  • MessageBox_UpdateStats_BizTalkMsgBoxDb - This agent job updates the statistics for the MessageBox database.
  • Operations_OperateOnInstance_OnMaster_BizTalkMsgBoxDb - this job is only needed if you have multiple MessageBoxes. The job performs asynchronously operation actions such as bulk terminate on the master MessageBox after those changes have been applied to the subordinate MessageBox databases.
  • PurgeSubscriptionsJob_BizTalkMsgBoxDb - This job purges unused subscription predicates from the MessageBox.
  • Rules_Database_Cleanup_BizTalkRuleEngineDb - This job purges automatically old audit data from the Rule Engine database every 90 days. It also purges old history data like deploy/undeploy information.
  • TrackedMessage_Copy_BizTalkMsgBoxDb - This job copies the message body of a tracked message to the tracking database.
  • Backup BizTalk Server - Maybe not needed to explain but it takes a full backup of databases and logs.
  • DTA Purge and Archive - This jobs archives data from the BizTalk Tracking database and purges obsolete data.

See also the TechNet Wiki BizTalk Databases: Survival Guide

Database autogrowth

SQL servers provide an auto-growth feature. All BizTalk databases should be MB not %. and no bigger den 100 MB(for large files), 10 MB(for medium-sized files) and 1 MB(for small files).

Disk separation for the BizTalk databases

To separate the disks will help for performance. It is recommended to put the data and log files on separate disk for the BizTalkMsgBoxDb and BizTalkDTADb.

Both BizTalk Servers and SQL Servers

These elements goes for Both BizTalk Servers and SQL Servers.

Move the Pagefile

If you run out of RAM when you process large amount of data, windows change some background processes to a pre-allocated location of the hard drive and making in at part of system RAM. You should therefore never have the pagefile on the system disk but on a dedicated or second disk.

Defrag your drive

Since BizTalk has a lot of disk I/O its vital to have a periodically defragmentation of your disk to maintain performance and stability. Recommended by me is once a week.

Move TEMP and TMP folders

As the page file (though different tasks) its important to not run this on the system disk, due to the issues when dealing with large files. The disk I\O when dealing with large files can impact the performance since BizTalk uses the temp folders. It is therefore recommended to move the TEMP and TMP files for the BizTalk user to its own dedicated disk.

Disable unused windows services

There is no need to run windows services other then required for Windows and BizTalk to run, services like "Print Spooler". Disable them from running so they don't take up necessary resources from the BizTalk server.

Antivirus

Disable the real-time scanning on files and folders used by BizTalk if its possible. Local disks, ASP.NET temp folder. (this will also increase performance)

Free disk space

Its recommended to have at least between 20%-25% of total free disk space. If it drops below this you could experience a disk I/O issue. The last data a disk has the faster it will operate. A disk write to the outside of the spindle and write towards the middle, the middle spindle is the slowest part of the disk.

Monitoring Health in production environment

Performance Bottlenecks

In case you experience performance bottlenecks with you BizTalk environment or experience issues you can use the PAL (Performance Analysis of Logs) tool and BizTalk MessageBox Viewer. PAL is is a powerful tool that reads in a performance monitor counter log and analyzes it using known thresholds. It basically automates the analysis of performance counter logs. PAL was originally created by Clint Huffman and recently updated to version 2.2.1.

Health Check with MBV

The BizTalk MessageBoxViewer created by Jean-Pierre Auconie. The MBV tool retrieves information from a BizTalk System and identifies all possible issues, which could be critical or need attention, and present them in a user friendly format.

A lot of these points can be found in the health check from Microsoft. This is by no means a copy of it and you should always use Microsoft for valid and good health check report of your environment, this is just a guidance for you to do a small health check yourself, Credit to Microsoft for this article also follows as they "created" most of this checks in some way or the other.

See Also

Read suggested related topics:

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

Sort by: Published Date | Most Recent | Most Useful
Comments
Page 1 of 2 (20 items) 12