Microsoft StreamInsight is more than just an API for writing streaming queries. Included with the StreamInsight installation are powerful tools that allow you to manage, monitor, tune, and troubleshoot your streaming data applications. These tools range
from diagnostic methods and properties built into the API to a full graphical debugging application.
StreamInsight is built upon a client/server architecture: a client deploys queries to a server where they are executed. In the simplest case, the server actually can be embedded within the client, keeping everything within a single application. But, more
typically, the StreamInsight server resides in a separate application, running on the same machine as the client or running on a separate machine or even in a server farm.
For more information, see
StreamInsight Server Deployment Models and
Deploying StreamInsight Entities to a StreamInsight Server.
This architecture provides you with many options for managing performance by giving you choices on where to deploy computation-intensive queries. It also allows for an external Event Flow Debugger with which you can monitor, troubleshoot, and manage active
queries within a running StreamInsight server.
StreamInsight includes a stand-alone Event Flow Debugger with a graphical user interface. The debugger enables you to inspect, debug, and reason about the flow of events through a StreamInsight query.
The debugger can be used for two primary purposes:
The following is an example “query graph” view in the debugger:
For more information on the Event Flow Debugger, including available options and examples, see
Using the StreamInsight Event Flow Debugger.
With StreamInsight, extended features are built into the API that make it easier for you to develop, troubleshoot, and manage your application.
StreamInsight supports resiliency against system failures during data stream processing by providing a checkpointing feature that can periodically save the state of a query to disk. After an outage, the query can then be restored to its state as of the checkpoint.
Note that if an outage does happen, events that occurred after the checkpoint need to be replayed and presented again to the query after the query has been restored to its checkpoint state. Events that were received after the checkpoint but before the outage,
when they are replayed, will be duplicates, and this needs to be accounted for.
Fortunately, StreamInsight takes these issues into account by providing three levels of resiliency. Selecting a level depends on your requirements and your ability to change existing applications, sources, and sinks.
For more information, including examples of creating and configuring a resilient server and setting up checkpoints, see
StreamInsight provides a diagnostic view API that allows you to monitor the server, queries, and entities in your StreamInsight application. You can find information such as the current state of a query (Initializing, Running, Suspended, Stopped, etc.),
query latency, or entity properties. You can use Windows PowerShell to access the manageability information or manage metadata for a running instance.
The following are some examples of the kinds of issues you can troubleshoot using diagnostic views.
You can focus on specific areas of event flow, including events incoming through sources, consumed by queries, produced by queries, and outgoing through sinks. This allows you to measure things such as latency or memory consumption at different specific
points in the event flow.
For a definition of the DiagnosticView class, see
Microsoft.ComplexEventProcessing.DiagnosticView. For more information on how to use diagnostic views, including examples, see
Monitoring the StreamInsight Server and Queries.
StreamInsight automatically installs and configures performance counters for StreamInsight servers, queries, input adapters, and for resiliency. Server counters are turned on by default and cannot be turned off. Query counters are off by default but you
can turn on counters for individual queries (and any associated adapters) using the
The following performance counters are available.
Input Adapter counters
Here are some example scenarios in which performance counters can help you understand, monitor, and troubleshoot your StreamInsight applications:
StreamInsight also logs events to the Windows Application event log from both the server and queries. These logs identify events such as the server being created or a query starting or being suspended. You can turn administrative logging on or off using
the SetDiagnosticSettings method.
The following are events you may see in the event log.
For more information, including examples of initializing and configuring performance counters and administrative event logging, see
Monitoring StreamInsight Performance Counters and Events