BizTalk Application Insights in depth – Part 1
In my previous blog, I explained about installing BTS 2016 feature pack 1 and configuring it for Application Insights integration. In this article, I want to go a bit deeper and try to demonstrate,
- Nature of tracking data sent to Application Insights
- Structure of the data
- Querying data in Application Insights
- Some practical examples of getting sensible analytics for BizTalk interfaces.
I am hoping to give a jump start to someone who wants to use the Application Insights for BizTalk Server 2016.
Tracking data
As you know the term tracking data in BizTalk refers to different types of data emitted from different artifacts. It could be in/out events from ports and orchestrations, pipeline components, it could be system context properties, could be custom properties tracked from custom property schemas, could be message body in various artifacts, could be events fired from rule engine etc. So we would like to know, whether we will be able to get all this data in Application Insights or is it just a subset. I will try to answer this question based on the POC I have created.
POC I created is pretty simple. It has one receive port which receives an order XML file, processes that in an orchestration and send it to two different send ports. It can be pictorially represented as below.
- I have enabled pipeline tracking on XML receive and XML transport pipelines.
- Enabled track message body and analytics in Receive ports (If you want to know about Analytics option please refer to my first article.
- Enabled the Track Events, Track Message Bodies, and Analytics on orchestration.
- Enabled the Analytics on Send Port
Note: I enabled a different level of tracking at different artifacts to see if it has an impact on the analytics data sent to Application Insights. Later I realized that different tracking levels do not have any impact on the analytics data.
Analytics Data in Application Insights
I placed a single file into the receive location and started observing the events pushed to Application Insights. In general, Applications integrated with Application Insights can send data belonging to various categories, such as traces, customEvents, pageViews, requests, dependencies, exceptions, availabilityResults, customMetrics, band browserTimings. With BizTalk, I have observed that data belongs to “CustomEvents” category. Following are the custom events which are ingested from my BizTalk interface.
- There are two events for a receive port.
- There is an event for every logical port inside the orchestration. And hence we can see three events in total for orchestration.
- There are two events for each send port.
All these events can be related to events logged into “Tracked events” query results which are shown below.
Structure of a BizTalk custom event
In the previous section, we saw that our BizTalk interface emitted various custom events for ports and orchestration. In this section, we will look into the structure of data which is captured in a custom event.
Event Metadata
Event metadata is the list of values which defines an event. Following are the event metadata in one of the custom events.
Custom Dimensions
Custom dimensions consist of the service instance details and context properties promoted in the messaging instance. Hence we can observe two different kinds of data under custom dimensions.
Service instance properties: These are the values specific to service instance associated with the messaging event.
Context properties: All the context properties which are non-integer type will be listed under the custom dimensions.
Custom Measurements
As per my observation, custom measurements only contain the context properties of integer type.
Since there is no proper documentation regarding this, I tried to prove this theory by creating three custom properties in a property schema and promoted the fields in the incoming message. Following is the property schema that I defined.
I observed that PartyID and AskPrice properties which are of type string and decimal respectively are moved to Custom Dimensions section. Property Quantity which is of type integer is moved to Custom measurement.
Querying data
As discussed in above section all the BizTalk events are tracked under the customEvents category. Hence our query will start with customEvents.
Query language in Application Insights is very straightforward and yet very powerful. If you want to find out all the construct of this query language please refer this link Application Insights Analytics Reference.
In this section, I would like to cover some concepts or techniques which are relevant for querying BizTalk events.
Convert the context property values to specific types
In Application Insights, the context property values are stored as dynamic types. When you directly use them into queries especially in aggregations, you will receive a type casting exception as shown below.
To overcome this error, you will need to convert the context property to a specific type as shown below.
Easy way to bring a context property key into the query
Since context properties are a combination of namespace and property name, it will be a bit of an effort to type them in the queries that we create. To bring the context property on to the query page easily, follow steps as below.
- Query for custom events and navigate to the property you are interested in the results section.
- When you hover the mouse on the desired property, we will get two buttons. ‘+’ for inclusion and ‘-’ for exclusion.
Selecting specific fields
If you already know app insights query language, this tip is not so special. But if you are new to it and trying to find out how to select a column, you will face some difficulty as I did. The main reason for this is there is no construct called “select”. Instead, you will have to use something called “project”. Below is an example query.
Some useful sample queries.
In this section, I will try to list some queries which I found useful.
Message count by port names
query
customEvents | where customDimensions.Direction == "Receive" | summarize count() by tostring(customDimensions.["PortName (http_//schemas.microsoft.com/BizTalk/2003/messagetracking-properties)"])
Chart
Messaging Volume by schema
Query
customEvents | where customDimensions.Direction == "Receive" | summarize count() by tostring(customDimensions.["MessageType (http_//schemas.microsoft.com/BizTalk/2003/system-properties)"])
Chart
Analytics with custom context properties.
Ability to generate analytics reports based on the custom promoted properties is a very powerful feature which really makes using application insights interesting. As I explained in previous sections I have created a custom property schema to track PartId, Quantity and AskPrice fields. Now we will see some example reports based on this.
Total quantity by part id
Query
customEvents | where customDimensions.PortType == "ReceivePort" | where customDimensions.Direction == "Send" |summarize sum(toint(customMeasurements.["Quantity (https_//SampleBizTalkApplication.PropertySchema)"])) by PartId = tostring(customDimensions.["PartID (https_//SampleBizTalkApplication.PropertySchema)"])
Chart
Total sales over period of time
Query
customEvents | where customDimensions.PortType == "ReceivePort" | where customDimensions.Direction == "Send" | summarize sum(todouble(customDimensions.["AskPrice (https_//SampleBizTalkApplication.PropertySchema)"])) by bin( timestamp,10m)
Chart
Pinning charts to Azure dashboard
All the charts that you have created can be pinned to an Azure dashboard and you can club these charts with other application dashboards as well. My dashboard with the charts that we created looks as below.
Summary
In summary BizTalk analytics option which is introduced in BizTalk Server 2016 Feature Pack 1 is useful to get analytics out of tracking data. I would like to conclude by stating following points.
- Only the tracked messaging events, service instance information and context properties of associated service instance are sent to App iInsightsby analytics feature. Message body, pipeline events, business rule engine events etc. are not being pushed out.
- Under messaging events, I was unable to find the Transmission Failure events for send ports. This will be useful for getting metrics on failure rates. If you agree with this observation please vote here.
- The different level of tracking on ports and orchestrations does not have an impact on the data being transmitted to app insights.
- Orchestration failures/suspended events are not pushed to application insights. It would be good if Microsoft provides an extensible feature to push exceptions from orchestrations. If you agree please vote here
- There is no control on what context properties published to app insights. It is all or nothing scenario. It would be good to have control on it. Especially when you are promoting and tracking business data. If you agree please vote here.
- Ability to perform analytics based on context property values can turn out to be a powerful feature for BizTalk implementations.
