Performance Data Collection Optimization

Performance Data Collection Optimization

BizTalk360 aims to offer capabilities out-of-the-box from tools like the Windows Performance Monitor. The BizTalk360 Analytics service collects the performance data for the various server types like BizTalk, SQL, IIS, and Windows. Based on the server type selection, the Analytics service will start collecting the performance data and that can be visualized in BizTalk360 Analytics widgets or the user can push that data to third-party APM’s (Application Performance Monitoring) like New Relic, AppDynamics or Dynatrace.

BizTalk360 also allows administrators to automatically execute queries against the Tracking database, at a specific interval, to view, analyze and troubleshoot the tracked data. For the Tracking data, BizTalk360 provides a similar user experience to the performance data collection. This allows the user to pick the metrics they want to collect.         

In this article we will look into detailed information about:

  1. Performance Data Collection of each type (BizTalk, Windows, SQL, IIS)
  2. Tracking Data Collection
  3. Analytics Custom widgets in BizTalk360

Performance Data Collection

To analyze the performance of a BizTalk environment, BizTalk360 is equipped with Performance Data Collection in the Analytics section. Once the user enabled any one of the server types (BizTalk, SQL, IIS, and Windows) in any selected environment, the BizTalk360 Analytics service will start to collect all the related counters.

For enabling the performance data, navigate to the Settings section of BizTalk360. In the Analytics configuration, you can manage the performance data collection in the Manage Analytics section.

Once the performance data collection is enabled for a server, the Analytics service will start to collect counter data on the next polling cycle.

For a larger environment, the Analytics service may need to collect multiple counter data, but the user might not require all those data. For instance, if a user wants only the BizTalk message related counter data to analyze the performance of the BizTalk environment, the Analytics service collects all the BizTalk related counters.


Performance Data collection Optimization

To address this challenge, BizTalk360 allows the user to manage the metrics collection from the v9.0 phase 2. To narrow down the data collection at a server level, the user can optimize the data collection using the “Manage Metrics” option for each server in an environment.

With the “Manage Metrics” option, the user can choose the required performance metrics based on types, so that BizTalk360 Analytics will start collecting data only for those metrics.

Let’s take a deep dive on how a user can manage the metrics collection at the server level.

In BizTalk360, each server’s counter is segregated into 4 types:

  1. BizTalk
  2. Windows
  3. IIS
  4. SQL

Most of the BizTalk environments will be in a multi-server setup where at least BizTalk Server and SQL Server are configured in different machines. A user may expect BizTalk related counters from BizTalk Server or SQL related counters from SQL Server. A user may not want SQL/IIS related performance metrics data in a BizTalk server machine. By using the “Manage Metrics” option in each server, the user can choose the metrics needed for each server. This improves the overall performance of the Analytics service and avoids the growth of the database.


Tracking Data Collection

BizTalk360 allows the user to run queries against the Tracking database and display the result in a widget. In some scenarios, retaining the tracking data longer than the actually required duration, causes the database to grow exponentially. In other scenarios, the environment doesn’t follow a strict purging policy. For both cases, it is hard to query against the Tracking database due to its size.

In a larger database, querying against the Tracking database and displaying the data, results in a graphical form in the widget. This could degrade the performance of the queries and impact on the user experience of the Analytics dashboard.

To overcome these challenges, we created a service in v8.3 that collects the tracking data periodically in small chunks and avoiding expensive queries on the Tracking database, and metrics are collected rather than pulling all the tracking data into BizTalk360 database.

Once decided to go with new this data collection method, BizTalk360 provides a customizable interface where user can pick the metrics they wanted to collect. A similar user experience to the performance data collection in BizTalk360 allows the user to fine-tune the data collection at an environment level.


Analytics Custom Widget in BizTalk360

BizTalk360 is loaded with customizable information and a variety of widgets to choose from, which can be added to your dashboards. The BizTalk360 Analytics dashboard is loaded with performance data widgets and tracking database widgets, like the Messaging performance and transmission failure rate widgets.


Users can add custom widgets to both the Analytics dashboard and custom dashboards in BizTalk360. The product provides rich options for adding widgets in these dashboards. Users can choose

  • Date Range (24 hours, 7 days, 30 days, custom date range)
  • Comparing data with previously generated data
  • Graph type (line, column, area)

Based on the selected date range, the performance data gets collected from the BizTalk servers.

Once the analytics components, for which the data needs to be collected, are enabled, the different metrics for the custom widgets will get enabled. With the custom widgets, you can choose different metrics for which you want to view the data. Users can choose messaging performance/transmission failure of tracking data or performance data from the drop-down.


Do you wish to see more information on the widgets or from the product itself? Then, please put in your suggestions and feedback in our user voice portal. The existing ideas can also be voted for. We at BizTalk360 aim at providing the features that fulfill the customer requirements. It is from this feedback portal that the features get picked up for every release.

For more information about how these widgets work, check our Documentation Portal.

The post Performance Data Collection Optimization appeared first on BizTalk360.

BizTalk Server Tips and Tricks: Take control of your environment: Tracking Data

BizTalk Server Tips and Tricks: Take control of your environment: Tracking Data

Welcome back to another entry on my blog post series about “BizTalk Server Tips and Tricks” for developers, administrators or business users and I couldn’t resist on speaking about a topic that normally divides BizTalk developers and BizTalk administrators: Tracking Data!


Normally Developers have in their environments Full tracking Enabled, why? Because it’s easier to debug, troubleshoot, analyze or validate and see if everything is running well or simply, what is happening with their new applications.

The important question is: Do Developers remember to disable tracking before they put the resources in production?

No! And actually… they don’t care about that! Is not their task to do it or control it. And if you ask them, you should always have them enabled! Sometimes, to be fair, they don’t know the right configurations that should be applied to production.

This can be an annoying and time-consuming operation. It will be the same as asking developers to change their way of being, and for them to remember each time they export an application to disable the Tracking data properties can be a big challenge… or even impossible!

Solution (or possible solutions)

My advice is, if you are a BizTalk Administrator, let them be happy thinking they are annoying you and take back the control of your environment by yourself.

These tasks can be easily automated and configured by easily creating or using PowerShell.

You should disable all Tracking or enable just the important settings at the application level. You may lose 1 day developing these scripts, but then you do not need to worry anymore about it.

As an example, with this script: BizTalk DevOps: How to Disable Tracking Settings in BizTalk Server Environment, you can easily disable all tracking settings for all the artefacts (orchestrations, schemas, send ports, receive ports, pipelines) in your BizTalk Server Environment

# Disable tracking settings in orchestrations     
$Application.orchestrations |  
%{ $_.Tracking = [Microsoft.BizTalk.ExplorerOM.OrchestrationTrackingTypes]::None } 
# Disable tracking settings in Send ports        
$disablePortsTracking = New-Object Microsoft.BizTalk.ExplorerOM.TrackingTypes 
$Application.SendPorts |  
%{ $_.Tracking = $disablePortsTracking } 
# Disable tracking settings in Receive ports 
$Application.ReceivePorts |  
%{ $_.Tracking = $disablePortsTracking } 
# Disable tracking settings in pipelines         
$Application.Pipelines |  
%{ $_.Tracking = [Microsoft.BizTalk.ExplorerOM.PipelineTrackingTypes]::None } 
# Disable tracking settings in Schemas 
$Application.schemas |  
    ?{ $_ -ne $null } | 
    ?{ $_.type -eq "document" } | 
    %{ $_.AlwaysTrackAllProperties = $false }

This can easily be edited by you to disable only one application or you can configure the right tracking setting that you want for your applications and environment.

If you are working with BizTalk Server 2016…

In previous versions of BizTalk Server, tracking settings were automatically imported with the rest of the application bindings. However, if you are working with BizTalk Server 2016, you have a new feature that allows you to have a better control while importing your BizTalk Applications: Import Tracking Settings.
If you are importing an MSI file, on the “Application Settings” tab, you will have a checkbox “Import Tracking Settings” that allows you to say: I don’t want to import the tracking from DEV or another environment in which the MSI was generated from.
BizTalk Server Import MSI Not Importing racking Data
If you are importing a Binding file you will also have this same option:
BizTalk Server Import Bindings Not Importing racking Data
Of course, if you want to properly define the correct or minimum tracking settings of your application, you need to do it manually or, once again, using a PowerShell script to accomplish that.
Quick, simple and practical.
Stay tuned for new tips and tricks!
Author: Sandro Pereira

Sandro Pereira is an Azure MVP and works as an Integration consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.