BizTalk Application Deployment Using Azure Pipeline with BizTalk360 API’s

BizTalk Application Deployment Using Azure Pipeline with BizTalk360 API’s

Introduction

The process of BizTalk Application deployment is a complex, time consuming and repetitive task. Microsoft BizTalk Server comes with capabilities to create deployment packages. However, the MSI packages you create with the out-of-the-box features of BizTalk Server have some shortcomings which lead to deployments which are error-prone.

BizTalk Deployment Framework

The BizTalk Deployment Framework (BTDF) covered these deployment issue.

  • Deploy complex BizTalk solutions containing BizTalk artifacts but also for example SSO settings, ESB itineraries and SQL Server scripts, with no human intervention
  • Consolidate all environment-specific configuration and runtime settings into one, easy-to-use Excel spreadsheet
    Maintain a single binding file that works for all deployment environments
  • Make automated deployment a native part of the BizTalk development cycle, then use the same script to deploy to BizTalk servers

To eliminate the manual process, organization’s start to adopt the DevOps practice. In this article, we will see how the BizTalk360 API’s are helpful in a BizTalk environment monitoring while deploying BizTalk Applications using Azure CI/CD Pipelines.

Deploy BizTalk Applications with Azure Pipelines

Microsoft has introduced automated deployment of BizTalk Applications in BizTalk Server 2016 Feature Packs. In BizTalk Server 2016 Feature Pack 1, automatic deployment and application lifecycle management (ALM) experience was introduced. Automatic deployment process has been improved with the release of BizTalk Server 2016 Feature Pack 2.

  1. Within Visual Studio for each project, set the Application Name of your BizTalk application
  2. In addition to using an agent-based deployment, you can also use deployment groups to deploy your BizTalk applications to multiple servers
  3. In the release task, you can install the BizTalk application, and enter the BizTalk management computer, and the path to the deployment package

Azure Pipelines (CICD)

With Automated build/release definitions you can set up the CI or CICD Pipelines based on environment (Development/QA/Staging/Production):

  1. Continuous Integration (CI) : Set up build task definitions rules to the VSTS Source code base
  2. Continuous Integration Continuous Deployment (CICD) : Configure both build task definition and release task definition to automate the build and test process

If you are new to Azure CI/CD Pipeline go through this article Configure automatic deployment with Visual Studio Team Services in BizTalk Server

Deployment Build Definitions

BizTalk Build Pipelines

In the BizTalk Application deployment process, you can create the following build/release definitions:

  1. Visual Studio Build

    The build and release definitions are Visual Studio Team Services tasks, and should be done by a VSTS admin. The build definition builds your project within your git or VSTS repository, and the release definitions deploys it to your BizTalk Server environment

  2. Publish Build Artifacts

    In Publish Build Artifacts stage configure the Path to publish (Application Project Folder)
    Publish Build Task

  3. Deploy BizTalk Server Application

    BizTalkApplicationDeployment
    A BizTalk Server application deployment build definition operation has three options available

    1. Create new BizTalk Application: Deploys a new application, if the application already exists, it uninstalls the current applications (full stop), and installs the new application. If continuous integration is enabled, it automatically redeploys the application when it is updated in the repository
    2. Update an existing BizTalk Application: Appends changes, such as schemas, to an already running application. It does not require a full redeploy of the application
    3. Install BizTalk Server Application: Install the application, enter the BizTalk management computer name and the deployment package path

BizTalk360 API’s in CI/CD

Customer Scenario: In Build pipeline stages the customer wants to utilize the BizTalk360 API to set the Stop Alert for Maintenance in a BizTalk environment. Once after the new build is deployed to BizTalk Server and then user wants to configure newly deployed artifacts to monitor in BizTalk360. As the last step revoke the Stop Alerts in BizTalk360 Application.

We can use the following steps in Build definitions to achieve the above scenario;
BizTalk Depolyment using BizTalk360 API

  1. Visual Studio Build (explained above)
  2. Set Stop Alert for Maintenance

    BizTalk360 customers can take advantage of using the BizTalk360 API’s to set the alert maintenance during the BizTalk Application deployment.
    Create the PowerShell Script stage to call the SetAlertMaintenance API method to set the BizTalk environment under the maintenance. Assign the PowerShell Script full path
    Set Alert Maintenance

    SetAlertMaintenance.ps1

    $Request = '{
      "context": {
        "callerReference": "SAMPLE_REFERENCE",
        "environmentSettings": {
          "id": "DC45E848-E485-41D7-92E5-7C9932D43CB8"
        }
      },
      "alertMaintenance": {
        "comment": "BizTalk Deploy",
        "maintenanceStartTime": "2018-11-22T15:26:37.000",
        "maintenanceTimeUnit": 0,
        "maintenanceTimeLength": 2,
        "isActive": false,
        "expiryDateTime": ""
      }
    }'
    $Request
    $body = ConvertFrom-Json -InputObject $Request
    $body
    $RepsonseSet = Invoke-RestMethod -Uri http://biztalk360webserver/Services.REST/AlertService.svc/SetAlertMaintenance -Method Post -ContentType "application/json" -Body $body -UseDefaultCredentials
    $RepsonseSet 
    
    

    Set Alert Maintenance API returns the alert maintenance object with maintenance Id (out parameter). You can refer this maintenance Id, when revoking the maintenance in the below step.

  3. Publish Build Artifacts (explained Above)
  4. Deploy BizTalk Server Application

    Once after BizTalk Application deployment is completed, next step you can configure the newly artifacts to monitor. For creating BizTalk360 alarms and BizTalk Application mappings during deployments with BTDF, you could consider to BT360Deploy command-line tool. Use this GitHub project to setup pipeline stage.

  5. Remove the Stop alert for Maintenance

    After the deployment of BizTalk Applications, the maintenance mode of BizTalk environment must be revoked in BizTalk360. To achieve this step, create a PowerShell Script stage and assign the PowerShell Script path(RemoveMaintenance.ps1)

    $RequestRem = '{"context":{"callerReference":"SAMPLE_REFERENCE","environmentSettings":{"id":"DC45E848-E485-41D7-92E5-7C9932D43CB8"}},"maintenanceId":"308d8b2c-b3e0-4024-b600-845113b06b80"}' 
    
    Invoke-RestMethod -Uri  http://biztalk360webserver/BizTalk360/Services.REST/AlertService.svc/RemoveAlertMaintenance -Method Post -ContentType "application/json" -Body $RequestRem -UseDefaultCredentials
    
    

    Remove Alert Maintenance

Summary

Automation of Build and/or Deployment will be helpful to reduce the turnaround time. Hopefully, this article has provided insight in how you can setup the BizTalk Application Deployment automation and how you can utilize BizTalk360 API’s to stop alert during deployment and how to set up monitoring of the newly deployed BizTalk Artifacts.

Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.

Optimizing the Data Collection in BizTalk360

Optimizing the Data Collection in BizTalk360

Introduction

BizTalk360 is a tool for operating, monitoring and getting analytical information for BizTalk environment(s). BizTalk360 comes with productivity tools like the Advanced Event Viewer, the Throttling Analyzer, the Backup/DR visualizer, the BizTalk Health Monitor etc. These tools are also designed to help support people to perform their activities in a seamless way. BizTalk Server, being middle-ware, high volumes of data passes through the Message Box. Based on the business requirements, Tracking is enabled for important BizTalk Integration transactions.

In similar fashion, BizTalk360 collects important information to perform various activities in operations, monitoring and analytics section. In this article, we show BizTalk360’s data collection features and how you can configure the settings to optimize the data collection.

Advanced Event Viewer

The Event Log is an important component which is used by BizTalk Administrators to investigate, if any configuration or transaction errors have happened in BizTalk Server. The Advanced Event Viewer is a powerful feature in BizTalk360 which in fact is a central repository for collecting the event logs from multiple BizTalk servers and SQL servers in BizTalk environment. It has the capability to filter the event logs by various parameters like Event Id, Event Type, Sources etc.

The BizTalk360 Monitoring service collects the event log information based on Event Logs and Event Sources which are configured in the BizTalk360 Advanced Event viewer settings.

BizTalk360 Monitoring IOPS

BizTalk360 uses Windows Management Instrument (WMI) queries to collect the event log details from BizTalk Servers and SQL Servers in an environment. In a multi-server BizTalk environment, few customers raised concern about BizTalk360 IOPS (input/output operations per second). In this set-up, there is change of high volume of IOPS in BizTalk360 Box (Monitoring Service).

BizTalk360 Recommendation
To avoid large volume of data collection or to improve the performance of the BizTalk360 monitoring services, following event log configuration are suggested
1. Enable the necessary servers to collect Event Logs and Event Sources
• Primary BizTalk Servers in Active – Passive scenario
• Multiple SQL server, enable the event log data collection for important nodes.

2. BizTalk360 provides a default set of Event Logs and Event Sources which are widely used in BizTalk and SQL servers. Configure the necessary Event Sources that are required as per the business requirement:

• BizTalk Event Log Sources(Default):
Bam Event Provider, BamManagementUtility,BAMWebServices,BizTalk DW Reporting, BizTalk Server, BizTalk Server Deployment, BizTalk Server EDI, BizTalk360 Monitor, BusinessRulesEngine, ENTSSO,MSDTC,MSDTC Client, RuleEngineUpdateService, System.ServiceModel 4.0.0.0, System.ServiceModel.Install 3.0.0.0,W3SVC-WP,XLANG/s

Configure required BizTalk Sources as
“BizTalk Server,BizTalk Server Deployment,BizTalk Server EDI,ENTSSO,MSDTC,MSDTC Client,RuleEngineUpdateService”

• SQL Sources(Default):
MSSQLSERVER, MSSQLSERVER$AUDIT, MSSQLServerOLAPService, SQLSERVERAGENT

Configure required SQL Sources as
“MSSQLSERVER, SQLSERVERAGENT”
Note: This configuration will reduce the BizTalk360 Input/Output operations. You can read this article to know more details about Event-Log Data collection configuration.

Analytics Performance Data

In a data-driven business setup, there will be scenarios where critical executive decisions are made based on the performance reports of various components of the listed servers in a BizTalk environment. BizTalk360 aims to offer an out of the box tool with similar capabilities as the Performance Monitor tool in Windows servers.
BizTalk360 Analytics offers visual display of the most important performance counters that are consolidated and arranged on a single screen so that the information can be monitored in a glance. Custom reports can be built in minutes with the metrics that really count for your business, and they are easy to share with other users. In addition, dashboards can give you a summary of many reports on a single page using drag-and-drop widgets for fast and easy customization.

AdvancedEventViewerByServer
Performance metrics data collection can be optimized by selecting the necessary metrics (Windows, SQL, BizTalk and IIS) based on the server type.

• BizTalk Server: Enable Windows, BizTalk by default. Web Application is configured in the server enable IIS and similarly to SQL.
• SQL Server: Enable Windows and SQL performance metrics

Tracking Data

Initial design of the tracking data widgets was simple; it ran the necessary queries directly against the Tracking database and displayed the results in a graphical form on the widget. A larger database degraded the performance of these queries and it had a direct impact on the user experience of the Analytics dashboard, causing slower widget loading and occasional time outs. Since Analytics widgets offered users the option to view data up to 30 days within the widget, an efficient approach was required to improve the performance and user experience.
In BizTalk360 v8.3 to overcome these challenges, we decided to create a service that runs in the background. This service collects the tracking data periodically in small chunks and thereby avoid running expensive queries on the Tracking database. Here, only the resulting metrics are collected rather than pulling all the tracking data to BizTalk360 database. This improves the overall performance of the Tracking widgets as they are not querying the Tracking database directly.
BizTalk360 Analytics needs the tracking data to represent data in custom widgets

• Failure Rate by Port and Schema
• Message Performance by Host, Orchestration, Pipeline, Adapter, Port and Schema

BizTalk360 suggests enabling the data collection for categories and counters needed in custom widgets. If you want custom widgets for Failure Rate and Message Performance of Port and schema, enable only those counters in Tracking data collection.

AnalyticsPerformanceData

Data Purging

Collecting the necessary data is important to be able to represent the data in BizTalk360. However, you don’t want unlimited growth of your BizaTalk360 comes out of box with the “Data Purging” feature to able to manage the size of BizTalk360 database. If you want to know more about Data Purging please read this article.

DataPurgingConfiguration

Constantly manual monitoring the database size is a tedious process. You can utilize the Database Query monitoring when the database size is grown over the threshold limit.

SELECT total_size_mb = CAST(SUM(size) * 8. / 1024 AS INT)
FROM sys.master_files WITH(NOWAIT)
WHERE database_id = DB_ID()

Conclusion

We hope this article is useful to optimize and manage the data collection for various features in BizTalk360.
Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.

Best Practices to Create Custom Widgets in BizTalk360

Best Practices to Create Custom Widgets in BizTalk360

Introduction

Custom widgets are powerful feature in BizTalk360, that allow the users to bring third-party integrations into BizTalk360.  It will be useful to bring the custom solutions into the BizTalk environment as it allows to view integrated solutions in one single tool.

Custom widgets can be used for various purposes, below list constitute few important scenarios;

  • Embed third-party portals like Power BI or internal portals
  • Monitor BizTalk Artifact Statuses
  • Display query results using Secure SQL Queries

In this article, we can see important best practices while creating Custom widgets.

Best Practices 

1. Hiding confidential information in Custom Widgets with Placeholders

Custom Widgets are created using API’s (e.g.: BizTalk360 or BizTalk Integration API), which might contain confidential information like the credentials of your BizTalk360 Service Account or API Authenticated credentials. Of course, you don’t want this information to be revealed in plain text in your widgets. To hide such confidential information, you can use Placeholders.

Beside using Placeholders to hide confidential information, you can also use them for providing frequently used data to your Custom Widgets. Think of for example the BizTalk360 “EnvironmentId”, which is a value which often needs to be provided to Custom Widgets. By creating a Placeholder for the “EnvironmentId”, you can easily re-use it and you don’t need to search for such values, each time you are creating a Custom Widget which needs such a value.

Place Holder

Read this article to know more about creating the place holders and how it can be used in widget.

2. Knockout JS Functions

Frequently, utility functions are used while binding the data to view. In these situations, the user can utilize the BizTalk360 referenced utility functions.

BizTalk Integration API methods returns its responses in XML format. To represent the data in client script(JavaScript/jQuery),it needs to be in JSON format.

x2js = new X2JS({ attributePrefix: '' });
bizTalkHostsList = function () 
{
  var _this = this;
 _this.getBizTalkHosts(function (data) 
  {
   _this.bizTalkHosts(x2js.xml_str2json(data.queryResult));
  });
};

When a view expects the API method to return the result as an array, occasionally a single record is produced. In this situation,the API result needs to be converted to an array. 

x2js = new X2JS({attributePrefix: '', arrayAccessForm:"property", 
arrayAccessFormPaths : ["root.records.record"]});
bizTalkSendPortsList = function () {
      var _this = this;                 
      _this.getbizTalkSendPorts(function (data) 
{
      var results = x2js.xml_str2json(data.queryResult);
      if (Array.isArray(results.root.records.record))
         _this.bizTalkSendPorts(results.root.records.record);
      else {
         _this.bizTalkSendPorts([results.root.records.record]);
         }
      });
   };

3. BizTalk360 Styles and Icons

 The styles of BizTalk360 can be used in custom widgets by inspecting the element in the BizTalk360 Application.  You can take advantage of the listed styles and controls while creating the custom widgets.

CSS Styles

Utilize BizTalk360 in-built CSS in custom widget creation

  • List the data in Table: table table-lists
  • Scroll Bar: WidgetScroll
<div id=”WidgetScroll” style=”top:30px;” data-bind=”addScrollBar: WidgetScroll, scrollCallback: ‘false'”> 
  <table class=”table table-lists”>
  </table>
</div>
  • Success/Info Tag: Use the Success or Information tag based the data to be presented in the list
 <span data-bind="text: HostType == 1 ? 'InProcess' : 'Isolated', css: HostType == 1 ? 'success-tag' : 'info-tag'">
 </span>

Boot Strap

Boot Strap styles and tags could be used during widget creation. BizTalk360 uses Boot Strap styles to design the base layouts.

  • Container Fluid
  • row
  • col-md-*
<div class="container-fluid">
  <div class="row">
    <div class="col-md-8"></div>
    <div class="col-md-4"></div>
  </div> 
</div>

Font Awesome Icons

Users can represent status or any information in graphical way, you can use Font Awesome Icons.

  • Success – fa fa-check-circle
  • Error – fa fa-power-off
  • Warning – fa fa-times-circle

KendoUI Controls

BizTalk360 uses KendoUI controls, users can utilize those controls in widget script

  • Drop Down
  • Grid
  • Switch Box
  • DateTime Picker
  • Numeric Text Box
  • Calendar
// View
<div class="form-group margin-t"> //Boot Strap Form Group
  <div class="col-md-10"> // Boot Strap Column
    <div class="col-md-offset-2">
      <div class="alert alert-info" role="alert"> //Boot Strap Information
        <b>Note :</b> Choose the environment and enable the servers for Event Log data to be collected.
      </div>
      <div class="form-horizontal">
        <div class="form-group" data-bind="validationElement: selectedEnvironment">
          <label class="col-md-3 control-label">Select Environment</label>
          <div class="cus-select" style="text-align: left;">
            <label>
              <select data-bind="options: configuredEnvironments, optionsText: 'name', value: selectedEnvironment" 
               class="form-control" style="min-width:300px;"></select> // Kendo Dropdown
             </label>
           </div>
         </div>
       </div>               
     </div>
   </div>
</div>    
        
//ViewModel     
 selectedEnvironment.subscribe(onEnvironmentChanges, this); // change event to the environment

4. Embedding Third-Party Scripts

High Chart Solid Gauge

BizTalk360 uses a component called High Charts to represent data in graphical way. Also, some of our customers are using the custom widgets to represent their BizTalk Integration in graphical chart. BizTalk360 supports High Charts (Basic Widgets) with High Chart JS.  To reference the third-party scripts, use the following JavaScript to reference the external libraries.

function injectScript(src) {
    return new Promise((resolve, reject) => {       
                for (var i = 0; i < src.length; i++) {
    var script = document.createElement('script');
    script.src = src[i];
    script.async = false; // This is required for synchronous execution
    document.head.appendChild(script);
  }        
        script.addEventListener('load', resolve);
        script.addEventListener('error', () => reject('Error loading script.'));
        script.addEventListener('abort', () => reject('Script loading aborted.'));
        document.head.appendChild(script);
    });
}
injectScript(['https://cdnjs.cloudflare.com/ajax/libs/highcharts/5.0.14/highcharts-more.js','https://cdnjs.cloudflare.com/ajax/libs/highcharts/5.0.14/js/modules/solid-gauge.js'])

Customer scenarios

Through our support channel, customers are explaining their scenarios which they require in their day to day activities. Here, we pick few common scenarios that will be useful in BizTalk Integration space.

Monitor Artifacts of a BizTalk application

BizTalk360 has the capability to monitor multiple application artifacts in a single alarm configuration. For instance, a specific group of users wants to monitor the status of specific application artifacts. To achieve this requirement, custom widget creation is one of the possible solutions.  Create the custom widget to monitor an application artifacts status and associate the widget to dashboard. You can download the custom widget of monitoring the application artifacts.

Secure SQL Queries

Secure SQL query is an important feature to fetch data based on the user’s business demand. However, it is hard to create all the possible transaction results out of box from predefined widgets. Custom Widgets will help the BizTalk360 users to address this gap in BizTalk360.

Please follow the article how to create custom widgets using Secure SQL Queries.

Represent the data using Cross Domain API and High Charts

A user wants to represent BizTalk Integration Transaction details in the graphical chart. The Integration API is hosted cross domain. To access the cross-domain API Methods through client script, you need to use the XMLHTTPRequest method

function createCORSRequest(method, url) {
var xhr = new XMLHttpRequest();
 if ("withCredentials" in xhr) {
 // XHR for Chrome/Firefox/Opera/Safari.
 xhr.open(method, url, true);
 } else if (typeof XDomainRequest != "undefined") {
 // XDomainRequest for IE.
 xhr = new XDomainRequest();
 xhr.open(method, url);
 } else {
 // CORS not supported
 xhr = null;
 return xhr;
}
//Make the actual CORS request
function makeCorsRequest(url,method) {
var xhr = createCORSRequest(method, url);
if (!xhr) {
 alert('CORS not supported');
return;
 }
// Response handlers
xhr.onload = function() {
var resultData = xhr.responseText;
expectedResults(x2js.xml_str2json( JSON.parse(resultData).MonitorQueryResult).root.records.record);
 }
 xhr.onerror = function() {
 alert('Woops, there was an error making the request.')
}
 xhr.send();
}
makeCorsRequest(‘https://YOURCROSSDOMAIN/SERVICENAME’,'GET');

In this case, the user wants to represent the data in High Charts solid-gauge pattern . For that, you can use the above mentioned suggestion “Embed Third-Party Script”.

Conclusion

Custom widget is a useful feature to bring the customized solution in a dashboard. Hope this article will be helpful to get start the custom widgets creation. Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail.