BizTalk Server 2020 – Audit Capabilities and View Auditing Activities in BizTalk360

BizTalk Server 2020 – Audit Capabilities and View Auditing Activities in BizTalk360

Introduction

Integrate is the premier event in the Microsoft Integration space. Every year it has happened in London, focusing on BizTalk & Azure. Due to Covid-19 pandemic uncertainty, Integrate 2020 is planned as a virtual conference.  In Integrate 2019, the Microsoft product group was announced the new version release of BizTalk Server 2020. Microsoft has announced the release of BizTalk Server 2020 early this year (January 15, 2020).

BizTalk Server 2020 has bundled with a lot of exciting features for BizTalk Administrators, operators, Developers, and Business users. Some of the Important features in BizTalk Server 2020 are

In this blog, we are focusing on the BizTalk Audit capabilities in BizTalk Server 2020. Different operations, that are performed in the BizTalk Admin console, are being audited. It could be useful to BizTalk Administrators and Deployment Teams to know what changes have happened in the BizTalk Group.

BizTalk Audit Activities

Auditing can be handy to backtrack changes that have happened in, among other things, BizTalk Applications. With this new auditing feature, the following activities are captured in the BizTalk Management Database (BizTalkMgmtDb). They are being stored in the table ‘bts_auditlog’.

BizTalk-Audit-Activities

Configure Audit Log Settings

Users can get started with audit capabilities, by enabling the Audit Management Operations property in the BizTalk group settings.

By default, 10.000 activities are audited. Users can increase or decrease the number of audit entries.

Configure-Audit-Log-Setting

Audit Operations

In BizTalk Server 2020, operation metrics on different Artifacts are audited. Check the table below, for the operations which are audited with BizTalk Server 2020. 

Audit-Operations-BizTalk-Server-2020

View Audit Logs

To be able to view Audit data, users must install the Operational Data Service in IIS. After that, users can retrieve Audit log data with data range, as shown below;

http://localhost/BizTalkOperationalDataService/AuditLogs?fromDate=2019-12-25T01:00:00&toDate=2020-01-10

Supported date formats are yyyy-MM-dd or yyyy-MM-ddThh:mm:ss.

So, in summary, you have two capabilities to view the Audit Log:

  • Via the Operational Data Services
  • Via the bts_auditlog table in the BizTalkMgmtDb

View BizTalk Audit Log in BizTalk360

BizTalk360 users can take advantage of custom widgets to visualize the audit logs that are captured in BizTalk Server 2020.  

Secure SQL Queries

BizTalk360 offers the Secure SQL Queries functionality as a secure platform to store and execute predefined queries. BizTalk360 comes pre-loaded with a set of queries that are of use for BizTalk administrators.

In addition, BizTalk360 allows you to manage the existing SQL queries and add/edit/delete your own queries.

Custom Widgets

The ability to create custom widgets is a powerful feature in BizTalk360 which enables you to integrate/visualize data from third-party sources using API methods or pull the data from databases through Secure SQL Queries.

We will have a look at how users can visualize the audit data from BizTalkMgmtDb in BizTalk360. The steps that are involved to create custom widgets are:

  1. Create the Secure SQL Query to pull the audit log from the “bts_auditlog” table. In this custom widget script, we will show Artifact Name, Artifact Type, User, Machine, Operation, and CreatedDate.
  2.  If you want to see the payload in the table, you can include the SQL script to visualize it in the operational dashboard.

    SELECT [UserPrincipal],[Machine],[ArtifactType],[ArtifactName],[OperationName],[Payload],[CreatedDate] FROM BTS_AUDITLOG  ORDER BY [CreatedDate] Desc 

    Create-SQL-Query

  3. The next step is to create a custom widget. If you are new to custom widgets,
    follow this article which explains how to create custom widgets.
  4. Add-Custom-Widget

    Custom Widgets Script

    <div id="WidgetScroll" style="top:30px;" data-bind="addScrollBar: WidgetScroll, scrollCallback: 'false'">
              <table class="table table-lists">
            <thead>
                <tr>
                    <th style="width:30%">Artifact Name</th>
                    <th>Artifact Type</th>
                    <th>User</th>
                    <th>Machine</th>
                    <th>Operation</th>            
                  <th>Audit Date</th>              
                </tr>
            </thead>
            <tbody>
              <!-- ko if: (auditActivities()) -->
                <!-- ko foreach: auditActivities().root.records.record -->
                <tr>
                    <td data-bind="text: ArtifactName"></td>               
                  <td data-bind="text: ArtifactType"></td>
                   <td data-bind="text: UserPrincipal"></td>
                   <td data-bind="text: Machine"></td>
                   <td data-bind="text: OperationName"></td>              
                   <td data-bind="text: CreatedDate"></td>
                </tr>
                <!-- /ko -->
              <!-- /ko -->
            </tbody>
        </table>
    </div>
      <script>
    	username = "`username_119251`";
    	password =  "`password_702990`";
    	environmentId = "`environmentid_346562`";
    	queryId = "408e4e00-dc6a-4d3b-a2dd-032ddd232ffe";
    	queryName = "BizTalk Server 2020 Operations";
    	sqlInstance = "KOVLTP013DEV";
    	database = "BizTalkMgmtDb";
    	sqlQuery = "SELECT [UserPrincipal],[Machine],[ArtifactType],[ArtifactName],[OperationName],[Payload],[CreatedDate] FROM BTS_AUDITLOG  ORDER BY [CreatedDate] Desc";
    	url = 'http://localhost/BizTalk360/Services.REST/BizTalkGroupService.svc/ExecuteCustomSQLQuery';
    
    	auditActivities = ko.observable();
    	x2js = new X2JS({ attributePrefix: '' }); 
    
    	auditActivitiesList = function () {
                var _this = this;			
                _this.getAuditActivities(function (data) {                
                _this.auditActivities(x2js.xml_str2json(data.queryResult));          
               });
    
            };
        getAuditActivities = function (callback) {
        var _this = this;			      
    	$.ajax({
    	dataType: "json",
    	url: _this.url,
    	type: "POST",
    	contentType: "application/json",
    	username: _this.username,
    	password: _this.password,
    	data: '{"context":{"environmentSettings":{"id":"'+ _this.environmentId + '","licenseEdition":0},"callerReference":"REST-SAMPLE"},"query":{"id":"'+ _this.queryId + '","name":"' + _this.queryName + '","sqlInstance":"' + _this.sqlInstance + '","database":"'+ _this.database +'","sqlQuery":"' + _this.sqlQuery + '","isGlobal":false}}',
    	cache: false,
    	success: function (data) {
    		callback(data);
    	},
    	error: function (xhr, ajaxOptions, thrownError) { 
    		alert(xhr.status);
    		alert(xhr.responseText);
    	},
    });
            };
      auditActivitiesList();
    </script>
    
  5. Associate the widgets to the Operational dashboard to visualize the custom widgets. After this configuration, we can see the audit activities, in the tabular format, in a BizTalk360 dashboard

BizTalk360-Audit-Dashboard

Conclusion

BizTalk360 has capabilities to audit the various operations (Applications Artifacts – Start/Stop, Host Instances – Enable/Disable, Service Instances – Resume/Terminate/Suspended, View Message Content, User Management- Add/Edit/Delete) which is performed in BizTalk360 web application. Thereby, this is more complete than the current features in BizTalk Server 2020. BizTalk360 users can take advantage of both auditing capabilities in a single tool.

Users can write more custom scenarios through Custom Widgets. This makes users more collaborative by using BizTalk360. Happy Integration!

The post BizTalk Server 2020 – Audit Capabilities and View Auditing Activities in BizTalk360 appeared first on BizTalk360.

BizTalk Server 2020 – Features that Developers/Administrators can leverage

BizTalk Server 2020 – Features that Developers/Administrators can leverage

Introduction

On January 15, 2020, Microsoft has announced the much-awaited release of BizTalk Server 2020 for public usage. Ever since Microsoft Product Team announced BizTalk Server 2020 in Integrate 2019 event, there has been a lot of expectations about the new release.  BizTalk Server 2020 is an important update with key features and addresses some of the existing challenges. This version has support for newer platforms like Visual Studio 2019, Windows Server 2019, Windows SQL Server 2019. BizTalk Server 2020 requires the following  Hardware and Software requirements.

New features that are shipped in this version are Analytical capabilities (Publish tracking data to Azure), Application Life Cycle Management with VSTS, Management APIs, Advanced Scheduling, TLS 1.2 Support, API Management, gMSA Account, Adapters (Event Hubs, Office 365, Blob) and more additional updates, like SSO Affiliate application support in SFTP Adapter. Partially disabled receive locations etc., Some of the features that are in the release have been updated with BizTalk Server 2016 feature packs (Automatic Deployment with VSTS in BizTalk Server and Manage API are part of Feature Pack 1 update)

These features are helpful to Administrators and BizTalk Developers/Deployment Team with common updates like Adapters support. In this blog, we are focusing on the features that Developers, Administrators /Deployment team can take advantage of.

  1. XSLT 3.0 Support and Custom XSLT Transform
  2. Automatic Deployment with VSTS in BizTalk Server
  3. Manage API
  4. Publish API Services to Azure API Management

Custom XSLT Transform

From this version of BizTalk Server on, the BizTalk Mapper has the ability to select the Custom XSLT Transform (Saxon 9 HE) or .Net Framework. BizTalk’s default XSLT engine implementation is based on .Net Framework, however, this support is limited to XSLT 1.0.  By using this new property “XSLT transform engine”, other XSL transform engines can be configured at the map level.

Custom-XSTL-Transform

XSLT 3.0

Support of XSLT 3.0 in BizTalk Server 2020 will help the developer to use any XSLT version in schema transformation in BizTalk Mapper. Developers can customize the XSLT based on their business requirements. XSL Functions can be defined, and it provides the option to choose the dynamic transform functionality.

What is Saxon-HE?

Saxon-HE is an open-source tool for processing XML documents using XSD schema and XSL Transform. Users can start using Saxon-HE 9 engines if they needed XSLT3.0 support during data transformation.

Let us take a business scenario in which EDI X12 850 data needs to be transformed into a purchase order. Saxon HE 9 XSLT will be used to transform the 850 by XSL Path and Custom extension XML to Purchase Order XML (XSLT3.0 will support the JSON Format in XPath Transform).

Saxon-Business-Scenario

Saxon-9

Note: Saxon 9 doesn’t support embedded scripting. As a result, functoids shipped as part of BizTalk may not function well with Saxon-HE  9.

Extend to other Custom XSLT Engines

Starting with BizTalk Server 2020, users can extend the custom XSLT transform engine in the BizTalk Mapper. Users can implement a custom XSLT transform engine by defining the XSLT transform implementation which is derived from abstract class Microsoft.XLANGs.BaseTypes.ITransform2 in assembly Microsoft.XLANGs.BaseTypes.dll.

Follow this article to know how to extend Custom XSLT with different Transform engines.

Automatic Deployment with VSTS in BizTalk Server

The deployment of BizTalk Applications can be a cumbersome process, as it can be hard to manage the artefacts binding information of different environments. Automatic Deployment of BizTalk Server with Visual Studio Team Services is released as part of the BizTalk Server 2016 Feature Pack1 update. Now, all the feature pack updates of BizTalk Server 2016 are clubbed into the BizTalk Server 2020 release. With Azure DevOps Service, users will be able to automate the deployment process through configuring a VSTS agent.

The VSTS Admin creates the build and releases definitions in Azure CI/CD pipelines. Developers will take advantage of the CI pipeline to build the BizTalk Applications and check-in the source code into Git or VSTS repositories. The CD pipeline will be used to deploy the BizTalk Applications into different environments (DEV, QA & Production).

You can follow the steps to create a deployment pipeline as mentioned in this article.

Manage API

BizTalk Server Developers can take advantage of REST APIs to customize their Business requirements through the exposed APIs covering the following areas in BizTalk Management Service. Management data APIs are endpoints that let you remotely update, add, and query the status of different Artifacts in your BizTalk Server environment.

  1. Parties & Agreements
  2. Applications
  3. Batches
  4. Business Profiles, Role Links
  5. Fallback Settings
  6. Hosts
  7. Operational Data
  8. Orchestrations, Receive Locations, Receive Ports, Send Ports and Send Port Groups
  9. Policies
  10. Protocol Types
  11. Resources
  12. Schemas and Transforms

Installation

REST APIs are installed as part of the BizTalk Server 2020 Setup. It has been configured in IIS.

  1. BizTalkManagementService
  2. BizTalkOperationalDataService

Manage-API-Installation

Swagger Definitions

The endpoints are added using REST and come with a swagger definition. Users can access the Swagger definitions of the installed Management Service APIs. With the swagger definitions, developers can benefit from knowing the input and output parameters of each API method. Users can try out/test API methods through Swagger definitions.

http://localhost/BizTalkManagementService/swagger

BizTalk-Management-Swagger

Publish WCF-Basic HTTP methods to Azure API Management

Starting from BizTalk Server 2020, WCF-Basic HTTP API endpoints can be published to Azure API Management. It will provide the option to monitor and manage the API methods from the Azure Portal.

API-Methods-Azure-Portal

The option to publish to API Management is only available for the Receive Locations of WCF-Basic Http Adapter type. Prior to publishing API methods, create the API Management Service in the Azure Portal.

Publish-API-Management

Users can publish the API methods to the Azure API Management Service:

  1. Right-click the WCF-Basic HTTP Adapters and select the “Publish to API Management “option
  2. In the Publishing Dialogue, sign in to the Azure Portal
  3. Select the Azure Subscriptions, Resource Group, and API Management Service
  4. WSDL Specifications can be published by choosing either WSDL file or Http URL with single WSDL
  5. Mention the API Name
  6. Then click on the Publish button

Once the WCF HTTP APIs are published successfully, you can view the published APIs under the selected API Management Service. Now the user has the facility to test or push the API methods to higher environments.

Publish-API

Conclusion

As integration solution has started to focus on Azure Serverless technologies, Microsoft has started to provide Hybrid integration capabilities.  It will provide the options for the users to choose the solution that suits their business case. With the release of BizTalk Server 2020, more Azure Service offerings like Backup the BizTalk databases, API Management Service are introduced. Along with Logic Apps Adapter, these new features will help you build flexible integration solutions. Happy Integration!

The post BizTalk Server 2020 – Features that Developers/Administrators can leverage appeared first on BizTalk360.

Querying BizTalk Subscriptions in BizTalk360

Querying BizTalk Subscriptions in BizTalk360

Introduction

The ability to query BizTalk Subscriptions is useful when you want to review how Orchestrations and Send Ports subscribe to messages etc. For example, when troubleshooting routing failures, you can review the existing subscriptions to see if any of them are improperly configured, thereby causing the routing failure. It is handy for the BizTalk users to view the publisher and subscriber when they troubleshoot or determine the use case.

Until now, BizTalk360 had all other queries (Message Box, Tracking Data and EDI), except Subscriptions. That feature is pending with respect to match the functionalities in BizTalk Admin Console. Based on customer feedback, we have implemented this feature in the upcoming release (9.0.3) of BizTalk360.

customer-feedback

Type of Subscriptions

Subscriptions are created by service classes in BizTalk Server, which is listed in the “adm_ServiceClass” table in the BizTalk Server Management database. These services include the caching service; in-process and isolated messaging, hosted by the Endpoint Manager; and orchestrations/XLANG hosted by the XLANG subservice. Each of these service classes can create subscriptions and receive published messages.

Subscribers use subscription in BizTalk to define the properties or criteria of the messages in which they are interested. A Subscription in BizTalk is like a filter or criteria or a condition that dictates BizTalk which message needs to be routed to which component or entity in BizTalk. There can be more than one subscriber to the same message. If that happens, each subscriber will get its own copy of the message.

In BizTalk Server, there are two subscription types are available. Both types are described below

Activation Subscription

An activation subscription specifies that when a message that fulfills the subscription, it should activate or create a new instance of the subscriber when such a message is received.

 Example: Activation subscriptions include send ports with filters or send ports that are bound to orchestrations, and orchestration’s receive shapes that have their Activate property set to true.

Instance Subscription

An instance subscription indicates that messages that fulfill the subscription should be routed to an already-running instance of the subscriber.

Example: Instance subscriptions are orchestrations instances with correlated receives and request/response-style receive ports waiting for a response from an external system.

Subscriptions queries in BizTalk360

The Subscription query feature can be found under Operations-> Data Access-> Message Box Queries. As like the other MessageBox queries (Suspended Service Instances), subscriptions are fetched from the BizTalk Message Box database(s) with filters and Max Matches similar as in the BizTalk Admin Console.

Clicking the Execute Query button will fetch the subscription list with both activation and instance subscriptions. BizTalk360 has the capability to export the results in excel format. Users can manage the filters with saved query functionality.  

saved-queries

In the Subscription list view, there are three buttons are available to display the following

1. Subscription Details & Predicates

When navigating to the subscription details page, it will present detailed information about subscriptions and predicates. Subscription General tab will display Subscription Priority, Ordered Delivery, Creation Time, Subscription Type, etc.

subscription-details

Predicates: The Message Agent calls the bts_FindSubscriptions stored procedure held in the BizTalk Message Box to establish which subscriptions if any, match this message. This stored procedure queries the Subscription and Predicate SQL tables and joins them with the properties inserted for the batch of messages to identify potential subscribers.

Different Predicate SQL tables are used to match all the possible subscription expression combinations.

  • EqualsPredicates
  • ExistsPredicates
  • GreaterThanPredicates
  • GreaterThanOrEqualPredicates
  • GreaterThanPredicates
  • LessThanOrEqualPredicates
  • LessThanPredicates
  • NotEqualPredicates

subscription-expression

2. Service Properties

Clicking the Service properties button will provide the service instance details, error information and Message details, Message content and Message context properties.

service-properties

When subscribing, it is not possible to subscribe to any content of the actual messages entering BizTalk, but only to what information is stored in the Context of the message. The message metadata is called Context Properties; on receiving the message, the Adapter, Pipeline, and Map will possibly add information to the Context.

Context Properties can either be Promoted or Not promoted. Properties that are promoted can be used for subscribing to the message. However, Not promoted properties cannot be used for subscribing to the message.

message-properties

3. Messages

When you click on the Message icon, it will populate the Messages query with Service Instance ID and Host Name related to the subscriptions.

Subscription processing is complex, so in short, you just need to remember that once a message arrives at the Message Agent, subscribers are evaluated, the message is inserted, and references are then added into the host instance queue, ready for processing by the subscriber hosted by the host instance.

query-expression

User Access Policy

When users are having the permissions for Message Box Queries, they are also able to access the Subscriptions. To view the Messages and Content, the user should have permission to view the Message Content/Context in the user access policy.

data-access

When the user does not have the privilege to view Message Content/Context, they will be restricted to view Message Content/Context from the Subscriptions List.

query-box

Note: Subscriptions are only implemented in the BizTalk360 Operations section. Subscriptions are not implemented in Data Monitoring as it is not applicable for the feature.

 Conclusion

We are constantly improving on the features based on feedback in the customer forum. BizTalk360 will continue to provide more useful features in every release. Why not give BizTalk360 a try! It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk environments. 

The post Querying BizTalk Subscriptions in BizTalk360 appeared first on BizTalk360.

BizTalk360 Application Server – Hardware Requirements

BizTalk360 Application Server – Hardware Requirements

Introduction

Managing the infrastructure of BizTalk/SQL Servers is a vital task to BizTalk/System Administrators. Manually monitoring the infrastructure of BizTalk Environment is a cumbersome task.  BizTalk360 is the operational, monitoring and analytics tool which is used to manage the health of BizTalk and SQL Servers in an efficient way. Many customers approach the BizTalk360 Support Team to provide the Hardware Requirements of BizTalk360.

In our Documentation portal, we have provided the BizTalk360 installation document with hardware requirements for a BizTalk360 instance which contains one BizTalk Group. We provided the prerequisites with both software and hardware requirements, considering the BizTalk360 is set up in a stand-alone server.

We also have BizTalk360 Installation guide to set up BizTalk360. This article is focused on the hardware requirements of Windows Server to install the BizTalk360 application.

BizTalk Group Configuration

In most organizations, BizTalk Groups are set up in at least 3 environments (Production, Staging, QA/Dev). Considering these configurations of BizTalk Groups, we are suggesting the BizTalk360 application servers hardware requirements with BizTalk360 Gold tier. Having the Gold tier, and not Platinum, means that you will not have the ability to use the BizTalk360 Analytics features, but this will also save you from needing additional resources which come along with the Platinum features. However, with the mentioned hardware requirements, we kept the possibility open for upgrading to the Platinum tier, without the need for additional hardware requirements.

BizTalk-Group-Configuration

BizTalk360 – Application Server Configuration

Goal: Create a separate instance of BizTalk360 Application Servers for twos version of BizTalk Server environments (BizTalk Server 2016 & 2013 R2).

Scenario 1

We have written down the requirements for the BizTalk360 Application servers for BizTalk Server 2013 R2 environments, as well as BizTalk Server 2016 environments. Each of these BizTalk360 Application servers will be used to operate and monitor 3 different BizTalk environments. The specifications for these servers have also been written with that requirement in mind.

You will also find the requirements for the SQL server, which will host 2 SQL Server instances. Each SQL instance will host a BizTalk360 database, which will connect to either the BizTalk360 application server for monitoring the BizTalk Server 2016 environments or the one for monitoring the BizTalk Server 2013 R2 environments.

The system requirements to install BizTalk360 are considered to configure 3 BizTalk Groups in per BizTalk360 installation.

BizTalk360 Application Server

Resource

Requirements

Computer and Processor

A computer with an Intel Pentium-compatible CPU that is 1 GHz or higher quad processors. The 64-bit versions of BizTalk Server require a 64-bit operating system running on an x64 based system.

Memory

16 GB or higher

Hard Disk

Minimum 20 GB of available hard-disk space for a complete installation including the operating system and all prerequisite software. The hard disk must be NTFS formatted.

 

BizTalk360 – Database Server (2 SQL Instances)

Resource

Requirements

Computer and Processor

A computer with an Intel Pentium-compatible CPU that is 1 GHz or higher quad processors. The 64-bit versions of BizTalk Server require a 64-bit operating system running on an x64 based system.

Memory

16 GB or higher

Hard Disk (Database)

Minimum 200 GB of available hard-disk space for a complete installation including the operating system and all prerequisite software. The hard disk must be NTFS formatted.

 

Why do we need these specifications?

In complex BizTalk environments, multiple BizTalk Servers and SQL Servers are configured. Based on these BizTalk Group configurations, we arrived at the BizTalk360 Application Server and BizTalk360 Database Server Configuration.

BizTalk360 Application Server

  1. Processor: To process the various data from (2 or more) BizTalk Servers and SQL Servers, BizTalk360 Application Server requires a high-end processor.
    • WMI Queries are used to collect the Event Log Sources from BizTalk and SQL Servers

    • Querying against the BizTalk databases like Message Box and Tracking Database

    • PerfMon is used to collect System Resources and Performance Metrics analytics data

  2. Hard Disk: Disk Storage of BizTalk360 Application Server is necessary to hold the following data
    • Message Content (Archived Messages)

    • User Profile Images

    • Processed Email Content

    • Custom Widget Scripts

BizTalk360 Database Server

BizTalk360 collects Event Log Sources and Performance Metrics from BizTalk and SQL Server and stores that data into the BizTalk360 database. It also collects Tracking Data to determine the Message Patterns and Transmission Failure Rates.

The BizTalk360 Administrator makes sure the Data Purging Jobs are in healthy status to manage the growth of BizTalk360 database.

BizTalk360 High Availability

Many customers are configuring the BizTalk360 Monitoring Service and Analytics Service in high availability with Windows server cluster setup.  In this case, you need two BizTalk360 Application servers, with the configuration mentioned in the above table.

Note: Customers can install all three components in high availability configuration

  • BizTalk360 Web Application
  • Monitoring Service
  • Analytics Service

High-Availability-Status

Users can manage the BizTalk360 Monitoring and Analytics services in the Settings -> BizTalk360 Health. Read this article to know more about BizTalk360 High Availability status.

Scenario 2

Some customers use different instances of BizTalk360 for the same version of a BizTalk Group in QA, Staging and Production environments. In this scenario, three different Windows servers are required for BizTalk360 Application configuration. Similarly, three SQL Servers are required to configure the BizTalk360 databases. If they want to set up high availability of BizTalk360 in production, then one BizTalk360 Application Server and one SQL Server are added to the list.

BizTalk360-Application-Configuration

Conclusion

Once all the setup and configurations are in place, it is a quick and seamless task to install/upgrade BizTalk360. The customer suggestion and feedback are always heard and addressed, which helps us to improve the product and provide better service.

The post BizTalk360 Application Server – Hardware Requirements appeared first on BizTalk360.

Challenges in determining the Message Pattern using tracking database

Challenges in determining the Message Pattern using tracking database

Introduction

In complex BizTalk environment, it is important to visualize the patterns of how messages are processed within the system or integrate with multiple disparate systems to handle the message transfer from one point to another. It could be a simple direct transfer between the two ports, or through a complex messaging route passing through orchestrations to perform certain validations, etc. It is helpful to BizTalk Administrators or Business users to track the flow of messages in different stages. For this purpose, the BizTalk Admin Console has the capability to view the workflow of messages, although that feature will mainly be of use for BizTalk Administrators.

BizTalk360 has the functionality to allow users to view the graphical flow of the messages within your BizTalk Group. However, the challenge with the Graphical Message Flow Viewer is that, for instance, if there are 100 messages passing through the system at a point of time, each message is tracked at the service instance level with a unique id. The problem with this approach is that every single transaction is logged into the database. In an environment where there are lots of messages passing through the BizTalk system, there is every chance that the database will get filled up quickly.

Message pattern is to visualize interactive message flows with additional capabilities like the number of messages processed and Average Execution Time in the artifacts.

Message Pattern using Tracking Data

In BizTalk360 v8.0 introduces the “Messaging Patterns” capability to visualize a list of unique message flows that are being processed by the system. The key difference from Graphical Message Flow Viewer is that, in the Message Flow list, only the unique messaging patterns get recorded. As in the previous case, if there are 100 messages passing through the system at a point of time, not all the 100 message flows are recorded in the Messaging patterns. Only the unique message patterns are recorded, which could be just, say, 15 or 20.

For Messaging Patterns to work, a backend service (called Mapping Data) is responsible for keeping track of the unique message flows. That Mapping data service will collect the tracked message in the DTA DB.

Note: Make sure you have enabled Tracking at the ‘Pipeline’ level for the unique data flow data to be captured.

message-patterns-biztalk360

Challenges in determining the Message Patterns

Determining the message flows using tracking data has some challenges. As BizTalk Developers and Administrators might be aware that the BizTalk DTADb has its own challenges. For example, few customers, face performance issues with the tracking database when the BizTalk360 Analytics service fetches message flows using the MessageInout table;

  1. Query Time out

    In the initial version of Message Pattern, we have seen that query timeout issues have happened while retrieving message service instance data.

  2. Temp Database Growth

    To address this issue of query time outs, since BizTalk360 Version 8.5, the relevant data is fetched in a temporary table in the tracking database. However, with a few customers, this approach causes temp database growth.

Alternate Technique (BizTalkMgmt Database)

To overcome this challenge, in BizTalk360 9v.0 Phase2, we bring an alternative mechanism to determine the message patterns. The alternate way of finding the message patterns uses the BizTalk Management database relational data. It has the advantage that here the flow is determined without enabling the pipeline tracking at the artifact level.

Message Flows have been determined by the following models:

  1. Publish-Subscribe Model

    This method is used to determine the message flow between ports without any orchestration being involved. It uses the Filter property in send ports to understand the start and endpoint of the flow.
    send-port-propertiessend-receive-port

  2. Orchestration Port Binding Model

    The Orchestration Port Binding model uses the orchestration port binding relational data to determine the message flows. Using BizTalk Artifact tables (adm_ReceiveLocation, bts_receiveport, bts_sendport, bts_pipeline, bts_application) with orchestration port and orchestration port binding.

SELECT 'Pipeline' as fromServiceType,FROMPL.Name as fromServiceName,   
RP.nvcName as fromPortName, 
RP.uidGUID as fromUIdServiceId, 
RPAPP.nvcName as fromAppName,
							   
'Orchestration' as toServiceType, ORCH.nvcFullName as toServiceName, ORCH.uidGUID as toUIdServiceId, OP.nvcName as toPortName, '' as toAPPName					    
								
FROM [dbo].[bts_orchestration] ORCH
                                
INNER JOIN  [dbo].[bts_orchestration_port] OP ON ORCH.nID = OP.nOrchestrationID
                                
INNER JOIN [dbo].bts_porttype PT ON PT.nID = OP.nPortTypeID
                                
INNER JOIN  [dbo].[bts_orchestration_port_binding] OPB ON OPB.nOrcPortID = OP.nID                                
                                
LEFT OUTER JOIN [dbo].[bts_receiveport] RP ON RP.nID = OPB.nReceivePortID
                                
LEFT OUTER JOIN adm_ReceiveLocation RL ON RL.ReceivePortId = RP.nID
                                
LEFT OUTER JOIN bts_pipeline FROMPL ON FROMPL.Id = RL.ReceivePipelineId                               
                                
LEFT OUTER JOIN bts_application RPAPP ON RPAPP.nID = RP.nApplicationID
WHERE OP.nPolarity = 2 AND RP.nvcName 
IS NOT NULL AND (OP.nvcName  IS NOT NULL AND RP.nvcName IS NOT NULL)

orchestration-port-bindingflow

Message Pattern Configuration

Users can choose the message flow determination technique by enabling or disabling the Dynamic pattern option.  When Dynamic pattern option is enabled, then the Message pattern will be determined using the tracking database.  If it is disabled, it will use the BizTalk Management database to find the message patterns for the BizTalk Group.

message-pattern-configurationFew Points to be remembered to use this new mechanism (Dynamic Pattern – Disabled).

  • Pipeline tracking needs to be enabled to see the message count and average execution time
  • This mechanism will be able to determine the possible flows. In case a Port uses other filter criteria than Port Names, they are not determined

Other Improvements

In the upcoming version of BizTalk360 v9.0 phase 2, we are bringing management capabilities like hiding and deleting of message flows.

The Hide Operation can be used on the message patterns that are decommissioned or less frequently used. The Delete Operation can be applied on the flows that are no longer used in BizTalk Group.

messaging-patterns-hide

Conclusion

We hope these improvements will bring a new way to determine the message flows in your BizTalk environments. You can expect these improvements to be available in the upcoming version of 9.0 phase 2. We are constantly improving on customer feedback. Please share your requests through the feedback portal

The post Challenges in determining the Message Pattern using tracking database appeared first on BizTalk360.

BizTalk360 GitHub Project

BizTalk360 GitHub Project

Introduction

BizTalk360 is the one-stop tool to manage operations, monitoring and analytics of your BizTalk environment.  Over 80+ features in the product bring all the BizTalk Tools like EDI, ESB, Business Rules, BAM, and the BizTalk Health Monitor. It is a matured product with these features included of which there have been 50+ iterations of product updates. User Access Policies with governance audit (Security), Operational Tool Integration, Monitoring, and Analytics are among the BizTalk360 core capabilities. BizTalk360 has brought important business scenarios into the product like:

  • Auto-healing of BizTalk Artifacts
  • Automated actions on Suspended Instances
  • Process Monitoring and Data Monitoring

 These capabilities make BizTalk360 the must-have tool to manage the BizTalk Server.

Customization in BizTalk360

BizTalk360 has provided solutions to common Integration scenarios (BizTalk, Azure). In addition to the above-mentioned features, BizTalk360 has provided the opportunity to integrate custom solutions into the product through:

  • Custom Notification Channels
  • Custom Widgets
  • Create Alarm Tool

BizTalk360-GitHub

Custom Notification Channels and Custom Widgets are bundled into the product, during the major update of v8.0.  Users are taking advantage of these features, by customizing alert notification to their preferred channels and visualize their important data in BizTalk360 Dashboard through Custom Widgets.

The source code of such customized solutions (Notification channels, Widgets and Alarm Creation Tools) is available in our GitHub Project.

Users can utilize the source code to create new customized solutions or extend their existing solution to meet their business requirements.

Users can commit their solutions into BizTalk360 GitHub which will benefit other users who follow this channel.

Custom Notification Channels

Custom Notification Channels provide a platform to send alerts to an external system, like your ticketing system, internal databases, calling a REST endpoint, executing a PowerShell script, etc. If you want to know how you can create, customize and configure new or existing custom notification channels, read this article.

 BizTalk360 has out of box solutions for the following Custom Notification Channel:

Users can create their own customized solution for their ticketing systems rather than ServiceNow or trigger activities (PowerShell) based on their business needs. We can see some of the use cases that users can take by customizing the notification channels:

  1. ServiceNow – Customers can extend the BizTalk360 ServiceNow notification channel to add more fields to meet the business process
  2. PowerShell Notification – Users are utilizing this capability to trigger actions when monitoring conditions are violated
    • Start/Stop Host Instances

    • Terminate dehydrated service Instances

    • Terminate messages after X number of days

    • Manage Maintenance mode during BizTalk Deployments

  3. Microsoft Teams – Enhance the Microsoft Teams notification channel to include a message card
  4. SMTP Notification – Users can improve on default notification by using the SMTP Notification Channel to configure Up, AutoCorrect Email separately.

The source code in the GitHub project can be pulled, customize the code to meet your business scenario and then configure it in BizTalk360 to be able to use your custom solutions.

Custom Widgets

Custom Widget is a powerful feature by which users can take advantage in BizTalk360. Customers can integrate their business process flows by using Custom widgets. This allows integrating solutions with BizTalk360 own API’s, as well as third-party API’s (like SalesForce, Power BI, Azure, etc.) and database queries by using the Secure SQL Queries.

Use Cases:

  1. Business Process – Few customers are integrating their business flows using their internal API’s and created custom widgets by which business users take advantage of BizTalk360 dashboard integration
  2. Graphical Representation – Visualize the reports in Solid gauge using High Charts, which is available in BizTalk360
  3. Consolidated View of Monitoring Dashboard – Users are able to customize their monitoring dashboard to a single view, by grouping type of monitors
  4. Secure SQL Query – This provides the opportunity to pull data from other databases into the BizTalk360 dashboard through Secure SQL Queries
  5. Monitoring Status– Many of our users are using dashboards to monitor their business flow integrations. They can create a custom widget and pin it to a dashboard to monitor the integration. You can think of showing the following:
    • Errors & Warnings of an Alarm in a Grid

    • Last Date Time of Errors & Warnings

    • Web Endpoint Monitoring status of an integration

    • Number of Suspended Instances in an alarm

    • Number of messages processed in an application

  6. Monitoring SQL JobsSQL Jobs are monitored through BizTalk360 API’s by creating a custom widget

custom-widget

status-monitor

Alarm Configuration

Another example of running third-party executables is the creation of BizTalk360 alarms. Normally, you’ll have to do this manually after each deployment of a new or changed BizTalk application. This too can be a time-consuming and error-prone task.

By using BTDF and a tool called BT360Deploy, you can automate the creation of alarms during the deployment of your BizTalk application.

In complex BizTalk environments, creating Alarms and mapping artifacts to monitor is a tedious process. To overcome this challenge, our Technical Lead (Lex) has written an article to configure BizTalk Artifacts for monitoring through the Alarm Creation Tool. In that article, he explained how the BizTalk Deployment Framework (BTDF) is integrated with BizTalk360 Monitoring.

Conclusion

Users can take advantage of custom solutions using Custom Notification Channel, Custom Widgets and Alarm Creation Tools.  Customers get the maximum benefits out of BizTalk360 when creating custom solutions.

The post BizTalk360 GitHub Project appeared first on BizTalk360.

BizTalk Server 2020 Announcement with Hybrid Integration updates

BizTalk Server 2020 Announcement with Hybrid Integration updates

Introduction

Integrate is the premier integration conference for everyone working in the Microsoft Integration Space. Integrate happens annually in London and this year it happened from 3 to 5 June. The 3-day event, with speakers from the Microsoft Product Group and from the Global Integration Community, is the ideal event to listen to leading Integration Specialists and learn what is coming next in Microsoft integration. This year a lot of newer people were joining the event. Initially, the event used to be a BizTalk Server focused event however, from last few years it expanded the scope to general integration technologies in Microsoft space, including Azure Integration capabilities.

The Value of Hybrid Integration

Paul Larsen, a Principal Program Manager for the Microsoft Corporation in Redmond, presented a session on “The Value of Hybrid Integration”. At Microsoft, Paul leads the Host Integration engineering team to deliver and support Azure, BizTalk Server, SQL Server, Office, and Windows connectors to existing IBM systems. At Integrate, he started the session by explaining the legacy of BizTalk Server and explained the meaning of the word Hybrid with some examples.

BizTalk Server 2020 Announcement

After that, Paul made the big announcement of “BizTalk Server 2020” and he also announced it will be released at the end of 2019.

BizTalk Server 2020 Announcement

Important feature updates in BizTalk Server 2020 are:

  • Integrate cloud Services with built-in adapters for cloud, Logic Apps to On-Premise Gateway, Service Bus, Event Hubs, API Management, O365 Mail, O365 Schedule, O365 Contact, Blob Storage (Dev) API management to all the resources (On-Premise and Cloud)
  • Manage BizTalk Server environments by using the new BizTalk Server REST API’s
  • The Blob Storage adapter is still under development. Using File adapters which are published as a UNC Endpoint is not straight forward, as pluggable. Using Blob storage adapter will be a useful feature that will help administrators to, for example, have the Backup BizTalk Server job write the backups to Azure Blob storage
  • Secure the BizTalk Server Infrastructure settings with Read-only administrator view
  • Deployment with VSTS – Application Life Cycle Management to deploy and update BizTalk Server instances using Visual Studio Team System
  • Transport Layer Security 1.2– Securely deploy BizTalk Server using TLS 1.2 authentication and encryption
  • SQL Server Encrypted Data– Read and write to SQL Server with Always encrypted columns using BizTalk Adapter for SQL Server
  • Advanced Scheduling -Set up recurrence on BizTalk Server receive locations with greater precision using time zones, months, weeks, and days
  • Application Insights– Send BizTalk Server tracking data to Application Insights for extensible analytics, to understand performance and usage of BizTalk Server
  • Event Hubs – Send BizTalk Server tracking data to Event Hubs
  • Power BI Integration – Analyze and report BizTalk Server tracking data using Power BI. BizTalk operational data template, for use with Power BI visualizations and dashboards
  • .NET components continue to evolve over the last release of the .NET MQ Client.
  • A new version of Enterprise Single Sign-On

BizTalk Server 2020 Adapters

BizTalk Server 2020 Deployment

BizTalk Server 2020 Advance Scheduling

Host Integration Services

Host Integration Services has improved a lot in this version of BizTalk Server 2020. In short, most of the packages in BizTalk Server are getting updated in this version:

  1. HIS as add on to BizTalk licensing
  2. New Microsoft platforms
  3. New IBM platforms – integrated feature packs

BizTalk Server Migration Tool

Microsoft IT develops the BizTalk Server Migration tool which they use to migrate their own BizTalk environments. Because the architecture of BizTalk Server is kept common between BizTalk Server 2010,2013, 2013 R2 and 2016, the migration does not affect the underlying layer of BizTalk Components. In addition, as Microsoft makes this tool available to BizTalk users, this tool will be helpful for them on dehydrating the configuration on an old BizTalk server and re-hydrate that configuration to new Azure IAAS or VM.

BizTalk Migration Tool References:

Integrate 2019 USA

Logic Apps IBM 3270 Connector

Currently, the Logic Apps connector for IBM 3270 emulation is available in preview. In short, the Microsoft Connector for 3270 integrates Azure applications with resources available via TN 3270 Screen Scraping.

IBM 3270 Connector

Robotic Process Automation (RPA) is an Industrial trend in cloud computing which offers screen scraping capabilities. Host Integration Server extends the Integration to the TN3270 connector with Telnet or TCP protocol to connect to IBM Mainframe systems. However, you require an Integration account to be able to connect the IBM 3270 adapters.

After that, Paul demonstrated the Integration of IBM mainframe programs and Azure Cloud application with 3270 Logic App connector. After the Demo, he made a couple of announcements about CICS, IMS screen clients for 3270/5050 connectors and VSAM, DDM .NET clients for IBM file systems.

Logic Apps IBM Connectors Road Map

In conclusion, Paul touched base on the Logic Apps IBM Connectors Road Map beyond the 2020 version. For instance, BizTalk Server vNext version will include improvement on a number of Logic Apps connectors:

  1. 3270 IBM Connector Improvement
  2. Midrange 5250 AS connector
  3. Mainframe SAM Connectors
  4. DB2 and Informix Support

IBM Connectors Road Map

Conclusion

The well-informed reader might have noticed that several BizTalk Server 2020 features also appear in the so-called Feature Packs for BizTalk Server 2016 which have been released by Microsoft. However, BizTalk Server 2020 also helps in aligning with the latest versions of the platform products. So, this release of BizTalk Server will be of use for all BizTalk Server customers.

Secondly, the release of BizTalk Server 2020 is good news for all existing BizTalk users, as it indicates that Microsoft covers them for the next couple of years. Similarly, we notice that customers who are just starting their integration journey, also choose for Microsoft BizTalk Server. In other words, this new release will be very helpful for them as it will help them with both their on-premise integrations, as well with their hybrid integrations.

The announcement of BizTalk Server 2020 was done during Integrate 2019 in London. However, in case you missed the event, there is no need to worry, because at the end of this month, Integrate will also come to the US! So, if you are still not sure if you should attend Integrate 2019 US, then this blog post, written by our CEO Saravana Kumar, might convince you to participate in the world-wide premier Microsoft Integration event.

Integrate 2019 USA

The post BizTalk Server 2020 Announcement with Hybrid Integration updates appeared first on BizTalk360.

System Alerts and Unmapped Application Artifacts in BizTalk360

System Alerts and Unmapped Application Artifacts in BizTalk360

Introduction

BizTalk360 has powerful monitoring features to manage the BizTalk Application Artifacts, Queues, Infrastructure, Health check tools, etc. Most of the customers are using BizTalk360 for its core monitoring capabilities. BizTalk360 is the single tool to manage operational activities, monitoring and analytics of the mission-critical BizTalk environment. In this transforming phase, it is a tedious process for a dedicated person to monitor all the configurations. To address this scenario, System Alert notification is implemented in BizTalk360 v9.0.

Why System Alerts?

Currently BizTalk360 use Alarms to send out the notifications of Application artifacts, queues, Infrastructure settings health of the BizTalk environment. In the same perspective there is need to know the health of BizTalk360, which will helpful to the administrator who takes care BizTalk360. The BizTalk360 Monitoring service will take care of sending system notifications to the configured admin user email boxes. These email boxes can be configured in the system settings. A new monitoring sub service called ‘System Alert’ has been introduced to send system related notifications. System Alerts can be enabled in the System Alert Settings under Monitoring and Notification tab of system settings. A user can configure multiple admin emails with semicolon separated values.

A System Alert notification can be pushed to the admin users in the two ways:

  • Scheduled
  • Automatically Triggered

The BizTalk360 Monitoring service has a polling interval of 60 seconds and it will check for system alerts to be pushed. Either these alerts will be pushed based on a trigger or on a schedule.

NOTE: Custom Notification channel alerts are not applicable for Systems Alerts in this version (9.0).

System Alert Schedules

System Alert Schedules will send periodic reports of unmapped application artifacts. This will be helpful to the BizTalk Operation Users/ Administrators who can take appropriate actions to configure the artifacts for monitoring.

System Alert configuration can be scheduled as follows:

  1. Daily – Daily schedule will push the notification at the specified time in a day for all the days of the week.
  2. Weekly – Weekly schedule can push the notification on the specified day and time in the week (for example, Wednesday, 12 PM)
  3. Monthly – Monthly schedule can push the notification on the specified day and time in the month (for example, the 15th day of the month, 10 AM)

System Alerts Schedule Configuration

Based on the configuration, the Monitoring Service will alert the Unmapped Application Artifacts in the BizTalk Environment. The Unmapped Application Artifacts list will be available as an attachment in the email notification.

Unmapped BizTalk Application Artifacts

The Unmapped Application Artifacts list feature can be used to manage the artifacts which are not mapped to any of the BizTalk360 Alarms for monitoring.

Scenario

It will be helpful for the administrators who are taking care of the application artifact’s health. When BizTalk Developers or a deployment team deploy new artifacts into the BizTalk Group, the BizTalk administrators might not exactly know the newly deployed artifacts. In such cases, the Unmapped Application Artifacts status will provide a warning indication to the administrators/operators.

The status of Unmapped artifacts is shown in the monitoring dashboard. Unmapped application artifacts status will be healthy when all the application artifacts in an environment are mapped to the alarm. It will show the unhealthy status, when any of the artifacts left unmapped for monitoring. “Unmapped Application Artifacts Link” will be listed with the application artifacts which are still unmapped.

Unmapped Application Artifacts

Unmapped Monitor State

From BizTalk360 version 9.0 on, the new Unmapped monitoring status is introduced. This is the default state for application artifacts. Users can set the expected state and start monitoring the artifacts.

NOTE: Users can use the Do Not Monitor state, if they don’t want to monitor an artifact. In such cases, that artifact will not be listed under Unmapped application artifacts.

Automatic Triggered Alerts

When the events or conditions meet the triggering rules, then the alerts will be triggered automatically. System Alerts can be of triggered based

  • BizTalk360 License Expiration
  • Database Sizes (BizTalk Database & BizTalk360)

In this version of BizTalk360, BizTalk360 License Expiration is implemented. Database Sizes triggered notification will be implemented in a future version of BizTalk360.

License Expiration

When your BizTalk360 license is about to expiry, BizTalk360 System Alerts service will notify the license expiration to the admin users. Notification can be sent on the 30, 15, 7, 2 days before the license expiry date.

System Alert History

Click on the ‘System Alert History’ button to view the historical System Alerts. Alert History is maintained for both alert types(Schedule and Trigger). Users can view the system alerts email notification which has been sent to admin users in HTML Format.

System Alerts History

Data Purging

System Alerts historical records are maintained based on the data purging policy of “Alert & Maintenance History”. BizTalk360 will maintain the system alert notification history up to the configured number of days/months.

Based on your requirement you can set the purge duration.

Conclusion

System Alerts notifications will be helpful to BizTalk Administrators to manage the BizTalk Group health and BizTalk360 environments.

Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

The post System Alerts and Unmapped Application Artifacts in BizTalk360 appeared first on BizTalk360.

Achieving a Consolidated Monitoring View Using Custom Widgets

Achieving a Consolidated Monitoring View Using Custom Widgets

BizTalk360 Monitoring

BizTalk360 comes with out of box capabilities to monitor BizTalk Artifacts, BizTalk Environment, Queues (IBM, Azure Service Bus and MSMQ), File Location (File, FTP and SFTP) and so on. The Monitoring Dashboard represents the Monitoring Status in a nice graphical chart view. Different set of users like BizTalk Administrators, Operators, Technical members and Business users are using the monitoring features for their day to day activities. Users create multiple alarms to suffice their needs to monitor the health of their BizTalk environment.

While users are monitoring the multiple alarms, they must keep an eye on multiple dashboards. It is a cumbersome task if a user takes care of more than two Alarms. Based on this requirement, few customers have requested for consolidate view of alarms. In this article, we are going through the process of how you can create a consolidated view of multiple alarms using Custom Widgets.

Scenarios

BizTalk360 users are configuring the alarms based on their business vertical or alignment to their process. There have been different patterns in alarm configurations. We can see some of the commonly used patterns;

  • Integration-based alarm configuration

Configuring BizTalk Artifacts, Web Endpoints, Queues, File Location etc. of integration in an alarm.

  • Role-based alarm configuration

BizTalk Administrators will look after infrastructure settings like Disk, System Resources, Host Instances and configure in an alarm BizTalk Operators can configure BizTalk Artifacts and other entities in a single alarm

  • Entity-based alarm configuration

For instance, users can configure all their Queues (IBM/Azure Service Bus) in an alarm. Similarly, they can configure Web Endpoints, Infrastructure (Disks, System Resources) in separate alarms.

Based on the above patterns, Role-based and Entity-based alarms can have a single view of multiple alarms in which you can monitor all the entities. When you follow the Integration-based alarm pattern, then entities like Application Artifacts, Azure Service Queues and BizTalk Server health activities are in different alarms.

Many organizations follow the Integration pattern by grouping all the related artefacts in an alarm. In this scenario, users (BizTalk Operators/BizTalk Technician) must monitor multiple alarms. For Instance, BizTalk Operators looking after BizTalk Artifacts or Developers responsible to monitor Azure Service Bus Queues will have to monitor the multiple alarms. Let’s see how to overcome this challenge by using custom widgets to group multiple alarms.

Consolidated Dashboard

The Monitoring Dashboard is one of the most used features among the users to monitor BizTalk Environments and Artifacts in BizTalk360. An enriched monitoring capability while using Custom widgets to group the multiple alarms to get a single view based on business or team members role.

Custom Widgets are a powerful feature in BizTalk360, as such widgets will provide the opportunity to meet their business scenarios. Many customers adopted Custom Widgets to provide solutions to end users.

Following are a few illustrated scenarios of grouping alarms when it is configured based on integrations

  1. Azure Service Bus Queues
  2. BizTalk Application Artifacts
  3. BizTalk Environment Health

In this article we take one of these scenarios, to show how users can achieve the grouping alarms using Custom Widgets

Azure Service Bus Queues

Custom Widgets used BizTalk360 Secure SQL Query API’s to fetch all Azure Service Bus Queues Monitoring Status from BizTalk360 Database. A user can select the filter with different alarms they want to group.

By default, the user can see only the unhealthy queues in an environment. If they set the Enable Healthy Queue option, they will be able to view all queues in the tree nodes. The Azure Service Bus Queues Custom Widget will fetch the information from Secure SQL Query with a refresh interval which is configured in the script of the custom widget.

Follow the below steps to achieve the Custom Widget’s script;

1. Secure SQL Query

Create the Secure SQL Query to fetch the monitoring status of Azure Service Bus Queues in the environment. To know more about how you can execute Secure SQL Queries using custom widgets, refer to this article.

2. Custom Widget Script

Create the initial variables, Place holders and SQL Query to call the BizTalk360 API method in the Custom Widget’s script

// BEGIN User variables
    azureQueueRefresh = 20;
    username = "`username_812414`"; 
    password = "`password_701422`"; 
    environmentId = "`productionenvid_189308`";    
 queryId = "8eea2771-10a7-44c6-8709-a597687434cf"; 
 queryName = "Azure ServiceBus Queues"; 
 sqlInstance = "kovai-bts";     
database = "BizTalk360"; 
    sqlQueryForAzureGraph = "Select AA.Id,AA.[Name],AME.MonitorStatus,AME.ExecutionResult from [dbo].[b360_alert_MonitorExecution] AME Inner Join[dbo].[b360_alert_Alarm] AA ON AME.AlarmId = AA.Id and AME.MonitorGroupType = 'Queues' WHERE AME.LastExecutionTime = (SELECT MAX(LastExecutionTime) from [dbo].[b360_alert_MonitorExecution] WHERE MonitorGroupType = 'Queues' and AA.IsAlertDisabled='false' and AlarmId = AA.Id and LastExecutionTime &amp;gt;= DATEADD(MINUTE,-60,GETUTCDATE())) AND AME.EnvironmentId ='`productionenvid_189308`'"; // The SQL Query which needs to be executed from the custom Widget
    bt360server = "biztalk360.westus.cloudapp.azure.com"; // Name of the BizTalk360 server (needed to do an API call to execute the SQL query
    //Mention the created alarm details and the respective partner name to display in the graph.
    AlarmDetails = [
        { AlarmName: "Threshold-1", PartnerName: "Air India" },
        { AlarmName: "Threshold-2", PartnerName: "British Airways" },
        { AlarmName: "Threshold-3", PartnerName: "DHL" },
		{ AlarmName: "BizTalk360 Default Alarm1", PartnerName: "Jet Airways" }
    ];
    // END User variables

3. GO JS Framework

BizTalk360 uses the GO JS Framework- Organization chart to represent the monitoring hierarchical view in the UI. It will display the organization chart structure as Azure Service Bus Queue node as a root node, Integration friendly name as second level node and Queue threshold violation details as a child node. You can control the nodes expand/collapse capability of through the custom widget code.

4. Filter Options

Users can able to filter the data based on the selected alarms or the option to show healthy queues. By default, users can see only unhealthy Azure Service Bus Queues. Similarly, other scenarios are implemented through custom widgets. To get the full source code of the Custom widgets you can download it from our GitHub Project.

Summary

We hope this article is useful in grouping the multiple alarms into a single view in an environment. This will bring you more control over the custom development you may want to achieve.

Get started with the free 30 days trial. For any queries/feedback please write to us support@biztalk360.com.

The post Achieving a Consolidated Monitoring View Using Custom Widgets appeared first on BizTalk360.

BizTalk360 V8.9.5 Enhanced Features

BizTalk360 V8.9.5 Enhanced Features

Introduction

BizTalk360 has many features that BizTalk Operators and BizTalk Administrators can take advantage to manage their BizTalk Environments. BizTalk360 is constantly improving the existing features based on the customer’s feedback in every release of the product. Similarly, in version 8.9.5 the existing features are improved based on the customer votes for the feature request. In this article, we can see the three different feature improvements like Filter Improvements, ESB Fault Delete Operation, Notification channel reconfiguration that has been implemented.

“The first step in exceeding your customer’s expectations is to know those expectations.” ~ Roy H. Williams, Author & founder of the Wizard Academy Institute.

Filter Improvements

The Filter Component is an important tool to get query results from BizTalk databases like BizTalkMsgBox, BizTalkDTADb and BAMPrimaryImport. In Message Box Queries and Tracking Queries (Graphical Message Flow), there were few filter operators missed in the previous version of BizTalk360. Due this missing filter operators’ customers faced challenges to filter data they expected get results out of it. To overcome this situation, BizTalk360 planned to implement the filter operators to match with BizTalk Admin Console. Please find the consolidated list of filter operators that are implemented in the latest version of BizTalk360.

Message Box Queries:

In Message Box Queries, to the filters Instance Status and Service Class “Not Equals” Operator are implemented in Operation and Data Monitoring Section

  1. All In-Progress Service Instances
  2. Running Service Instances

Similarly, other operators like IS Null, IS NOT NULL and Does Not Contains are implemented. While choosing the filter operators “IS NULL and Is NOT NULL“ there is no need of filter inputs.

Graphical Message Flow:

ESB Filters:

In the ESB Module, “Machine Name and Scope” are implemented in Operation (ESB Exception Data) and Data Monitoring Section (ESB Data) Machine Name filter has operators:

  1. Equals
  2. Not Equals

Scope filter has operators:

  1. Equals
  2. Not Equals
  3. Contains
  4. Does Not Contains

Custom Notification Channel Reconfiguration

Custom Notification channel is a powerful mechanism to notify BizTalk360 monitoring alerts to various channels like Slack, Service Now, Microsoft Teams etc. There is even a possibility that a customer can write their own custom notification channel.

Scenario:

A customer has written their own custom notification channel like PowerShell notification to execute the PowerShell scripts to restart the Windows service when an Error or a Warning occurred. Based on the business case, they want to enhance the custom notification channel implementation.

In this situation, the customer must remove the configured notification channel from the alarms to which it is associated. Then reconfigure custom notification channel to the alarms. If the number of alarms is large, then it is tedious process to remove and reconfigure the notification channel from all the alarms. To Overcome this challenge, in this version 8.9.5 Custom Notification Channel Reconfigure capability has been introduced. By this feature, users don’t need to remove the configured channels from the alarms. Instead the user can click on Reconfigure button and choose the latest DLL to update the existing channel object stream in the database.

The automatically reconfigured notification channel will affect from the next monitoring cycle. The users must reconfigure the notification channel properties value, if they introduce a new alarm or global properties in the updated notification channel. The user can view the list of alarms to which a custom notification channel is associated. This will give insight in the number of alarms the notification channel is associated with.

ESB Fault Delete Operation

From the Feedback portal we understood that customers are interested to have Delete Operation on ESB Fault Exceptions.

Scenario:

A Send Port that you are using in an application fails unexpectedly, therefore both the service instance and the message become suspended and the fault information is written into the ESB exception database. Once the exception is corrected and resubmitted for further go, there may be two scenarios which will come into picture:

  1. Messages which are submitted successfully are residing in the ESB Exception database but are of no further use
  2. Messages are rerouted to the ESB Exception database due to recurrent failure. In this specific case, the original message is also available in the ESB portal

ESB Fault Deletion is introduced in the ESB Exception Management section. The user has the option to select multiple faults, which can be deleted in the one go. The delete operation will remove the data from the following tables in the ESB Exception database.

  1. [MessageData]
  2. [ContextProperty]
  3. [Message]
  4. [Fault]

The Delete operation will delete Multi-Part message or multiple messages and context properties related to the faults from the corresponding tables.

Governance Audit:

BizTalk360 will capture the governance audit about the message delete operation. It will audit the user who performs the delete operation and message Id’s are displayed in governance audit section. Similarly, these operations are notified in Live Feed section.

Summary

We hope that the enhanced features will give you more control over the filters, Custom notification channels and ESB Faults. We always keep track of our Feedback Portal and take up the valid suggestions and feedback. If you would like to know more about how BizTalk360 can help your organization to manage your BizTalk Server middle ware platform, feel free to contact us.

Author: Senthil Kumar Palanisamy

Senthil Kumar Palanisamy is the Technical Lead at BizTalk360 having 14 years of experience in Microsoft Technologies. Worked various products across domains like Health Care, Energy and Retail. View all posts by Senthil Kumar Palanisamy