January 2018 - Page 2 of 3

Integration Down Under Webcast Launched

by Bill Chesnut | Jan 22, 2018 | BizTalk Community Blogs via Syndication

Martin Abbott, Dan Toomey, Wagner Silveira, Rene Brauwers and myself have decide we need a Australian/New Zealand time zone Integration focused webcast similar to Integration Mondays from the UK.

Out 1st webcast will be February 8th at 8pm ADST to register – https://register.gotowebinar.com/register/4469184202097473281

Thanks to SixPivot for providing the GoToWebinar

We will spend some time introducing the leaders of the group and then each person will present a short presentation

Bill Chesnut – API Management REST to SOAP

Martin Abbott – In this session, Martin will walk through the new visual tooling available in Azure Data Factory v2. He’ll look at what you can do, set up source control, install an Integration Runtime for on premises fun, and do a simple data copy to give a flavour of how quick and easy it is to get going.

Wagner Silveira – A Lap around Azure Functions Proxy, Azure Functions Proxy is a simple API Toolkit embedded in Azure Functions, enable quick composition of APIs from various sources. In this lightning talk, Wagner Silveira will show the main features and how to quickly compose an API from various sources.

Dan Toomey – Microsoft recently released the public preview of Azure Event Grid – a hyper-scalable serverless platform for routing events with intelligent filtering. No more polling for events – Event Grid is a reactive programming platform for pushing events out to interested subscribers. Support for massive scalability and minimal latency makes this an ideal solution for a number of scenarios, including monitoring, governance, IoT, or general integration. This talk will demonstrate how easy it is to configure the capture of events with Azure Event Grid.

Rene Brauwers – a reactive integration primer

Azure Cosmos DB Performance – Throughput

by Howard Edidin | Jan 22, 2018 | BizTalk Community Blogs via Syndication

Recently I was designing a Cosmos DB SQL API solution for a client. During testing we were constantly getting http status code 429 – Too many Requests errors. We manually had to increase the Throughput for a collection. This got to be tedious.

In production, this was unacceptable.

We needed to automate the setting of the Cosmos DB Collection Throughput.

Architecture

The solution architecture I built looks like the diagram below and utilizes the Throttling Cloud Design Pattern.

Steps

Create an Cosmos DB Azure Alert for http status code 429 – Too many Requests. The collection has exceeded the provisioned throughput limit.
When the condition is met, an Alert is sent to a Function App.
Function App increases the Request Units for the Collection by 100.
Update Cosmos DB Collection Offer (Request Units).

Throughput SLA

“Throughput Failed Requests” are requests which are throttled by the Azure Cosmos DB Collection resulting in an Error Code, before Consumed RUs have exceeded the Provisioned RUs for a partition in the Collection for a given second.

NOTE: The Metric we want to check for is Status Code 429 – Too many Requests . The collection has exceeded the provisioned throughput limit.

Create a new Azure Cosmos DB Alert

We need to create an Alert for our Azure Cosmos DB Database as shown in the following figure.

The following are the configuration settings.

Select our Resource.
Name for our Alert Rule.
Description of our Alert Rule.
Select Throttled Requests for the Metric.
The condition, threshold, and period that determine when the alert activates.
Condition set to greater than.
Threshold Count set to 1.
For Period we will start with Over the last hour.
Check if the service administrator and co administrators are emailed when the alert fires.

NOTE: We will configure our WebHook setting later.

Create our Function App

We will create our Function App using the Azure Portal. Optionally we could do this in Visual Studio 2017. I choose the Portal because I had planned on add additional functionality.

INFO: We could download the Function App code with a Visual Studio project File. This will allow us to modify the source code, use Visual Studio Team Services (VSTS) for source control, and incorporate a CI/CD pipeline.

The Azure Function App template I used is a genericJson Webhook. Application Insights was added.

The following is the code from the ProcessEvent Function App.

using System;
using System.Net;
using Newtonsoft.Json;
using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
using Microsoft.Azure.Documents.Linq;
using System.Configuration;
using System.Collections.Generic;

public static async Task<object> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info($"Webhook was triggered!");

    var endPoint = ConfigurationManager.AppSettings["Endpoint"];
    var authKey = ConfigurationManager.AppSettings["AuthKey"];
    var databaseId = ConfigurationManager.AppSettings["databaseName"];
   
    var list = new Dictionary<string, int>();
    var newThroughput = 0;

    string jsonContent = await req.Content.ReadAsStringAsync();
    dynamic data = JsonConvert.DeserializeObject(jsonContent);

    
  

    DocumentClient client = new DocumentClient(
        new Uri(endPoint),
                authKey,
                new ConnectionPolicy
                {
                    ConnectionMode = ConnectionMode.Direct,
                    ConnectionProtocol = Protocol.Tcp
                });

    var colls = await client.ReadDocumentCollectionFeedAsync(UriFactory.CreateDatabaseUri(databaseId));
    
    foreach (var coll in colls)
    {
        Microsoft.Azure.Documents.Offer offer = client.CreateOfferQuery().Where(r => r.ResourceLink == coll.SelfLink)
                    .AsEnumerable().SingleOrDefault();

        //Get the current Throughput for the collection
        var result = ((OfferV2) offer).Content.OfferThroughput;

        // Increase the Throughput by 100
        newThroughput = result + 100;

        // Modify the offer
        offer = new OfferV2(offer, newThroughput);
        
        
        list.Add(coll.Id, result);

        await client.ReplaceOfferAsync(offer);
    }
   
    // format the response content
    var results = string.Join("; ", list.Select(x => x.Key + "=" + x.Value));
   
    var res = new HttpResponseMessage(HttpStatusCode.OK)
    {
        
        Content = new StringContent($"Collection(s): {results} Request Units increased to {newThroughput}")
    };

    return res;   
   
}

NOTE: We are not using the ‘HttpRequestMessage’ data in our Function App. It is only triggered when an Event is received.

The following is an example of an Offer

offer {{
  "id": "gCK7",
  "_rid": "gCK7",
  "_self": "offers/gCK7/",
  "_etag": ""00002b01-0000-0000-0000-5a5d0b6f0000"",
  "offerVersion": "V2",
  "resource": "dbs/jdUKAA==/colls/jdUKAIUe1QA=/",
  "offerType": "Invalid",
  "offerResourceId": "jdUKAIUe1QA=",
  "content": {
    "offerThroughput": 10600,
    "offerIsRUPerMinuteThroughputEnabled": false
  },
  "_ts": 1516047215
}}  Microsoft.Azure.Documents.Offer {Microsoft.Azure.Documents.OfferV2}

Next we need to set our Azure Cosmos DB Connection settings

We need to modify the Function App Application Setting. The Cosmos DB connection settings need to be added.
* Endpoint
* AuthKey
* databaseName

The following figure shows the these settings.

Next we will test our Function App as shown in the following figure.

Our response is Collection(s): resources=1000; leases=1000 Request Units increased to 1100

We can see that the Collections resources and leases request units have been increased to 1100.

When we run it again, the request units increase by 100.

Modify Event Rule

Finally, we need modify our event rule.

We first need to copy our Function App URL, as shown in the following figure.

2. We then edit our Event Rule by pasting the Function App URL into the Webhook text box, as shown below.

3. Save the Event Rule.

Summary

Azure Event Rules can be used to send events to a Function App
Using Azure Cosmos DB SQL Api Microsoft.Azure.Documents.Offer builtin functionality to Get and Set the OfferThroughput is easy to use.
Increased the Request Units for all Collections in a Database.
We were able to provide the client with an automated process.

Next Steps

Extend functionality by using Logic Apps
Include Unlimited Collections
Add ability to lower Throughput
Use the Event data to get the collection Id.

BizTalk Server Teach me something new about Flat Files (or not) Part II video and slides are available at Integration Monday

by Sandro Pereira | Jan 22, 2018 | BizTalk Community Blogs via Syndication

Last year I presented several sessions on Integration Monday community but I never had the chance, for several and different reasons, to proper highlight on my personal blog. The second session that I delivered last year was the second part of the topic BizTalk Server: Teach me something new about Flat Files (or not) Part II.

In that session I tried to address and solve the following questions:

How to suppress Empty Lines everywhere (end or middle)?
My positional flat-file contains data that does not match the length expected (and they are not filled with empty spaces). How can I handle that?
What if we don’t want to remove Headers but… in fact we want to deal with Headers and Trailers?
Do I need to always create a custom pipeline for dealing with Flat-Files? Or it is possible to create a Generic Pipeline?

That was the sixth session that I deliver for that community:

About my session

Session Name: BizTalk Server: Teach me something new about Flat Files (or not) Part II

Session Overview: This is the second part of Sandro Pereira’s earlier presentation on flat files on Integration Monday. You can watch the video recording of the previous session here.

Despite over the year’s new protocols, formats or patterns emerged like Web Services, WCF RESTful services, XML, JSON, among others. The use of text files (Flat Files) as CSV (Comma Separated Values) or TXT, one of the oldest common patterns for exchanging messages, still remains today one of the most used standards in systems integration and/or communication with business partners.

While tools like Excel can help us interpret such files, this type of process is always iterative and requires few user tips so that software can determine where there is a need to separate the fields/columns as well the data type of each field. But for a system integration (Enterprise Application Integration) like BizTalk Server, you must reduce any ambiguity, so that these kinds of operations can be performed thousands of times with confidence and without having recourse to a manual operator.

You can see the recording session here: BizTalk Server: Teach me something new about Flat Files (or not) Part II.

About Integration Monday

Integration Monday is full of great sessions that you can watch and I will also take this opportunity to invite you all to join us next Monday.

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community. View all posts by Sandro Pereira

Why did we built User Access Policy to Manage BizTalk Server Security?

by Saravana Kumar | Jan 22, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Being a middleware product, BizTalk Server is often going to sit right in the center of the organisation. All your critical backend systems like SAP, CRM, Oracle, SQL, and so on are connected to each other via BizTalk Server. So in theory, if someone has full access to your middleware platform like BizTalk Server, technically they can also get access to any of the underlying systems.

When it comes to security or performance, it is determined by the weakest point in the link chain. You might have created a highly protective security wall for each one of those critical backend systems, but when there is a punch whole for an external system (BizTalk Server) to get through, it’s important to protect the security of that external system to the same strength as your backend system.

We felt the current Administrative Security capabilities of BizTalk Server is very limited (will see in detail) and hence the birth of Advanced User Access Policy & BizTalk Server Security feature in BizTalk360.

What is the current limitation in BizTalk Server?

When it comes to day-to-day BizTalk Server Administration and Security, currently the scenario is like this. The Administrator will access the BizTalk Environment using the BizTalk Admin Console, depending on whether he/she belongs to either “BizTalk Server Administrators” or “BizTalk Server Operators” Window NT Group. Certain features will be enabled or disabled in BizTalk Admin Console based on these user roles.

There are few challenges in this approach. If you are a member of the BizTalk Server Administrators group, you have all the access required wide open. However, if you are a member of the BizTalk Server Operators group, your options are limited and the rules are hard coded by Microsoft in terms of what you can and cannot do.

The table below shows the actions that can and cannot be performed by members of BizTalk Operators Group

Can Do	Cannot Do
View service state and message flow	Modify the configuration for BizTalk Server.
Start or stop applications (Send Ports, Send Port Groups, Orchestrations, Receive Locations)	View message context properties classified as Personally Identifiable Information (PII) or message bodies.
Terminate/Resume Service Instances	Modify the course of message routing, such as removing or adding new subscriptions to the running system, including the ability to publish messages into the BizTalk Server runtime.

In pretty much every case, those rules are not practical for the day-to-day administrative work and pretty much within days every single person supporting/administering the BizTalk Server environment will be working with Administrative privileges.

Common challenges in BizTalk Server security

Let’s highlight some of the common day-to-day challenges that are not addressed by the out of the box BizTalk Server security mechanism.

Application Level Security: Your BizTalk Server environment will be a shared environment in many organisations. For various practical and cost reasons, you are not going to have many BizTalk Server environments. However, many business units in the organisation will be deploying their integration solutions into the platform. In such scenarios, for example Team A might need access to Applications A,B,C and Team B might need access to Applications X, Y, Z. You cannot set this level of Application isolation security using the standard BizTalk Server security mechanism.

Read Only Access: In some cases, you might want to give read-only access to your BizTalk Server environments. Example: It’s very common to allow few developers who worked on the project to have access to production environments. You wanted to make sure developers do not modify or deploy anything without change control. This is not possible with the standard BizTalk Server security mechanism.

Mixed Privilege Scenario: For practical reasons, there will be scenarios where you wanted the support person to have some level of mixed privilege. Example: They might need to start and stop the BizTalk Host Instances periodically, they may need to turn on/off tracking settings, or you may NOT want them to terminate or resume service instances etc. If you want to achieve this, there is no other option than making that person (or team) part of BizTalk Server Administrators group. This basically results in no security. As you can see, hard coded rules are not practical and it results in elevated privilege. Sadly this is the scenario with most organisations.

Security for different tools: The default security mechanism only covers the BizTalk Admin console and WMI access. However, when you are working with BizTalk Server, there are at least 5-8 other tools you’ll be using such as BAM Portal, ESB Portal, Event Viewers, Azure Services etc. Currently they all need their own security mechanisms, making it super complex and often vulnerable.

Multiple BizTalk Environments: Typically, organisations will have 2-3 BizTalk Server environments. Example: Production, DR, QA (System Integration, User Acceptance) etc. You can simply multiply the above mentioned problems by number of BizTalk Server environments you have.

How does BizTalk360 solves these problems?

From the above points discussed, it is pretty clear and obvious that a better security mechanism is required when it comes to day-to-day BizTalk Server Administration and Operations. Let’s take a quick look at how BizTalk360 addresses the challenges in a seamless way.

The below screenshot shows how you can add BizTalk Administrator/Operator to your BizTalk environment. There are few things to note — in the first place, BizTalk360 supports multiple BizTalk Server environment management from a single console; so you can set up security and access rights from a single place. You can either configure security for individuals or as team (ex: Create an NT Group called “BizTalk Production Support”)

In the second screenshot below, you can see that you do not need to give access for the administrator to complete BizTalk Server environment. You can carefully choose required BizTalk Applications for which a BizTalk Server administrator requires access. The entire BizTalk360 application is context aware, for example, if you have given permission for Application A,B,C and they depend on specific Host and Host Instance, only those related host/host instances will be visible for the user.

In the final screenshot, you can see how BizTalk360 provides full flexibility in terms of what level of permission you wanted to assign for the user. There are no hard coded rules; this makes it practical to give the correct level of permission to BizTalk Administrators/support people. BizTalk360 also comes with pre-defined security templates which you can apply as shown below. The “Can Action” section basically allows users to take some actions in the BizTalk Server environment, for example, if you do not want the user to terminate/resume service instances you can simply uncheck the check box “Service Instances”.

What is the Business Benefit?

Every business likes to run their systems in a secure way. You do not want your critical business systems like BizTalk Server to run with security challenges. This particular feature in BizTalk360 was one of the first key feature built in the product after the original web based BizTalk Admin Console functionality is completed.

Get started with a Free Trial today!

If you are not using BizTalk360, it is pretty clear that you are compromising on certain Administrative security aspects in your BizTalk Environment. So, why don’t you download and try BizTalk360 on your own environment. We provide 30 days free trial of the fully functional product.

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments. View all posts by Saravana Kumar

Microsoft Integration Weekly Update: Jan 22, 2018

by Gautam | Jan 21, 2018 | BizTalk Community Blogs via Syndication

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

Feedback

Hope this would be helpful. Please feel free to provide any feedback on the Integration weekly series.

Microsoft Integration (Azure and much more) Stencils Pack v3.0.0 for Visio

by Sandro Pereira | Jan 19, 2018 | BizTalk Community Blogs via Syndication

Microsoft Integration (Azure and much more) Stencils Pack it’s a Visio package that contains fully resizable Visio shapes (symbols/icons) that will help you to visually represent On-premise, Cloud or Hybrid Integration and Enterprise architectures scenarios (BizTalk Server, API Management, Logic Apps, Service Bus, Event Hub…), solutions diagrams and features or systems that use Microsoft Azure and related cloud and on-premises technologies in Visio 2016/2013:

BizTalk Server
Microsoft Azure
- BizTalk Services
- Azure App Service (API Apps, Web Apps, Mobile Apps and Logic Apps)
- Event Hubs
- Service Bus
- API Management, IoT, and Docker
- Machine Learning, Stream Analytics, Data Factory, Data Pipelines
- and so on
Microsoft Flow
PowerApps
Power BI
PowerShell
Infrastructure, IaaS
And many more…

I start this project because at the time I didn’t find nice shapes – graphically beautiful and resizable shapes – to produce BizTalk Server topologies diagrams and high-level overview of integrating processes. The project grew as community member asked for new shapes and during the last few years I have been updating and publishing new shapes, particularly associated with Azure services, which has a very low release cadence.

This time I cannot say it was an update because was actually a complete makeover and the reasons behind this decision are mainly these 2:

The Project Become Huge: more than 1000 shapes, and due to the project structure that I decide to implement at the time, it became a little difficult to maintain since even I had difficulty finding and organizing all the shapes and there were several duplicate shapes (some were purposely duplicated and still are).
A Fresh New Look: at the time, almost all the shapes were blue – not a beautiful blue but an obsolete annoying blue – so I decide to use, in almost the cases, a monochrome approach opting for a darker color – but after all these years it was already a little worn and needing for a new modern look and this time I decided to follow the look that Microsoft is implementing in Microsoft Docs – in fact, several stencils were collect from there – a more light and multicolored approach.
- You liked the old aspect? Do not worry, I still kept the old (monochrome) shapes but moved to support files.

What’s new in this version?

Is this version all about a fresh and modern new look? No, it is not. That was indeed one of the main tasks, but in addition:

New shapes: 571 new forms have been added – many of them are in fact a redesign of the existing features to have a modern look – but it is still an impressive number. Making a total of 1883 shapes available in this package.
The package structure changed: It is more organized – went from 13 files to 20 – which means that more categories were created and for that reason, I think it will be easier to find the shapes you are looking for. The Microsoft Integration (Azure and much more) Stencils Pack v3.0.0 is now composed of 20 files:

- Microsoft Integration Stencils v3.0.0

- MIS Additional or Support Stencils v3.0.0

- MIS Apps and Systems Logo Stencils v3.0.0

- MIS Azure Additional or Support Stencils v3.0.0

- MIS Azure Others Stencils v3.0.0

- MIS Azure Stencils v3.0.0

- MIS Buildings Stencils v3.0.0

- MIS Databases Stencils v3.0.0

- MIS Deprecated Stencils v3.0.0

- MIS Developer Stencils v3.0.0

- MIS Devices Stencils v3.0.0

- MIS Files Stencils v3.0.0

- MIS Generic Stencils v3.0.0

- MIS Infrastructure Stencils v3.0.0

- MIS Integration Patterns Stencils v3.0.0

- MIS IoT Devices Stencils v3.0.0

- MIS Power BI Stencils v3.0.0

- MIS PowerApps and Flows Stencils v3.0.0

- MIS Servers (HEX) Stencils v3.0.0

- MIS Users and Roles Stencils v3.0.0

You can download Microsoft Integration (Azure and much more) Stencils Pack from:
Microsoft Integration Stencils Pack for Visio 2016/2013 v3.0.0 (16,6 MB)
Microsoft | TechNet Gallery

Author: Sandro Pereira

Why did we built BizTalk Host Throttling Monitoring?

by Saravana Kumar | Jan 19, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

BizTalk Server being a Middleware product connected to various legacy backend systems, it needs to make sure the entire ecosystem can work in an optimal way. If one of the legacy system connected to BizTalk is slow for any reason, then BizTalk Server needs to act sensibly and not to overload that system with messages more than what it can handle. In such scenarios, BizTalk Server will throttle itself (slow down itself) and make sure the messages are delivered to the backend in an optimal rate.

BizTalk Server achieves this capability by continuously monitoring various performance counters (memory footprint, thread count, message publishing rates, database size etc.) and self tuning itself. There are over 50 performance counters related to throttling in BizTalk Server which monitors both inbound and outbound traffic.

So it’s important to understand whether your BizTalk Environment is working correctly or under throttling condition.

What are the current challenges?

BizTalk Expert Required: If you wanted to analyse and detect a throttling condition in your BizTalk Environment, an experienced BizTalk person is required. There are no out of the box tooling from Microsoft to understand throttling conditions. The person should configure all the required Perfmon counters, collect data, analyse it and predict any throttling condition.

Time Consuming: Even if you have an experience person, setting up Perfmon counters, data collection and analysis is time consuming process. In our experience, it takes anywhere from a day to a week to fine tune Throttling in BizTalk Environment.

No Monitoring: There are no out of the box monitoring solution for BizTalk Throttling conditions.

How BizTalk360 Solves the problem?

BizTalk360 addresses this problem by introducing two important features called BizTalk360 Throttling Analyser and BizTalk Host Throttling Monitor.

Throttling Analyser periodically collects all the required Performance Counters relevant to throttling and stores the values in a SQL database, so there is no necessity for setting up Perfmon as and when required. Once the data is collected, it presents the status in a very interactive graphical dashboard. The BizTalk Administrators can easily visualize the throttling condition and take appropriate action. The throttling data is kept for last 7 days, so you can login to BizTalk360 anytime and see historic throttling condition as well.

Throttling Monitoring is the new addition to the product where the Administrator can enable BizTalk Host Monitoring in almost a single click or fine tune it according their requirements. Once configured, BizTalk360 will periodically check for any throttling condition violations and alert appropriate users via various notification channels like Email, SMS, Slack, etc.

Interested to try this feature?

Download and try BizTalk360 on your own environments free for 30 days. The installation will not take more than 5-10 minutes.

Author: Saravana Kumar

Why did we build the Operations Dashboard in BizTalk360?

by Lex Hegt | Jan 18, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

As a BizTalk Administrator, it is critical to constantly have a good overview of the health of the BizTalk environments for which the BizTalk Administrator is responsible. To be able to consider the health of BizTalk Groups, it does not stop by knowing whether all ports and orchestrations are started. There are so many other aspects which are relevant to get the full picture of the health. Think of, for example, the state of the Host Instances and SQL Server jobs, the number of suspended service instances, all kind of Windows services, ESB Exceptions, EDI transactions, etc.

It would take BizTalk Administrators a lot of their precious time to constantly check the actual health status of all these different artifacts.

What are the current challenges?

One of the main challenges with the out-of-the-box tooling of BizTalk Server is that it is not possible to have the needed information available in one view. To get that overview you need to look into multiple screens and browse through multiple applications. For example, multiple screens of the BizTalk Server Administration console, SQL Server Management Studio, the portal of the ESB toolkit, Remote Desktop sessions to access the Windows server, etc.

At BizTalk360, we think this is far from efficient as it would take the BizTalk Administrators too much time just to get the overview information. In our opinion, BizTalk Administrator should spend his time on better things, than constantly going from one console to another, just to find out the current health of the system.

How BizTalk360 solves this problem?

BizTalk360 has the concept of Dashboards. These highly customizable Dashboards can show different types of data. Amongst these dashboards are:

The Operations Dashboard
The ESB Dashboard
The EDI Dashboard
The Monitoring Dashboard
The Data Monitoring Dashboard
The Analytics Dashboard

On a number of dashboards, you can add widgets from the collection of widgets that are provided by BizTalk360, or you can create your own widgets and add them to the dashboards. In certain cases, certain data can be viewed from different kinds of dashboards. For example, the EDI and ESB dashboards contain widgets which can be shown on their own dashboard, but also on the Operations Dashboard.

In most cases, the Operations Dashboard will be the first screen a BizTalk Administrator will see, after he has opened BizTalk360.

The Operations Dashboard, by default, already shows a number of widgets, but BizTalk Administrators can completely modify the layout of that dashboard to suit their needs. If the collection of widgets, which comes out of the box, is not enough, BizTalk360 allows administrators to create Custom widgets and add them to your dashboard. Besides the widgets, the Operations Dashboard can also contain pinpoints which are like shortcuts and provide you with easy access to all the different screens of the application.

Additionally, you can also create multiple dashboards if you like, to fit multiple purposes. For example, say, you can have Dashboards for particular interfaces or setting them up for different kind of roles.

Download and Try BizTalk360 Free for 30 days today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360. View all posts by Lex Hegt

Why did we build monitoring for IBM MQ, MSMQ, Azure Service Bus Queue?

by Saravana Kumar | Jan 17, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

BizTalk and Queuing Platforms

In any Enterprise integration scenario, message queuing technologies like IBM MQ, MSMQ, Azure Service Bus Queues etc., play a significant role. Message Queuing provides the solution to some critical business scenarios like the store and forward, first in first out, batch processing and so on.

Often times, queuing platform works more or less like a database — it accumulates the messages that was received from the source systems. The real power of the platform is realized when messages are pulled and utilized by the downstream systems like order processing, shipping etc.

BizTalk Server comes with native adapters for IBM MQ, MSMQ, Service Bus making it a powerful platform when your integration solution needs to communicate to Queuing endpoints.

Why do we need this feature?

Given the above scenario, the importance of a queuing platform in an Enterprise integration solution is pretty clear. In an integration scenario, it’s important to make sure both the BizTalk Server platform as well as the systems to which it is connected are all working seamlessly without any downtime. If an order receiving IBM MQ is down, then the entire integration journey of end-to-end order processing is also down. Hence it’s pretty important, you monitor those external queuing systems in addition to your BizTalk Server platform.

What are the current challenges?

No Monitoring: There is no out of the box monitoring solution that comes with BizTalk Server to monitor either BizTalk Server platform or the external sources like IBM MQ, MSMQ, Azure Service Bus Queue etc.

Custom Solutions: Often, developers write some C# or PowerShell scripts to monitor the external Queues which is hard to maintain, which will not have a good configuration experience, and will go out of sync over a period of time. Even though at the face it will look like few lines of code, real implementation will require correct type of client libraries, handling SSL certificates etc.,

Limited 3rd party solutions: There are third party options available to monitor the queues, however, they come with limited functionality. It’s not tightly integrated with your BizTalk Solution and the configuration is not seamless.

How BizTalk360 solves this problem?

The monitoring for Queuing technologies that work with BizTalk Server is built from the ground up in BizTalk360. The user experience of configuring the monitoring for Queues is seamless. If your environment uses any Queue based BizTalk Receive Port/adapter or Send Port/adapter, BizTalk360 picks up all the configuration from them. You only need to specify the conditions you are interested in monitoring.

Here is the list of supported thresholds for different Queuing technologies that comes along with BizTalk360

IBM MQ:

Current Queue Depth
Current Queue Usage %
Backout Queue Depth
Backout Queue Usage %

MSMQ:

Queue Size
Active Message Count
Journal Message Count
Dead Letter Message Count

Azure Service Bus Queue:

Queue Status
Active Message Count
Dead Letter Message Count
Transfer Message Count
Queue Size
Message Count
Transfer Dead Letter Message Count
Scheduled Message Count
Queue Size

You can combine multiple check conditions using a logical AND/OR as shown in below, this makes it extremely powerful

General Monitoring Benefits

BizTalk360 monitoring comes with a set of standard features that’s applicable to all monitoring plugins. The Queue monitoring can also take advantage of the following capabilities.

Import/Export: The import/export function allows you to move your monitoring configuration from one environment to another. Example: From your UAT/Staging environment to Production.

Notification Channel: BizTalk360 comes with a list of Notification Channels like email, SMS, slack, HP Operations Manager etc., making it powerful to notify administrators as soon as something goes wrong.

Monitoring Dashboard: BizTalk360 comes with rich monitoring dashboard that visualizes the system health in a graphical way.

Notification History: You can also keep track of all the notification history that’s been sent to the support people.

Author: Saravana Kumar

Why did we build a web based BizTalk Admin Console?

by Saravana Kumar | Jan 16, 2018 | BizTalk Community Blogs via Syndication

Back in 2010, the original idea of BizTalk360 was very simple — to create a web based version of the BizTalk Admin Console (BAC). However, over the years, the product has evolved into a full blown Operations, Monitoring and Analytics solution for Microsoft BizTalk Server; thanks to constant customer and community feedback.

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

History of BizTalk Server and Admin tools

If you look at the history of BizTalk Server, the first version was released in 2000 and then an updated version was released in 2002. Both these initial versions were very preliminary. They used Visio as the Orchestration designer, had the concept of channels, worker queues etc., and the standard MMC snap (like any other Microsoft Product).

In 2004, Microsoft released BizTalk Server 2004 — one of the major step forward in the history of BizTalk Server. The product was developed from scratch using .NET technologies since the previous two versions (BizTalk Server 2000 and BizTalk Server 2002) were built on top of COM/COM+ technology (most of the young generations now probably won’t know much about it).

BizTalk Server 2004 came with the integrated Orchestration and Map designer within Visual Studio and also introduced some major new concepts like Business Activity Monitoring, Enterprise Single Sign on, Business Activity Services, Business Rules Engine so on. It also came with a management tool called “Health and Activity Tracker (HAT)”. HAT was the only tool available for BizTalk Administrators/Support people to run the day-to-day operations of your BizTalk Environment.

In BizTalk Server 2004, there was no concept of BizTalk Applications (to group/segregate things together). You simply deploy your solutions containing schemas, maps, orchestrations etc., to the environment and soon the environment became too complex.

Two years later, Microsoft released BizTalk Server 2006. One of the major update to this edition of BizTalk Server was the new architecture when compared to BizTalk 2004. Microsoft addressed some of the shortcomings in the previous version of the product. One of the major functionality of BizTalk Server 2006 was the introduction of “BizTalk Admin Console” and the concept of Applications where you can group related artifacts like Schemas, Maps, Orchestrations, .NET dlls, batch files, Rules Policies etc., together. This version also introduced the ability to package applications into MSI’s for easy deployment.

Why did we build a web based BizTalk Admin Console?

After 2006, Microsoft constantly released new versions of BizTalk Server pretty much every 2 years once. BizTalk Server 2006 R2, BizTalk Server 2009 and BizTalk Server 2010. However, there was no major improvement that was made on the BizTalk Admin Console.

Being active in the BizTalk community, we constantly started to hear people’s frustration of both not modernizing the BizTalk Admin Console to a web based console and a lot of missing capabilities when it comes to Enterprise requirements like enhanced security, auditing, governance etc.

This is when we realized there is a clear need for an Enterprise Grade Web Based BizTalk Admin console to help and support the thousands of BizTalk Server Enterprise customers out there.

Having worked with Microsoft for nearly 2 decades, we pretty much knew how the company operates. We understood that Microsoft will not invest their valuable time in creating a web based management solution. Microsoft is always focused on creating a scalable/robust platform and it always embraces the partner ecosystem to build tooling around it. If you look at other popular products in Microsoft stack like SQL Server, IIS, Active Directory Manager, Windows Event Log etc, the default tooling will be pretty basic, none of them have a web based solution.

Birth of BizTalk360

In 2010, for all the above mentioned reasons, we took the big bet on building a complete web based BizTalk Admin console i.e the birth of BizTalk360. The below screen shot shows how BizTalk360 is simply a web based BizTalk Admin Console.

Like any software product, BizTalk360 has evolved and matured in the past 6 years. Even though the original idea was to build just a web based BizTalk Admin Console, over the years, more and more features were added to the product and today BizTalk360 has become a single operations, monitoring and analytics solution for Microsoft BizTalk Server.

As a mid sized company (around 50 people), it’s super important for us to carefully select the features we wanted to build. We cannot afford to build features that no-one wants. In the series of blog articles over the new few weeks, we’ll reveal similar stories of why we built specific features and what’s the reasoning behind each one of them.

As you can see, if you are still using only the standard BizTalk Admin Console for your day-to-day operations, then you are at least 10 years behind in terms of tooling.

Download and Try BizTalk360 Free for 30 days today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

Author: Saravana Kumar

View all posts by Saravana Kumar

« Older Entries

Next Entries »

Architecture

Steps

Throughput SLA

Create a new Azure Cosmos DB Alert

Create our Function App

The following is an example of an Offer

Next we need to set our Azure Cosmos DB Connection settings

Modify Event Rule

Summary

Next Steps

Share this:

Like this:

Related

About my session

About Integration Monday

Author: Sandro Pereira

Why do we need this feature?

What is the current limitation in BizTalk Server?

Common challenges in BizTalk Server security

How does BizTalk360 solves these problems?

What is the Business Benefit?

Get started with a Free Trial today!

Author: Saravana Kumar

Feedback

Share this:

Like this:

What’s new in this version?

Author: Sandro Pereira

Why do we need this feature?

What are the current challenges?

How BizTalk360 Solves the problem?

Interested to try this feature?

Author: Saravana Kumar

Why do we need this feature?

What are the current challenges?

How BizTalk360 solves this problem?

Download and Try BizTalk360 Free for 30 days today!

Author: Lex Hegt

BizTalk and Queuing Platforms

Why do we need this feature?

What are the current challenges?

How BizTalk360 solves this problem?

General Monitoring Benefits

Author: Saravana Kumar

History of BizTalk Server and Admin tools

Why did we build a web based BizTalk Admin Console?

Birth of BizTalk360

Download and Try BizTalk360 Free for 30 days today!

Author: Saravana Kumar

Search this Site:

Recent Posts

Recent Topics