Consolidated monitoring with the new BizTalk Group Dashboard

Consolidated monitoring with the new BizTalk Group Dashboard

Introduction

BizTalk users who use BizTalk360 can manage their BizTalk environment in a more efficient way. Monitoring and Notification is amongst the most important features in BizTalk360, The first and foremost step for monitoring is mapping the artifacts to an alarm. In which user can use this on a daily  basis to monitor their BizTalk environment and get notified if any violation occurs.

The Monitoring Dashboard is the one stop point for users to view the status of the artifacts which are mapped to an alarm. Typically a customer displays the monitoring dashboard in big screens and looks for the health of their BizTalk environment visually.  The monitoring dashboard is structured at alarm level, which lists the status of all the artifacts mapped to the respective alarm. However, you cannot see all the alarms and their mapped artifacts status in a single view, which was a usability problem persist from our side. To overcome these we are implementing the “BizTalk Group Dashboard” for the v9.0 (phase 2). With this dashboard you can view the status of all the artifacts mapped to any number of alarm in a single view and you can easily customize it using the rich filter option provided  

Let’s deep dive into BizTalk Group Dashboard and see how it solves the problem.

BizTalk Group Dashboard

The BizTalk Group Dashboard is designed to monitor all the artifacts (and view all errors) which are mapped to all the configured alarms in BizTalk360, in a single view. The BizTalk Group Dashboard can be found under the Home of Monitoring section. The “BizTalk Group Dashboard” will automatically pick up all the artifacts (which are mapped to any of the BizTalk360 alarms) and displays the status of the artifacts in a graphical manner . Also the error details of the displayed artifacts  are shown in a grid view . The BizTalk Group dashboard checks the status of the artifacts in every 60 seconds .

For better usability, BizTalk Group Dashboard splits the windows in two sections:

  1. Displays the mapped artifacts in a graph
  2. Show the Error/Warning details in an artifact level segregation

The dashboard has a draggable separator so that the user can adjust the width of the graphical window and the error/warning pane, based on their resolution. They can even minimize or maximize the error/warning pane to the leftmost section.

There is a chance of mapping the same artifact to multiple alarms. In such cases, the BizTalk Group Dashboard picks only the positive state of artifacts for monitoring. For instance, a Receive Location is mapped to an alarm1 with the expected state as Enabled and the same receive location is mapped to another alarm, say alarm2, with the expected state as Disabled; in this scenario, the BizTalk Group dashboard will only consider the positive state mapping such as expecting state as Enabled for monitoring.

Note: BizTalk group Dashboard is only designed for visual monitoring of the BizTalk environment and artifacts. It will not trigger any alert notifications.

Advantages of BizTalk Group Dashboard

Comparing to the already existing Monitoring Dashboard, the BizTalk Group dashboard has quite some advantages. Some of the main advantages of the BizTalk Group Dashboard are:

  • It consolidates the status of all the artifacts which are mapped to any alarm and there is no need to switch alarms for viewing the status of any other artifact
  • The BizTalk Group Dashboard can be split into two windows; one for the artifact graph and the other for viewing the Error/Warning pane. For usability, the split windows can be adjusted based on the monitor resolution
  • For better monitoring, the user can apply filters (at the top of the graph) based on the artifact and/or based on the event type filter option
  • The user can view the Error/Warning details along with the graph

BizTalk_Group_Dashboard

Scenarios

Many organizations use different alarm configurations and patterns for their processes. Some of the commonly used alarm configuration patterns are:

  1. Integration-based alarm configuration
  2. Role-based alarm configuration
  3. Entity-based alarm configuration

Integration-based alarm configuration means configuring BizTalk artifacts, File location, Queues, Web endpoints, etc. in an alarm, which all belong to the same integration.
Role-based alarm configuration is configuring alarms for people in different roles. For example, a platform administrator will be interested in infrastructure settings like Disk, system resources, host instances in an alarm.A Database Administrator (DBA), on the other hand, will be more interested in the SQL Server instances and the jobs which have been deployed in it.
Entity-based alarm configuration is configuring similar components in the same alarm. For instance, putting all the queues like IBM Queue, Azure service bus queue, MSMQ in one alarm.

Irrespective of the alarm configuration/pattern, now the user can customise the BizTalk group dashboard using the filter capablity .Say Administrator can monitor the infrastructure components which are mapped to all the alarm .) and queues in a single dashboard window by filtering only the Infrastructure components.

Conclusion

With this feature, users are able to visually monitor all the artifacts and view error/warning details in a single dashboard with a much more comprehensive view. If you have any feedback or suggestions, please write to us at support@biztalk360.com. You can get started with monitoring your BizTalk environment via the BizTalk Group Dashboard by downloading the 30-day free trial of BizTalk360.

The post Consolidated monitoring with the new BizTalk Group Dashboard appeared first on BizTalk360.

Comparing BizTalk360 to DynaTrace

Comparing BizTalk360 to DynaTrace

Introduction

From time to time we hear that organisations are using DynaTrace for operating and monitoring their BizTalk environments. When we talk to such organisations, they ask us about the benefit of using BizTalk360 over Dynatrace for administering their BizTalk environments. In this blog post we try to give a fair comparison between both products.

BizTalk360 and Dynatrace in short

BizTalk360 is a comprehensive product for operations, monitoring and application performance management (APM) of BizTalk Server environments. Unlike other APM products, BizTalk360 is an agent less, non-intrusive software which requires zero configuration on BizTalk servers. This makes it an easy to use and operations-friendly tool. Customers have a lot of influence in the roadmap and the vendor is very receptive to feedback via http://feedback.biztalk360.com. Furthermore, Support is very responsive. This means the product is always up to date to support the latest releases of BizTalk Server.

BizTalk360 is a market leader in BizTalk Server space, has a large customer base and healthy partner ecosystem. The company understands that supporting BizTalk Server applications for business continuity and efficient operations requires a tool which can go beyond being a traditional application performance management tool. Therefore, the product provides BizTalk specific capabilities such as managing BizTalk Applications, proactive monitoring of throttling conditions, role-based user access policies, audit logs, data monitoring etc. In some instances, BizTalk360 has helped customers to meet SOX compliance.

DynaTrace, on the other hand, is a typical Application Performance Management tool with generic capabilities to monitor operating systems, services, network protocols, system metrics, network infrastructure, services, applications etc. There is no out of the box DynaTrace plugin or agent for BizTalk Server. Either the customer needs to create one of their own or use an open source plugin Fastpack. This plugin must be installed on all the BizTalk Servers and some configuration changes must be made to the BizTalk host instance’s configuration file (btsntsvc.exe.config). With an agent it brings the ability to analyse and monitor the performance counters for BizTalk host instances, databases, orchestrations, pipelines etc. The Fastpack plugin seems to have been updated a couple of years back and might not be up to date with the latest BizTalk releases; this is a typical challenge with using open source plugins from individual contributors.

Typical requirements

In this section we will compare the 2 products, BizTalk360 and DynaTrace, for capabilities in achieving BizTalk Server monitoring requirements.

Getting a proactive alarm to warn before a problem occurs

In the BizTalk environment problems can occur in the following situations:

  • Important artefacts such as receive locations, send ports, orchestrations, host instances etc get disabled or stopped, either due to errors or due to unintentionally performed tasks. This leads to outages and stops the flow of mission critical messages.
  • If the SQL Server jobs are stopped, the size of databases increases over time and impacts the messaging performance.
  • Due to several reasons BizTalk can start throttling messages leading to an impact on performance.
  • If suspended messages are not cleaned up, the size of the message box increases resulting in performance issues.

BizTalk360 has features to monitor state-based artefacts, which ensures that all BizTalk application artefacts such as receive locations and orchestrations are up and running and generate notifications in case of aberrations. State based monitoring also ensures that essential SQL jobs and Windows services are running as expected. Features like suspended queue monitoring and throttling monitoring ensure that the BizTalk Server is running in a healthy state.

DynaTrace has no out-of-the box support for BizTalk Server. The Open Source plug-in does not have features such as state-based monitoring, throttling monitoring and suspended queue monitoring. However, it does have monitoring capabilities based on some BizTalk and SQL performance counters. In conclusion, it lags behind BizTalk360 in proactively warning of potential issues.

Problem-solving actions

In some situations, issues do occur and the operations team needs easy to use tooling to correct the faults. For example, if receive locations get disabled, they will have to be re-enabled. If messages get suspended, they will have to be terminated or resumed. If we have a software which auto-corrects these faults and automates the operational tasks then it is a big bonus as it brings the down time to its minimum levels and ensures operational efficiency.

BizTalk360 has a powerful feature called Auto Healing, which allows the users to define the expected state of an artefact and in case there is an aberration, arestores the expected state without any human intervention. With another feature Operational Automation, the day to day monotonous tasks such as resuming/terminating suspended messages can be automated.

DynaTrace on the other hand has no operational capabilities. Auto-healing and Operational Automation are not features you can expect from a typical Application Performance Management system. Hence, DynaTrace will not be the right tool to be used for BizTalk related problem solving actions.

Access to the BizTalk servers

BizTalk Server has a very limited number of user groups. The BizTalk Administrators group has the highest level of privileges and the BizTalk Application Operators group has highly restricted access. There are no access levels to give intermediate level access to BizTalk applications and artefacts. Many customers have ended up providing their users with the unrestricted access to production boxes. It is typical of organisations to provide all the users the access to production servers for business continuity.

One of the widely used features of BizTalk360 is Granular User Access Control. With this feature, user access can be controlled at a very granular level. Users do not need to RDP to production boxes to support BizTalk applications. BizTalk360’s web portal allows them access to all the features without the need for them to log on to production boxes unless they are doing deployments.

Again, access control is the feature which can be expected from an Application Performance Management tool. Hence, DynaTrace clearly has no capabilities which to help to restrict access to BizTalk Servers.

One tool for operations, monitoring and analytics

If a support professional needs to use multiple tools daily, operational efficiency takes a big hit due to all the context switching he must do between the tools. This is one of the main challenges in supporting BizTalk Server applications. Every operations team needs a single tool to perform operations, monitoring and analytics.

BizTalk360 brings all the capabilities of operations, monitoring and analytics into a single web portal. Apart from this, it allows the users to access BAM data, ESB data, EDI reports from the same web portal. This means users do not have to switch to different tools.

DynaTrace being an Application Performance tool, does have the ability to monitor performance counters and detect some anomalies. However, the user will be required to switch to different tools such as admin console, event log, ESB portal, BAM portal etc. to perform various other tasks. In a true sense, it adds another tool to the list.

Tracking

Depending on the performance needs, organisations enable or disable BizTalk tracking. Some organisations enable only event tracking and disable message body and property tracking. Hence each organisation has different tracking needs. Regardless, it is important to have good control to manage the tracking options at various artefacts.

BizTalk360 has a feature called Tracking Manager, which brings great control to manage tracking options at various artefacts. This helps customers to ensure they are following the tracking standards set at the organisation. BizTalk360 also has a feature called Graphical Flow, which is an analytics feature providing a visualisation of message flow across various artefacts such as receive ports, orchestrations and send ports. Enabling pipeline tracking is a minimum requirement for the feature. Not many of our customers use it due to performance requirements. This is the only feature which is dependent on tracking. Most of the earlier discussed features of BizTalk360 are not dependent on the tracking.

DynaTrace, like BizTalk360, will not be able to provide analytics related to ports and orchestrations if the tracking is not enabled. It surely is not a tracking management tool either.

Summary

Comparing the capabilities of BizTalk360 and Dynatrace against the day-to-day requirements necessary to work with BizTalk Server, BizTalk360 comes out as a clear winner. Dynatrace is a generic APM tool and will not match most of the capabilities of BizTalk360. The features such as State-Based Monitoring, Auto Healing, Operational Automation, Throttling Monitoring etc. are essential to bring the operational efficiency and ensure the health of BizTalk environments and applications.

Why not give BizTalk360 a try! A trial version of BizTalk360 can be requested here.

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360. View all posts by Lex Hegt

Why did we build Azure API Apps Monitoring in BizTalk360?

Why did we build Azure API Apps Monitoring in BizTalk360?

Azure API Apps Monitoring BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Azure API Apps is a modern way of building web services hosted in Microsoft Azure. Out of the box it provides capabilities such as security, load balancing, auto-scaling, and automated management. You can also take advantage of its DevOps capabilities such as continuous deployment from VSTS, GitHub, Docker Hub, and other sources, package management, staging environments, custom domain, and SSL certificates.

In the context of Azure Logic Apps connectors, Azure API Apps plays a vital role. Sometimes there may not be an out of the box connector to an external system and you may need to wrap a web service on top of that external system to create a custom Azure Logic Apps connector. This article explains how you can easily Create a custom connector for a web API.

The Azure API App can contain multiple endpoints to be able to perform multiple actions like creating an order or requesting the status of an order.

The Azure API Apps are powerful building blocks for your hybrid integration scenarios. As these API Apps can be of critical importance, you should be well aware of the status of these API Apps.

What are the current challenges?

We have already seen why Azure API Apps are important building blocks and can be of vital importance for your integration scenarios. So, whether you are consuming the external Azure API Apps from other parties or you are exposing your own API Apps, depending on the importance of the Azure API App, it is equally important to constantly be aware of the state of the Azure API Apps and their endpoints.

The main challenge at the moment is the management and monitoring of Azure API Apps in isolation via the Azure Portal. You’ll miss the end to end context of the importance of the API App in your integration scenario.

For example, you might have an order processing system that uses few BizTalk Receive Locations, Orchestrations, Send Ports and few Azure API Apps connected via BizTalk WCF-REST/HTTP adapters and are part of the same end-to-end solution. If the Azure API App is down or returning errors, then your end-to-end purchase order solution is broken. It’s important to monitor the API Apps and understand the end to end context.

We address this challenge in BizTalk360 by providing a unified platform to monitor and manage both BizTalk artifacts and some of the key Azure artifacts which are important from the integration point of view.

How BizTalk360 solves this problem?

Depending on whether you want to monitor your own Azure API Apps or the API Apps which are exposed by others, BizTalk360 offers multiple methods to setup monitoring. This is due to the fact that to be able to monitor whether API Apps are enabled, you need access to the Azure subscription which contains these API Apps. In case of API Apps which are exposed by others, you won’t have access to that Azure subscription, so you’ll only be able to monitor the HTTP Endpoints of the API Apps.

To monitor API Apps from within BizTalk360, the product offers the following features:

  • Azure API App of others – Web Endpoint monitoring
  • Your own Azure API Apps – Azure API App monitoring

Web Endpoint Monitoring

If you are simply consuming the Azure API Apps from external parties, then they are not any different from a standard REST/HTTP web services. In this case, the BizTalk360 web endpoints monitoring solution will be sufficient.  The features and capabilities of BizTalk360 Web Endpoint monitoring are described in the Why did we build Web Endpoints Monitoring in BizTalk360? article.

Azure API Apps Monitoring

If you are owning the Azure API Apps, then BizTalk360 provides a rich set of capabilities to monitor the Azure API Apps like auto-discovery of multiple endpoints and granular control on monitoring setup.

Monitoring API Apps from BizTalk360 consists of the following types of monitoring:

  • State Monitoring – monitor whether the API Apps are Enabled/Disabled
  • Endpoint Monitoring – monitor if the HTTP Endpoints of the API App are accessible

State Monitoring API Apps
From BizTalk360, you can monitor the state of API Apps. An API App can have the following 2 states:

  • Enabled – the API App is active and can receive requests
  • Disabled – the API App is disabled and cannot receive requests

With BizTalk360, you can monitor for both states and of course, we want to provide a monitoring experience which as seamless as possible. After you have selected the correct alarm and Azure subscription, it is just a matter of selecting the Azure API Apps and select the required expected state from the drop-down box. Once that’s done, BizTalk360 immediately starts to monitor the API App(s).

Azure API Apps Monitoring in BizTalk360Endpoint Monitoring
Azure API Apps basically are REST services which use JSON for data exchange. These REST services are the actual endpoints of the API Apps, which can be called by other systems. Each Azure API App has at least one endpoint but can contain many more endpoints. 

BizTalk360 has a quite extended feature set for Web Endpoint Monitoring, and it comes with advanced capabilities like adding custom GET parameters and HTTP headers:

  • Custom GET parameters – provide required/optional parameters
  • HTTP headers – provide additional HTTP headers (for example for authorization)

When the Endpoint sends a response, you can perform multiple checks to find out whether the Endpoint is in good health. You can check, for example:

  • HTTP Return codes – like HTTP 200 must exist, else throw warning/error
  • Existing and Non Existing keywords – like ‘success’ must exist in the response message
  • Response times – if it takes longer than, say, 5 seconds before the response is received, something might be wrong

HTTP Return codes
You can add multiple monitoring rules for HTTP return codes. This allows you to check for the existence, but also the non-existence of certain HTTP return codes. Besides that, you can also determine whether a Warning or an Error will have to be thrown in case the monitoring rules have not been met.

Existing and Non-Existing keywords
This is a very strong capability, as it allows you to check whether certain keywords exist or do not exist. As the response message might be in different formats, BizTalk360 allows you to check for the following formats and throw a Warning/Error based on the outcome of the monitoring rule:

  • Plain Text – search for keywords which should/should not exist
  • XML – enter an XPath query and the expected value which should/should not exist
  • JSON – enter an XPath query and the expected value which should/should not exist

For each format, you can search for just one keyword or multiple keywords to (not) exist.

Response times
Bad response times of endpoints might indicate that the endpoints (or the system/service behind it) are under stress. So monitoring on response times helps you to be aware of such situations.

As all of the above mentioned checks can be combined with each other, this provides very strong monitoring capabilities.

BizTalk360 API Apps Monitoring Response Alert

Multiple Azure Subscriptions
You might have multiple Azure subscriptions which contain different Azure API Apps. You can register all the relevant subscription and attach them to corresponding BizTalk Server environments within BizTalk360. This provides a seamless management/monitoring experience.

Conclusion

With Azure API Apps monitoring, we make it easy to constantly be aware of the well-being of your API Apps, without having to check that yourself manually. Besides monitoring whether the Azure API App is Enabled, we bring strong features for monitoring the actual HTTP endpoints.

You can read more about Azure API App monitoring and the other Azure services for which BizTalk360 provides support.

Get started with a Free Trial today!

If you want to monitor your Azure API Apps or have other BizTalk Monitoring related challenges, why not give BizTalk360 a try! It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments. Get started with the free 30 days trial.

BizTalk360-Free-Trial

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360. View all posts by Lex Hegt

Will Artificial Intelligence and Machine Learning help BizTalk Server Monitoring?

Will Artificial Intelligence and Machine Learning help BizTalk Server Monitoring?

Artificial Intelligence (AI), Machine Learning (ML), Big Data, and BlockChain are some of the buzz words we hear more and more in recent times.

As a technology enthusiast, I’m personally fascinated by some of the technology advances that are happening in these areas and it’s mind-blowing to see the pace at which they are advancing. At the same time, it becomes almost a commodity these days and is available for everyone to take advantage of it with a fraction of the cost — thanks to all the public cloud providers like Microsoft, Amazon, Google, and IBM.

Can AI & ML be used for BizTalk Server Monitoring?

These days customers expect every software product to have some level of AI & ML. In this article, let’s take a closer look at whether AI & ML can be used for BizTalk Server Monitoring and what are the practical challenges.

About a year ago, we looked into what AI & ML can do for BizTalk Server Monitoring in BizTalk360. I was not completely convinced that we can magically solve the BizTalk Server Monitoring problems using machine learning alone at that stage. Here are some of the scenarios that strike me and the answer to these challenges are not straightforward.

I don’t want to sound too negative about using AI and ML for the monitoring solution. There are certain use cases where AI and ML based solution will be far superior to manual solutions (check out the last section of this article). However, I just wanted to highlight some real-world challenges from my 15+ years of experience working with BizTalk Server.

What is Machine Learning?

Let’s first understand the basics. Machine learning is a class of algorithms that can learn from and make predictions on data. Generally speaking, the more data you have, the better the outcome for machine learning techniques. Machine learning doesn’t require users to set explicit rules like “if this, then that.” It will make that determination on its own, based on the data and algorithms. There are three things here, the data, compute power and types of algorithms (models) which helps you to predict the outcome.

Data is the King

The quality of data you have determines the quality of judgment in machine learning. So typically, you’ll start collecting all the available data metrics over a period of time to plot a predictive graph. You will start having a baseline (higher and lower bound for a period) as acceptable values for each measurable metric.

If there is any drastic change in the underlying conditions, then the whole algorithm needs to relearn.

Is time factor vs. message volume a business reality?

Let’s start with an example. In BizTalk (or Middleware world), the volume of messages passing through the system is crucial and one of the important things to monitor. We encounter the following scenario quite often. If you are using an AI based system, the period marked in red (in the below image) will be triggered as a violation and you’ll get alerted since it’s deviating from the baseline significantly.

The system basically works on the intelligence of the data it collected over the past weeks/months and blindly assumes that the volume of messages is getting deviated from the baseline volume.

In theory, this looks fantastic. However, in reality, this could be a big problem in a middleware solution. Situations like this where the volume of messages are either higher than normal or lower than normal are common on a middleware platform and we call this a “floodgate or drain scenario“.

We can probably keep training the system saying this is a known volume, but how often can you do it. So, you’ll get the false alert first, the admin person needs to teach the system that it is the expected volume, the system understands it next time. However, what happens if the floodgate happens in another time instance, again the system admin needs to correct it.

In case your business has unpredictable burst scenarios frequently, the AI system will struggle to cope.  It’s one of those things, like a demo scenario where everything runs smoothly, the system performs fantastic, but the real world is far from the demo scenario.

How about changes in hardware configuration?

I mentioned in the beginning “data is the king” in an AI based system. It’s like teaching a child — the child keeps learning every day based on what they see, what they hear, what they experience and so on. In a similar way, the AI system (if implemented well) will keep learning all the new patterns that emerge in the system, keep learning and tune itself. But, there is a challenge when the underlying platform changes dramatically then all the old learning might be completely invalid.

Let’s take an example! The production BizTalk Environment was set up and tuned for six months or so, there were a lot of manual override corrections for AI learning and everything is working fine. Suddenly, there is a business requirement saying they are expecting a huge increase in the volume of transactions and they wanted to increase the hardware capacity. This could simply be adding more servers to the BizTalk group (BizTalk or SQL Servers), increasing the processor capacity like higher performing CPU’s, more memory on BizTalk or SQL servers, changing the SAN disks and so on.

Now all the learning and tuning you have done for the last 6 months is completely wasted and the system has to be re-trained for a period of another 6 months to get to the same level with all human intensive manual tuning.

No one can simply judge what level of performance the new system configuration is going to provide; it can only be learned gradually.

How about BizTalk Server upgrade scenarios?

Another common and unavoidable scenario is the BizTalk Server upgrade. Example: Moving from BizTalk Server 2013 or 2013 R2 to BizTalk Server 2016. In fact, this is something that will be forced on you, since the mainstream support for every version will elapse after 5 years (10 years extended support).

Let’s say you are using BizTalk Server 2013 and depending on an AI powered monitoring solution. You are using the platform for 4 years and then decided to migrate to BizTalk Server 2016. In most of the cases, you will not just migrate your BizTalk Server; you’ll also migrate the OS and database from Windows Server from 2012 to 2016, SQL Server from 2012 to 2016.

This will change the complete underlying platform and make all of your AI learning so far for 4+ years completely invalid and you need to start from scratch. Can you afford it? By the time the team who had put together and tuned the solution in the first place might have left the company.

How about Christmas and Black Friday?

Pretty much every business has a special seasonal period in a year where things go upside down. In retail industries, it’s Black Friday and Christmas period and in Financial services industries, it’s typically month-end or year-end closing dates. Typically, most of these high-end businesses will have a system frozen/lock-down period where no changes are allowed.

When I used to work for a financial services client, the whole of March is locked down for 5th April, the year-end deadline in the UK. Almost 50-60% of the personal investments will be made during this period since everyone wants to take advantage of their personal tax allowances and customers typically leave it until the last minute.

How will an AI based monitoring system work under these circumstances? I can see there are only 2 choices — one, you educate the system saying it’s a special period, for this, it must have seen the scenarios earlier i.e at least one or two years period. In the first year it would have raised a false alarm all over the place, someone might have tuned it saying this is the expected behavior and in the second year, it will validate the condition and finally you might have a system that understands this unusual period and load provided nothing changed (like our hardware change) during this period.

The second option is simple — you simply turn off the monitoring for this period to avoid getting flooded with alarms.

How about manual intervention and performance tuning?

A platform like BizTalk Environment is never a “set once and run it forever” platform. It will require regular manual interventions either for performance tuning or for operational reasons. Example: for performance reasons, you may be re-configuring the SQL Server database Auto-growth parameter, SAN configuration, purging and archiving settings for tracking and BAM databases and so on.

In a similar way, for operations reasons you may be periodically turning on and off certain things like BizTalk Send Ports, Receive Locations, Orchestrations, BizTalk Host instances etc for various reasons. A more concrete example, let’s say one of your external partners’ FTP server is down for maintenance. You will turn off the FTP/SFTP Send Port or Receive Location to that partner for time until the problem gets resolved.

In situations like these, what will happen to the AI based monitoring system? As I mentioned before “data is the king”, any changes to the system will have an impact on the AI interpretation of past data and it will start sensing wrong thresholds. The AI system needs to forget the past learning and relearn the new configuration (for few weeks/months), or someone needs to manually overwrite.

How about fine control on monitoring metrics?

In a platform like BizTalk Server, monitoring just the high-level metrics like CPU%, Memory, Message Volume, Event Log error counts etc may not be sufficient. Let’s take an example scenario. You are integrating a bunch of external REST/HTTP or SOAP based web services with your BizTalk Orchestration. It’s important for you to keep an eye on the health of those external web endpoints like return status codes, response times, looking for specific value in the response JSON/XML content, you might need to verify multiple conditions in the response, you may need to wait for x-seconds/minutes before marking the system as down and so on.

Let’s take one more example. You may want to watch the growth of certain tables in SQL Server. Example: “select count(*) from spool” in the BizTalk MessageBox database. This is very specific and only a human BizTalk admin will know the importance of such custom metric.

This level of fine control on certain systems/metric will be challenging to achieve using an AI based monitoring system, since AI systems work on the volume of data and predictability. It might predict that the external system is down and responding slower, but it cannot have the granularity of control required as explained here.

Where is your data?

Now comes the million-dollar question — where is my data going to reside? As I mentioned repeatedly, “data is the king” in any AI based system to deliver the promises. It needs to understand various patterns that emerge out of historical data to apply its intelligent algorithms. So where is the BizTalk Server metrics data going to reside, is it on-premise or in the cloud?

Even though it’s technically possible to collect and keep the data on-premise and use the AI system on-premise, most of the AI based systems are well suited for the cloud. Mainly because of the three important factors, requires huge data store, requires scalable compute power and finally, the algorithms will require constant tweaking and tuning.

If the data is hosted on-premise, then maintaining such a system with constantly updating algorithms become even challenging.

If the data is hosted in the cloud, it introduces regulatory challenges in industries like Healthcare, Financial Services etc. even though the AI based providers can claim only metadata information gets transmitted, how confident the customers will be.

Why Microsoft does not have an AI powered Monitoring offering?

It is surprising to see a company like Microsoft who are market leaders in machine learning and artificial intelligence not having any AI powered monitoring solutions. None of their monitoring products like Azure OMS (Operations Management Suite), Azure Log Analytics, Azure Application Insight, SCOM (System Center Operations Manager) etc., claim they can automatically detect and fix the problem using AI.

I highlighted Microsoft here, however, it’s true for other big players like Google, Amazon, and IBM (Watson) as well. They have an AI platform, but none of them focus on AI powered monitoring solution.

I’m sure it’s just a matter of time before these platforms get matured and we will see AI powered offering (there are already few bits and pieces), but it’s not going to be 100% fully automated replacement anytime soon. Still, you need that product and business knowledge to have a solid monitoring solution.

Is it all negative with AI and BizTalk Server Monitoring?

No, not at all. It’s important to embrace new technologies and move things forward. However, it’s also important to understand the limitations of such systems from a practical business point of view. There are some scenarios well suited for AI based predictive analysis and notification like identifying security vulnerabilities, unusual threat analysis like DOS attack, face & voice recognition, some element of monitoring which are static like general message volume, response times and so on.

Another area where I can see a real use case of an AI based solution is when it’s working on top of your business data. Example: volume of purchase orders you receive from various vendors over various periods. The AI system can learn about the volume and are not impacted by underlying platform changes (and keep re-learning).

In case of BizTalk Server Monitoring, I don’t see a clear answer to the challenges I highlighted above. If your platform is stable, nothing changes (like no hardware changes, no tuning, no new deployments, no burst scenarios), then AI will have a good chance of base lining your platform and detect and notify variations from baseline. But again, this is not practical for a live BizTalk Server Environment.

How does BizTalk360 solve these problems?

Being a market leader for the past 7 years in BizTalk Server Monitoring space, we consistently improve the product and add value to our customers. We invested heavily in a lot of areas which are practical and usable for enterprise BizTalk customers TODAY. We focused primarily on customer feedback (check out our BizTalk360 feedback portal) and most of our feature offering were customer driven.

I’ve written an article explaining how BizTalk360 addresses the challenges highlighted in this article to provide a robust monitoring solution for your BizTalk Server environments. You can download it here.

BizTalk360 Whitepaper

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.

View all posts by Saravana Kumar

Alarm Configuration Changes in BizTalk360 v8.4

Alarm Configuration Changes in BizTalk360 v8.4

Here comes BizTalk360 version 8.4 with a bunch of exciting new features and enhancements as usual. As per the below quotes,

Customers often know more about your products than you do. Use them as a source of inspiration and ideas for product development. – David J. Greer

we always listen to the ideas of our customers and implement them for the product development. We make sure their suggestions are addressed and update them in the feedback portal. Based on the priority of the voting by the customers, the features are added to the product in the upcoming releases.

The Monitoring capability is the key feature of BizTalk360. Setting up monitoring is a very easy and quick task with two step process, create an alarm and map the artefact to be monitored. We can configure three different types of alarms namely Threshold, Health check and Data monitoring alarms.

In BizTalk360 v8.4, there are certain enhancements done as part of the alarms. The following are the features as part of the alarm configuration.

  • Alarm renaming capability
  • Segregation of Data monitoring alarms
  • Alarm bulk status update.

In this article, we will have a look into each one of them.

Alarm renaming capability

Prior to v8.4, we cannot rename an alarm and it would be grayed when we edit an alarm. As per the feedback from many of the customers, we have included this ability in our latest release. So now you can rename the alarm as per the business requirement. This renaming applies to Data monitoring alarms also. Once the data monitor alarm is renamed, it will be reflected in the Data monitoring section as well.

With respect to the database changes, earlier, only the Alarm Name was referred in all the places and now AlarmId is being referred everywhere.

Limitation:

  • You cannot have same names for two different alarms in the same environment. The alarms can have the same names across the environments.

Segregation of Data Monitoring alarms

In BizTalk360 v7.9 the Data Monitoring functionality was introduced. This also came as a requirement from the customers. It was designed to monitor the data transfer happening in the BizTalk server and trigger alerts based on the conditions set.

In the monitoring home dashboard screen, all the alarms are displayed in a drop down. There might be a chance of unintendingly mapping an artefact to the wrong data monitoring. This confusion can be avoided if the data monitoring alarms are separated from the other alarms. Hence the data monitoring alarms are now segregated and they will be available only in the Data Monitoring section. They will no more be available in the Monitoring home dashboard section and under the Manage Mapping drop-down.

Alarm bulk status update

Under Settings -> Stop alert for maintenance, BizTalk360 has the capability to stop all the alarms from triggering emails. This was as per the feedback from the customer for which we received many votes.

From a business perspective, customers can have created many alarms and a situation might arise that they want to disable certain alarms for maintenance purposes, instead of stopping all of them at once.

In this version of BizTalk360, we can select multiple alarms and enable/disable them at once by clicking on the “Status” button. This is a newly added option for alarm status update. It might look like a small improvement done, but it will be very helpful when doing large implementations and transitions.

What happens during the migration?

Migration scenarios occur when a customer upgrades from a lower version to the latest one. Lot of testing efforts is required to ensure that all the information is retained after migrating from a lower version to higher one. We need to take care that all the data gets migrated without any problem and there is no data loss happening from each of the versions. Also, new changes must get reflected. Extensive testing efforts need to be put to see that this is taken care of and we have assured this.

When the customers migrate from the old versions to the latest one, the ability to edit the alarm name will be shown. The following picture contains some of the migration scenarios that were tested.

One important thing to be noted is the listing of data monitoring alarms. In the older versions, we can see the data monitoring alarms getting listed along with the other types of alarms. Now in the new version, they will get displayed only in the data monitoring section. However, we can edit the names of those alarms in the Manage Alarms page.

Conclusion

Apart from the alarm configuration changes, we have a lot of new features like Logic Apps Data monitoring, IBM MQ monitoring and folder monitoring. We always hear the feedback and suggestions from the customers and try hard to fulfill them to make our product better and at the same time to achieve 100% customer satisfaction.

You can write to us at support@biztalk360.com. Have a try at our latest version by downloading a 14-day free trial of BizTalk360.

Author: Praveena Jayanarayanan

I am working as Senior Support Engineer at BizTalk360. I always believe in team work leading to success because “We all cannot do everything or solve every issue. ‘It’s impossible’. However, if we each simply do our part, make our own contribution, regardless of how small we may think it is…. together it adds up and great things get accomplished.” View all posts by Praveena Jayanarayanan

Alarm Configuration Changes in BizTalk360 v8.4

Alarm Configuration Changes in BizTalk360 v8.4

Here comes BizTalk360 version 8.4 with a bunch of exciting new features and enhancements as usual. As per the below quotes,

Customers often know more about your products than you do. Use them as a source of inspiration and ideas for product development. – David J. Greer

we always listen to the ideas of our customers and implement them for the product development. We make sure their suggestions are addressed and update them in the feedback portal. Based on the priority of the voting by the customers, the features are added to the product in the upcoming releases.

The Monitoring capability is the key feature of BizTalk360. Setting up monitoring is a very easy and quick task with two step process, create an alarm and map the artefact to be monitored. We can configure three different types of alarms namely Threshold, Health check and Data monitoring alarms.

In BizTalk360 v8.4, there are certain enhancements done as part of the alarms. The following are the features as part of the alarm configuration.

  • Alarm renaming capability
  • Segregation of Data monitoring alarms
  • Alarm bulk status update.

In this article, we will have a look into each one of them.

Alarm renaming capability

Prior to v8.4, we cannot rename an alarm and it would be grayed when we edit an alarm. As per the feedback from many of the customers, we have included this ability in our latest release. So now you can rename the alarm as per the business requirement. This renaming applies to Data monitoring alarms also. Once the data monitor alarm is renamed, it will be reflected in the Data monitoring section as well.

With respect to the database changes, earlier, only the Alarm Name was referred in all the places and now AlarmId is being referred everywhere.

Limitation:

  • You cannot have same names for two different alarms in the same environment. The alarms can have the same names across the environments.

Segregation of Data Monitoring alarms

In BizTalk360 v7.9 the Data Monitoring functionality was introduced. This also came as a requirement from the customers. It was designed to monitor the data transfer happening in the BizTalk server and trigger alerts based on the conditions set.

In the monitoring home dashboard screen, all the alarms are displayed in a drop down. There might be a chance of unintendingly mapping an artefact to the wrong data monitoring. This confusion can be avoided if the data monitoring alarms are separated from the other alarms. Hence the data monitoring alarms are now segregated and they will be available only in the Data Monitoring section. They will no more be available in the Monitoring home dashboard section and under the Manage Mapping drop-down.

Alarm bulk status update

Under Settings -> Stop alert for maintenance, BizTalk360 has the capability to stop all the alarms from triggering emails. This was as per the feedback from the customer for which we received many votes.

From a business perspective, customers can have created many alarms and a situation might arise that they want to disable certain alarms for maintenance purposes, instead of stopping all of them at once.

In this version of BizTalk360, we can select multiple alarms and enable/disable them at once by clicking on the “Status” button. This is a newly added option for alarm status update. It might look like a small improvement done, but it will be very helpful when doing large implementations and transitions.

What happens during the migration?

Migration scenarios occur when a customer upgrades from a lower version to the latest one. Lot of testing efforts is required to ensure that all the information is retained after migrating from a lower version to higher one. We need to take care that all the data gets migrated without any problem and there is no data loss happening from each of the versions. Also, new changes must get reflected. Extensive testing efforts need to be put to see that this is taken care of and we have assured this.

When the customers migrate from the old versions to the latest one, the ability to edit the alarm name will be shown. The following picture contains some of the migration scenarios that were tested.

One important thing to be noted is the listing of data monitoring alarms. In the older versions, we can see the data monitoring alarms getting listed along with the other types of alarms. Now in the new version, they will get displayed only in the data monitoring section. However, we can edit the names of those alarms in the Manage Alarms page.

Conclusion

Apart from the alarm configuration changes, we have a lot of new features like Logic Apps Data monitoring, IBM MQ monitoring and folder monitoring. We always hear the feedback and suggestions from the customers and try hard to fulfill them to make our product better and at the same time to achieve 100% customer satisfaction.

You can write to us at support@biztalk360.com. Have a try at our latest version by downloading a 14-day free trial of BizTalk360.

Author: Praveena Jayanarayanan

I am working as Senior Support Engineer at BizTalk360. I always believe in team work leading to success because “We all cannot do everything or solve every issue. ‘It’s impossible’. However, if we each simply do our part, make our own contribution, regardless of how small we may think it is…. together it adds up and great things get accomplished.” View all posts by Praveena Jayanarayanan

Alarm Configuration Changes in BizTalk360 v8.4

Alarm Configuration Changes in BizTalk360 v8.4

Here comes BizTalk360 version 8.4 with a bunch of exciting new features and enhancements as usual. As per the below quotes,

Customers often know more about your products than you do. Use them as a source of inspiration and ideas for product development. – David J. Greer

we always listen to the ideas of our customers and implement them for the product development. We make sure their suggestions are addressed and update them in the feedback portal. Based on the priority of the voting by the customers, the features are added to the product in the upcoming releases.

The Monitoring capability is the key feature of BizTalk360. Setting up monitoring is a very easy and quick task with two step process, create an alarm and map the artefact to be monitored. We can configure three different types of alarms namely Threshold, Health check and Data monitoring alarms.

In BizTalk360 v8.4, there are certain enhancements done as part of the alarms. The following are the features as part of the alarm configuration.

  • Alarm renaming capability
  • Segregation of Data monitoring alarms
  • Alarm bulk status update.

In this article, we will have a look into each one of them.

Alarm renaming capability

Prior to v8.4, we cannot rename an alarm and it would be grayed when we edit an alarm. As per the feedback from many of the customers, we have included this ability in our latest release. So now you can rename the alarm as per the business requirement. This renaming applies to Data monitoring alarms also. Once the data monitor alarm is renamed, it will be reflected in the Data monitoring section as well.

With respect to the database changes, earlier, only the Alarm Name was referred in all the places and now AlarmId is being referred everywhere.

Limitation:

  • You cannot have same names for two different alarms in the same environment. The alarms can have the same names across the environments.

Segregation of Data Monitoring alarms

In BizTalk360 v7.9 the Data Monitoring functionality was introduced. This also came as a requirement from the customers. It was designed to monitor the data transfer happening in the BizTalk server and trigger alerts based on the conditions set.

In the monitoring home dashboard screen, all the alarms are displayed in a drop down. There might be a chance of unintendingly mapping an artefact to the wrong data monitoring. This confusion can be avoided if the data monitoring alarms are separated from the other alarms. Hence the data monitoring alarms are now segregated and they will be available only in the Data Monitoring section. They will no more be available in the Monitoring home dashboard section and under the Manage Mapping drop-down.

Alarm bulk status update

Under Settings -> Stop alert for maintenance, BizTalk360 has the capability to stop all the alarms from triggering emails. This was as per the feedback from the customer for which we received many votes.

From a business perspective, customers can have created many alarms and a situation might arise that they want to disable certain alarms for maintenance purposes, instead of stopping all of them at once.

In this version of BizTalk360, we can select multiple alarms and enable/disable them at once by clicking on the “Status” button. This is a newly added option for alarm status update. It might look like a small improvement done, but it will be very helpful when doing large implementations and transitions.

What happens during the migration?

Migration scenarios occur when a customer upgrades from a lower version to the latest one. Lot of testing efforts is required to ensure that all the information is retained after migrating from a lower version to higher one. We need to take care that all the data gets migrated without any problem and there is no data loss happening from each of the versions. Also, new changes must get reflected. Extensive testing efforts need to be put to see that this is taken care of and we have assured this.

When the customers migrate from the old versions to the latest one, the ability to edit the alarm name will be shown. The following picture contains some of the migration scenarios that were tested.

One important thing to be noted is the listing of data monitoring alarms. In the older versions, we can see the data monitoring alarms getting listed along with the other types of alarms. Now in the new version, they will get displayed only in the data monitoring section. However, we can edit the names of those alarms in the Manage Alarms page.

Conclusion

Apart from the alarm configuration changes, we have a lot of new features like Logic Apps Data monitoring, IBM MQ monitoring and folder monitoring. We always hear the feedback and suggestions from the customers and try hard to fulfill them to make our product better and at the same time to achieve 100% customer satisfaction.

You can write to us at support@biztalk360.com. Have a try at our latest version by downloading a 14-day free trial of BizTalk360.

Author: Praveena Jayanarayanan

I am working as Senior Support Engineer at BizTalk360. I always believe in team work leading to success because “We all cannot do everything or solve every issue. ‘It’s impossible’. However, if we each simply do our part, make our own contribution, regardless of how small we may think it is…. together it adds up and great things get accomplished.” View all posts by Praveena Jayanarayanan

Introducing BizTalk360 Version 8.4 – BizTalk Server Licensing Widget, Data Monitoring for Azure Logic Apps, Folder Monitoring, FTP/FTPS/SFTP, IBM MQ Monitoring, BizTalk Health Monitoring Integration

Introducing BizTalk360 Version 8.4 – BizTalk Server Licensing Widget, Data Monitoring for Azure Logic Apps, Folder Monitoring, FTP/FTPS/SFTP, IBM MQ Monitoring, BizTalk Health Monitoring Integration

We are continuing our tradition of one new release every 3-4 months once continuously for the past 6 years. In every release, we wanted to make sure we add 5-6 meaningful features that will help BizTalk Server administrators with Operations, Monitoring and Analytics capabilities. In BizTalk360 version 8.4, we added some exciting new features.

  • BizTalk Server Licensing Widget
  • Data Monitoring for Azure Logic Apps
  • Folder Monitoring
  • FTP/FTPS/SFTP Monitoring
  • IBM MQ Monitoring
  • BizTalk Health Monitoring (BHM) Integration
  • Manage NT Services directly in the Web Console
  • Manage SQL Jobs directly in the Web Console
  • Few enhancements/bug fixes

BizTalk Server Licensing Widget

From our experience dealing with 100’s of BizTalk Server customers, we noticed that a lot of them struggle to understand how many BizTalk Server licenses are required for their servers. About 4 years ago I wrote an article called Understanding BizTalk Server Licensing and that’s one of the popular articles on our blog. We just thought it will be a good idea to transform that knowledge into a small widget in BizTalk360 so that people will easily understand how many licenses are required for your BizTalk Server.

Out of the box we ship this new widget in BizTalk360 version 8.4 called “BizTalk Server License” as shown below. This widget will display key information like BizTalk edition, server type, processor type, number of cores, processor manufacturer, retail license cost per server, how many servers to be licensed and total cost for the environment.

Data Monitoring for Azure Logic Apps

Data monitoring in BizTalk360 is one of the key capabilities that allows you to monitor and trigger alerts based on historical events. Some of the common use cases of data monitoring include “No Event Alerting”. Ex: If you are expected to process 5 purchase orders from an FTP location every hour and you haven’t received the expected volume, then BizTalk360 can alert you with a message.

We are taking the exact same concept to Azure Logic Apps to alert you based on historical transactions within a time window.

There are various interesting use cases. For example, as shown above, you may want to get alerted if one of your Logic App has gone crazy and firing thousands of executions and costing so many $$$. We will cover in detail about this feature in a separate blog article.

Here is the detailed list of metrics you can set data monitoring on.

Folder Monitoring

Sometimes as the product matures and you look at adding more features, you tend to miss out on the basic things. Folder monitoring is one such thing we left for so long! It’s better late than never. In BizTalk360 version 8.4 we are bringing the capability of folder monitoring. The technology might have improved significantly like Micro Services, REST API’s etc, however, file based integration is there to stay. It’s very common in the integration world where you drop a file (purchase order, batch file, EDI transactions etc) and the integration kicks in. One of the common challenges in such integration is that what happens when the integration is broken and the files started to pile up in the pickup folder.

We solve the exact problem using the folder monitoring capability.

One of the core values of BizTalk360 is to make monitoring configuration seamless and that’s how we differentiate ourselves from general purpose monitoring products. In the folder monitoring case, we automatically list down all the receive locations and send ports that use BizTalk FILE adapter and allow the user to configure in pretty much 2-3 clicks, we pick up all the values like folder location from existing configuration as shown above.

FTP/FTPS/SFTP

On the similar lines of normal Folder monitoring, FTP based integrations are key in most of the enterprise integration scenarios. BizTalk Server comes with three different adapters FTP, FTPs and SFTP to tackle FTP scenarios where the variations are mainly around the security capabilities of the FTP server.

With BizTalk360 version 8.4, you can monitor FTP locations for data pile up. We put a lot of efforts to make the configuration experience as seamless as possible, the screens will automatically list all the FTP based send ports/receive locations and all the available values like location, username etc are automatically picked up from those configurations.

IBM MQ Monitoring

For the past few releases, we are slowly bringing in the capability of Queue monitoring into BizTalk360. Queues play a vital role in enterprise integration especially for robustness, store, and forward patterns. In the previous releases, we introduced MSMQ and Azure Service Bus Queue. In BizTalk360 version 8.4, we are bringing in support for IBM MQ. We support both MQSC and MQS based configurations.

In any IBM MQ queues, you can monitor for the following 4 parameters – queue depth, backout queue depth, queue usage % and backout queue usage %.

BizTalk Health Monitoring Integration

For many years, we had support for Message Box Viewer inside BizTalk360. We periodically run MBV in configured environments, parse and store the result and display it in the BizTalk360 web console. We also got the ability to monitor and alert users based on Message Box Viewer raised errors and warning. Two years ago, Microsoft made some major changes to Message Box Viewer, re-branded it as BizTalk Health Monitor with few additional functionalities and deprecated MBV.

Support for BHM is one of the top requested features on our feedback portal. In BizTalk360 version 8.4 we introduced support for BHM and deprecated MBV support.

NT Services Operation in BizTalk and SQL Servers

One of the main objective for us from the security perspective is to stop people logging on/off into production BizTalk and SQL servers during business hours. We also wanted to audit any activities performed by BizTalk support people. Even though you shouldn’t start/stop services in your environment, we noticed in some cases NT services like world wide web, enterprise single sign on, BizTalk host instances, SQL Agent, etc needs to be started/stopped for the variety of reasons. Currently, users will RDP or have remote MMC snap-in to manage NT services in BizTalk and SQL servers. With BizTalk360 version 8.4 you can manage them directly in the web console. In addition, the activities will be audited by BizTalk360.

Manage SQL Jobs Operation from web console

In a similar concept to managing NT Services in both BizTalk and SQL Servers, SQL jobs play a vital role in a BizTalk Server environment. SQL Jobs are responsible for keeping your BizTalk environment healthy. They take care of routine housekeeping activities like moving data from Message Box database to Tracking database, purging/archiving tracked data, backup/disaster log shipping etc.

In BizTalk360 version 8.4, we brought in capabilities to manage SQL jobs directly from the web console. This functionality of monitoring SQL Jobs has been in the product for a very long time.

Few Enhancements and Bug Fixes

This article mainly covers all the new and exciting features we shipped in BizTalk360 version 8.4. In every release, we also allocate time to enhance current features and to address any top priority bugs we received from our existing customers.  Here are some of the key enhancements and bug fixes

Alarm Management Improvements: Now you have the ability to bulk enable/disable monitoring alarms. You can also rename the alarms (again, one of the most requested features on our feedback portal.)

Webhook Notification Channel: Now you have the ability to specify (override) a new endpoint URL at the alarm level.

Logic Apps Monitoring Performance Improvement: Few critical updates been made to improve the performance of monitoring Azure Logic Apps

Get started today

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments. View all posts by Saravana Kumar

Introducing BizTalk360 Version 8.4 – BizTalk Server Licensing Widget, Data Monitoring for Azure Logic Apps, Folder Monitoring, FTP/FTPS/SFTP, IBM MQ Monitoring, BizTalk Health Monitoring Integration

Introducing BizTalk360 Version 8.4 – BizTalk Server Licensing Widget, Data Monitoring for Azure Logic Apps, Folder Monitoring, FTP/FTPS/SFTP, IBM MQ Monitoring, BizTalk Health Monitoring Integration

We are continuing our tradition of one new release every 3-4 months once continuously for the past 6 years. In every release, we wanted to make sure we add 5-6 meaningful features that will help BizTalk Server administrators with Operations, Monitoring and Analytics capabilities. In BizTalk360 version 8.4, we added some exciting new features.

  • BizTalk Server Licensing Widget
  • Data Monitoring for Azure Logic Apps
  • Folder Monitoring
  • FTP/FTPS/SFTP Monitoring
  • IBM MQ Monitoring
  • BizTalk Health Monitoring (BHM) Integration
  • Manage NT Services directly in the Web Console
  • Manage SQL Jobs directly in the Web Console
  • Few enhancements/bug fixes

BizTalk Server Licensing Widget

From our experience dealing with 100’s of BizTalk Server customers, we noticed that a lot of them struggle to understand how many BizTalk Server licenses are required for their servers. About 4 years ago I wrote an article called Understanding BizTalk Server Licensing and that’s one of the popular articles on our blog. We just thought it will be a good idea to transform that knowledge into a small widget in BizTalk360 so that people will easily understand how many licenses are required for your BizTalk Server.

Out of the box we ship this new widget in BizTalk360 version 8.4 called “BizTalk Server License” as shown below. This widget will display key information like BizTalk edition, server type, processor type, number of cores, processor manufacturer, retail license cost per server, how many servers to be licensed and total cost for the environment.

Data Monitoring for Azure Logic Apps

Data monitoring in BizTalk360 is one of the key capabilities that allows you to monitor and trigger alerts based on historical events. Some of the common use cases of data monitoring include “No Event Alerting”. Ex: If you are expected to process 5 purchase orders from an FTP location every hour and you haven’t received the expected volume, then BizTalk360 can alert you with a message.

We are taking the exact same concept to Azure Logic Apps to alert you based on historical transactions within a time window.

There are various interesting use cases. For example, as shown above, you may want to get alerted if one of your Logic App has gone crazy and firing thousands of executions and costing so many $$$. We will cover in detail about this feature in a separate blog article.

Here is the detailed list of metrics you can set data monitoring on.

Folder Monitoring

Sometimes as the product matures and you look at adding more features, you tend to miss out on the basic things. Folder monitoring is one such thing we left for so long! It’s better late than never. In BizTalk360 version 8.4 we are bringing the capability of folder monitoring. The technology might have improved significantly like Micro Services, REST API’s etc, however, file based integration is there to stay. It’s very common in the integration world where you drop a file (purchase order, batch file, EDI transactions etc) and the integration kicks in. One of the common challenges in such integration is that what happens when the integration is broken and the files started to pile up in the pickup folder.

We solve the exact problem using the folder monitoring capability.

One of the core values of BizTalk360 is to make monitoring configuration seamless and that’s how we differentiate ourselves from general purpose monitoring products. In the folder monitoring case, we automatically list down all the receive locations and send ports that use BizTalk FILE adapter and allow the user to configure in pretty much 2-3 clicks, we pick up all the values like folder location from existing configuration as shown above.

FTP/FTPS/SFTP

On the similar lines of normal Folder monitoring, FTP based integrations are key in most of the enterprise integration scenarios. BizTalk Server comes with three different adapters FTP, FTPs and SFTP to tackle FTP scenarios where the variations are mainly around the security capabilities of the FTP server.

With BizTalk360 version 8.4, you can monitor FTP locations for data pile up. We put a lot of efforts to make the configuration experience as seamless as possible, the screens will automatically list all the FTP based send ports/receive locations and all the available values like location, username etc are automatically picked up from those configurations.

IBM MQ Monitoring

For the past few releases, we are slowly bringing in the capability of Queue monitoring into BizTalk360. Queues play a vital role in enterprise integration especially for robustness, store, and forward patterns. In the previous releases, we introduced MSMQ and Azure Service Bus Queue. In BizTalk360 version 8.4, we are bringing in support for IBM MQ. We support both MQSC and MQS based configurations.

In any IBM MQ queues, you can monitor for the following 4 parameters – queue depth, backout queue depth, queue usage % and backout queue usage %.

BizTalk Health Monitoring Integration

For many years, we had support for Message Box Viewer inside BizTalk360. We periodically run MBV in configured environments, parse and store the result and display it in the BizTalk360 web console. We also got the ability to monitor and alert users based on Message Box Viewer raised errors and warning. Two years ago, Microsoft made some major changes to Message Box Viewer, re-branded it as BizTalk Health Monitor with few additional functionalities and deprecated MBV.

Support for BHM is one of the top requested features on our feedback portal. In BizTalk360 version 8.4 we introduced support for BHM and deprecated MBV support.

NT Services Operation in BizTalk and SQL Servers

One of the main objective for us from the security perspective is to stop people logging on/off into production BizTalk and SQL servers during business hours. We also wanted to audit any activities performed by BizTalk support people. Even though you shouldn’t start/stop services in your environment, we noticed in some cases NT services like world wide web, enterprise single sign on, BizTalk host instances, SQL Agent, etc needs to be started/stopped for the variety of reasons. Currently, users will RDP or have remote MMC snap-in to manage NT services in BizTalk and SQL servers. With BizTalk360 version 8.4 you can manage them directly in the web console. In addition, the activities will be audited by BizTalk360.

Manage SQL Jobs Operation from web console

In a similar concept to managing NT Services in both BizTalk and SQL Servers, SQL jobs play a vital role in a BizTalk Server environment. SQL Jobs are responsible for keeping your BizTalk environment healthy. They take care of routine housekeeping activities like moving data from Message Box database to Tracking database, purging/archiving tracked data, backup/disaster log shipping etc.

In BizTalk360 version 8.4, we brought in capabilities to manage SQL jobs directly from the web console. This functionality of monitoring SQL Jobs has been in the product for a very long time.

Few Enhancements and Bug Fixes

This article mainly covers all the new and exciting features we shipped in BizTalk360 version 8.4. In every release, we also allocate time to enhance current features and to address any top priority bugs we received from our existing customers.  Here are some of the key enhancements and bug fixes

Alarm Management Improvements: Now you have the ability to bulk enable/disable monitoring alarms. You can also rename the alarms (again, one of the most requested features on our feedback portal.)

Webhook Notification Channel: Now you have the ability to specify (override) a new endpoint URL at the alarm level.

Logic Apps Monitoring Performance Improvement: Few critical updates been made to improve the performance of monitoring Azure Logic Apps

Get started today

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments. View all posts by Saravana Kumar

Introduction to File Locations Monitoring in BizTalk360

Introduction

File Locations Monitoring is one of the new Monitoring capabilities introduced in BizTalk360 v8.4. From our customers, we received many requests to introduce the ability to monitor Folder Level into BizTalk360. We are happy to announce that in BizTalk360’s upcoming release 8.4, we are bringing a feature to monitor file count for File Locations (File, FTP and SFTP), configured in BizTalk receive locations and send ports.

Need for File Locations Monitoring

Messaging or data exchange between business can be done in various ways. Frequent data communication process is done through for example File and FTP with XML or EDI as the message format. From BizTalk 2013 onwards, SFTP is also included in BizTalk Server which was available as Out-of-box feature in prior versions. Even though BizTalk works seamlessly with File Adapters, it has some known issues which occur due to incorrect configurations.

The BizTalk File Location Adapters (File, FTP, SFTP) fail to perform the operations on following scenarios:

  • The File receive adapter cannot access the receive location on the file system or network share because the specified path does not exist. For a network share, the File receive adapter disables the receive location after all retry attempts have been exhausted.
  • The File receive adapter cannot access the receive location on the file system or network share because the account used by the associated host instance does not have read-write permission for that location. For a network share, the File receive adapter disables the receive location after all retry attempts have been exhausted
  • Files with names longer than 256 characters are encountered in the receive location

To resolve the above issues, we need to ensure that the specified path or share exists and the account used as the Logon should have read-write access. Additional to this, if you configure schedule/service window for your receive locations, messages will be accepted only during that time window, all other times BizTalk won’t pick up messages. Any violation to this scenario also needs to be monitored.

We often experience that organizations facing these kinds of challenges used custom solutions for this kind of monitoring. To overcome this, BizTalk360 added the File Locations Monitoring capabilities out of the box.

File Locations (File, FTP, SFTP)

In BizTalk360 v8.4, we are introducing support to monitor the File, FTP and SFTP servers under File Location Monitoring Section. File Location Monitoring will list all the locations configured in the BizTalk Artifacts (Send Ports and Receive Locations) for the Transport Types (File, FTP, SFTP) respectively, which helps users to monitor all the File Locations mapped with Receive Locations/Send Ports.

File Monitoring

To get familiar with File Adapter configuration, kindly refer below link:

https://msdn.microsoft.com/en-us/library/aa559105.aspx

In BizTalk360 File Monitoring Configuration contains three sections: Basic Information, Authentication, and File Monitoring Configurations.

  • The Basic Information Section contains Folder Location and File Mask configured in BizTalk.
  • The Authentication section is Optional. By default, authentication could be processed by the BizTalk360 Monitoring Service account, when credentials are not given.
  • In the File Configurations Section, we can configure the Thresholds with the metric File Count to monitor

When a File Location is in the Orphaned State, BizTalk360 would let the users know about the cause of the failure on hovering the warning icon

FTP Monitoring:

To get familiar with FTP Adapter configuration, kindly refer to below link:

https://msdn.microsoft.com/en-us/library/aa561032.aspx

The FTP Configuration UI is categorized into three sections: FTP Details, Firewall Details and FTP Monitoring Configurations

  • The FTP Details Section contains the details about the FTP Location, Authentication, and SSL
  • The Firewall Details contains the configurations to connect FTP Server through a Firewall
  • In the FTP Monitoring Config section, we can configure the monitor with Threshold Conditions for the metric File Count

SFTP Monitoring:

To get familiar with the SFTP Adapter Configuration in BizTalk, kindly refer the below link

https://msdn.microsoft.com/en-us/library/jj684551.aspx

The SFTP Monitor Tab in BizTalk360 lists the SFTP Locations which are configured in BizTalk. It contains four sections:

  • SSH Server Section has the details about the SFTP Location
  • The Proxy Details Section is optional to connect SFTP Server behind a firewall

Note: In BizTalk, Proxy details are available from BizTalk 2013 R2

  • Security Details Section has the authentication details
  • In the SFTP Monitoring Config Section, we can configure the monitor with threshold conditions for the metric File Count

Conclusion

With this latest release 8.4, BizTalk360 brings the File Locations Monitoring with the ability to monitor the File Count. In the future, we will be adding support to monitor Folder Size and Access permissions. If you have any feedback or suggestions, please write to us at support@biztalk360.com.