Why did we build Receive Locations, Send Ports and Orchestrations Monitoring for BizTalk?

by Lex Hegt | Apr 18, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360“.

Why do we need this feature?

Microsoft BizTalk Server is a powerful middleware product which allows integrating all kind of information systems, independent of whether these systems are on-premise or in the cloud. BizTalk Server uses Receive and Send ports to communicate with a broad set of on-premise/cloud systems, both on the technology level (FILE, FTP, POP3, SMTP, etc.) as on the business level (ERP, CRM, etc.).

Besides just receiving and transmitting messages, BizTalk solutions might also contain Orchestrations, which can process all kind of business logic,  for example, Order processing, Credit card validation, Claims processing etc.

As BizTalk Server is very scalable, a BizTalk environment can contain many deployments for all kind of different integrations. All together a BizTalk environment may contain hundreds of ports and orchestrations. The well-being of these artifacts is of vital importance to have the integrations working properly. When not running properly, these interruptions may cause serious damage to the business processes they support.

What are the current challenges?

To be well aware of the state of the BizTalk artifacts, the BizTalk administrator needs to consult the BizTalk Administration console multiple times per day. However, there are few challenges when depending on the BizTalk Administration console for checking the state of these artifacts.

Access to the Administration console – Often, the BizTalk Administration console is only installed on the BizTalk servers. Besides that there is the risk of unintended damaging the BizTalk server while accessing the server, you need to be authorized to access the servers. Not to be forgotten, is the fact that it constantly takes time to setup Remote Desktop connections to access the BizTalk server and the Administration console. Time which can easily be used for something more tangible.

Not suited for less experienced administrators – The Administration console is such a powerful tool, which is great when you are an experienced BizTalk Administrator. However, in the hands of less experienced administrators, unintentionally a lot of damage can be done. Think of for example stopping ports or terminating processes!

Not an efficient way of monitoring – Checking the state of the ports and orchestrations manually, multiple times per day, is just not efficient. It can easily be forgotten in the rush of the day, thereby putting the wellbeing of your integrations at stake.

Too many artifacts to monitor efficiently – BizTalk environments can contain hundreds of ports and orchestrations. In most scenarios, the artifacts should be in the Enabled/Started state. But there can be several reasons why certain artifacts are not in the expected state. Just to mention a few:

• After the deployment of an update of a BizTalk application the artifacts are in a wrong state
• Due to a temporary maintenance window, some of the artifacts need not be in the Enabled/Started state, to prevent suspended instances
• Artifacts might be Disabled as a result of a temporary outage
• Certain artifacts need only be Enabled/Started as a fall back scenario

By the given examples, it must be clear, that it can be very challenging to constantly be aware of the desired state of all the artifacts. This makes it hard to manually monitor the state of the artifacts in an efficient way.

Instead of having to monitor BizTalk manually, it is much more efficient to have automated, state-bound, artifact monitoring. Automated BizTalk monitoring will prevent you of constantly having to check for the state manually, enabling you to spend your time on more important activities. In the next section, we’ll show which features BizTalk360 has, to allow you to automatically monitor your BizTalk artifacts.

How BizTalk360 solves this problem?

To give you that comforting feeling that you are in control over the state of your ports and orchestrations, BizTalk360 provides multiple features.

Threshold Monitoring – While using the concept of Threshold Monitoring, BizTalk360 allows you to setup state-bound monitoring. This means that you can easily setup monitoring to have the product check for the Expected state of your ports and orchestrations.

In above screenshot, you can see how some Send Ports are being monitored against their Expected State. Once, the Current State deviates from the Expected State, a threshold violation has occurred and BizTalk360 will notify you of that fact.

Positive and Negative monitoring – In most scenarios, the Expected State of your Ports and Orchestrations will be Enabled/Started; which is called Positive Monitoring. However, there might be valid reasons that certain ports/orchestrations need NOT be Enabled/Started. Setting up that kind of monitoring is called Negative Monitoring. BizTalk360 allows to setup for both Positive and negative Monitoring.

Receiving Notifications via multiple channels – Once a threshold violation has occurred, by default, you will receive notifications by email. Moving forward, BizTalk360 also allows you to receive notifications via multiple other channels.

We are constantly considering new ways to send notifications, but currently, the following Notification Channels are provided out-of-the-box: Email, SMS, Event Log, Slack, Microsoft Teams, ServiceNow, HP Operations Manager, Webhook. In addition, you can build your own custom notification channel like writing to a database, calling an internal system etc.

Auto Correct – BizTalk360 not just monitors your artifacts, it can also try to bring the artifacts back to their Expected state, once there is a mismatch between the Current State and the Expected State. Think of for example that FTP Receive Location which goes down at night or during the weekend.

The Auto-Correct feature works for the following artifacts:

• Receive Locations
• Orchestrations
• Send Ports
• BizTalk Host Instances
• Windows NT Services
• SQL Server Agent Jobs
• Azure Logic Apps

There is an extensive article on BizTalk360 Auto correct, which also explains in which kind of scenarios this feature comes at hand. You can that article here:

Monitoring Dashboard – Although receiving notifications of any mismatches between the Current State and the Expected State is very handy, it will equally be handy to have a large screen, at for example the Support Desk, which simply shows the current health of your artifacts. For this, we bring the Monitoring Dashboard, which does exactly that.

As you can see from below screenshot, it brings you a nice Expandable/Collapsible and filterable treeview, with automatic refresh, thereby constantly showing the current state of the artifacts.

Health Check Reports – Last but not least, are the Health Check Reports you can receive at your convenient timing. These reports function as your daily health check, providing you with an overview of the state of all the artifacts from your alarm. Very handy to receive for example at the beginning of your working day!

Conclusion

In this article, we have seen why being aware of the state of your ports and orchestrations is important. We have also seen why manually monitoring that state is very inefficient and sometimes very complex. As a company which knows these challenges from our own experiences, we have brought multiple features which will help the BizTalk administrator being in control, without having to manually check the state.

Get started with a Free Trial today!

Download and try BizTalk360 on your own environments free for 30 days. The installation will not take more than 5-10 minutes.

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360. View all posts by Lex Hegt

Why did we build FTP/SFTP Monitoring for BizTalk Server?

by Lex Hegt | Apr 11, 2018 | BizTalk Community Blogs via Syndication

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”.

Why do we need this feature?

In the day to day activities of a BizTalk administrator, you might come across integrations where FTP sites are used for receiving and transmitting messages. FTP sites are often used for cross-platform integrations. For example, when you have an SAP system on Unix that has to be integrated, via BizTalk Server, with other systems, you might use FTP for receiving and transmitting of messages.

SFTP & FTPS are just the secured version of FTP with advanced transport encryption mechanisms, so your end-to-end data transmission is secure and safe.

To keep the business process going, it can be of vital importance that the FTP/SFTP sites are online and the messages are being picked up. So, when a BizTalk administrator needs to be constantly aware whether the FTP/SFTP sites are online and working properly, the administrator needs to monitor the sites and the activities which take place on these sites.

What are the current challenges?

BizTalk Server offers no monitoring capabilities, not for Receive Locations / Send Ports and also not for endpoints like FTP, SFTP and FTPS sites. So, using just the out-of-the-box features of BizTalk Server, a BizTalk administrator will have to manually check whether the FTP sites are online and whether all (appropriate) files are being picked up for further processing.

Manual monitoring

This kind of manual monitoring can be quite cumbersome and time-consuming. The administrator will probably use multiple pieces of software to be able to perform these tasks. Think of for example the BizTalk Administration console to check whether the Receive Locations/Send Ports are up and some FTP client to check whether files are being picked up.
It is obvious that this is not a very efficient scenario, which could easily be automated by setting up monitoring.

Maintaining scripts for monitoring FTP sites

To reduce their workload, we experience that BizTalk administrators are creating their own scripts to monitor FTP sites and all kind of other resources. Although this kind of scripts certainly can be of help, we still think this does not fully solve the problem.

For example, often these kinds of scripts need maintenance when FTP sites need to be added, changed or deleted from monitoring. This kind of tasks can be easily forgotten.

Also from a knowledge transfer perspective, it’s easy to forget to update new colleagues about the existence of this kind of scripts, as they will probably be installed on some (monitoring) server.

Another challenge with solving this kind of problems with scripts is that not each administrator is capable to write this kind of scripts, which makes knowledge transfer even harder.

To keep the overview, we think that it is easier to use software, like BizTalk360, to have everything in one easily accessible place, with good visibility of all the features/capabilities, fine-grained security/auditing and without the need to maintain custom scripts etc..

How BizTalk360 solves this problem?

With BizTalk360, we make monitoring of FTP/SFTP/FTPS sites a lot easier. For a very long time, the product offers monitoring of Receive Locations and Send Ports, but for some time now, BizTalk360 also offers to monitor of the physical FTP/SFTP/FTPS endpoints.

We wanted to make setting up this kind of endpoint monitoring as seamless as possible and therefore we simply show all the ports in the current BizTalk group which make use of the FTP/SFTP/FTPS adapter.

In BizTalk360, you can find FTP monitoring under Monitoring => Manage Mapping => File Locations (File, FTP, SFTP).

Next, you can set up monitoring rules based on File Count and Directory Size and have BizTalk360 send Warning or Error notifications through the notification channels which are configured on the associated alarm.

A fully monitored FTP endpoint might look like shown below.

Of course, besides the greater-than-or-equals operator, also other common operators are available.

Conclusion

As a final point, we see from time to time that administration teams maintain a administrators handbook, which contains all the tasks a (BizTalk) administrator should take care of. We think that by using software like BizTalk360, we can reduce the number of pages in such handbooks, as the kind of scripts we mentioned no more have to be described in that kind of books.

This description could be replaced by, for example, a general guideline on how FTP sites should become monitored and the monitoring rules with BizTalk360.

As a result, we hope to make the work of BizTalk administrators a bit easier so the team can focus on the more exciting parts of the job of BizTalk administrators, instead of constantly having to update their handbooks.

So, we think that we make the day to day life of a BizTalk administrator, who needs to monitor the well-being of FTP sites, a little bit easier by bringing this feature.

If you want to read more on FTP/SFTP monitoring in much more detail, you can check the following article:

FTP, FTPS, SFTP Location monitoring

BizTalk360 also offers an advanced monitoring capability called “Data Monitoring” which allows monitoring the traffic/volume of messages going through the ports for a given period, ex: expected 50 PO orders from our partner via FTP/SFTP. Please check out this article. Introducing BizTalk Server Data Monitoring in BizTalk360

Get started with a Free Trial today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360. View all posts by Lex Hegt