Introduction to File Locations Monitoring in BizTalk360

Introduction

File Locations Monitoring is one of the new Monitoring capabilities introduced in BizTalk360 v8.4. From our customers, we received many requests to introduce the ability to monitor Folder Level into BizTalk360. We are happy to announce that in BizTalk360’s upcoming release 8.4, we are bringing a feature to monitor file count for File Locations (File, FTP and SFTP), configured in BizTalk receive locations and send ports.

Need for File Locations Monitoring

Messaging or data exchange between business can be done in various ways. Frequent data communication process is done through for example File and FTP with XML or EDI as the message format. From BizTalk 2013 onwards, SFTP is also included in BizTalk Server which was available as Out-of-box feature in prior versions. Even though BizTalk works seamlessly with File Adapters, it has some known issues which occur due to incorrect configurations.

The BizTalk File Location Adapters (File, FTP, SFTP) fail to perform the operations on following scenarios:

  • The File receive adapter cannot access the receive location on the file system or network share because the specified path does not exist. For a network share, the File receive adapter disables the receive location after all retry attempts have been exhausted.
  • The File receive adapter cannot access the receive location on the file system or network share because the account used by the associated host instance does not have read-write permission for that location. For a network share, the File receive adapter disables the receive location after all retry attempts have been exhausted
  • Files with names longer than 256 characters are encountered in the receive location

To resolve the above issues, we need to ensure that the specified path or share exists and the account used as the Logon should have read-write access. Additional to this, if you configure schedule/service window for your receive locations, messages will be accepted only during that time window, all other times BizTalk won’t pick up messages. Any violation to this scenario also needs to be monitored.

We often experience that organizations facing these kinds of challenges used custom solutions for this kind of monitoring. To overcome this, BizTalk360 added the File Locations Monitoring capabilities out of the box.

File Locations (File, FTP, SFTP)

In BizTalk360 v8.4, we are introducing support to monitor the File, FTP and SFTP servers under File Location Monitoring Section. File Location Monitoring will list all the locations configured in the BizTalk Artifacts (Send Ports and Receive Locations) for the Transport Types (File, FTP, SFTP) respectively, which helps users to monitor all the File Locations mapped with Receive Locations/Send Ports.

File Monitoring

To get familiar with File Adapter configuration, kindly refer below link:

https://msdn.microsoft.com/en-us/library/aa559105.aspx

In BizTalk360 File Monitoring Configuration contains three sections: Basic Information, Authentication, and File Monitoring Configurations.

  • The Basic Information Section contains Folder Location and File Mask configured in BizTalk.
  • The Authentication section is Optional. By default, authentication could be processed by the BizTalk360 Monitoring Service account, when credentials are not given.
  • In the File Configurations Section, we can configure the Thresholds with the metric File Count to monitor

When a File Location is in the Orphaned State, BizTalk360 would let the users know about the cause of the failure on hovering the warning icon

FTP Monitoring:

To get familiar with FTP Adapter configuration, kindly refer to below link:

https://msdn.microsoft.com/en-us/library/aa561032.aspx

The FTP Configuration UI is categorized into three sections: FTP Details, Firewall Details and FTP Monitoring Configurations

  • The FTP Details Section contains the details about the FTP Location, Authentication, and SSL
  • The Firewall Details contains the configurations to connect FTP Server through a Firewall
  • In the FTP Monitoring Config section, we can configure the monitor with Threshold Conditions for the metric File Count

SFTP Monitoring:

To get familiar with the SFTP Adapter Configuration in BizTalk, kindly refer the below link

https://msdn.microsoft.com/en-us/library/jj684551.aspx

The SFTP Monitor Tab in BizTalk360 lists the SFTP Locations which are configured in BizTalk. It contains four sections:

  • SSH Server Section has the details about the SFTP Location
  • The Proxy Details Section is optional to connect SFTP Server behind a firewall

Note: In BizTalk, Proxy details are available from BizTalk 2013 R2

  • Security Details Section has the authentication details
  • In the SFTP Monitoring Config Section, we can configure the monitor with threshold conditions for the metric File Count

Conclusion

With this latest release 8.4, BizTalk360 brings the File Locations Monitoring with the ability to monitor the File Count. In the future, we will be adding support to monitor Folder Size and Access permissions. If you have any feedback or suggestions, please write to us at support@biztalk360.com.

BizTalk360 Version 8.0 Review

BizTalk360 Version 8.0 Review

A new version of BizTalk360 has been released, version 8 which comes with a boatload of features and a complete overhaul of its user interface to provide an exceptional user experience. The product adopted a concept of widgets in the dashboard similar to the new Microsoft Azure portal. A BizTalk 360 user can now create and customize dashboards for different use cases.

The User Interface, Dashboards and Widgets.

In this post I will review a few of the new features and let’s kick off with the UI, which has improved dramatically. Intially BizTalk360 was a Silverlight application before it migrated to HTML 5. And in version 8.0 it has adjusted to what you experience on the Azure Portal and what you see with UI in devices. Below you see a customized dashboard with several widgets.

Figure 1. An example of my customized monitoring dashboard.

The end user has also the ability to create his own custom widgets via the settings page.

Picture 2. An example of creating a custom widget.

Furthermore the separation of monitoring and analytics is based upon audience targeting. An analyst or support engineer can interpret information of messaging and processes within a BizTalk Group from a different perspective as shown below demonstrating a customized dashboard.

Figure 3. An example of a customized analytics dashboard..

BizTalk360 API’s

Another incredible addition are the API’s, which offer a tremendous amount of value to enterprise customers. It brings extreme extensibility for its end users and enables the for instance to pull information from several environments. The API’s (REST web services) are the core of the BizTalk360 capabilities and the BizTalk360 front-end uses them. And as a BizTalk360 end user you can leverage all of these API’s to have access to your BizTalk environment. In the setting menu under API Documentation you can find the descriptions of each API and operation. For instance if you select BizTalkApplicationService you can observe a list of operations belonging to it. Subsequently select one of the operations, Services.REST/BizTalkApplicationService.svc/GetOrchestrations, specify the environmentId and applicationName and hit “Try it out!”.

Figure 4. BizTalk360 API’s document page..

You can call the API’s from different machines, by setting the appropriate authorization in the user access policy. Calling the API operation from SoapUI will result in the same response as from BizTalk360 API Documentation.

Figure 5. Calling BizTalk360 API from SoapUI.

The look-and-feel are similar when you publish an (Web) API configured with swagger documentation describing it.

Business Rules in a browser

Yet another new feature added to BizTalk360 is the rule engine capability of BizTalk Server i.e. the business rule and composer are accessible from a browser. You can view what type rules are deployed.

Figure 6. Deployed rules in a BizTalk Group.

And one of the capabilities is that you as end user can compose new rules, edit or test them. Below you see an example of testing a deployed rule, which you can do in a browser!

Figure 7. Test of a deployed business rule.

The version of the business rules composer in BizTalk360 only supports pre-created vocabularies. Therefore, to ensure a smooth user experience, you need to create all the vocabularies you will require to develop or edit your rules in the BizTalk Business Rules Composer. In a nutshell follow the instructions on the BizTalk360 portal with regards to rules.

Live feeds

In an enormous environment various operation people perform various actions in a production or UAT environment, and it can be valuable means of real-time tracking of what happens. This new version of BizTalk360 offers live feeds and you view on the right side after click the icon (indicated in red in the picture below).

Figure 8. Live feed in BizTalk Group.

Alarms and notifications

In BizTalk360 you can set alarms, which can result in an email being sent to you.

Figure 9. An example of email indicating a receive location is down.

Or you can leverage one of the out of the box notification channels like Slack and configure an alarm using a notification channel.

Figure 10. Enable the SlackNotification on a custom alarm.

Once you have set up Slack following the instructions on BizTalk360 UI and the alarm you will receive notifications in Slack.

Figure 11. Notification in Slack.

Final words

The BizTalk360 team has worked intensely the last 12 months to bring this to the table. Not just the flexibility with widgets, yet also with the separation of monitoring and analytics, the 360 plus API’s on top of your BizTalk Group, live feeds, notifications and the rules engine. This product has definitely evolved to the next level. Expect the release of 8.0 to change the way BizTalk is being monitored!

Cheers,

Steef-Jan

BizTalk360 – An easy way to evaluate

BizTalk360 – An easy way to evaluate

BizTalk360 is a full featured BizTalk Server Monitoring & Operations product. The product has matured from a small web based monitoring tool to a full blown feature rich product for monitoring BizTalk Server. The product includes features for BizTalk Operation support people like the Throttling analyzer, Topology Viewer, and Backup DR Monitor. Recently a new version 7.8 has been released with another set of valuable features:

  • Advanced Process Monitoring and dashboard
  • EDI reporting capabilities in line with BizTalk Admin Console.
  • System Profile: Ability to remove features you don’t want in BizTalk360.
  • User Profile: Flexibility to set custom time zone and date/time format for each user.
  • Removed Orphaned artifacts from Monitoring: Ability to remove old orphaned artifacts like Applications, which no longer exist in the environment.

If you haven’t heard of BizTalk360 than you will like this blog post as it will explain how you can easily evaluate the tool by yourself and witness the great value this product can offer. Now this will be not an extensive post to showcase each feature. I will focus on how you can leverage the features: Graphical Flow (Tracking), Business Activity Monitoring (Portal), and the Tracking Manager.

Easy way to see these features in action is by deploying the BamEndToEnd example you can find in the <Samples Path>BamEndToEndInput folder of the installation of your BizTalk Server. This sample stems from version 2004 and can be used from 2006 up to 2013 R2. The demonstrations in this post is the BamEndToEnd example with BizTalk Server 2013 R2, and the latest BizTalk360 Version 7.8.

After deploying the BamEndToEnd example you will need to create the tracking profile yourself and deploy it. Next you will to add/verify the tracking is enable on the send, receive ports and the orchestrations.

Note that tracking can be very useful for troubleshooting and traceability in general. However in production environments you beware that extensive tracking can influence performance. For best practices for tracking see blog post on BizTalkAdminsBlogging Best Practices for Tracking. With regards to orchestration tracking is that Event Tracking is enabled by default. This can be helpful during development, testing and troubleshooting with the Orchestration Debugger. In Production, these tracking events should be disabled, because these events are written to the TrackingData_x_x tables in the BizTalkMsgBoxDb database and then moved to the dta_DebugTrace table in the BizTalkDTADb database by the Tracking host. If the TrackingData_x_x tables (BizTalkMsgBoxDb database) and/or the dta_DebugTrace table (BizTalkDTADb database) get too large*, performance will be impacted, see How to maintain and troubleshoot BizTalk Server databases.

After deployment and configuration is done you can drop the test messages in the folder belonging to the receive location. In SQL Studio Manager you can execute the query below to examine if data is present in the BAMPrimairyImport database tables belonging to the BamEndToEndActivity.

When that is in order you can switch to BizTalk360 product. The product is available for free trail. The installation is straightforward described on the site on this page. Once you have installed and configured BizTalk360 you can start looking at some of the features. One is the Graphical Flow (Viewer). You can select this viewer and leave everything default and click Execute Query.

Next you can select one of the tracked service instances.

On right hand side you can see the properties of the third orchestration of the BAMEndToEnd example. You can click the blue dot (where the message is received) and see a line going from the green dot (message being send from a location) of the second orchestration. Now you can click in the right hand pane the context and/or content tab. You will see the context of the message going from Orchestration 2 to 3 and the content.

clip_image010

You can click the blue dot again and reach orchestration 1 the first orchestration message are flowing through. This feature is a very valuable feature in case you need to troubleshoot a process that for instance consists of multiple orchestrations. From a messaging perspective you can look at messages coming in and out of BizTalk. In the dashboard you can select Tracked Message Events instead of Tracked Service Instances. Select one of the Send Type the eye icon.

A dialog will appear for the message you selected. You can see the general properties, context properties and the content. Note this depends on how tracking is configured. For both Tracked Message Events and Tracked Service Instances you add filters that can enhance search ability for a certain message(s) and instance(s).

Next we will look at the Business Activity Monitoring. BizTalk360 offers a view/dashboard for BAM that is similar to the BAM Portal. One of the concepts of BizTalk360 is offering a single operation and monitoring experience. This means one product for accessing data and user experience (UI) for BizTalk, BAM, BRE, EventViewer, BizTalk databases and ESB Toolkit.

If you go to Business Activity Monitoring Dashboard in BizTalk360 you can do similar things one can do in the BAM Portal. In The BAM Dashboard you can select a BAM View. In this case the BAMEndToEnd View. Select columns and execute a query.

As you can see the data can be viewed from the dashboard. There is no need to go to a separate Portal in case your BizTalk solutions include BAM. The BAM Dashboard in BizTalk360 will also give you a view of the sizes of the BizTalk BAM Databases.

Sizes of your BAM databases matter. In case BizTalk system is under a huge load a significant amount will be persisted to the BAM database. Therefor information on size and growth are valuable. You can monitor this by setting alarms in BizTalk360.

The final feature I like to discuss in this context is the Tracking manager. For the BAMEndToEnd example you can enable tracking on send/receive ports and the orchestrations. By selecting the Tracking Manager in BizTalk360 you can examine how tracking is enable on the receive ports, send ports and orchestrations.

You can select different applications in this dashboard, examine if global tracking is enabled and how per application the tracking is set on receive/send ports and orchestrations. This manager or dashboard if you like gives you a quick view on how tracking is set up in your BizTalk environment. This can be useful in cases you have not direct insight of the bindings of the applications that are being deployed. The bindings can contain tracking settings, see for instance binding collection of a receive port on MSDN.

In this post I quickly showed three powerful features in BizTalk360. However, there are many more which you can examine by yourself or by reviewing the feature tour on the BizTalk360 site. In case you like to further explore the features BizTalk360 offers you can do so by using the BizTalk out-of-the box examples on a VM and the evaluation edition of the product.

Cheers,

Steef-Jan