Cross-domain file monitoring

Cross-domain file monitoring

One of the core features of BizTalk360 is it’s monitoring and notification capability. BizTalk Server is used in mission-critical environments to integrate different applications within the enterprise. While BizTalk Server can integrate with multiple applications and provide a one-stop solution for enterprises, it lacks the ability to allow the administrators to monitor, amongst others, the health of connected applications.

BizTalk360 understands the problems faced by enterprises and offers an out-of-the-box monitoring solution that assists administrators to monitor the BizTalk environments. BizTalk360 solves the purpose of being able to monitor BizTalk Server solutions in a simple and easy manner.

So, What Can BizTalk360 Monitor?

BizTalk360 allows administrators to set up alerts on specific components of BizTalk Server, such as:

  • BizTalk Applications and their Send Ports, Receive Locations, Orchestrations, Service Instances
  • Disks
  • Event Logs
  • NT Services
  • System Resources (CPU, Memory)
  • SQL Jobs
  • Web Endpoints
  • BizTalk Health Monitor, Errors and Warnings
  • Host instances (normal, clustered)
  • Azure services
  • Queues – MSMQ, Azure Service Bus, IBM MQ
  • File Location – File, FTP and SFTP
  • Database queries

I would like to explain the feature of File Location monitoring usage and the challenges we faced.

File Location Monitoring

File Locations Monitoring is one of the core monitoring capabilities which we had introduced in BizTalk360 v8.4. From our customers, we received many requests to introduce the ability to monitor Folder Level into BizTalk360.

How File Location monitoring works?

BizTalk360 has the capability of fetching the file locations which are configured in the BizTalk Admin console.

File Location Monitoring lists all the locations configured in the BizTalk Artifacts (Send Ports and Receive Locations) for the Transport Types (File, FTP, SFTP) respectively and shows them in the UI. Users can map only the file location of BizTalk Artifacts; file locations which are not mapped with the artifacts will not be monitored from BizTalk360.

Cross-domain

Just to give an overview of a domain

A domain is a group of computers and devices on a network that is administered as a unit with common rules and procedures. Within the Internet, domains are defined by the IP address. All devices sharing a common part of the IP address are said to be in the same domain.

Overview of Cross-domain

By cross-domain, we refer to the communication between different domains by creating trusts.

A trust is a relationship established between two different domains that enable users in one domain to be authenticated by a domain controller in the other domain. There are different types of trusts in the Microsoft Active Directory domain such as External, Realm, Forest and shortcut. An external trust is necessary when users of two different domains of two different business units want to utilize resources such as printers and file servers of trusted domains.

Cross-domain in BizTalk

In BizTalk, the receive locations or the send ports can be configured with the file locations present in the server. These locations may be in the same server or in a different server in the different domain. With the trust created between the domains, the file transfer would happen, and the message gets transferred.

File Location monitoring in BizTalk360

As said earlier, in BizTalk360 we have the capability to monitor file locations and trigger alerts based on the file count and directory size configuration.

The BizTalk360 monitoring service will check for the file count and directory size and trigger alert emails. So, as per the environment configuration, the service account running the monitoring service will need access permissions to the BizTalk environment to fetch the data of the BizTalk artifacts.

Often the support tickets raised by customers provide different learning experiences to the support team. One such ticket was an issue with the file monitoring.

Configure File Location monitoring

Configuring File Location monitoring is a very straightforward activity in BizTalk360. You have the option to configure with authentication mode as well as anonymous mode. BizTalk360 will automatically list down all the available file locations on the server.

Click the Gear icon to open the File Details blade and toggle on the Authentication button to enter the username and password. If the username and password columns are blank, it will be considered as anonymous authentication. Configure the warning and error rules for monitoring under the File Monitoring Configurations settings. Click Save Configurations to complete the process.

Scenario 1: Cross-domain file share

One of our customers raised an issue that a file share is getting Orphaned once after configuring with the alarm. We started with the basic troubleshooting steps.

We understood that customer configured a File path in BizTalk server which is a cross domain. In detail for the specific send port, they used a file adapter and while they provide a path they have provided a path where the folder is available in a different domain.

Troubleshooting the problem

We asked the customer to share if any exception appeared and we received that “the username and password are incorrect”.

Technically, BizTalk360 should monitor the specific file without any issues. This is the reason why we are using BizTalk360’s monitoring service account to fetch the details. If BizTalk360 monitoring service needs access across the domain, it should monitor without any problem.

To find the exact root cause of the issue we have developed a console application(with logs enabled) and provided it to the customer. Developing back and forth, we found that it’s an authentication issue and solved it in version 8.9 of the product.

Scenario 2 – Monitoring a cross-domain Linux server

It’s again an interesting case once after the release of version 8.9 (fix for cross domain file monitoring). One of our customers came with a feedback that he is facing a problem with file monitoring.

Once we engaged a call with our customer, we found that they are using a Linux machine for a folder path in the BizTalk server, which is a cross domain. In detail, for the specific send port, they used a File adapter and while they provided the path, they have provided a folder which is available in a different domain in a Linux machine.

Troubleshooting the problem

We asked the customer to share if any exception appeared and we received the same error that “username and password is incorrect”.

To connect the Linux folder location from the Windows machine, the customer used a tool called Samba share. We have enhanced our code to authenticate and connect the Linux machines with encrypt and decrypt, along with NT service mechanism.


Later, we have provided a console application to our customer which solved the problem. We have planned to include the fix in our upcoming version.

What about FTP and SFTP?

By default, FTP and SFTP are hosted in a web(domain)and we support cross-domain in FTP and SFTP file monitoring.

Conclusion

This was another support case which was interesting for me as it helped me to understand the code from a Developer perspective. I have explored a different kind of cross-domains from the Tester perspective. Working along with the customer gave satisfaction as a Support engineer. Hope I will receive “an Awesome” review from the customer side 😊

Author: Sivaramakrishnan Arumugam

Sivaramakrishnan is our Support Engineer with quite a few certifications under his belt. He has been instrumental in handling the customer support area. He believes Travelling makes happy of anyone. View all posts by Sivaramakrishnan Arumugam

An interesting support experience – Troubleshooting External Notification Channel(Slack)

An interesting support experience – Troubleshooting External Notification Channel(Slack)

I can say last month was one another interesting month for me as a Support and Test engineer, as we have received quite a few interesting tickets.

Every support ticket gives us a different experience in its own way. But, some of the support tickets seem interesting, because it helps us to explore and learn new things. One of such tickets was related to the Slack Notification Channel.

Short intro about Notification Channels

BizTalk360 is embedded with a feature called Notification Channel. Not only does BizTalk360 support internal notification like emails and SMS, along with it, we support external notifications so that customers can write their own code to send alerts to the Notification Channel they use as per their organization policy.

External Notification Channel is another customer’s feature request, which we have developed and included in our product. As it can be very helpful to Support teams, we decided to create some inbuilt Notification Channels. Further, customers can create their own Notification Channel using their own code. Kindly find the blog for Custom Notification channel.

Feedback from the customer about custom Notification Channel

We have got very positive feedback from our customers about external Notification Channels and received quite a few requests to include some of the Notification Channels as an inbuilt feature. And we did so.

Also, we haven’t kept support aside; whenever customers face challenges with Notification Channels, we helped them to solve the issue at hand to our best. One such classic case is ‘Using BizTalk360’s Notification Channel to perform some action via a PowerShell script’.

Default Notification Channels

During the initial point of time, along with the version 8.0, we have provided BizTalk360 with some inbuilt Notification Channel such as:

1. Slack and
2. Webhook Notification Channel

Later as per the customer’s request, we have included two more Notification Channels and we have 4 now.

3. ServiceNow
4. Teams
Notification channel configuration in our latest version 8.8 looks like below.


Also, we have enhanced the Notification Channel feature as per the customer’s request. Such as,

Challenge in External notification #Slack

Last month, we received a support case regarding the Slack Notification Channel, that it’s not receiving any notifications from BizTalk360 and they faced the below exception in the log file:

2018-06-08 12:01:40 ERROR – Unable to connect to the remote server

We started to approach the case with the basic troubleshooting steps such as:

• Does the user have proper credentials?
• Any ports are blocked?
• Any security tools are blocking the connection?
• The URL what they used to connect slack?
•Does the proxy is blocking the internet connectivity?

The basic troubleshooting did not help us to solve the problem. So, we went into the screen-sharing session to investigate furthermore. At that point of time, we found that admin scope is missing for the application.

To give more insight about Slack:

In Slack,

• Groups are considered as #Channels
• Applications are predefined functionality, that is considered as #Apps
• Permissions are called as #Scopes

The flow is like:

1. You must create a Channel inside Slack
2. App creation will be followed by channel
3. Add necessary permissions/scopes to the created app
4. Generate Access Key/Token for the app
5. Add app inside the channel
6. Add the Access token for the app inside BizTalk360

We asked the customer to provide the admin scope and left the call. After few days customer came back with the same problem even though they have provided admin scope.

Creation of test environment

For Testing purposes, I have created a new Slack account and Channel, tested from the scratch and we have provided the same permission as the customer had.We got the same issue as the customer encountered, though the application has the scope admin.

What our code does?

We have analyzed with the source code level to find if anything is missing at our side. But found, the code is written in a generic way to push the plaintext message to any Notification Channel. So that, it could be sent to any platform. we have found that it is not a problem with sending a notification from BizTalk360, but it could be a permission issue.

After some testing and researching, we were not able to find the minimum permission what needs to be provided for the application in order to solve the customer issue. Later, we have decided to seek help from the Slack support team for further investigation.

Working with Slack Support

Daniel from the slack support team who worked with me throughout the way. I have created the support ticket along with the sample test application which had been created for testing. The conversation started with the basic questionaries and made sure the basic configuration is configured as per the actual expectation. However, everything is correct, still, we couldn’t make the app work. The direction of an investigation gone little deep and we will look it in the coming section in detail.

Troubleshooting with the permissions

Daniel conveyed, installed app in the test workspace has a valid token. But, looking through the logs, the application has enough permission (both admin & User identity) to receive the messages. Permissions are as below,

  1. https://api.slack.com/scopes/admin – Got Admin scope
  2. https://api.slack.com/scopes/identify – And got Identify Scope

Additionally, he suggested providing 

  1. Conversiontions:write &
  2. Converstions:read scopes

After following the suggestion, the app was not working. But the next suggestion made a trick.

What is the trick?!!

Later, we got a reply from Daniel “I took a closer look at your app, and it appears you haven’t added the chat: write scope. You’ll need this to make any chat:postMessage (https://api.slack.com/methods/chat.postMessage) calls with your app.
Providing the chat: write permission to the application indeed did the trick! The application writes a message in the Slack from BizTalk360. I have updated the same to the customer and that helped them to resolve their issue.

Satisfaction does it!

This was another support case which was interesting for me as it helped me to understand the code of Notification Channel from a Developer perspective. I have explored Slack along with BizTalk360 which provided more insight from the Tester perspective. Working along with the customer and Slack Support team gave satisfaction as a Support engineer. I have received “an Awesome” review from the customer side 😊

Author: Sivaramakrishnan Arumugam

Sivaramakrishnan is our Support Engineer with quite a few certifications under his belt. He has been instrumental in handling the customer support area. He believes Travelling makes happy of anyone. View all posts by Sivaramakrishnan Arumugam