August 12, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

August 12, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

 

 

Community Blog Posts

 

Videos

 

Podcasts

 

How get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

BizTalk360 – Ready to function

BizTalk360 – Ready to function

BizTalk360 is one-stop monitoring tool for monitoring your BizTalk environment. The BizTalk360 installation is a very quick process and does not take much time when all the prerequisites are met. For any software product, there would be some basic requirements that should be satisfied for its successful installation. BizTalk360 also has its own hardware and software requirements. In addition to these requirements, there are also some permissions and rules that need to be provided.

We have different documents that contain this information. But I thought it would be even more helpful for our customers if we have all this information in a single place. In this blog article, I would like to give a consolidation of all the requirements that one should make sure of before installing BizTalk360.

There are many considerations that need to be taken care of before installing BizTalk360, like where do you want to install BizTalk360, the service account permissions, the database creation, etc. Let’s look at each of them.

Installing guide for BizTalk360

BizTalk360, being a separate integration tool, always arises a query from the customers as to where we must install it. What is the best practice to install BizTalk360? It can be either installed on the same server where BizTalk is installed or in a standalone server.

The recommendation would be to install BizTalk360 on a separate server, rather than in the BizTalk server.

This will be useful in the performance perspective. The BizTalk server performance would not be hindered and BizTalk360 will notify the users when the BizTalk server is down. The important prerequisite for installing in a standalone server is that the BizTalk Admin Components must be installed beforehand in the standalone server.

BizTalk-Server-Installation

If these components are not installed, then the BizTalk360 installer will fail with the following error.

BizTalk360-Prerequisite-Check

The role of the service account in BizTalk360

One of the most important prerequisites for BizTalk360 is to configure a separate service account. The service account will be one that is used for the IIS App pool that will be created during the installation and also for running the BizTalk360 Monitor and Analytics service.

The recommendation is to create a dedicated service account, the reason being there are some permissions that need to be provided for this account.

The permissions would be:

  • Local administrator permission in the BizTalk360 server for starting the Monitor and Analytics service
  • Local administrator in the BizTalk and SQL server(s) for collecting the event log data and for disk and system resource monitoring
  • Sysadmin permission to the SQL Server for creating and updating the BizTalk360 database
  • Sysadmin permission for the BizTalk databases for monitoring and autocorrecting the SQL jobs. In case if you do not prefer to provide Sysadmin permission, you can provide SQLAgentOperator role for the service account for monitoring the SQL jobs and performing the autocorrect functionality
  • Member of all the BizTalk groups

Once you have created the service account, you will have to log in to the BizTalk360 server with this service account to install BizTalk360. Logging in with other users may result in other permissions related to the database as other users may not have the expected permissions as the service account.

Firewall ports and protocols to be checked

When you plan to install BizTalk360 in a separate standalone server, there are some configurations that need to be taken care in the security perspective. For example, there may be firewall rules that may restrict access to the server. For BizTalk 360 to be able to contact BizTalk Server, the MSDTC connection needs to be enabled in all the servers. BizTalk360 should be enabled with certain ports/protocols for communicating with the external servers.

Service-Feature

Next, make sure the below services are running in the server:

RPC

Also, make sure if custom ports are utilized for any service, unblock them as well from the firewall for the seamless working.

Ready to install BizTalk360

Once all the prerequisites are in place, you are now ready to install BizTalk360. The installation is a seamless process which would roughly take about 10-15 minutes when all permissions are in place.

You will have to make sure that you have logged into the server with the BizTalk360 service account. The reason for this is that when BizTalk360 is installed, the logged-in user will be created as a Super User in BizTalk360.

This Super User permission is required for the service account to perform certain automated processes like terminating/resuming service instances through Data Monitoring alarms and also for Analytics operations.

If for some reason, the installation fails, the first thing to be looked for the error, would be the installation logs.

Important configurations for BizTalk360

For BizTalk360 to be up and running, there are some configurations that need to be done. For example, the SMTP setting would be required for triggering the alert emails. You will need to create different users for accessing BizTalk360.

You can refer to the implementation of best practices here.

Best practices on a performance perspective

For any software product, the performance would be the foremost priority. Some of the performance measures that can be taken up for enhancing BizTalk360 would be:

  • Configuring only the required sources in the Advanced Event Viewer so that the database growth is controlled
  • Maintaining the appropriate purge duration to remove the historical data based on the business needs
  • Enabling only the relevant performance counters for Analytics performance data collection

Upgrade and Migration Scenarios

  • What happens when you upgrade BizTalk360 to a new version?
  • What happens when you migrate your BizTalk server?
  • Let’s have a look!

Upgrading BizTalk360

A new version of BizTalk360 is released every quarter with new features and enhancements picked up based on customer feedback. The release would also include certain bug fixes as well.

It is important that customers upgrade their BizTalk360 to the latest version to be in line with the release and enjoy the new features.

Things to be taken care while upgrading BizTalk360:

  • You should log in with the BizTalk360 service account to the BizTalk360 server
  • The license needs to be deactivated
  • The database should be backed up so that in case of any errors, the old database can be restored
  • Once after the successful upgrade, it is important to clear the browser cache and then launch BizTalk360

In a scenario where there might be some problems during the upgrade, the customer could have uninstalled and reinstalled BizTalk360. In this case, they would have restored the old database and the following error would occur.

restore-database-error

Here, we can see that the application has been upgraded but not the database since the old version of the database has been restored. The suggestion here would be to run the SQL scripts from the next version seen in the error from C:Program Files (x86)Kovai LtdBizTalk360ServiceSQL folder. Please note, if there is more than one script that needs to be executed, then they should be run in the same sequence.

Upgrading BizTalk Server

Customers might migrate from a lower version of BizTalk Server to a higher version, say for example from BizTalk Server 2013 to BizTalk Server 2016. There are some configuration changes that need to be done at the BizTalk360 side as well to point to the new BizTalk environment. The below steps need to be followed in the BizTalk360 box when you migrate the BizTalk server.

  1. Deactivate the license in BizTalk360 first before migration
  2. Install the new version of BizTalk admin components in BizTalk360 server
  3. After migration, update the environment details to point to the SQL instance which contains the BizTalk databases
  4. The below tables need to have the BizTalk version changed appropriately and the MS_BizTalk_install_location should be updated

b360_admin_BizTalkEnvironment -> BizTalk Version

b360_admin_GlobalProperties -> MS_BIZTALK_VERSION & MS_BIZTALK_INSTALL_LOCATION

You can export the alarms from the old BizTalk360 environment and import them to the new environment. The mappings will also get imported to the new environment. If you are using BizTalk360 v8.8 or later, then you can export the user settings as well.

  1. Modify the alarm mapping to point to the artifacts in the new environment

Migrating BizTalk360 database and application

When the BizTalk360 database is moved to a new server, the connection string needs to be modified in the following config files:

  • Web.config
  • BizTalk360.Monitor.exe.config
  • BizTalk360.Analytics.exe.config

For more details, refer to the article.

BizTalk360 additional details

The integrated Troubleshooter in the BizTalk360 UI will provide you with the complete details of BizTalk360 like the connection string, installation location, and the database.

biztalk360-components

biztalk360-component

Conclusion

Once all the setup and configurations are in place, it is a quick and seamless task to install/upgrade BizTalk360. The customer suggestion and feedback are always heard and addressed, which helps us to improve the product and provide better service.

Happy monitoring with BizTalk360! 😊

The post BizTalk360 – Ready to function appeared first on BizTalk360.

New Dev SKU for Azure Integration Service Environment (ISE)

New Dev SKU for Azure Integration Service Environment (ISE)

Last week, Microsoft responded to numerous requests from the community by announcing a new developer tier offering for Integration Service Environments (ISE). The ISE has been generally available for several weeks, but the single available SKU prior to this announcement carried a hefty price tag.

(fromEldert)WhatsApp Image 2019-06-27 at 10.59.51 AMI had the great honour and privilege of speaking about ISE at the INTEGRATE 2019 conference at Microsoft headquarters in Redmond, WA (USA) last month.  My topic was Four Scenarios for Using an Integration Service Environment, which attempted to shed some light on what type of situations would justify using this flat-cost product as opposed to the consumption-based serverless offering of Logic Apps.

While this presentation hopefully piqued interest in the offering, one of the burning questions from the attendees was “When will a lower cost developer SKU be available so we can try it out?”  All Microsoft was able to say at that point was, “Soon.”  Well, at least they were right, as it is now available only a few weeks later!

What is an Integration Service Environment?

The ISE is a relatively recent offering from the within Microsoft’s Azure Integration Services that enables you to provision Logic Apps and Integration Accounts within a dedicated environment – much like App Service Environments enable you to do the same for App Services. The advantages of this include:

  • Predictable performance
  • VNET connectivity
  • Isolated storage
  • Higher data transfer rates (especially for hybrid scenarios)
  • Flat cost

The VNET connectivity is a large driving factor for using an ISE. Consumption-based Logic Apps do not offer any direct integration with VNETs – they operate within a multi-tenant context inside of each Azure location.  This means that all connectors share the same pool of static outbound IP addresses for all instances deployed to that region. It also means that performance can potentially be affected by “noisy neighbours”.

image

Because an ISE is injected into a VNET that you create, all the runtime components now operate within that VNET and as such are isolated from other resources in that region. It also gives you the ability to put network-level security controls around your Logic Apps (for example using Network Security Groups), and also explicitly control the scaling of your application as opposed to relying on the reactive scaling offered by the serverless option (mind you, auto-scaling rules are still available within an ISE).

Flat-Cost Model

Unlike the consumption-based serverless offering of Logic Apps which charges per action execution, ISE runs at a flat hourly cost. The cost is calculated at a scale unit level. The base offering includes a single scale unit which can accommodate roughly 160 million action executions per month. Additional scale units will add capacity for about 80 million action executions each, and each costs 1/2 the price of the base unit.

While it may seem that the cost benefit is only realised in very high-volume scenarios, you need to consider that the base unit also includes a Standard Integration Account (worth ~US$1000/month) as well as an Enterprise Connector with unlimited connectivity (these connectors are charged at a higher rate in the consumption-based model). And that’s not considering the other benefits of an ISE (VNET connectivity, isolation, etc) that may justify the investment.  It is worth noting that VNET integration for most other Azure services generally requires a premium tier or plan (e.g. Service Bus, API Management, Functions, Event Hubs).

Developer SKU

Still, the cost of an ISE in production is a significant investment, just as it is for an ASE. That can make it an expensive proposition, particularly for deployment into non-production environments (DEV, TEST).  Initially, the ISE was only available in one SKU.

That is why the announcement last week was so greatly anticipated. The new developer pricing tier allows organisations to deploy and test an ISE in non-production environments at approximately one-sixth of the base cost of the production SKU (please refer to the Logic Apps pricing page). This makes it a much more affordable proposition and opens up the ISE to many organisations that would otherwise be unable to afford it.

Or course, this developer tier offers no SLA, does not support additional scale units, and includes only a free integration account – so this is definitely not suited for production use. It is a good idea to review the artefact limits imposed on integration accounts.

To choose the developer tier, simply select the non-premium option from the SKU drop-down list when creating your ISE:

image

For detailed instructions on how to provision an ISE including all of the pre-requisites, please refer to the Microsoft documentation.

Summary

Azure Integration Service Environments offer a solid solution for integrating your Logic Apps with a VNET, enabling your entire integration solution to be managed within the controls of a private network. It enables additional security controls and better performance in hybrid scenarios over the On-Premises Data Gateway, whilst reducing the impact of “noisy neighbours”. The introduction of the developer pricing tier now makes it much more affordable for enterprises to build and test solutions using an ISE before committing to production.

This article has been cross-posted. The original post may be found here.

August 5, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

August 5, 2019 Weekly Update on Microsoft Integration Platform & Azure iPaaS

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform and Azure iPaaS?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

 

Microsoft Announcements and Updates

Community Blog Posts

 

Videos

Podcasts

 

How get started with iPaaS design & development in Azure?

Feedback

Hope this would be helpful. Please feel free to reach out to me with your feedback and questions.

Best Practices for BizTalk360 Alarm Configuration

Best Practices for BizTalk360 Alarm Configuration

After the installation of BizTalk360, the implementation phase starts. One of the steps you will want to do is taking benefit of the rich automated monitoring features the product provides. For that, you must set up alarms in BizTalk360. In this article, we intend to give you some best practices around Alarm Configuration. Firstly, however, we’ll shortly explain the different monitoring concepts which are available in BizTalk360.

A short explanation of the supported monitoring types

BizTalk360 has multiple types of monitoring, which are Threshold monitoring, Health Monitoring, and Data Monitoring. Each alarm you create in BizTalk360 can support all types of monitoring, either as separate alarms or as combined alarms.

Threshold-Monitoring

As you can see from the above screen, BizTalk360 enables you to configure an alarm for multiple purposes. So, in case you want the same people to receive the same notifications for both Thresholds as for Health Checks, you only need to create one alarm.

Threshold monitoring 

With this kind of monitoring, you get notified in case a threshold occurs. For example, that can be a Receive Location which was Disabled, while it should be Enabled or any other artefact which is mapped in an alarm and is in an unexpected state. 

At the Alarm creation level, you can configure the following:

  • Name – each alarm needs to be given a name. The more descriptive the name, the easier it is for identifying the purpose of the alarm
  • Description – optionally, you can provide a detailed description of the purpose of the alarm
  • Email Ids – each alarm can send notifications to multiple recipients
  • High Priority – in case you want specific notifications (for example when anything goes wrong with the BizTalk platform) to jump out in your email box, you can enable the High Priority switch
  • Violation persistence – to configure after how much time and with which interval you want to be notified of exceptions
  • Limit the number of notifications per exception – to prevent you from being bombarded with notifications and lose the overview as a result
  • Back to Normal notification – receive a notification when the alarm no more contains artifacts in the wrong state
  • Alert reset – start receiving notifications again after a specific time in case you have received the maximum number of notifications
  • Alert schedule – configure when the alarm should perform monitoring. Handy when you know that specific artifacts are not available all the time, due to, for example, maintenance

Health monitoring 

It can be helpful if you receive a report with the actual status of the monitored artifacts, for example at 9AM every business day. For such scenarios, you can configure alarms for Health monitoring.

BizTalk360 Health Monitoring alarms allow you to easily configure when you want to receive these status reports via an intuitive interface.

Below screenshot shows the screen where you can select when you want to receive these status reports.

health-monitor

This alarm has not only been configured to send a report at the beginning of each working day but also towards the end of the business day. This helps in being aware of the status of your environment before you leave home.

Data Monitoring

A detailed explanation of Data Monitoring falls outside the scope of this article, therefore we provide a somewhat better overview of its capabilities here.

The concept of Data Monitoring allows you to set up monitoring of your data flows. As an example, your BizTalk platform and applications might be in perfectly good health, but when a misconfigured firewall prevents messages from arriving in BizTalk, no processing will be taking place. This interrupts your business processes, with maybe bad consequences for your company. Data Monitoring can be used to monitor this kind of scenarios.

Data Monitoring provides monitoring at the following levels:

  • Process Monitoring – monitor whether your processes run as expected
  • MessageBox Monitoring – monitor the active processes and (optionally) take automated actions
  • Tracking Data Monitoring – monitor completed processes
  • BAM Monitoring – monitor your deployed BAM Views and Activities
  • EDI Data Monitoring – monitor your EDI processes
  • ESB Data Monitoring – monitor your ESB Exceptions and resubmissions
  • Logic Apps Monitoring – monitor tens of metrics about the processing of your Azure Logic Apps
  • Event Log Monitoring – monitor your consolidated Event Log entries of your BizTalk and/or SQL servers

After selecting which kind of Data Monitoring you want to set up, it now comes down to configure what exactly you want to monitor and when monitoring needs to be done.

Below screenshot gives an example of Process Monitoring.

process-monitoring

Once you have set up Data Monitoring, the output of all the different monitor runs will be shown in a consolidated Data Monitoring dashboard. This dashboard helps in answering the questions from your business/functional users whether a specific process has taken place.

data-monitoring-dashboard

You can refer to the Data Monitoring section in the Documentation portal for a more detailed explanation of this way of monitoring in BizTalk360.

Some more information about BizTalk360 Alarms can be found here.

BizTalk platform versus BizTalk application monitoring 

When setting up monitoring in BizTalk360, it is handy to make a difference between monitoring the actual BizTalk platform and the BizTalk applications which are deployed on it. Making this distinction gives several advantages, like:

  • Notifying different people, depending on their role and needs
  • Use different monitoring settings like:
    • The monitoring interval for early notifications in case of important issues
    • The number of notifications you want to receive per exception
    • Receive a ‘back to normal’ notification in case everything is healthy again
    • Set the High Priority flag to easily spot important notifications in your email box
    • Run PowerShell scripts for immediate actions

Even within platform monitoring, you can make segregations for different administrators. For example, a SQL Server DBA has different needs than a System (Windows) Administrator, so other artefacts need to be monitored. BizTalk360 enables you to set up different Alarms for different needs.

What to setup for Platform monitoring 

When you are creating an alarm for platform monitoring, you want to be notified of unexpected situations which might occur on your BizTalk platform. For this purpose, you will create a Threshold Monitoring Alarm. If you also want to receive a daily status report at say every business day at 8:00 AM, you can configure an alarm for Health Check Notification.

See below for a list of artefacts you could consider for monitoring BizTalk platform health:

  • Host Instances (clustered and non-clustered)
  • Host Throttling
  • BizTalk Server availability
  • SQL Server Jobs
  • Windows NT Services
    • Enterprise Single Sign-On service
    • Internet Information Services
  • BizTalk Health Monitor output
  • Eventlog entries
    • BizTalk Criticals, Errors, and Warnings
    • SQL Server Criticals, Errors, and Warnings
    • ESSO Criticals, Errors and Warnings
    • Internet Information Services Criticals, Errors, and Warnings
    • Server restarts
    • Certificate expiration
  • BizTalk and SQL related stuff
    • Disk space
    • CPU/Memory usage
    • Windows NT Services

Below screen shows the Monitoring Dashboard for a Platform alarm.

monitor-dashboard

What to setup for BizTalk Application monitoring 

Before we move to discuss which artifacts to monitor from BizTalk Application alarms, let’s firstly discuss how to organize your BizTalk Application-oriented alarms. Basically, there are two ways you can set up your BizTalk Application alarms.

The Alarm equals Application approach

When deploying BizTalk applications, it is often handy to create an alarm per BizTalk application for Threshold Monitoring and/or Health Check Notification. The main motivation for this is, that a BizTalk Application is the unit which you will deploy, so setting up alarms around them will help in keeping the overview. For easy reference, you could give the alarms the same name as the BizTalk application it will be monitoring. This can be considered as a bit more technical approach for your monitoring.

The Alarm equals Interface approach

Another approach to set up your BizTalk Application-oriented alarms is around the interfaces which are being processed by BizTalk. Say, your BizTalk environment contains multiple applications of which several their contained artifacts together are part of a Purchase Order. To keep the overview of such an interface, it makes sense to monitor all artifacts related to that interface. This is totally fine and BizTalk360 allows you to set up your alarm in such a way. Ideally, you can give the alarm the name of the interface. This approach is a bit more business-wise way of monitoring and can be helpful if you need to notify your business/functional users in case of any issues.

However, there is a downside to monitoring your BizTalk artifacts around interfaces. Because artifacts from multiple BizTalk Applications need to be monitored, it might be a bit harder, especially when many artifacts are involved, to keep the overview and be sure that you are really monitoring all the relevant artifacts.

No ‘Always Right’ way

Summarizing, there is no ‘always right’ way to set up your BizTalk Application monitoring. Even a ‘hybrid’ setup, which follows both just mentioned practices, of your monitoring is totally fine and not uncommon. In the end, it all depends on what works the best for you and your organization. BizTalk360 gives you all the freedom you need.

Artifacts to be monitored

When setting up monitoring for your BizTalk Applications, you will set it up for your ports and orchestrations of your BizTalk Applications. However, you can also consider the following artefacts for monitoring: 

  • State of Services Instances
    • Suspended (Resumable/Not Resumable)
    • Active
    • Dehydrated
    • Ready to Run
    • Scheduled
  • Endpoints of your Receive/Send ports
    • Web Endpoints
    • Queues (MSMQ, IBM MQ, Azure Service Bus)
    • File shares
    • FTP sites
    • API Apps
    • Logic Apps

Tip: To be able to find out if files/messages are being picked up, you can monitor your File shares, FTP sites, and Queues. Below screen shows an example of monitoring a File share.

file-monitoring-configuration

As a side note, BizTalk Server Best Practices suggest that you have a proper Host configuration in place, where separation takes place based on receiving/sending messages, processing of orchestrations and tracking of processed messages/service instances. However, sometimes Host configuration has been set up for a specific process or BizTalk application. In that case, it is valid to monitor your Host Instance(s) in your BizTalk application-oriented alarm, instead (or on top) of your BizTalk platform alarm.

Advanced Monitoring setup

Besides deciding on how to setup your alarms and which artifacts to monitor, there are a couple of other topics you could consider take benefit of. You can think of:

  • Auto-Correct – the ability to automatically recover artifacts
  • Notification Channels – inform people not just via email but also via different channels
  • Customized Notification Templates – provide additional information or brand the templates

Using such features not just makes your life as an administrator easier, it will also help you in making the best of your investment in BizTalk360. Let’s have a better look at all these options.

Auto-Correct

For state-related artefacts like BizTalk Ports and Orchestrations, Host Instances and SQL Server jobs, you could consider using the Auto-Correct feature, which, after a threshold situation has been met, tries to get the artifacts back to the expected state.

Tip: You can use this feature to automatically Enable your Receive Locations, in case they failed to connect to their endpoints due to a temporary failure!

You can read more about the Auto-Correct feature in this article.

Below screen shows that BizTalk360 will try to automatically try to recover the Host Instances once they are no more in the Started state.

BizTalk360-host-instance

Notification Channels

The default way of BizTalk360 for sending notifications is via email. However, sending notifications does not end there. The product has a number of so-called Notification Channels you can use. Examples are:

  • Slack – send notifications to a Slack channel
  • ServiceNow – have tickets created, directly in ServiceNow
  • Microsoft Teams – send notifications to a Teams channel
  • PowerShell – trigger a PowerShell script to perform an action
  • SMTP – have more control over the way notifications are being sent
  • Webhook – connect to an already existing REST API

Besides this rich set of notification channels, there is also an easy to use API at your convenience. By implementing the API, you can, for example, connect to your ticketing system or other important backend systems.

Below screen shows the information you can provide when using the ServiceNow Notification channel.

ServiceNow

More information about all the Notification channels can be found here.

Customized Notification Templates

BizTalk360 allows you to customize the notifications it sends. This allows you for example to provide some additional information/instructions for the people who receive the notifications. However, it can also be used for branding purposes.

Although some notification information can simply be provided via text input fields, the actual templates are based on XSLT, so some skills with editing such files will be helpful.

More information about customizing the notification templates can be found here.

Conclusion

With this article, we intended to provide you some help and guidelines with setting up your alarms. Both the Documentation portal and our Blog contain many articles on monitoring your BizTalk environment with BizTalk360. However, if you feel you need some more support with setting up your monitoring, feel free to reach out. We are here to help you!

The post Best Practices for BizTalk360 Alarm Configuration appeared first on BizTalk360.