Microsoft Integration Weekly Update: Feb 12, 2018

Microsoft Integration Weekly Update: Feb 12, 2018

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

Feedback

Hope this would be helpful. Please feel free to provide any feedback on this Integration weekly series.

Advertisements

When data in CRM is updated I want to send it to another application

When data in CRM is updated I want to send it to another application

Having worked a lot with Dynamics CRM/365 over the last few years I thought it would be interesting to discuss a common use case and some of the architecture patterns you may consider to implement the solution.

Lets imagine a scenario where the business requirement is as follows:

  • The user will be updating a customers record in Dynamics 365
  • When the user saves the change we need the change to be synchronised with the billing system

Now at this point I am going to deliberately ignore flushing out these requirements too much. Any experiences integration person will now be thinking of a number of functional and non-functional questions they would want to get more information about, but the above is the typical first requirement. We will use this vagueness to allow us to explore some of the considerations when we look at the options that are available to solve the problem. One thing to note is I am going to consider this to be a 1 way interface for this discussion.

Option 1 – CRM Custom Plugin – Synchronous

In option 1 the CRM developer would use the extensibility features of Dynamics. This allows you to write C# code which will execute within the CRM runtime environment as a plugin. With a plugin you can configure when the code will execute. Options include things like:

  • When an entity is updated but before the save is made
  • When the entity is updated but after the save is made
  • As above but on other commands such as created/deleted

The below picture shows what this scenario will look like

Good things:

  • This is probably the quickest way you can get the data from the commit in CRM to the other application
  • This is probably the simplest way you can do this integration with the minimum number of network hops
  • This solution probably only needs the skill set of the CRM developer

Things to consider:

  • You would be very tightly coupling the two applications
  • You would have some potential challenges around error scenarios
    • What happens if the save to the other app works but the save to CRM doesn’t or visa-versa
  • The custom plugin is probably going to block the CRM users thread while it makes the external call which is asking for performance issues
  • You would need to consider if you would do the call to the other application before or after saving the data to CRM
  • You would need to consider where to store the configuration for the plugin
  • There would be error and retry scenarios to consider
  • There would be the typical considerations of tightly coupled apps
    • What if the other app is broken
    • What if it has a service window
  • Errors are likely to bubble up to the end user
  • You will have OOTB (out of the box) CRM plugin tracing diagnostics but this may require some custom code to ensure it logs appropriate diagnostic information

Option 1.5 – CRM Custom Plugin – Asynchronous

In this option the solution is very similar to the above solution with the exception that the developer has chosen to take advantage of the asynchronous system jobs feature in CRM. The plugin that was developed is probably the same code but this time the configuration of the plugin in CRM has indicated that the plugin should be executed out of process from the transaction where the user is saving a change. This means that the commit of the change will trigger a system job which will be added to the processing queue and it will execute the plugin which will send data to the other application.

The below picture illustrates this option.

Good things:

  • The synchronize transaction will no longer block the users thread when they save data
  • The system jobs gives a degree of troubleshooting and retry options if the other system was down compared to option 1
  • This only required CRM developer skills

Things to consider:

  • There may be other things on the processing queue so there is no guarantee how long it will take to synchronize
  • You may get race conditions if another transaction updates the entity and you haven’t appropriately covered these scenarios in your design
    • Also think about the concurrency of system jobs and other plugins
  • I have seen a few times where option 1 is implemented then flipped to option 2 due to performance concerns as a workaround
    • This needs to be thought about upfront
  • You may struggle to control the load on the downstream system
  • Again there is a tight coupling of systems. CRM has explicit knowledge of the other application and a heavy dependency on it
    • What if the app is down
    • What if there are service windows
  • Error scenarios are highly likely and there could be lots of failed jobs

Option 2 – CRM out of the Box Publishing to Azure Service Bus

Option 1 and 1.5 are common ways a CRM developer will attempt to solve the problem. Typically they have a CRM toolset and they try to use a tool from that toolset to solve the problem as bringing in other things was traditionally a big deal.

With the wide adoption of Azure we are starting to see a major shift in this space. Now many Dynamics projects are also including Azure by default in their toolset. This means CRM developers are also gaining experience with tooling on Azure and have a wider set of options available. This allows a shift in the mindset that not everything has to be solved in CRM and actually doing stuff outside of CRM offers many more opportunities to build better solutions while at the same time keeping the CRM implementation pure and focused on its core aim.

In this solution the CRM developer has chosen to add an Azure Service Bus instance to the solution. This means they can use the OOTB plugin (not a custom one) in CRM which will publish messages from CRM to a queue or topic when an entity changes. From here the architecture can choose some other tools to get messages from Service Bus to the destination application. For simplicity in this case I may choose an Azure Function which could allow me to write a simple bit of C# to do the job.

The below solution illustrates this:

Good things:

  • No custom coding in CRM
  • The Service Bus plugin will be much more reliable than the custom one
  • The Service Bus plugin will get a lot of messages out to Service Bus very fast by comparison to the custom plugin in 1.5 which will bottleneck on the downstream system probably
  • Service Bus supports pub/sub so you can plugin routing of messages to other systems
  • The Azure Function could be developed by the CRM developer quite easily with a basic C# skillset
  • Service Bus offers lots of retry capabilities
  • The queue offers a buffer between the applications so there is no dependency between them
  • The function could be paused in downtime so that CRM can keep pumping out changes and they will be loaded when the other app is back online
  • The solution will be pretty cheap, you will pay a small cost for the service bus instance and per execution for the function. Unless you have very high load this should be a cheap option

Things to consider:

  • The key thing to remember here is that the solution is near realtime. It is not an instant synch. In most cases it is likely the sync will happen very quickly but the CRM System Jobs could be one bottleneck if you have lots of changes or jobs in CRM. Also the capability of the downstream system may be a bottleneck so you may need to consider how fast you want to load changes
  • The only bad thing is that there are quite a few moving parts in this solution so you may want to ensure you are using appropriate management and monitoring for the solution. In addition too CRM System jobs you may want to consider Service Bus 360 to manage and monitor your queues and also Application Insights for your Azure Functions

Option 3 – Logic App Integration

In option 3 the developer has chosen to use a Logic App to detect changes in CRM and to push them over to the other application. This means that the CRM solution is very vanilla, it doesn’t even really know that changes are going elsewhere. In the above options a change in CRM triggered a process to push the data elsewhere. In this option the Logic App is outside CRM and is periodically checking for changes and pulling them out.

Typically the Logic App will check every 3 minutes (this is configurable) and it will pull out a collection of changes and then 1 instance of the logic app will be triggered for each change detected.

The logic app will then use an appropriate connector to pass the message to the downstream application.

The below picture shows what this looks like.

Good things:

  • There is nothing to do in CRM
  • The Logic App will need monitoring and managing separate to CRM
  • The Logic App is not part of the CRM developers core skill set, but they are very simple to use so it should be easy to pick this up
  • The Logic App has a lot of features if you run into more advanced scenarios
  • The Logic App has connectors for lots of applications
  • You may be able to develop the solution with no custom code
  • The Logic App has some excellent diagnostics features to help you develop and manage the solution
  • The Logic App has retry and resubmit capabilities
  • The solution will be pretty cheap with no upfront capital cost. You just pay per execution. Unless you have very high load this should be a cheap option
  • This option can also be combined with Service Bus and BizTalk Server for very advanced integration scenarios

Things to consider:

  • Is the polling interval going to be often enough
  • Only the most recent change will be extracted, if a particular row has been updated 3 times since the last trigger you will get the latest stage
  • It may require some more advanced patterns to control the load if the downstream system is a bottleneck. This may be beyond the CRM developers Logic App skills

Option 4 – SSIS Integration

The next option to consider is an ETL based approach using SSIS. This approach is quite common for CRM projects because they often have people with SQL skills. The solution would involve setting up an SSIS capability and then purchasing the 3rd party Kingswaysoft SSIS connectors which includes support for Dynamics.

The solution would then pull out data from CRM via the API using a fetch xml query or OData Query. It would then push the changes to the destination system. Often SSIS would be integrating at database level which is its sweetspot but it does have the capability to call HTTP endpoints and API’s.

Although the diagrams look similar, the big difference between the Logic App approach and SSIS is that SSIS is treating the records as a batch of data which it is attempting to process in bulk. The Logic App is attempting to execute a separate transaction for each row it pulls out from the CRM changes. Each solution has its own way of dealing with errors which makes this comparison slightly more complex, but typically think of the idea of a batch of changes vs individual changes.

In the SSIS solution it is also very common for the solution to include a staging database between the systems where the developer will attempt to create some separation of concern and create deltas to minimize the size of the data being sent to downstream systems.

Good things:

  • You can process a lot of data very quickly
  • Common approach on CRM projects
  • Kingswaysoft product is mature
  • Predominantly configuration based solution
  • Sometimes error scenarios can be complex

Things to consider:

  • Capital cost for 3rd party software and probably maintenance too
  • Need to consider where to host SSIS (Azure VM or On Premise VM) – Cost associated with this
  • Possible license cost for SQL depending on organisation setup
  • You will sync on a schedule, how often does it need to be
    • The more frequent the less data each time
    • Cant be too frequent
  • How will you monitor and schedule the SSIS package

There is no right or wrong answer based on the original 2 line requirement we got, but you can see each solution has a lot to think about.

This emphasises the importance of asking questions and elaborating on the requirements and working out the capabilities of the applications you will integrate with before choosing which option to take. As a general rule I would recommend not to jump too quickly to option 1 or 1.5. As an integration guy we usually frown upon these kind of options because of the way they couple applications and create long term problems even though they might work initially. I think the other 3 options (2-4) will be relatively easy to choose between depending on the requirements elaboration but with option 1 and 1.5 I would only choose these in niche cases and I would do so only with full buy in from your architecture team that you have a justifiable reason for choosing it that has been documented enough to be able to explain later when someone comes along and asks WTF?

One other factor to consider which we didn’t touch on too much above. I have kind of assumed you have an open toolset on todays typical CRM and Azure project. It may also be the case that your project has some constraints which may influence your decision to choose one option over the other. I hope in these cases the above considerations will help you to validate the choice you make or also give you some ammunition if you feel that you should challenge the constraint and consider another option.

The birth of a new SSO Application Configuration Tool for BizTalk Server 2016

The birth of a new SSO Application Configuration Tool for BizTalk Server 2016

Happy to announce the birth of a new SSO Application Configuration tool that will provide the ability to easily add and manage configuration applications, add and manage key-value pairs in the SSO database, as well as securely import and export configuration applications so that they can be deployed to different environments.

SSO Application Configuration Tool for BizTalk Server 2016

BizTalk Server leverages the Enterprise Single Sign-On (SSO) capabilities for securely storing critical information such as secure configuration properties (for example, the proxy user ID, and proxy password) for the BizTalk adapters. Therefore, BizTalk Server requires SSO to work properly. BizTalk Server automatically installs SSO on every computer where you install the BizTalk Server runtime.

But it also can keep your own application configuration data in SSO database, let say the usual configurations that we normally keep in a configuration file (“app.config”)). If you’ve been in the BizTalk world long enough, you’ve probably faced this challenge or need and until 2009 there wasn’t an easy way to archive that and Richard Seroter’s BizTalk SSO Configuration Data Storage Tool was the go tool to store and manage Single Sign-On (SSO) applications – this is still a valid tool and if you rebuild the code in the last version of BizTalk Server it still works perfectly.

In mid-2009 Microsoft released an MMC snap-in to tackle this exact issue: SSO Configuration Application MMC Snap-In provides the ability to add and manage applications, add and manage key-value pairs in the SSO database, as well as import and export configuration applications so that they can be deployed to different environments. It wasn’t nor is it the perfect tool in my opinion since it as several UI limitations but it worked perfectly until… a new version of BizTalk Server was released.

Unfortunately, this tool will not work properly at least from BizTalk Server 2013 forward. At first sight, it seems that everything is working properly but when you try to create a key-value pair you will see that nothing happens and no key is created.

At the time I published a hotfix for the tool:

And M.R.Ashwin Prabhu published the same hotfix for BizTalk Server 2016: BizTalk Server 2016: Fix for SSO Configuration Application MMC Snap-In.

In part, these hotfixes solved the issue but I recently realized that the tool even with the hotfix doesn’t work properly in multiple environments and to try to uninstall the Microsoft snap-in after the hotfix is “installed” it is a nightmare.

Again, Richard Seroter tool is a great tool but is not fully compatible with Microsoft tool and in some parts, I liked that tool. So me and my team decided to “recreate” and improve SSO Application Configuration and the result is this:

  • Fully resizable windows (you will understand if you are a BizTalk Developer);
  • You can securely export and import Application configurations and it is compatible with MSFT SSO snap-in;
  • You can duplicate Applications (copy and past);
  • You can rename Applications;
  • You can easily add new key-values without the need to always perform a right click and select new key option;
  • You can easily add edit key-values without the need to always perform a double-click to open a new window;
  • You can refresh the Applications tree view
  • You can configure you system settings

SSO Application Configuration Tool for BizTalk Server 2016: Settings

  • You can search!

THIS TOOL IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.

You can download BizTalk Server SSO Application Configuration Tool for BizTalk Server 2016 from:
BizTalk Server SSO Application Configuration Tool for BizTalk Server 2016 (45 KB)
Microsoft | TechNet Gallery

Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

BizTalk360 Customer Support — 2017 Achievement

BizTalk360 Customer Support — 2017 Achievement

BizTalk360 Customer Support Statistics

The stats are in. Our continuous striving for improvement and dedication have resulted in some really positive numbers in BizTalk360 Customer Support. Here’s some statistics that we are happy to share. These statistics are taken from data provided by our customer support platform – Freshdesk.

  • 5514 customer queries addressed in 2017
  • Tickets ranging across technical support, licensing and sales enquiries
  • The busiest month for the team was the month of May 2017. We received about 1300 support tickets.
  • We managed to respond to 97% of the tickets and resolve 84% of the tickets within the SLA
  • We resolved 73% of the tickets with just one response to the customer
  • Our top performing support agent was Praveena Jayanarayanan
  • We received support tickets on
    • Email
    • Support Portal
    • Feedback widget
  • 86% of our customers rated our support as awesome and were happy with the support offered to them

What’s coming in 2018?

Here’s what is expected from BizTalk360 in 2018 –

  1. Launch of two new products – Atomic Scope and Document360
  2. INTEGRATE 2018 – June 4 – 6 at etc.Venues, 155 Bishopsgate, London. Registrations are open. Grab your tickets today.

BizTalk360 Customer Support

Video

Here’s a short video on our customer support statistics.

Author: Rochelle Saldanha

Rochelle Saldanha is currently working in the Customer Support & Client Relationship Teams at BizTalk360. She loves travelling and watching movies.

Why did we build Azure API Apps Monitoring in BizTalk360?

Why did we build Azure API Apps Monitoring in BizTalk360?

Azure API Apps Monitoring BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Azure API Apps is a modern way of building web services hosted in Microsoft Azure. Out of the box it provides capabilities such as security, load balancing, auto-scaling, and automated management. You can also take advantage of its DevOps capabilities such as continuous deployment from VSTS, GitHub, Docker Hub, and other sources, package management, staging environments, custom domain, and SSL certificates.

In the context of Azure Logic Apps connectors, Azure API Apps plays a vital role. Sometimes there may not be an out of the box connector to an external system and you may need to wrap a web service on top of that external system to create a custom Azure Logic Apps connector. This article explains how you can easily Create a custom connector for a web API.

The Azure API App can contain multiple endpoints to be able to perform multiple actions like creating an order or requesting the status of an order.

The Azure API Apps are powerful building blocks for your hybrid integration scenarios. As these API Apps can be of critical importance, you should be well aware of the status of these API Apps.

What are the current challenges?

We have already seen why Azure API Apps are important building blocks and can be of vital importance for your integration scenarios. So, whether you are consuming the external Azure API Apps from other parties or you are exposing your own API Apps, depending on the importance of the Azure API App, it is equally important to constantly be aware of the state of the Azure API Apps and their endpoints.

The main challenge at the moment is the management and monitoring of Azure API Apps in isolation via the Azure Portal. You’ll miss the end to end context of the importance of the API App in your integration scenario.

For example, you might have an order processing system that uses few BizTalk Receive Locations, Orchestrations, Send Ports and few Azure API Apps connected via BizTalk WCF-REST/HTTP adapters and are part of the same end-to-end solution. If the Azure API App is down or returning errors, then your end-to-end purchase order solution is broken. It’s important to monitor the API Apps and understand the end to end context.

We address this challenge in BizTalk360 by providing a unified platform to monitor and manage both BizTalk artifacts and some of the key Azure artifacts which are important from the integration point of view.

How BizTalk360 solves this problem?

Depending on whether you want to monitor your own Azure API Apps or the API Apps which are exposed by others, BizTalk360 offers multiple methods to setup monitoring. This is due to the fact that to be able to monitor whether API Apps are enabled, you need access to the Azure subscription which contains these API Apps. In case of API Apps which are exposed by others, you won’t have access to that Azure subscription, so you’ll only be able to monitor the HTTP Endpoints of the API Apps.

To monitor API Apps from within BizTalk360, the product offers the following features:

  • Azure API App of others – Web Endpoint monitoring
  • Your own Azure API Apps – Azure API App monitoring

Web Endpoint Monitoring

If you are simply consuming the Azure API Apps from external parties, then they are not any different from a standard REST/HTTP web services. In this case, the BizTalk360 web endpoints monitoring solution will be sufficient.  The features and capabilities of BizTalk360 Web Endpoint monitoring are described in the Why did we build Web Endpoints Monitoring in BizTalk360? article.

Azure API Apps Monitoring

If you are owning the Azure API Apps, then BizTalk360 provides a rich set of capabilities to monitor the Azure API Apps like auto-discovery of multiple endpoints and granular control on monitoring setup.

Monitoring API Apps from BizTalk360 consists of the following types of monitoring:

  • State Monitoring – monitor whether the API Apps are Enabled/Disabled
  • Endpoint Monitoring – monitor if the HTTP Endpoints of the API App are accessible

State Monitoring API Apps
From BizTalk360, you can monitor the state of API Apps. An API App can have the following 2 states:

  • Enabled – the API App is active and can receive requests
  • Disabled – the API App is disabled and cannot receive requests

With BizTalk360, you can monitor for both states and of course, we want to provide a monitoring experience which as seamless as possible. After you have selected the correct alarm and Azure subscription, it is just a matter of selecting the Azure API Apps and select the required expected state from the drop-down box. Once that’s done, BizTalk360 immediately starts to monitor the API App(s).

Azure API Apps Monitoring in BizTalk360Endpoint Monitoring
Azure API Apps basically are REST services which use JSON for data exchange. These REST services are the actual endpoints of the API Apps, which can be called by other systems. Each Azure API App has at least one endpoint but can contain many more endpoints. 

BizTalk360 has a quite extended feature set for Web Endpoint Monitoring, and it comes with advanced capabilities like adding custom GET parameters and HTTP headers:

  • Custom GET parameters – provide required/optional parameters
  • HTTP headers – provide additional HTTP headers (for example for authorization)

When the Endpoint sends a response, you can perform multiple checks to find out whether the Endpoint is in good health. You can check, for example:

  • HTTP Return codes – like HTTP 200 must exist, else throw warning/error
  • Existing and Non Existing keywords – like ‘success’ must exist in the response message
  • Response times – if it takes longer than, say, 5 seconds before the response is received, something might be wrong

HTTP Return codes
You can add multiple monitoring rules for HTTP return codes. This allows you to check for the existence, but also the non-existence of certain HTTP return codes. Besides that, you can also determine whether a Warning or an Error will have to be thrown in case the monitoring rules have not been met.

Existing and Non-Existing keywords
This is a very strong capability, as it allows you to check whether certain keywords exist or do not exist. As the response message might be in different formats, BizTalk360 allows you to check for the following formats and throw a Warning/Error based on the outcome of the monitoring rule:

  • Plain Text – search for keywords which should/should not exist
  • XML – enter an XPath query and the expected value which should/should not exist
  • JSON – enter an XPath query and the expected value which should/should not exist

For each format, you can search for just one keyword or multiple keywords to (not) exist.

Response times
Bad response times of endpoints might indicate that the endpoints (or the system/service behind it) are under stress. So monitoring on response times helps you to be aware of such situations.

As all of the above mentioned checks can be combined with each other, this provides very strong monitoring capabilities.

BizTalk360 API Apps Monitoring Response Alert

Multiple Azure Subscriptions
You might have multiple Azure subscriptions which contain different Azure API Apps. You can register all the relevant subscription and attach them to corresponding BizTalk Server environments within BizTalk360. This provides a seamless management/monitoring experience.

Conclusion

With Azure API Apps monitoring, we make it easy to constantly be aware of the well-being of your API Apps, without having to check that yourself manually. Besides monitoring whether the Azure API App is Enabled, we bring strong features for monitoring the actual HTTP endpoints.

You can read more about Azure API App monitoring and the other Azure services for which BizTalk360 provides support.

Get started with a Free Trial today!

If you want to monitor your Azure API Apps or have other BizTalk Monitoring related challenges, why not give BizTalk360 a try! It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments. Get started with the free 30 days trial.

BizTalk360-Free-Trial

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.

Why did we build Azure Logic Apps Monitoring in BizTalk360?

Why did we build Azure Logic Apps Monitoring in BizTalk360?

Azure Logic Apps Monitoring BizTalk360

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Traditionally, a lot of system integration happened on-premise. When integration with partners or external systems was the need of the hour, this was achieved using SOAP/REST/HTTP/FTP/SFTP etc.

With the rise and adaption of cloud technologies like Amazon AWS and Microsoft Azure, cloud solutions became more and more capable to support scenarios where on-premise systems needed to interface to the on-premise systems of your partner(s) or with SaaS solutions like Office365 and SalesForce.

Microsoft is positioning Azure Logic Apps as the integration platform in the cloud mainly for connecting SaaS based solutions like Dynamics 365, Salesforce, ServiceNow, Workday etc.

Since the introduction of the Azure Logic Apps adapter for BizTalk Server 2016 and Azure Gateway, it is possible to integrate your on-premise systems via BizTalk Server with Azure Logic Apps seamlessly. This opens up great possibilities — it opens up some 300+ SaaS connectors that are available to Azure Logic Apps that can be utilized from BizTalk Server. Also, vice versa you can make the 100’s of external SaaS systems to talk to your on-premise BizTalk Server.

So it becomes very important to have a management/monitoring platform that’s capable of supporting both on-premise BizTalk Server and Azure Logic Apps from a single place.

What are the current challenges?

Azure Logic Apps provide a great way to extend your integration scenarios and capabilities to the cloud. However, from an operational and monitoring perspective, the Azure Portal is yet another portal the BizTalk Administrators need to be aware of and fit in their daily tasks.

So, besides their on-premises oriented BizTalk Server monitoring, organizations now also need to monitor their Hybrid solutions (which involve Azure Logic Apps) as events might occur they want to know about.

Think of, for example, somebody could have disabled your Azure Logic App  (that’s part of your end-to-end integration scenario), which prevents the Azure Logic Apps from being executed. Also, executions of the Azure Logic Apps might fail or run very slow, indicating that some other system might be under stress.

As it would take too much time to regularly check this kind of scenarios to take place, an administrator just wants to be informed automatically when these kind of events occur. You can use some of the monitoring features of Azure Portal, but you’ll not get the end to end visibility and it will operate on silo without giving the full context.

How BizTalk360 solves this problem?

At BizTalk360, we are closely following the trends in the integration landscape. We can clearly spot the necessity of using Azure Logic Apps in enterprise integration scenario along with BizTalk Server.

Because of this, we brought the support for Azure Logic Apps with BizTalk360. In addition to Azure Logic Apps, we also support other relevant integration technologies in Azure like Azure Service Bus, Azure API Apps and Azure Integration Account with BizTalk360.

In this article, we will just focus on how you can monitor your Azure Logic Apps from BizTalk360. Azure Logic Apps Monitoring with BizTalk360 consists of the following two parts:

  • State Monitoring – Monitor whether the Logic Apps are in the expected state
  • Data Monitoring – Monitor whether the executions of Logic Apps take place as expected

State Monitoring of Logic Apps

A Logic App can have the following two states:

  • Enabled – an instance of the Logic App can be started by means of a trigger
  • Disabled – the Logic App cannot be instantiated by means of a trigger

Azure Logic Apps monitoring with BizTalk360 consists of checking whether the state of a Logic App reflects the Expected state. In most cases, a Logic App will have to be in the Enabled state, but there might be scenarios where you want to monitor for the Logic App to be in the Disabled state, which is often referred to as Negative Monitoring. BizTalk360 allows you to monitor for both states.

Azure Logic Apps Monitoring in BizTalk360

Auto Correct
BizTalk360 has developed the Auto Correct feature for state-bound artifact monitoring. The feature allows you to let BizTalk360 try to bring the state bound artifact back to the expected state, once the artifact gets into the undesired state. This feature exists for the following artifact types:

  • Receive Locations, Orchestrations and Send Ports
  • Host Instances
  • Windows NT Services
  • SQL Server jobs
  • Logic Apps

As you can see from the list above, Azure Logic Apps monitoring can be decorated with Auto Correct. In case the Azure Logic App can’t be get back to the expected state, BizTalk360 will try to bring it to the desired state. You can also configure a maximum number of retries in the BizTalk360 portal. This is to prevent BizTalk360 to constantly keep on trying to bring the Azure Logic App back to the expected state, in case of some problem.

Multiple Azure Subscriptions
You might have multiple Azure subscriptions which contain different Azure Logic Apps. You can register all the relevant subscription and attach them to corresponding BizTalk Server environments within BizTalk360. This provides a seamless management/monitoring experience.

Data Monitoring of Logic Apps

With Data Monitoring of Azure Logic Apps, you can monitor for example whether:

  • expected number of runs are met in a certain time frame
  • a number of failures occur during a certain time frame
  • number of billable trigger executions in a certain time frame

The above scenarios are just an example – there are many more combinations.

Data Monitoring of Logic Apps will be discussed in more detail in another article, but here we just wanted to reveal a bit of that feature.

Conclusion

With the rise of Hybrid integration solutions, the job of BizTalk administrators has extended to the (Azure) cloud. BizTalk360 helps these administrators by providing operational and monitoring support for a number of Azure services, thereby taking away the need to constantly have to check the Azure portal to be aware of the health of these services.

In this article, we discussed the state monitoring capability of Azure Logic Apps. If you want to read more about the other Azure services BizTalk360 supports, just follow the articles in the Why did we built series.

You can read more on setting up Azure Logic Apps monitoring on our Documentation portal.

Try BizTalk360 Free for 30 days today!

Download and try BizTalk360 on your own environments free for 30 days. Installation will not take more than 5-10 minutes.

try biztalk360 for free

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.

Why did we build SQL Jobs Monitoring for BizTalk Environment?

Why did we build SQL Jobs Monitoring for BizTalk Environment?

BizTalk360 SQL Jobs Monitoring

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

As you will probably be aware of, BizTalk Server heavily relies on SQL Server databases for storing, for example, the configuration of BizTalk artifacts and the state of in-flight and completed processes. After installing and configuring BizTalk Server, few databases are created. The most important databases are the Configuration database, the MessageBox database and the Tracking database, but depending on the configured features, other databases might be created for BAM, BRE and SSO.

Further, there are a number of SQL jobs that gets created when you install and configure BizTalk Server. These SQL jobs are maintenance jobs and are of vital importance for the health of the BizTalk Server environment. The BizTalk SQL Jobs take care of making backups of the BizTalk databases, transfer tracking data from the MessageBox database to the Tracking database and all kind of cleaning up data from the databases.

Without having these jobs enabled (in cases disabled, not all the jobs are supposed to run) and properly running, it is just a matter of time, before you will get into trouble. This kind of trouble might lead to interruptions of the business process(es) which are being processed through BizTalk. For that reason, it is important to be aware of the (health) status of the SQL Jobs.

What are the current challenges?

There are two basic challenges at the moment with the current default tooling.

  • There is no out of the box support for monitoring SQL Jobs and keeping an eye on the health
  • Accessing SQL jobs for viewing, changing state, etc requires access to SQL Server

Although for experienced BizTalk administrators, it is obvious that they need to be well aware of the health of their BizTalk databases, for organizations with less experience on administrating BizTalk, this might come as a surprise.

We have seen that BizTalk administrators did not have access or insufficient access to the SQL Server management tools. This could especially be true, in case BizTalk Server has to share its SQL Server instance with databases of other systems, which is not a good idea anyway.

Even if the administrator has access to the SQL Management tools, having to manually monitor these SQL jobs is a time consuming and adds up to all the other assignments of the BizTalk administrator.

How BizTalk360 solves this problem?

As we, at BizTalk360, understand the importance of the SQL jobs, we brought both operational and monitoring features related to these jobs. In this section, we will discuss the Monitoring features.

BizTalk360 allows you to set up monitoring on SQL jobs seamlessly. You need to simply register the name of the SQL Server instances where your SQL jobs are running via the Settings -> Monitoring -> SQL Server Instances and all the SQL jobs will be ready for monitoring configuration.

BizTalk360 Add SQL servers for monitoring

Once configured, BizTalk360 enables you to achieve the following:

  • Monitor the Expected Job State (both enabled and disabled)
  • Configure Auto-correct
  • Monitor the Expected Last Job Run State

Below, all these capabilities will be discussed.

Monitor list of SQL jobs BizTalk360

Monitor the Expected Job State

With this kind of monitoring, you can check whether the SQL job is in the correct state. In most cases, the expected state will be Enabled, but in certain cases, the expected state must be Disabled. BizTalk360 allows you to monitor for both states.

The possibility of negative monitoring comes in handy for BizTalk Server, as the MessageBox_Message_Cleanup_BizTalkMsgBoxDb jobs need to be in disabled state. This job works like a stored procedure and used internally by other jobs. It should not be manually started by a DBA or BizTalk Administrator.

Configure Auto-Correct

The Auto-Correct feature is an extension of Expected State Monitoring. Basically, it allows you to bring State-bound artifacts back to the expected state after they reached the wrong state.

Auto-correction works 2 ways. If an artifact, say a SQL Job, is expected to be in the Enabled state but reached the Disabled state, the Auto-Correct feature will try to bring the SQL job back to the  Enabled state.

However, when a SQL job is expected to be in the Disabled state (like the MessageBox_Message_Cleanup_BizTalkMsgBoxDb job) but got Enabled, the Auto-Correct feature will try to bring the job back to the Disabled state.

This helps to maintain the state of the SQL jobs in the desired state even if there is a human error.

This feature works not just for SQL Server jobs, but also for:

  • Receive Locations, Orchestrations and Send Ports
  • Host Instances
  • Windows NT Services
  • Azure Logic Apps

Monitor the Expected Last Job Run State

This allows you to check whether the job runs successfully. Once you have setup monitoring for Expected Last Run state, you will be notified in case the Last Run State was unsuccessful.

Conclusion

By bringing SQL job monitoring, including the auto-correct feature, we think that this will make the life of both BizTalk administrators and SQL Server DBA’s a bit easier.

If you want to read more about SQL job monitoring, please read this article.

Get started with a Free Trial today!

If you are struggling with all the above-mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity of your own BizTalk Environments. Get started with the free 30 days trial.

BizTalk360-Free-Trial

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.

Microsoft Integration Weekly Update: Feb 5, 2018

Microsoft Integration Weekly Update: Feb 5, 2018

Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?

Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.

If you want to receive these updates weekly, then don’t forget to Subscribe!

Feedback

Hope this would be helpful. Please feel free to provide any feedback on this Integration weekly series.

Advertisements

Why did we build Web Endpoints Monitoring in BizTalk360?

Why did we build Web Endpoints Monitoring in BizTalk360?

BizTalk360 Web Endpoints Monitoring

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

SOAP & REST based Web Services are one of the most used technologies in integration scenarios. In the last few years, the need for HTTP based communication with external vendors/systems has increased significantly. Pretty much it became a norm to expose the API’s of the system for external consumption. Examples: Salesforce API, Microsoft Graph API for Office365 and so on. Also, the companies want to reuse their legacy system as much as possible internally and they expose them as internal web services.

BizTalk Server offers great support for HTTP(S) based web services, both on the Receive and the Send side of interfaces.

When you are integrating multiple endpoints via HTTP(s), it is important to know whether the web services are available, responsive, returning expected values etc., to be able to communicate with them and to support the business process.

What are the current challenges?

The use of web services (web endpoints) comes into play on both receive side and send side in BizTalk Server. Both comes with its own significance and it’s important to monitor them.

Receive Side web services

BizTalk Server allows you to publish schemas and orchestrations as WCF services, which are then published in the IIS web server on the BizTalk Server. In case, you have multiple BizTalk servers in the same BizTalk group, you can set up an NLB (Network Load Balancing) cluster to achieve Load Balancing and High Availability.

The published WCF services are associated with Receive Locations. As a result, if something is wrong with a particular WCF service, for example, the Application Pool in IIS is down, the Receive Location will automatically be set to Disabled.

If the Receive Location/Port is down, then it won’t accept any incoming messages via this channel. Therefore, it’s important to make sure it’s up and running all the time.

Send Side web services

On the outgoing side, you configure either HTTP, SOAP, REST, BizTalk Send port to connect to external web endpoints. If something is wrong with the external web endpoints, the state of the Send Port will remain the same inspite of the state of the external web endpoint.

So, to be able to know the actual state of the web endpoint, the well-being of the endpoint will have to be monitored and checked. This helps to be sure that when a real request to the web endpoint is made, that the actual endpoint can be accessed and the request is properly being processed.

When a web service is down and requests are being sent to that web service, the failing requests will lead to suspended messages in BizTalk, resulting in an interruption of the business process and some work for the BizTalk administrator to have the issue solved and to resume the suspended messages.

An external web service can be down for a variety of reasons.

  • Internal web server/service issues – you might get for example the famous, though non-informative, HTTP 500 return codes
  • Certificate issues – expired certificates, incorrectly configured or unaccepted client certificates
  • Firewall issues – web server/service cannot be contacted at all due to wrongly configured firewall rules
  • Flooding – there’s more work than the web server/service can handle

All this kind of errors will lead to unavailability of the web endpoint and interruptions of the business process. This is exactly what we want to prevent, and as it is impossible for a BizTalk administrator to manually monitor web endpoints 24/7, it is way more efficient to have monitoring in place.

How BizTalk360 solves this problem?

For monitoring the physical HTTP web endpoints, BizTalk360 offers Advanced Web Endpoint Monitoring. Once set up, this feature will periodically fire requests against the web service which is being monitored and report any issues identified.

The feature is quite rich and offers, amongst others, support of:

    • HTTP and HTTPS web services (REST/SOAP)
    • Using Gateway proxies
    • Providing credentials
    • GET and POST methods
    • Multiple content types like
      • Text
      • XML
      • JSON
    • providing custom
      • HTTP headers
      • GET parameters
      • POST payload

Advanced Web Endpoints Monitoring BizTalk360

These features allow you to monitor for example HTTP return codes (200 is okay, 500 is warning/error). But you can also check for certain keywords to exist in the XML/JSON response message, by entering XPath/JSON queries.
You can even monitor for response times of the web service, as this might be an indication that the web service is under stress.

BizTalk360 Web Endpoint Monitoring - Endpoint details

So with the Advanced Web Endpoint Monitoring feature of BizTalk360, you will be able to cover pretty much all the web endpoint monitoring scenarios, whether these web services reside inside your organization or outside and whether these services use XML or JSON.

Conclusion

To underline the importance of web endpoint monitoring, let’s shortly discuss the often forgotten concept of Chain Monitoring.
It is a bit of a human behavior, to just focus on your own work and systems, meanwhile forgetting the impact of your work on other systems.

A simple example: a Technical/Functional Application administrator is responsible for the administration of a CRM system. On a day to day basis, he is very busy with functional questions, which results in not always having the overview of the health of the web services of his CRM system, while these services received data on a regular base from an ERP system, via BizTalk Server.

It was with this kind of scenario where we have experienced that the BizTalk administrator who was using BizTalk360 and had set up Web Endpoint Monitoring, was notified by BizTalk360 of problems with a web endpoint. The administrator contacted the owner of the web endpoint, who was not yet aware of the issue! The issue was solved by the owner of the web service and no interruption of the business process took place.

The above example was based on a scenario which really took place in one of our customer and a very clear example of the purpose of Advanced Web Endpoint monitoring.

If you want to read more on the nitty/gritty of the Advanced Web Endpoint Monitoring feature in BizTalk360, please read this article.

Get started with a Free Trial today!

If you are struggling with the above mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check this feature on your own BizTalk Environments. Get started with the free 30 days trial.

try biztalk360 for free

Author: Lex Hegt

Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.

Why did we build ESB Exception Management Portal in BizTalk360?

Why did we build ESB Exception Management Portal in BizTalk360?

BizTalk360 ESB Exception Management Portal

This blog is a part of the series of blog articles we are publishing on the topic “Why we built XYZ feature in BizTalk360”. Read the main article here.

Why do we need this feature?

Microsoft shipped the ESB Toolkit back in 2007-2008 that extends the functionality of Microsoft BizTalk Server to provide a range of new capabilities for building SOA/ESB applications that incorporate things like itinerary based invocation for lightweight service composition without using Orchestrations, dynamic resolution of endpoints maps, Web services, exception management and reporting. They also provided a sample “ESB Exception Management” web application along with the toolkit.

The ESB Toolkit created a love/hate relationship with a lot of BizTalk Server customers even though it added a lot of value addition to BizTalk Server (as an add-on) while in some cases it simply complicated the solution.

The one thing that everyone loved about ESB Toolkit is the Exception Management framework and the sample web application that shipped with the Toolkit. Since the Exception Management part addressed two important challenges in an integration solution in an end-to-end Exception management framework, visualizing it in a web portal and ability to edit/resubmit failed messages, which were missing in BizTalk Server core.

What are the current challenges?

Sample Portal: The Exception Management Framework itself is a stable offering and fully supported by Microsoft. However, the portal that shipped with the ESB Toolkit is a “sample web application” built on top of the ESB Exception Database. It’s not fully supported by Microsoft and it’s kind of a half-baked solution with a lot of bugs.

Difficult to Install and Configure: Typically it takes few hours to one or two days to install and configure the ESB Exception management portal. It’s not maintained or updated as required over the years, and it uses certain components like older versions of “.NET Logging Application Blocks” that makes server level changes and affects your main BizTalk Solutions.

Missing Functionalities: As the ESB Exception Portal was shipped as a simple sample web application, it’s not matured and misses some important capabilities like “Bulk Edit/Resubmit” and functional alerting. Bulk edit/resubmit is very important since when there is a failure in your environment, you’ll typically have 10’s-100’s of failed messages for the same reason and you wanted to take bulk action.

The other important missing aspect is restricting users by permission and auditing, an example – you probably do not want all of your support people to have the ability to edit and resubmit messages. Even if they do, you need to have the traceability of who performed that action.

How BizTalk360 solves this problem?

We wanted to address the challenges highlighted above and also wanted to give a rich unified tooling experience for BizTalk Administrators. Hence we built the ESB Exception Management portal within BizTalk360 from the ground up. All you need to configure the ESB Exception management within BizTalk360 is to simply provide the connection string to your ESB exception database relevant to your BizTalk Environment. That’s it! You are set (takes about 3 minutes).

ESB Exception Management Portal BizTalk360

Since we built the ESB Exception Management portal from the ground up, we have thought through all the challenges in the sample portal and addressed them. It comes with the following set of features

  • Rich ESB Exception Dashboard (utilizing our powerful customizable dashboard framework)
  • Full Search/Filter/Display of exception details
  • Edit Resubmit – both single and multiple records
  • Download Messages – you can either download or email exception messages directly from the portal.
  • Integrated Knowledgebase – you can associate a Knowledgebase article with known exceptions
  • Security – ability to restrict user either to the full ESB portal section or allow them to do specific tasks like edit/resubmit
  • Governance & Auditing – all the core activities like editing/resubmit by the users are audited.
  • Rich Functional alerting – ex: if there are over 30 errors matching a specific error code in an application alert the administrator.

One of the other important objectives of BizTalk360 is to reduce the number of different tools the BizTalk Administrator has to use to support their BizTalk solution – tools such as Admin Console, BAM Portal, ESB Portal, SQL Management Studio, Perfmon and so on. This makes them totally unproductive and switch context between different applications. It is also difficult to on-board new people and bring them up to speed.

By bringing the ESB Exception Management portal within BizTalk360, we eliminate the need to use the sample ESB portal that comes with the toolkit.

Get started with a Free Trial today!

If you are struggling with all the above mentioned challenges, why not give BizTalk360 a try. It takes about 10 minutes to install on your BizTalk environments and you can witness and check the security and productivity on your own BizTalk Environments. Get started with the free 30 days trial.

BizTalk360-Free-Trial

Author: Saravana Kumar

Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.