by Saravana Kumar | Jan 15, 2018 | BizTalk Community Blogs via Syndication
BizTalk360 is pretty much a matured product now from its inception back in May 2011. Every year, we consistently released between 5 to 10 releases. In each release, we target anywhere from 3-5 good (useful) features and constant improvements, bug fixes, and usability enhancements. Over the last 6 years, we made 2 major/complete user interface revamp to give the product a face-lift in terms of user experience. First, when we moved the platform from Microsoft SilverLight to HTML5 (version 7.0) and the second when we revamped the complete user interface (version 8.0) to categorize the UI clearly into BizTalk Server Operations, Monitoring and Analytics sections.
Every feature we add to the product is carefully selected. We have clear objectives for each one of them. Before starting to work on a new feature, we ask the following questions like why are we building a particular feature, what problem we are trying to solve, and whether we can make BizTalk Server operations more streamlined/productive/secure.
In this new blog series “Why did we build XYZ feature in BizTalk360”, I wanted to explain clearly the reasoning behind each one of the core feature in the product, why we built it, what business problem it solves when it comes to running your BizTalk environment smoothly.
Here is the list of topics we are going to cover in this series. You can book mark this article, we will add links to each article as and when we publish the blog.
Operations
- Why did we build a web-based BizTalk Admin Console?
- Why did we build User Access Policy?
- Why did we build Auditing & Governance?
- Why did we build Advanced Event Viewer?
- Why did we build Custom SQL Queries?
- Why did we build Team Knowledgebase?
- Why did we build Operations Dashboard?
- Why did we build Search and Action on BizTalk Artifacts?
- Why did we build Advanced Tracking Manager?
- Why did we build Business Rules Composer?
- Why did we build ESB Management Portal?
- Why did we build EDI Dashboard?
- Why did we build Message Box Queries (MBV)/ BizTalk Health Monitor (BHM) Integration?
- Why did we build Graphical Message Flow viewer?
- Why did we build dedicated dashboard pages for BizTalk and SQL Servers?
- Why did we expose all of our internal REST API’s?
General BizTalk Server Monitoring
- Why did we build FTP/SFTP Monitoring?
- Why did we build MSMQ, IBM MQ, Azure Service Bus Queue Monitoring?
- Why did we build Web Endpoints Monitoring?
- Why did we build Database SQL Queries Monitoring?
- Why did we build Message Box Queries (MBV)/ BizTalk Health Monitor (BHM) Monitoring?
- Why did we build Azure Logic Apps Monitoring?
- Why did we build Azure API Apps Monitoring?
- Why did we build Receive Locations, Send Ports and Orchestrations Monitoring?
- Why did we build CPU, Memory and Disk Monitoring on BizTalk and SQL Servers?
- Why did we build BizTalk Host Instances Monitoring?
- Why did we build Auto Healing capability in Monitoring?
- Why did we build Windows Event Log Monitoring?
- Why did we build Window NT Service Monitoring?
- Why did we build Export and Import capability for Monitoring Configuration?
- Why did we build SQL Jobs Monitoring?
- Why did we build Notification History capability for BizTalk Monitoring?
- Why did we build Custom Notification Channels in Monitoring?
- Why did we build Webhook Channel out of the box?
- Why did we build ServiceNow Integration out of the box?
- Why did we build Slack Integration out of the box?
Data/Transaction Monitoring in BizTalk Server
- Why did we build Data Monitoring for Receive Locations and Send Ports?
- Why did we build Data Monitoring for BizTalk Message Box Data?
- Why did we build Data Monitoring for BizTalk Tracking Data?
- Why did we build Data Monitoring for BizTalk EDI Data?
- Why did we build Data Monitoring for BizTalk Business Activity Monitoring Data?
- Why did we build Data Monitoring for Azure Logic Apps Data?
- Why did we build a dedicated Transaction/Process Monitoring Dashboard?
Analytics
- Why did we build Analytics Dashboard?
- Why did we build Messaging Patterns Visualizer?
- Why did we build Throttling Analyser?
- Why did we build New Relic Integration?
Author: Saravana Kumar
Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.
View all posts by Saravana Kumar
by Gautam | Jan 15, 2018 | BizTalk Community Blogs via Syndication
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?
Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
If you want to receive these updates weekly, then don’t forget to Subscribe!
Feedback
Hope this would be helpful. Please feel free to let me know your feedback on the Integration weekly series.
by Saravana Kumar | Jan 10, 2018 | BizTalk Community Blogs via Syndication
Artificial Intelligence (AI), Machine Learning (ML), Big Data, and BlockChain are some of the buzz words we hear more and more in recent times.
As a technology enthusiast, I’m personally fascinated by some of the technology advances that are happening in these areas and it’s mind-blowing to see the pace at which they are advancing. At the same time, it becomes almost a commodity these days and is available for everyone to take advantage of it with a fraction of the cost — thanks to all the public cloud providers like Microsoft, Amazon, Google, and IBM.
Can AI & ML be used for BizTalk Server Monitoring?
These days customers expect every software product to have some level of AI & ML. In this article, let’s take a closer look at whether AI & ML can be used for BizTalk Server Monitoring and what are the practical challenges.
About a year ago, we looked into what AI & ML can do for BizTalk Server Monitoring in BizTalk360. I was not completely convinced that we can magically solve the BizTalk Server Monitoring problems using machine learning alone at that stage. Here are some of the scenarios that strike me and the answer to these challenges are not straightforward.
I don’t want to sound too negative about using AI and ML for the monitoring solution. There are certain use cases where AI and ML based solution will be far superior to manual solutions (check out the last section of this article). However, I just wanted to highlight some real-world challenges from my 15+ years of experience working with BizTalk Server.
What is Machine Learning?
Let’s first understand the basics. Machine learning is a class of algorithms that can learn from and make predictions on data. Generally speaking, the more data you have, the better the outcome for machine learning techniques. Machine learning doesn’t require users to set explicit rules like “if this, then that.” It will make that determination on its own, based on the data and algorithms. There are three things here, the data, compute power and types of algorithms (models) which helps you to predict the outcome.
Data is the King
The quality of data you have determines the quality of judgment in machine learning. So typically, you’ll start collecting all the available data metrics over a period of time to plot a predictive graph. You will start having a baseline (higher and lower bound for a period) as acceptable values for each measurable metric.
If there is any drastic change in the underlying conditions, then the whole algorithm needs to relearn.
Is time factor vs. message volume a business reality?
Let’s start with an example. In BizTalk (or Middleware world), the volume of messages passing through the system is crucial and one of the important things to monitor. We encounter the following scenario quite often. If you are using an AI based system, the period marked in red (in the below image) will be triggered as a violation and you’ll get alerted since it’s deviating from the baseline significantly.
The system basically works on the intelligence of the data it collected over the past weeks/months and blindly assumes that the volume of messages is getting deviated from the baseline volume.
In theory, this looks fantastic. However, in reality, this could be a big problem in a middleware solution. Situations like this where the volume of messages are either higher than normal or lower than normal are common on a middleware platform and we call this a “floodgate or drain scenario“.
We can probably keep training the system saying this is a known volume, but how often can you do it. So, you’ll get the false alert first, the admin person needs to teach the system that it is the expected volume, the system understands it next time. However, what happens if the floodgate happens in another time instance, again the system admin needs to correct it.
In case your business has unpredictable burst scenarios frequently, the AI system will struggle to cope. It’s one of those things, like a demo scenario where everything runs smoothly, the system performs fantastic, but the real world is far from the demo scenario.
How about changes in hardware configuration?
I mentioned in the beginning “data is the king” in an AI based system. It’s like teaching a child — the child keeps learning every day based on what they see, what they hear, what they experience and so on. In a similar way, the AI system (if implemented well) will keep learning all the new patterns that emerge in the system, keep learning and tune itself. But, there is a challenge when the underlying platform changes dramatically then all the old learning might be completely invalid.
Let’s take an example! The production BizTalk Environment was set up and tuned for six months or so, there were a lot of manual override corrections for AI learning and everything is working fine. Suddenly, there is a business requirement saying they are expecting a huge increase in the volume of transactions and they wanted to increase the hardware capacity. This could simply be adding more servers to the BizTalk group (BizTalk or SQL Servers), increasing the processor capacity like higher performing CPU’s, more memory on BizTalk or SQL servers, changing the SAN disks and so on.
Now all the learning and tuning you have done for the last 6 months is completely wasted and the system has to be re-trained for a period of another 6 months to get to the same level with all human intensive manual tuning.
No one can simply judge what level of performance the new system configuration is going to provide; it can only be learned gradually.
How about BizTalk Server upgrade scenarios?
Another common and unavoidable scenario is the BizTalk Server upgrade. Example: Moving from BizTalk Server 2013 or 2013 R2 to BizTalk Server 2016. In fact, this is something that will be forced on you, since the mainstream support for every version will elapse after 5 years (10 years extended support).
Let’s say you are using BizTalk Server 2013 and depending on an AI powered monitoring solution. You are using the platform for 4 years and then decided to migrate to BizTalk Server 2016. In most of the cases, you will not just migrate your BizTalk Server; you’ll also migrate the OS and database from Windows Server from 2012 to 2016, SQL Server from 2012 to 2016.
This will change the complete underlying platform and make all of your AI learning so far for 4+ years completely invalid and you need to start from scratch. Can you afford it? By the time the team who had put together and tuned the solution in the first place might have left the company.
How about Christmas and Black Friday?
Pretty much every business has a special seasonal period in a year where things go upside down. In retail industries, it’s Black Friday and Christmas period and in Financial services industries, it’s typically month-end or year-end closing dates. Typically, most of these high-end businesses will have a system frozen/lock-down period where no changes are allowed.
When I used to work for a financial services client, the whole of March is locked down for 5th April, the year-end deadline in the UK. Almost 50-60% of the personal investments will be made during this period since everyone wants to take advantage of their personal tax allowances and customers typically leave it until the last minute.
How will an AI based monitoring system work under these circumstances? I can see there are only 2 choices — one, you educate the system saying it’s a special period, for this, it must have seen the scenarios earlier i.e at least one or two years period. In the first year it would have raised a false alarm all over the place, someone might have tuned it saying this is the expected behavior and in the second year, it will validate the condition and finally you might have a system that understands this unusual period and load provided nothing changed (like our hardware change) during this period.
The second option is simple — you simply turn off the monitoring for this period to avoid getting flooded with alarms.
How about manual intervention and performance tuning?
A platform like BizTalk Environment is never a “set once and run it forever” platform. It will require regular manual interventions either for performance tuning or for operational reasons. Example: for performance reasons, you may be re-configuring the SQL Server database Auto-growth parameter, SAN configuration, purging and archiving settings for tracking and BAM databases and so on.
In a similar way, for operations reasons you may be periodically turning on and off certain things like BizTalk Send Ports, Receive Locations, Orchestrations, BizTalk Host instances etc for various reasons. A more concrete example, let’s say one of your external partners’ FTP server is down for maintenance. You will turn off the FTP/SFTP Send Port or Receive Location to that partner for time until the problem gets resolved.
In situations like these, what will happen to the AI based monitoring system? As I mentioned before “data is the king”, any changes to the system will have an impact on the AI interpretation of past data and it will start sensing wrong thresholds. The AI system needs to forget the past learning and relearn the new configuration (for few weeks/months), or someone needs to manually overwrite.
How about fine control on monitoring metrics?
In a platform like BizTalk Server, monitoring just the high-level metrics like CPU%, Memory, Message Volume, Event Log error counts etc may not be sufficient. Let’s take an example scenario. You are integrating a bunch of external REST/HTTP or SOAP based web services with your BizTalk Orchestration. It’s important for you to keep an eye on the health of those external web endpoints like return status codes, response times, looking for specific value in the response JSON/XML content, you might need to verify multiple conditions in the response, you may need to wait for x-seconds/minutes before marking the system as down and so on.
Let’s take one more example. You may want to watch the growth of certain tables in SQL Server. Example: “select count(*) from spool” in the BizTalk MessageBox database. This is very specific and only a human BizTalk admin will know the importance of such custom metric.
This level of fine control on certain systems/metric will be challenging to achieve using an AI based monitoring system, since AI systems work on the volume of data and predictability. It might predict that the external system is down and responding slower, but it cannot have the granularity of control required as explained here.
Where is your data?
Now comes the million-dollar question — where is my data going to reside? As I mentioned repeatedly, “data is the king” in any AI based system to deliver the promises. It needs to understand various patterns that emerge out of historical data to apply its intelligent algorithms. So where is the BizTalk Server metrics data going to reside, is it on-premise or in the cloud?
Even though it’s technically possible to collect and keep the data on-premise and use the AI system on-premise, most of the AI based systems are well suited for the cloud. Mainly because of the three important factors, requires huge data store, requires scalable compute power and finally, the algorithms will require constant tweaking and tuning.
If the data is hosted on-premise, then maintaining such a system with constantly updating algorithms become even challenging.
If the data is hosted in the cloud, it introduces regulatory challenges in industries like Healthcare, Financial Services etc. even though the AI based providers can claim only metadata information gets transmitted, how confident the customers will be.
Why Microsoft does not have an AI powered Monitoring offering?
It is surprising to see a company like Microsoft who are market leaders in machine learning and artificial intelligence not having any AI powered monitoring solutions. None of their monitoring products like Azure OMS (Operations Management Suite), Azure Log Analytics, Azure Application Insight, SCOM (System Center Operations Manager) etc., claim they can automatically detect and fix the problem using AI.
I highlighted Microsoft here, however, it’s true for other big players like Google, Amazon, and IBM (Watson) as well. They have an AI platform, but none of them focus on AI powered monitoring solution.
I’m sure it’s just a matter of time before these platforms get matured and we will see AI powered offering (there are already few bits and pieces), but it’s not going to be 100% fully automated replacement anytime soon. Still, you need that product and business knowledge to have a solid monitoring solution.
Is it all negative with AI and BizTalk Server Monitoring?
No, not at all. It’s important to embrace new technologies and move things forward. However, it’s also important to understand the limitations of such systems from a practical business point of view. There are some scenarios well suited for AI based predictive analysis and notification like identifying security vulnerabilities, unusual threat analysis like DOS attack, face & voice recognition, some element of monitoring which are static like general message volume, response times and so on.
Another area where I can see a real use case of an AI based solution is when it’s working on top of your business data. Example: volume of purchase orders you receive from various vendors over various periods. The AI system can learn about the volume and are not impacted by underlying platform changes (and keep re-learning).
In case of BizTalk Server Monitoring, I don’t see a clear answer to the challenges I highlighted above. If your platform is stable, nothing changes (like no hardware changes, no tuning, no new deployments, no burst scenarios), then AI will have a good chance of base lining your platform and detect and notify variations from baseline. But again, this is not practical for a live BizTalk Server Environment.
How does BizTalk360 solve these problems?
Being a market leader for the past 7 years in BizTalk Server Monitoring space, we consistently improve the product and add value to our customers. We invested heavily in a lot of areas which are practical and usable for enterprise BizTalk customers TODAY. We focused primarily on customer feedback (check out our BizTalk360 feedback portal) and most of our feature offering were customer driven.
I’ve written an article explaining how BizTalk360 addresses the challenges highlighted in this article to provide a robust monitoring solution for your BizTalk Server environments. You can download it here.
Author: Saravana Kumar
Saravana Kumar is the Founder and CTO of BizTalk360, an enterprise software that acts as an all-in-one solution for better administration, operation, support and monitoring of Microsoft BizTalk Server environments.
View all posts by Saravana Kumar
by Sandro Pereira | Jan 8, 2018 | BizTalk Community Blogs via Syndication
We cannot rely on documentation if they exist, to be accurate, especially regarding the status of the machines present in the environment that tell us what is installed on the machine or not. This script follows the sequence of scripts that I release in the past to check what Cumulative Updates where installed in the machines.
However, Microsoft introduces a new concept within BizTalk Server 2016, it calls Feature Packs.
BizTalk Server 2016 will use Feature Pack (FP) approach to providing new functionalities to the product at a faster pace. Now new features (or at least non-breaking features) will be delivered when they’re ready we no longer need to wait 2 years for the next major release of the product to have new features!
BizTalk Server uses the feature pack to provide improvements, features, and closer integration with Azure. Feature Pack 1 extends functionality in key areas, such as deployment, analytics, and runtime. – https://msdn.microsoft.com/en-us/library/mt800834.aspx
Feature Pack’s will be available for Software Assurance customers running BizTalk Server 2016 Developer and Enterprise editions or customers running BizTalk Server 2016 in Azure under an Enterprise Agreement
With this, people in charge of monitoring and maintaining BizTalk Server environments will have to not only check if all the Cumulative Updates (nevertheless, this is the most critical operation) but if their organization decide to install FP, they also need to check if and what feature packs are installed in which machine.
Although it seems simple, this operation is just or more painful to perform as the cumulative updates.
Of course, there are some ways to check that, for example:
- you can do it manually by checking “Control PanelProgramsPrograms and Features” and then view the “View Installed Updates”, however, this can be a very annoying task and sometimes time-consuming.
- you can use Windows Registry but still if you only want to check what FPs that are installed this will be an annoying and time-consuming task.
Probably there are other ways, nevertheless, I just want a quick and very easy way, because this is a basic and very simple task, to know what are the BizTalk Server 2016 Feature Packs installed like:
Microsoft BizTalk Server 2016 Feature Pack 1 is installed
Microsoft BizTalk Server 2016 Feature Pack 2 is installed
How to check what BizTalk Server 2016 Feature Packs
So how can we easily automate tasks? and reuse them whenever necessary and at the same time saving significant time for other tasks?
Using PowerShell is a good option. Windows PowerShell is a Windows command-line shell designed especially for system administrators and can be used by BizTalk administrators to help them in automating repetitive tasks or tasks that are time-consuming to perform manually.
This is a simple script that allows you to configure the template name of the feature packs, that will may change from version to version (FP1, FP2, …), and will give you the list of all Feature Packs installed in your machine:
$keys = Get-ChildItem -Path Registry::'HKEY_LOCAL_MACHINESOFTWAREWOW6432NodeMicrosoftE-Business Servers Updates'
#...
foreach ($key in $keys)
{
if($findF1 -eq $true)
{
break
}
foreach ($Property in $key.Property)
{
if ($Property -like '*Microsoft BizTalk Server 2016 Feature Pack 1*')
{
$findF1 = 1
Write-Host 'Microsoft BizTalk Server 2016 Feature Pack 1 is installed'
break
}
}
}
#...
foreach ($Property in $key.Property)
{
if ($Property -like '*Microsoft BizTalk Server 2016 Feature Update 2*')
{
$findF2 = 1
Write-Host 'Microsoft BizTalk Server 2016 Feature Pack 2 is installed'
break
}
}
#...
if(($findF1 -eq $false) -And ($findF2 -eq $false))
{
Write-Host 'Microsoft BizTalk Server 2016 Feature Pack is not installed'
}
THIS SQL SCRIPT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND.
Author: Sandro Pereira
Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.
He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira
by Gautam | Jan 8, 2018 | BizTalk Community Blogs via Syndication
Do you feel difficult to keep up to date on all the frequent updates and announcements in the Microsoft Integration platform?
Integration weekly update can be your solution. It’s a weekly update on the topics related to Integration – enterprise integration, robust & scalable messaging capabilities and Citizen Integration capabilities empowered by Microsoft platform to deliver value to the business.
If you want to receive these updates weekly, then don’t forget to Subscribe!
Feedback
Hope this would be helpful. Please feel free to let me know your feedback on the Integration weekly series.
by Lex Hegt | Jan 7, 2018 | BizTalk Community Blogs via Syndication
A new year has just began and at BizTalk360 we are excited to find out what 2018 will bring us as a company. We are delighted that we will release more versions of our flagship product BizTalk360, further evolve ServiceBus360 and bring new products like Atomic Scope and Document360.
Year in review
In this article however, we want to look back at what 2017 brought from a Microsoft Integration space perspective. Although we did not see a new version of BizTalk Server, we did see new features arrive for BizTalk Server 2016, by means of the two feature packs which have been released. Of course, we have also seen new CU’s appear for the currently supported BizTalk Server versions, which are 2013, 2013R2 and 2016.
Note however, that in case you are on BizTalk Server 2013 or 2013 R2, mainstream support will end July 10th, this year! So you should consider upgrading to BizTalk Server 2016. As migrating to another BizTalk platform is not something to take lightly, Microsoft offers the BizTalk Migration tool. Read more about this tool later in this article.
An important move Microsoft made last year, was embracing the Open Source community. Of course, also in the Logic Apps and related technologies a lot has happened. Keep on reading, when you want to read about most important developments in Microsoft Integration space.
BizTalk Server 2016 Feature Packs
The concept of Feature Packs is new to BizTalk Server. The idea of bringing Feature Packs is to bring new features once they are ready, instead of having to wait until the next major release.
For BizTalk Server, these packs can be installed on top of (only) BizTalk Server 2016 and bring new functionality, without breaking existing functionality. As far as we know, there are no plans to bring Feature Packs for other BizTalk version than 2016.
Last year Microsoft brought 2 Feature Packs for BizTalk Server 2016. Let’s have a look at both.
Feature Pack 1
This first Feature pack was released on April 26th. It could only be installed on top of the base version of BizTalk Server 2016 (3.12.774.0), so you could not install it after you have applied any CU’s for BizTalk Server 2016. This issue his been fixed on September 21st, when Microsoft released a version of FP1 which was compatible with Cumulative Update 3 (KB4039664), the most recent CU at that time.
Feature Pack 1 added the following features to BizTalk Server 2016:
Deployment
- Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS)
- New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support
Analytics
- Application Insights – Tap into the power of Azure for tracking valuable application performance and to better understand your workflow
- Leverage operational data – View operational data from anywhere and with any device using Power BI
Runtime
- Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns
- Advanced Scheduling – Set up advanced schedules for Receive Locations using the new capabilities with Feature Pack 1
More information on Feature Pack 1 can be found here: https://blogs.biztalk360.com/microsoft-biztalk-server-2016-feature-pack-1/
Download Feature Pack 1 here: https://www.microsoft.com/en-us/download/details.aspx?id=55100
Feature Pack 2
This pack was released on November 21st and can be installed immediately on top of the BizTalk Server 2016 Cumulative Updates 1, 2 and 3 and Feature Pack 1.
In this Feature Pack, the following features were brought:
Deployment and Administration
- Application Lifecycle Management with VSTS – Using Visual Studio Team Services, you can define multi-server deployments of BizTalk Server 2016, and then maintain those systems throughout the application lifecycle.
- Backup to Azure Blob Storage – When deploying BizTalk Server to Azure VMs, you can backup BizTalk Server databases to Azure blob storage.
Server Runtime
- Adapter for Service Bus v2 – When using the Service Bus Adapter, you can utilize Azure Service Bus Premium for enterprise-scale workloads.
- Transport Layer Security 1.2 – Securely deploy BizTalk Server using industry-standard TLS 1.2 authentication and encryption.
- API Management – Publish Orchestration endpoints using Azure API Management, enabling organizations to publish APIs to external, partner and internal developers to unlock the potential of their data and services.
- Event Hubs – Using the new Event Hub Adapter, BizTalk Server can send and receive messages with Azure Event Hubs, where BizTalk Server can function as both an event publisher and subscriber, as part of a new Azure cloud-based event-driven application.
Analytics and Reporting
- Event Hubs – Send BizTalk Server tracking data to Azure Event Hubs, a hyper-scale telemetry ingestion service that collects, transforms, and stores millions of events.
- Application Insights – When preparing BizTalk Server to send tracking data to Application Insights, released in FP1, you can use the new Azure sign-in dialog to simplify configuration and named instances of SQL Server.
More information on Feature Pack 2 can be found here: https://blogs.biztalk360.com/microsoft-biztalk-server-2016-feature-pack-2-released/
Download Feature Pack 2 here: https://www.microsoft.com/en-us/download/details.aspx?id=56262
Cumulative updates
In 2017, Microsoft also brought a number of Cumulative Updates for the different versions of BizTalk Server. See below for an overview which contains links to the download pages:
BizTalk Server 2013
NOTE: this version is going out of mainstream support July 10th, 2018
July 28th, 2017 – Cumulative Update package 6 (build 3.10.359.2) – KB4032678
BizTalk Server 2013 R2
NOTE: this version is going out of mainstream support July 10th, 2018
May 8th, 2017 – Cumulative Update package 6 (build 3.11.280.2) – KB4020020
October 12, 2017 – Cumulative Update package 7 (build 3.11.285.2) – KB4038891
BizTalk Server 2016
January 26, 2017 – Cumulative Update package 1 (build 3.12.796.2) – KB3208238
May 26, 2017 – Cumulative Update package 2 (build 3.12.807.2) – KB4021095
September 1, 2017 – Cumulative Update package 3 (build 3.12.815.2) – KB4039664
Embracing Open Source
In September Microsoft announced that BizTalk Server has joined the Open Source community. This is motivated by the increased commitment that Microsoft wants to provide a better product to their customers.
So far, Microsoft has open sourced the following:
- all schemas which are shipped with the product. These are over 3500 files, which are divided over the following standards:
- EANCOM
- EDIFACT
- HIPAA
- HIX
- X12
- the BizTalk Migration tool. This tool is developed to simplify the following migrations:
- BizTalk Server 2010 to BizTalk 2013R2
- BizTalk Server 2013R2 to BizTalk 2016For Host Integration Server
For Host Integration Server (HIS), Microsoft has provided a PowerShell script that helps creating .hidx files, which are used by the HIS runtime to communicate with the backend systems.
The repository which contains above mentioned stuff, can be found at GitHub:
https://github.com/Microsoft/Integration/
Microsoft’s intention is to continue their open source commitment. Community contributions are more than welcome and can contain artifacts like Schemas, Samples, Adapters, pipeline components or anything else you feel the community can take advantage of.
Azure Integration updates
It is amazing to see what’s all happening on Azure Integration! The Logic Apps team regularly post their updates on YouTube. At our blog, we publish recaps of these updates. Below is a picture of the entire Microsoft Pro Integration team on a bright, sunny Friday morning at Gold Coast, Australia, during #MSAUIgnite.
See below, just a couple of all the developments which have taken place:
Logic App-adapter – this adapter enables you to receive or send messages from BizTalk Server 2016 to your Logic Apps, via the On-Premise Data Gateway
Numerous improvements to the Logic Apps Designer – just a few which are worth to mention: editing Azure Functions directly from Logic Apps, generating JSON schema from sample payload, Parallel creation in the designer, use of Webhooks in Foreach loops, use of variables, batch processing, advanced scheduling of Logic App execution
Cognitive Vision APIs – to be able to use the OCR capability to understand the characters and text within an image
Azure Event Grid – The newest and hottest kid in town; technical preview version was released by Microsoft on August 16th
Azure Event Grid Connectors for Logic Apps – Of course the Event Grid offering is accompanied with a connector
More than 80 new SaaS adapters – few of the added connectors are: Azure Table Storage, Cognitive Services, Outlook – webhook trigger, SQL – dynamic schema for stored procedures, Azure Container Instances , Pitney Bowes Data Validation, Log Analytics Data Collector, ServiceNow, Skype for Business, LinkedIn, Azure Security Center Trigger, MySQL (R/W)
You can follow the Logic Apps team yourself, via their YouTube channel: http://aka.ms/logicappslive
The most recent recap of the Logic Apps update can be found here:
https://blogs.biztalk360.com/azure-logic-apps-monthly-update-november-2017/
What can we expect in 2018?
Of course Microsoft will keep on bringing new and exciting features and improvements. What we currently know is that we can expect a new feature pack for BizTalk Server 2016, being Feature Pack 3. But also on the Azure side we can expect new stuff and enhancements of existing features.
BizTalk Server 2016 Feature Pack 3
During the first half year, we can expect the release of BizTalk Server 2016 Feature Pack 3. What we currently know, is that we can expect multiple Office 365 adapters, with the following capabilities:
- Send and receive messages using Office 365 e-mail
- Set and update appointments using Office 365 schedules
- Define people and groups using Office 365 contacts
Further, we can expect:
- Web authentication – authenticate with Azure Active Directory and OAuth using Microsoft Enterprise Single Sign-On
- Advanced Scheduling – Set up recurrence on BizTalk Server receive locations with greater precision using hours, minutes, second
Besides that, Microsoft will keep on bringing CU’s for the supported BizTalk Server versions, including the final CU’s for BizTalk 2013 and 2013R2.
Further, Microsoft is also working on BizTalk Server vNext, so we can also expect CTP’s (Community Technology Previews), which will lead to a new release of BizTalk Server and more Feature Packs. However, a timeline towards a new release of BizTalk Server is not yet made public.
Azure Integration
In Azure, we can expect of course more adapters. The Logic Apps team has already mentioned amongst others: SOAP Passthrough, Office365 Excel, K2, Kronos, Citrix ShareFile and PostgreSQL.
We’ll also see more improvements in the designer. Features like Complex Conditions within the designer, Configurable lifetime and Degrees of parallelism for split-on and request triggers have already been mentioned by the team.
All in all, we can conclude that 2017 has been an awesome year for Microsoft Integration and we can expect that 2018 will be a fantastic year as well!
INTEGRATE 2018
You can get to know more about these updates directly from the Microsoft Product Group at INTEGRATE 2018. The dates and the venue for the event has been finalized. The website will be going live by this week. Stay tuned!
Author: Lex Hegt
Lex Hegt works in the IT sector for more than 25 years, mainly in roles as developer and administrator. He works with BizTalk since BizTalk Server 2004. Currently he is a Technical Lead at BizTalk360.
View all posts by Lex Hegt
by Sandro Pereira | Jan 4, 2018 | BizTalk Community Blogs via Syndication
If there is a year when it will be difficult to select the best moments, this could be one of them. There are so many that’s hard to begin describing what happened in 2017?
In personal terms, as I mention in my last post, the family is growing this time with a baby boy and I’m currently moving to a brand new a bigger house… so, exciting times! In professional terms, it was a year full of projects and traveling. But it was a year that my blog got a new and modern look that made it more professional… and I even got a sponsor, BizTalk360, which allows me to pay the costs of hosting the blog in my own domain.
Despite all of this changes in my blog, I still was able to publish 63 new posts in my blog the country that most visited is still the United States, followed by India, United Kingdom, and Canada.
And I had the opportunity, as a speaker, to present 16 sessions! A new personal record! 2 more than last year
So once again let’s look back and reflect what happened in 2017. In a few words, it was an interesting and challenging year both personal and professional!
My selected Highlights of 2017
Sharing my book with the community
Once again, sharing my book with the community is a moment that I want to highlight and I have no words to thank you!
First speaking event in the USA
I have been doing a lot of conferences in Europe in the last few years, but I have never had the opportunity to do it on the other side of the Atlantic Ocean, not because of lack of opportunities, but mainly because I have not been able to consolidate my professional agenda with this speaking opportunities. Fortunately, this year I was able to do it and be present at INTEGRATE USA in Redmond, presenting a session about BizTalk Server performance and optimizations:
And I have to say, you guys make me fill special! What amazing reception and feedback you provide me! THANKS!
TUGA IT 2017
Once again, I was in charge of organizing the Integration track at TUGA IT event, one of the major biggest events in Portugal that targets several distinct technologic areas like SharePoint and Office 365, Azure Infrastructure, Data Platform, Programming and Integration (BizTalk, IoT, and Azure) and much more.
Being responsible for the Integration track, I was able to invite some of my good friends to speak at this event: Nino Crudele, Steef-Jan Wiggers, Tomasso Groenendijk, Ricardo Torre and Eldert Grootenboer.
One of the top contributors to Microsoft Documentation
BizTalk Server documentation is now on DOCS and everyone can contribute and improve the existing documentation of which I’m glad to be one of the non-Microsoft top contributors.
A year of speaking engagements…
Another year full of speaking engagements… I had the opportunity to be present 16 sessions!!, speaking about different topics: BizTalk Server, Logic Apps, API Management, App Service, Azure Functions, Microsoft Flow or PowerApps, in multiple events (the UK, USA, and Portugal) and some of them through live meetings. New personal record!
Some of them I already mentioned above but here is the complete list of speaking engagements that I did in 2017:
- XXVIII Porto.Data Event: SharePoint integration: How can PowerApps and Microsoft Flow give power to your SharePoint users
- DevScope Academy: LogicApps, SmartDocumentor, DynamicCRM & PowerBI
- SQL Saturday #583 Lisbon: BizTalk Server and SQL Server Together: For Developers & Admins
- Oporto Global Integration Bootcamp: THE SPEAKER NIGHTMARE: Eval Forms & OCR & Logic Apps & Power BI
- Global Azure Bootcamp 2017 Lisbon: Halt and Catch Fire: Azure Functions
- Cloud Pro: Processing Feedback Evaluations (paper) automagically with SmartDocumentor OCR, Logic Apps, Azure Functions & Power BI
- Cloud Pro: Introduction to PowerApps and Microsoft Flow
- TUGA IT 2017: How can PowerApps and Microsoft Flow allows your power users to quickly build Enterprise Mobile Apps
- TUGA IT 2017: THE SPEAKER NIGHTMARE: Eval Forms & OCR & Logic Apps & Power BI
- Integration User Group: BizTalk Server: Teach me something new about Flat Files (or not)
- INTEGRATE 2017 London: BizTalk Server Fast & Loud
- Integration User Group: BizTalk Server: Teach me something new about Flat Files (or not) – Part 2
- Integration User Group: The Speaker Nightmare: Eval Forms, OCR, Logic Apps & Power BI
- INTEGRATE 2017 USA: BizTalk Server Fast & Loud
- Integration User Group: How can Power Apps and Microsoft Flow allow your Power Users to quickly build Enterprise Mobile Apps
- XXXVI Porto.Data Event: The Speaker Nightmare: Eval Forms, OCR, Logic Apps & Power BI
For me is always a great honor and joy to be participating in these type of events, either as speaker or attendee, be with old friends, meet new people and share knowledge!
My 3 favorite’s posts
My 3 top posts that I enjoyed writing last year was:
For all of you…
Thanks for following me, for reading my blog and I promise that I will continue to share my knowledge during 2018.
Author: Sandro Pereira
Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc.
He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.
View all posts by Sandro Pereira
by Sriram Hariharan | Jan 3, 2018 | BizTalk Community Blogs via Syndication
In 2017, we released 4 major releases and 3 patch releases of BizTalk360 with various new capabilities, usability improvements and bug fixes. We wanted to summarize the 15 best features we released in 2017 in this blog
1. EDI Capabilities
We enhanced the Electronic Data Interchange (EDI) capabilities of BizTalk360 with new features such as —
- EDI Reporting Manager
- EDI Reporting Dashboard
- EDI Functional Acknowledgement Status
EDI Reporting Manager
With the EDI reporting manager capability, you can turn on/off reporting for each agreement in a single click. This becomes a very tedious process when performed using the BizTalk Administration Console (BAC), especially when there are multiple EDI parties and agreements. Users can also perform bulk enable/disable operations on the NRR configuration. Administrators have the option of adding the Host Partner information that is required to configure the host party for NRR configurations.
EDI Reporting Dashboard
We added a rich out-of-the-box EDI dashboard that will aggregate the different EDI transactions to help business users to visualize the EDI data in a better way. The different widgets available in the EDI Reporting Dashboard are EDI Interchange Aggregation widgets, EDI Transaction Set aggregation widgets, AS2 messaging aggregation reports.
We also added few new aggregations (as widgets) that are not available in the BizTalk Administration Console such as Interchange Count by Agreement Name (Top 10), Interchange count by Partner Name (Top 10), Transaction count by ACK Status (Filtered by partner id), AS2 messaging aggregation reports.
EDI Functional Acknowledgement Status
In any common B2B scenario, when a message is sent from the source to destination, an ACK will be sent back to the source with the status of the message (say, a technical acknowledgment TA1 or 997 functional ACK). Say, a message is sent from source to destination and the TA1 ACK is successfully received, but the 997 functional ACK is not received. With BizTalk Administration Console, it is not easy to identify the reason why the 997 functional ACK was not received by the sender. BizTalk360 brings in the capability to allow administrators to easily view the status of functional ACK within the UI and set up data monitoring alerts to get notified of negative functional ACKs.
2. ESB Dashboard
We added a new ESB Dashboard that will allow users to aggregate ESB reports into a single graphical dashboard. There are total 13 widgets that will help users understand their ESB integrations and to better analyse the data to improve performance. The widgets are categorized into 3 groups — Fault code widgets (based on application, service or error type), Fault code over time widgets, Itinerary Widgets.
3. Notification Channels (Webhook, Microsoft Teams)
We added two new notification channels — Webhook and Microsoft Teams in addition to the existing notification channels such as Email, HP Operations Manager, Slack, ServiceNow. The new webhook notification channel allows you to send structured alert notifications (JSON format) to any REST endpoints whenever there is a monitoring violation.
Additionally, users can now connect a Microsoft Teams channel to their respective BizTalk360 alarms with the proper Webhook URL and receive alert notifications directly to their Teams channel.
4. Azure Logic Apps (Operations, Data Monitoring)
We brought in the capability into BizTalk360 to be able to manage and monitor the Azure Logic Apps. Therefore, if you are using only the Logic Apps offering from Azure, you need not switch to Azure Portal to manage your Logic App. You can perform all operations on the Logic App such as Enable, Disable, Run Trigger, and Delete. The Trigger history and Run history details are available in both graphical and grid view.
In addition to Logic App Operational capabilities, users can configure Data monitoring to monitor and trigger alerts based on historical events.
5. IBM MQ Monitoring
In addition to MSMQ and Azure Service Bus Queue monitoring capabilities, we added the IBM MQ monitoring capability. IBM MQ monitoring supports both MQSC and MQS based configurations; you can monitor for the following 4 parameters – queue depth, backout queue depth, queue usage percentage and backout queue percentage.
6. Folder Monitoring, FTP/FTPS/SFTP Monitoring
With BizTalk360, we wanted to make monitoring of BizTalk resources very seamless in order to differentiate BizTalk360 from other general purpose monitoring solutions. We introduced the capability of Folder monitoring where users get to view all the configured receive locations and send ports that make use of the FILE Adapter and configure monitoring on those specific folders.
Similar to Folder monitoring, BizTalk360 also supports the monitoring of FTP, FTPS and SFTP adapters. The main purpose of configuring monitoring of these adapters is to monitor for data pile up!
7. Infrastructure Settings – New Additions
We added new options to the Infrastructure Settings — BizTalk and SQL NT Services and SQL Server Jobs. Users can perform operations like Start, Stop, Restart, Pause and Resume on the BizTalk and SQL NT Services. Similarly, users can manage SQL Server Jobs (start and stop SQL jobs) in BizTalk360.
8. BizTalk Server Licensing Information Widget
We added a new widget on the Home Dashboard screen that will display key information of the BizTalk server such as BizTalk edition, Server type, Processor, Manufacturer, Number of BizTalk Servers, Total number of license needed and estimated pricing.
For more details about BizTalk Server Licensing, please refer this blog article.
9. BizTalk Health Monitor Integration
BizTalk360 now supports BizTalk Health Monitor (BHM), previously known as Message Box Viewer (MBV). For many years, we had support for Message Box Viewer where we periodically run MBV in configured environments, parse and store the result and display it in the BizTalk360 web console.
10. BizTalk Server Host Throttling Monitoring & Availability Monitoring
The ability to monitor throttling is not something new in BizTalk360. With Throttling Analyser, you can easily identify when your BizTalk environment is throttling and in what particular state. With the new host throttling capability, you can visualize and get alerted whenever your BizTalk environment is in the throttling state.
In addition to host throttling monitoring, we also added the capability to monitor the availability of your BizTalk server, say, all your BizTalk servers are up and running. If something goes wrong with one of your BizTalk server, you will automatically get notified about this through the notification channel.
11. Email Template
You can now set up different email templates for different alarms — customize the colour of the email body, font, logo, background, footer background etc. You also have the option to create an email template with an in-built XSLT validator with preview options. We have provided an option for you to choose the notification template in a light/dark theme.
12. Azure Integration Account
In an effort to support Hybrid Integration, we have integrated Azure Integration Account within BizTalk360. This allows users to access Maps, Schemas, Certificates, Parties and Agreements stored in Azure Integration account in configured Azure subscriptions all withing BizTalk360 instead of logging into the Azure portal.
13. BizTalk Reports (Analytics)
We added this new feature “BizTalk Reports” under Analytics that offers capabilities for users to create schedules and generate PDF reports of performance metrics at specific time intervals (eg., daily/weekly/monthly). The PDF will contain the key performance metrics of BizTalk, SQL and IIS servers, messaging performance and so on.
14. Event Log Data Monitoring
In addition to the existing data monitoring configurations, users can now monitor the event logs from BizTalk and SQL server in a particular environment. Users can take advantage of the rich querying capabilities of BizTalk360 Data Monitoring to monitor the event sources from different servers in a single place.
15. BAM Related Activities and Documents
We enhanced the existing Business Activity Monitoring (BAM) feature to accommodate the Related Activities and Related Documents functionality similar to BAM portal. We brought in this feature due to the repeated requests from our customers. You can view the detailed information for the request, view the related activities for the request, download the related documents and view the response activity.
Author: Sriram Hariharan
Sriram Hariharan is the Senior Technical and Content Writer at BizTalk360. He has over 9 years of experience working as documentation specialist for different products and domains. Writing is his passion and he believes in the following quote – “As wings are for an aircraft, a technical document is for a product — be it a product document, user guide, or release notes”.
View all posts by Sriram Hariharan
by Eldert Grootenboer | Jan 2, 2018 | BizTalk Community Blogs via Syndication
Another year has gone by, and looking back, it has been an amazing year for me. The year started with becoming a MVP, for which I am thankful and honored. I want to once again thank everyone who has helped me reaching this astounding accomplishment, with special thanks going out to my buddy Steef-Jan, who has been like a mentor to me.
Became a MVP
Looking back on 2017, it was also a year of lots of traveling, including trips either for speaking or for attending conferences. Most of these trips have been together with some great friends, like Steef-Jan, Tomasso, Sandro, Nino and many more. Last year I have visited Gold Coast, Brisbane, Sydney, Melbourne, Gothenborg, Stockholm, Ghent, Lisbon, London, Oslo, Mechelen, Seattle, Bellevue and Dutch Harbor.
Lisbon
Oslo
Dutch Harbor
I love to speak, and in 2017 I got the chance to speak at a lot of different locations and events. This includes speaking at conferences, user groups and webcasts, which allowed me to spread knowledge, meet old friends, make new friends, and interact with people from all around the world. With the rise of Azure, I have spoken on various subjects around its great technologies and services, for example IoT, Logic Apps, Event Grid, Bot Framework and many more.
IoT
Bot Framework
Last year we did the first Global Integration Bootcamp, for which I am one of the founders and board members. With a total of 12 countries, 16 locations, 55 speakers and over 650 attendees, it was a great success!
Global Integration Bootcamp
Over the past year I have once again written several blog posts, both on my own blog as well as a guest writer for BizTalk360. I love sharing this way, and have gotten a lot of great feedback from all of you, so you can certainly expect more to come.
Blog Posts
Not only have I been blogging, but I also wrote another ebook together with Steef-Jan, Tomasso and Rob, covering modern integration, as an addition to the Global Integration Bootcamp.
Modern Integration eBook
Of course, becoming a MVP was the most amazing reward I have gotten last year, but I am also very honored to have received the BizTalk360 Product Specialist award for the third time in a row.
BizTalk360 Product Specialist
I also have gotten my TOGAF certification, continuing my path into architecture.
TOGAF 9 Certified
As I said, 2017 has been a great year, having spent time with old friends, made many new friends, traveled all over the world, and accomplishing many personal goals. For 2018, I intend to continue doing this, and keep doing all these things I enjoy most. The year 2018 is already starting great, with ny first trip will be coming up in two weeks, as I honored to be speaking in Helsinki at IglooConf.
IglooConf
We are also working hard on another Global Integration Bootcamp, so make sure to attend as well! I am in the works of planning more sessions and trips at the moment as well, so if you ever see me around, come and say hello. You are the ones who give me the motivation and opportunities to do everything I get to do, thank you!