Stef’s Monthly Update – April 2017

Stef’s Monthly Update – April 2017

The first three months of 2017 was a rollecoster ride, like the trip to Australia and New-Zealand, the very succesful Global Integration Bootcamp, the publication of the labs, the guest blogs I wrote for BizTalk360 blog, and the collobaration with Kent on Middleware Friday. And the Month April was not different in experience.

Month April

The month kicked of with a marathon run in Rotterdam. If there was any marathon in the Netherlands I wanted to do it was Rotterdam. The 9th of April was a very sunny beautiful day, however for running 42K it was a challenge. Yet I made it after 5:05:45.

Besides running a marathon I prepared myself for a trip to Sweden to talk to the Swedisch BizTalk Usergroup in Gothenburg and Stockholm. Thanks to Johan Hedberg we (Eldert and myself) were able to travel to these nice cities in Sweden and do our talks (see Elderts post on the recap of that trip). Moreover, in Stockholm Eldert and myself joined an very interactive architecture session organised my Mikael Sand together with Mikael Hakansson, Johan Hedberg, Robin Hultman, Mattias Logdberg, and Joakim Schütt at Enfo.

We talked about hybrid scenario’s, connectivity, workload driven design, Logic Apps/Functions and exchanged ideas, experiences and thoughts. It was a very useful sessions for all of us.

During the Usergroup sessions I did four new interviews for my YouTube series “Talking with Integration Pros”. And this time I interviewed:

Finally I also wrote an article on LinkedIn about Logic Apps and Functions:

Bet on Serverless: Azure Functions and Logic Apps

Music

My favorite albums that were released in April were:

  • Aryeon – The Source
  • Life Of Agony – A Place Where There’s No More Pain
  • Ghost Bath – Starmourner
  • Novembers Doom – Hamartia
  • Royal Thunder – WICK
  • Deep Purple – Infinite

Afbeeldingsresultaat voor Ayreon - The Source

Prague

After my trip to Sweden I went with my family to Prague, the capital of the Czech Republic. We spend a couple of days here to visit some remarkable areas of this beautiful city. The food, city and vibe was great.

There you have it Stef’s fourth Monthly Update and I can look back again with great satisfaction. The marathon was a challenge, however I am happy I finished and completed my third marathon. I’m looking forward to my trip to Portugal, where I will be speaking at TUGA IT.

Cheers,

Steef-Jan

Author: Steef-Jan Wiggers

Steef-Jan Wiggers is all in on Microsoft Azure, Integration, and Data Science. He has over 15 years’ experience in a wide variety of scenarios such as custom .NET solution development, overseeing large enterprise integrations, building web services, managing projects, designing web services, experimenting with data, SQL Server database administration, and consulting. Steef-Jan loves challenges in the Microsoft playing field combining it with his domain knowledge in energy, utility, banking, insurance, health care, agriculture, (local) government, bio-sciences, retail, travel and logistics. He is very active in the community as a blogger, TechNet Wiki author, book author, and global public speaker. For these efforts, Microsoft has recognized him a Microsoft MVP for the past 6 years.

BizTalk Server 2016 Feature Pack 1

BizTalk Server 2016 Feature Pack 1

On April 26th 2017, Microsoft released Feature Pack 1 (FP1) for BizTalk Server 2016 and it’s been a while since I was this excited for a BizTalk Server release. Yeah, I just said that. I’m more excited for this feature pack than I was for BizTalk Server 2016 or even 2013 and 2013 R2, and here’s why… this is the first ever Feature Pack for any release of BizTalk Server, and is setting a precedent that we have never seen before in the 16+ years of the product.

A feature pack is a release of new non-breaking features for the product. These are not bug fixes or anything like that (those are distributed quarterly through Cumulative Updates). These are brand-new features that extend the product in new ways and help customers get the most out of their BizTalk Server investment.

The product team has confirmed that other feature packs are in the works, but they have not publicly confirmed when we can expect them. In discussions I’ve had with Tord Glad Nordahl (a program manager at Microsoft and longtime lover of BizTalk Server), he said:

“If it takes 6 months to build new features there will be another feature pack in 6 months, and if it takes 2 months there will be a new one in 2 months.”

My takeaway is that the team’s goal is to offer real answers to problems that customers face in the timeliest manner possible.

I’m excited that the BizTalk Engineering team at Microsoft will be releasing new features on a more regular cadence. It’s awesome that they are building new features to address long-overlooked issues and not making us wait another 2 years to get our hands on them.

FP1 is available to customers with Software Assurance who are using the Developer or Enterprise editions of BizTalk Server 2016.

FP1 introduces some innovative new features to BizTalk Server and addresses some longstanding concerns that many customers have had. The new features break down into three categories.

Deployment

Anyone who’s worked with BizTalk Server knows that the deployment/ALM story has left something to be desired. For years, the “official” deployment story has been to deploy applications using BizTalk MSI Packages. Although the BizTalk MSIs are pretty easy to use and they work well for simple applications, they tend to be inflexible and break with the complexities of a real application in the real world.

For example, to create a BizTalk MSI Package I have to deploy all my assets to the BizTalk Server Management database and then I can generate an MSI with those assets. It sounds easy and it is. The issue is that if I introduce a new assembly, or port, or anything at all, I have to add that new resource to the BizTalkMgmtDb and then generate a new MSI. In the modern world of DevOps and continuous integration/deployment, the standard MSI-based deployment is pretty cumbersome and most teams wanting to adopt those types of strategies need a different answer.

In the past, those teams have used community-designed tools such as the BizTalk Deployment Framework to automate building an MSI from source code repositories (like most modern ALM solutions) and therefore eliminating the need to deploy to a BizTalk Server system to create the deployment package. Feature Pack 1 for BizTalk Server 2016 introduces two new features that will serve as a foundation for more sophisticated deployment strategies in the future (which my inside sources at Microsoft have confirmed are coming in later feature packs).

  • Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS).
    For anyone who has used the build/release features of TFS or VSTS, this will be immediately familiar. This is a deployment task that you add to your release pipelines to deploy new or redeploy/update existing BizTalk Server applications.
    If you haven’t used the build/release features of TFS or VSTS check out this post where I explain how to use those features to enable continuous release for a Logic App.
  • New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support.
    Imagine having RESTful web APIs for updating, adding, or querying the status of your BizTalk Server applications and their resources… now stop imagining it because it’s a real thing!

Analytics

The tracking capabilities in BizTalk Server are extensive, but the configuration is often unintuitive, and no one likes digging through the BizTalkDTADb for the instance data they need.

FP1 enables you to send your tracking data to Azure Application Insights and feed operational data (subscriptions, batching status, message instance counts, etc.) to Power BI.

  • Application Insights – Tap into the power of Azure for tracking valuable application performance, usage, diagnostics, and availability.
    Enabling this is super easy: after creating an Application Insights instance, in the BizTalk Settings Dashboard there’s a new section for enabling analytics.

    Enabling Application Insights for the BizTalk Server group as introduced in BizTalk Server 2016 Feature Pack 1

    Once you’ve enabled analytics and provided your App Insights instrumentation key, in your ports and orchestrations you will have a new setting to output the tracking data to App Insights.

    Enabling Application Insights for a BizTalk Server orchestration as introduced in BizTalk Server 2016 Feature Pack 1

  • Leverage operational data – View operational data from anywhere and with any device using Power BI.
    This operational data is the same kind of information that you’d typically view using the BizTalk Group Hub (suspended instances, subscriptions, tracked events, etc.). If you can build a query for it using the Group Hub, you can output that to Power BI… but why would you?
    Because Power BI gives you the ability to view that data from anywhere (without having direct access to the BizTalk Group), and the tools in Power BI make querying that data surprisingly easy. With Power BI, you can ask questions in plain English and have MDX-style queries created for you in the background. FP1 comes with a pre-built Power BI template, but you of course have the ability to build your own. I’m interested to see what the BizTalk community can create using these tools.

Runtime

If I’m being completely honest, the two features in this runtime category weren’t really on my radar at all until Tord Glad Nordahl stopped by one of my classes last month and discussed them with the students. But now that I’ve seen them, I’m excited for the potential and happy that customers with these requirements are getting some much needed love.

  • Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns.
    Basically, SQL Server 2016 introduced a feature that enables client applications to read/write encrypted data within a SQL table without actually providing the encryption keys to SQL Server. This gives a new level of data security since the owners of the secure data (i.e., the client applications) can see it, but the manager of the data (i.e., SQL Server) cannot.
    This ensures that on-premises or cloud database administrators or other high-privileged (but unauthorized) users cannot access the sensitive data.
    With Feature Pack 1 of BizTalk Server 2016, the WCF-SQL adapter now offers an Always Encrypted property where you can simply enable or disable the feature as your needs dictate.

    WCF-SQL Adapter Always Encrypted property as introduced in BizTalk Server 2016 Feature Pack 1

  • Advanced Scheduling – Set up advanced schedules for BizTalk receive locations.
    The Schedule page of receive locations has additional options for shifting time zones and setting up recurrence schedules.

    BizTalk Server receive location advanced scheduling options as introduced in BizTalk Server 2016 Feature Pack 1

As always, the QuickLearn Training team is already looking for the best ways to incorporate these new features into our courses, but until we do you should grab the Feature Pack for yourself and give these new features a spin for yourself. While you’re at it go to the BizTalk Server User Voice page and vote for the features that you’d like to see in the next feature pack, or if you have an original idea for a feature add it there and see how love it gets.

BizTalk Server 2016 Feature Pack 1

BizTalk Server 2016 Feature Pack 1

On April 26th 2017, Microsoft released Feature Pack 1 (FP1) for BizTalk Server 2016 and it’s been a while since I was this excited for a BizTalk Server release. Yeah, I just said that. I’m more excited for this feature pack than I was for BizTalk Server 2016 or even 2013 and 2013 R2, and here’s why… this is the first ever Feature Pack for any release of BizTalk Server, and is setting a precedent that we have never seen before in the 16+ years of the product.

A feature pack is a release of new non-breaking features for the product. These are not bug fixes or anything like that (those are distributed quarterly through Cumulative Updates). These are brand-new features that extend the product in new ways and help customers get the most out of their BizTalk Server investment.

The product team has confirmed that other feature packs are in the works, but they have not publicly confirmed when we can expect them. In discussions I’ve had with Tord Glad Nordahl (a program manager at Microsoft and longtime lover of BizTalk Server), he said:

“If it takes 6 months to build new features there will be another feature pack in 6 months, and if it takes 2 months there will be a new one in 2 months.”

My takeaway is that the team’s goal is to offer real answers to problems that customers face in the timeliest manner possible.

I’m excited that the BizTalk Engineering team at Microsoft will be releasing new features on a more regular cadence. It’s awesome that they are building new features to address long-overlooked issues and not making us wait another 2 years to get our hands on them.

FP1 is available to customers with Software Assurance who are using the Developer or Enterprise editions of BizTalk Server 2016.

FP1 introduces some innovative new features to BizTalk Server and addresses some longstanding concerns that many customers have had. The new features break down into three categories.

Deployment

Anyone who’s worked with BizTalk Server knows that the deployment/ALM story has left something to be desired. For years, the “official” deployment story has been to deploy applications using BizTalk MSI Packages. Although the BizTalk MSIs are pretty easy to use and they work well for simple applications, they tend to be inflexible and break with the complexities of a real application in the real world.

For example, to create a BizTalk MSI Package I have to deploy all my assets to the BizTalk Server Management database and then I can generate an MSI with those assets. It sounds easy and it is. The issue is that if I introduce a new assembly, or port, or anything at all, I have to add that new resource to the BizTalkMgmtDb and then generate a new MSI. In the modern world of DevOps and continuous integration/deployment, the standard MSI-based deployment is pretty cumbersome and most teams wanting to adopt those types of strategies need a different answer.

In the past, those teams have used community-designed tools such as the BizTalk Deployment Framework to automate building an MSI from source code repositories (like most modern ALM solutions) and therefore eliminating the need to deploy to a BizTalk Server system to create the deployment package. Feature Pack 1 for BizTalk Server 2016 introduces two new features that will serve as a foundation for more sophisticated deployment strategies in the future (which my inside sources at Microsoft have confirmed are coming in later feature packs).

  • Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS).
    For anyone who has used the build/release features of TFS or VSTS, this will be immediately familiar. This is a deployment task that you add to your release pipelines to deploy new or redeploy/update existing BizTalk Server applications.
    If you haven’t used the build/release features of TFS or VSTS check out this post where I explain how to use those features to enable continuous release for a Logic App.
  • New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support.
    Imagine having RESTful web APIs for updating, adding, or querying the status of your BizTalk Server applications and their resources… now stop imagining it because it’s a real thing!

Analytics

The tracking capabilities in BizTalk Server are extensive, but the configuration is often unintuitive, and no one likes digging through the BizTalkDTADb for the instance data they need.

FP1 enables you to send your tracking data to Azure Application Insights and feed operational data (subscriptions, batching status, message instance counts, etc.) to Power BI.

  • Application Insights – Tap into the power of Azure for tracking valuable application performance, usage, diagnostics, and availability.
    Enabling this is super easy: after creating an Application Insights instance, in the BizTalk Settings Dashboard there’s a new section for enabling analytics.

    Enabling Application Insights for the BizTalk Server group as introduced in BizTalk Server 2016 Feature Pack 1

    Once you’ve enabled analytics and provided your App Insights instrumentation key, in your ports and orchestrations you will have a new setting to output the tracking data to App Insights.

    Enabling Application Insights for a BizTalk Server orchestration as introduced in BizTalk Server 2016 Feature Pack 1

  • Leverage operational data – View operational data from anywhere and with any device using Power BI.
    This operational data is the same kind of information that you’d typically view using the BizTalk Group Hub (suspended instances, subscriptions, tracked events, etc.). If you can build a query for it using the Group Hub, you can output that to Power BI… but why would you?
    Because Power BI gives you the ability to view that data from anywhere (without having direct access to the BizTalk Group), and the tools in Power BI make querying that data surprisingly easy. With Power BI, you can ask questions in plain English and have MDX-style queries created for you in the background. FP1 comes with a pre-built Power BI template, but you of course have the ability to build your own. I’m interested to see what the BizTalk community can create using these tools.

Runtime

If I’m being completely honest, the two features in this runtime category weren’t really on my radar at all until Tord Glad Nordahl stopped by one of my classes last month and discussed them with the students. But now that I’ve seen them, I’m excited for the potential and happy that customers with these requirements are getting some much needed love.

  • Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns.
    Basically, SQL Server 2016 introduced a feature that enables client applications to read/write encrypted data within a SQL table without actually providing the encryption keys to SQL Server. This gives a new level of data security since the owners of the secure data (i.e., the client applications) can see it, but the manager of the data (i.e., SQL Server) cannot.
    This ensures that on-premises or cloud database administrators or other high-privileged (but unauthorized) users cannot access the sensitive data.
    With Feature Pack 1 of BizTalk Server 2016, the WCF-SQL adapter now offers an Always Encrypted property where you can simply enable or disable the feature as your needs dictate.

    WCF-SQL Adapter Always Encrypted property as introduced in BizTalk Server 2016 Feature Pack 1

  • Advanced Scheduling – Set up advanced schedules for BizTalk receive locations.
    The Schedule page of receive locations has additional options for shifting time zones and setting up recurrence schedules.

    BizTalk Server receive location advanced scheduling options as introduced in BizTalk Server 2016 Feature Pack 1

As always, the QuickLearn Training team is already looking for the best ways to incorporate these new features into our courses, but until we do you should grab the Feature Pack for yourself and give these new features a spin for yourself. While you’re at it go to the BizTalk Server User Voice page and vote for the features that you’d like to see in the next feature pack, or if you have an original idea for a feature add it there and see how love it gets.

BizTalk and Azure Functions

BizTalk and Azure Functions

Recently I had the opportunity to look at the options for combining BizTalk and Azure Functions for some real world scenarios. The most likely scenarios today are:

  1. Call a function using the HTTP trigger
  2. Add a message to a service bus queue and allow the function to execute via the Service Bus trigger

This is great because this gives us support for synchronous and asynchronous execution of a function from BizTalk.

The queue option doesn’t really need any elaboration, there is an adapter to call Service Bus from BizTalk and there are lots of resources out there to show you how to use a Queue trigger in a function.

What I did want to show however was an example of how to call a function using the http trigger. First question though is why would you want to do this?

I think the best example would be when using a façade pattern. This is a pattern we use a lot where we often create a web service component to encapsulate some logic to call an external system and then we call that web service from BizTalk. The idea is that we make the interaction from BizTalk to the web service simple and the web service hides the complexity. In the real world we usually do this for the following reasons:

  • It makes the solution simpler
  • The performance of the solution will be improved
  • It is cheaper and easier to maintain in some cases

Azure Functions offers us an alternative way to implement the façade component compared to using a normal web service which we might host in IIS or on Azure App Service. Lets take a look at an imaginary architecture showing this in action.

In this example imagine that BizTalk makes a call to the function. In the function it makes multiple calls to CRM to lookup various entity relationships and then create a final entity to insert. This would be a typical use of the façade pattern for this type of integration.

Demo

In the video below I show an example of consuming a function from BizTalk.

[embedded content]

What would I like to see Microsoft do with BizTalk + Functions?

There are lots of ways you can use Functions today from within BizTalk, but Microsoft are now investing quite heavily in enhancing BizTalk and combining it with cloud features. I would love to see some “Call Function” out of the box capabilities.

Imagine if BizTalk included some platform settings where you could register a function. This would mean a list of functions which had a friendly name, the url to call the function and a key for calling it. Maybe there was a couple of variations on this register function capability so you could use it from different places. First off:

  • Call Azure Function shape in an Orchestration

This would be a shape where we could pass 1 message into it a bit like the transform shape we have now and get 1 message out. Under the hood the shape would convert the messages to xml or json then call the function. In the orchestration you would specify the function to call from a drop down choosing from the registered functions. This would abstract the orchestration from the functions changeable information (url/key).

This would be like the expression shape in some ways but by externalising the logic into a function it would become much easier to change the logic if needed.

  • Call Azure Function mapping functoid

We drag the azure function functoid onto the map designer. We wire up the input parameters and destination. You supply your friendly named function from the functions you have registered with BizTalk. When the map executes again it makes an http call to the function. The function would probably have to make some assumptions about the data it was getting, maybe a collection of strings coming in and a single string coming out which the map could then further convert to other data types

The use of a function in mapping would be great for reference data mapping where you want an easy way to change the code at runtime.

Microsoft BizTalk Server 2016 Feature Pack 1: Step-by-step Installation

Microsoft BizTalk Server 2016 Feature Pack 1: Step-by-step Installation

We are used to having Service Pack (I think the last one was in 2006 R2) and Cumulative Updates for BizTalk Server (see the full list here) that basically brings hotfixes for BizTalk Server and adapter pack issues that were resolved after the release of that specific BizTalk Server version. Now Microsoft introduces a new concept in BizTalk Server 2016: Feature Packs.

BizTalk Server 2016 will use feature pack approach to providing new functionalities to the product at a faster pace. Now new features (or at least non-breaking features) will be delivered when they’re ready we no longer need to wait 2 years for the next major release of the product to have new features!

BizTalk Server uses the feature pack to provide improvements, features, and closer integration with Azure. Feature Pack 1 extends functionality in key areas, such as deployment, analytics, and runtime. – https://msdn.microsoft.com/en-us/library/mt800834.aspx

Feature Pack’s will be available for Software Assurance customers running BizTalk Server 2016 Developer and Enterprise editions or customers running BizTalk Server 2016 in Azure under an Enterprise Agreement

Should we expect new BizTalk Server Cumulative Updates?

Yes, Microsoft will continue to release CU to provide hotfixes for issues that were resolved in the same cadence as they have done in the past

Microsoft BizTalk Server 2016 Feature Pack 1: System Requirements

BizTalk Server 2016 Feature Pack 1 requirements:

  • Microsoft BizTalk Server 2016 Enterprise Edition
  • Microsoft BizTalk Server 2016 Developer Edition

Important note: is required to restart BizTalk Server machine to finish the FP1 installation process.

Note: Feature Packs are not supported in BizTalk Server 2016 Standard Edition.

Microsoft BizTalk Server 2016 Feature Pack 1: Step-by-step Installation Instructions

After you download the Microsoft BizTalk Server 2016 Feature Pack 1 You should:

  • Run the downloaded .EXE file – BTS2016-KB4014788-ENU.exe – from your hard drive.
  • In the “Welcome to Microsoft BizTalk Server Feature Pack 1 Installation Wizard” screen, click “Next”

BizTalk Server 2016 Feature Pack 1 Welcome

  • In the “License agreement” screen, select the option “I accept the terms of this license agreement” and then click “Next”

BizTalk Server 2016 Feature Pack 1 License Agreement

  • In the “Ready to Install”, it will be provided a list of products that will be enhanced, you should click “Next” to continue with the installation.

BizTalk Server 2016 Feature Pack 1 Ready To Install

  • The FP1 installation will prompt a window with the list of services that will be restarted during the installation process, click “Yes” to continue with the installation.

BizTalk Server 2016 Feature Pack 1 services to restart

  • The installation process may take some minutes to complete, and you probably see some “background” windows to appear and to be completed, you should wait until the “Installation Wizard has completed” screen to appear
  • In the “Installation Wizard has completed” screen, click “Finish”

BizTalk Server 2016 Feature Pack 1 Complete

    • Note: that you must restart BizTalk Server machine to finish the FP1 installation process. Click “Yes” to restart the server.

BizTalk Server 2016 Feature Pack 1 restart now

Once the restart is complete, you will be able to see and use the new following features:

  • Deployment:
    • Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS)
    • New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support
  • Analytics:
    • Application Insights – Tap into the power of Azure for tracking valuable application performance and to better understand your workflow
    • Leverage operational data – View operational data from anywhere and with any device using Power BI
  • Runtime:
    • Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns
    • Advanced Scheduling – Set up advanced schedules using the new capabilities with Feature Pack 1
Author: Sandro Pereira

Sandro Pereira lives in Portugal and works as a consultant at DevScope. In the past years, he has been working on implementing Integration scenarios both on-premises and cloud for various clients, each with different scenarios from a technical point of view, size, and criticality, using Microsoft Azure, Microsoft BizTalk Server and different technologies like AS2, EDI, RosettaNet, SAP, TIBCO etc. He is a regular blogger, international speaker, and technical reviewer of several BizTalk books all focused on Integration. He is also the author of the book “BizTalk Mapping Patterns & Best Practices”. He has been awarded MVP since 2011 for his contributions to the integration community.

Logic Apps Batching

Logic Apps Batching

Scenario

For this blog post, I decided to try to batch the following XML message.  As Logic Apps supports JSON natively, we can assume that a similar setup will work quite easily for JSON messages.  Remark that the XML snippet below contains an XML declaration, so pure string appending won’t work.  Also namespaces are included.

Requirements

I came up with the following requirements for my batching solution:

  • External message store: in integration I like to avoid long-running workflow instances at all time. Therefore I prefer messages to be stored somewhere out-of-the-process, waiting to be batched, instead of keeping them active in a singleton workflow instance (e.g. BizTalk sequential convoy).
  • Message and metadata together: I want to avoid to store the message in a specific place and the metadata in another one. Keep them together, to simplify development and maintenance.
  • Native Logic Apps integration: preferably I can leverage an Azure service, that has native and smooth integration with Azure Logic Apps. It must ensure we can reliably assign messages to a specific batch and we must be able to remove them easily from the message store.
  • Multiple batch release triggers: I want to support multiple ways to decide when a batch can be released.
    > # Messages: send out batches containing each X messages
    > Time: send out a batch at a specific time of the day
    > External Trigger: release the batch when an external trigger is receive

Solution

After some analysis, I was convinced that Azure Service Bus queues are a good fit:

  • External message store: the messages can be queued for a long time in an Azure Service Bus queue.
  • Message and metadata together: the message is placed together with its properties on the queue. Each batch configuration can have its own queue assigned.
  • Native Logic Apps integration: there is a Service Bus connector to receive multiple messages inside one Logic App instance. With the peak-lock pattern, you can reliably assign messages to a batch and remove them from the queue.
  • Multiple batch release triggers:
    > # Messages: In the Service Bus connector, you can choose how many messages you want to receive in one Logic App instance

    > Time
    : Service Bus has a great property ScheduledEnqueueTimeUtc, which ensures that a message becomes only visible on the queue from a specific moment in time. This is a great way to schedule messages to be releases at a specific time, without the need for an external scheduler.

    > External Trigger
    : The Logic App can be easily instantiated via the native HTTP Request trigger

 

Implementation

Batching Store

The goal of this workflow is to put the message on a specific queue for batching purpose.  This Logic App is very straightforward to implement. Add a Request trigger to receive the messages that need to be batched and use the Send Message Service Bus connector to send the message to a specific queue.

In case you want to release the batch only at a specific moment in time, you must provide a value for the ScheduledEnqueueTimeUtc property in the advanced settings.

Batching Release

This is the more complex part of the solution. The first challenge is to receive for example 3 messages in one Logic App instance. My first attempt failed, because there is apparently a different behaviour in the Service Bus receive trigger and action:

  • When one or more messages arrive in a queue: this trigger receives messages in a batch from a Service Bus queue, but it creates for every message a specific Logic App instance. This is not desired for our scenario, but can be very useful in high throughput scenarios.
  • Get messages from a queue: this action can receive multiple messages in batch from a Service Bus queue. This results in an array of Service Bus messages, inside one Logic App instance. This is the result that we want for this batching exercise!

Let’s use the peak-lock pattern to ensure reliability and receive 3 messages in one batch:

As a result, we get this JSON array back from the Service Bus connector:

The challenge is to parse this array, decode the base64 content in the ContentData and create a valid XML batch message from it.  I tried several complex Logic App expressions, but realized soon that Azure Functions is better suited to take care of this complicated parsing.  I created the following Azure Fuction, as a Generic Webhook C# type:

Let’s consume this function now from within our Logic App.  There is seamless integration with Logic Apps, which is really great!


As an output of the GetBatchMessage Azure Funtion, I get the following XML 🙂

Large Messages

This solution is very nice, but what with large messages? Recently, I wrote a Service Bus connector that uses the claim check pattern, which exchanges large payloads via Blob Storage. In this batching scenario we can also leverage this functionality. When I have open sourced this project, I’ll update this blog with a working example.  Stay tuned for more!

Conclusion

This is a great and flexible way to perform batching within Logic Apps. It really demonstrates the power of the Better Together story with Azure Logic Apps, Service Bus and Functions. I’m sure this is not the only way to perform batching in Logic Apps, so do not hesitate to share your solution for this common integration challenge in the comments section below!

I hope this gave you some fresh insights in the capabilities of Azure Logic Apps!
Toon

Configuring the Management REST-API in BizTalk 2016 Service-Feature Pack 1

Configuring the Management REST-API in BizTalk 2016 Service-Feature Pack 1

Today Microsoft release the BizTalk Server 2016 Feature Pack 1: http://biztalkfp1.azurewebsites.net/

I’m was particularly interested in the Management REST APIs, because at the current customer our team has spent a lot of time developing a custom API on top of the ExplorerOM and Operations DLLs. So maybe we could learn something the way Microsoft did this.

Starting with BizTalk Server 2016 Feature Pack 1, there’s a Windows PowerShell script that installs the REST APIs, and their swagger definitions. Using these APIs, make REST calls to remotely manage your artifacts, including ports, orchestrations, partners, agreements, pipelines, and more. Provides full support of API REST management endpoints with full swagger support.

I went to this page on MSDN for the installer instructions. Also BizTalk360 has a nice write up about this topic here.

Bot guidelines are NOT complete and are missing the part were you also need to install the ‘OperationalData’ service via a second command.

To install both the ManagmentService and the OperationalDataService, you need to do it like this:

1. Open up a Powershell(x86) prompt

2. Install the ManagementService

FeaturePack.ConfigureServices.ps1 
 -Service Management 
 -WebSiteName 'Default Web Site' 
 -ApplicationPool BtsMgmtAppPool 
 -ApplicationPoolUser BtsAdmin 
 -ApplicationPoolUserPassword Azerty123 
 -AuthorizationRoles 'BizTalk Server Administrators'

3. Install the OperationDataService

FeaturePack.ConfigureServices.ps1 
 -Service OperationalData 
 -WebSiteName 'Default Web Site' 
 -ApplicationPool BtsMgmtAppPool 
 -ApplicationPoolUser BtsAdmin 
 -ApplicationPoolUserPassword Azerty123 
 -AuthorizationRoles 'BizTalk Server Administrators'

After that, you can open the Swagger of the ManagementService at http://localhost/BizTalkManagementService/swagger to start exploring the available methods:

The complete list of available operations is also viewable on MSDN here.

After giving it a quick spin, i can conclude that it works great and all methods are doing as they are supposed.
But what about HostInstances? It would be very nice to be able to manage them as well via this new API. Just my 2 cents!

BizTalk 2016 Gets Real Love With Feature Pack 1

BizTalk 2016 Gets Real Love With Feature Pack 1

For quite some time, BizTalk was not getting the love it deserved. Sure, we got our platform alignment, and sometimes a new or updated adapter, but all in all, there were not many exciting new features. That changes now, with the just released Feature Pack 1 BizTalk 2016. In this feature pack, we are seeing more new features than we have in a long time, and shows that the product team, with our very own Tord at the helm, really is caring about the product once again. If you look at the user-voice page for BizTalk, you will notice a lot of suggestions are being made by the community, and this feature pack shows that we are actually being listened to as well! In this post, I will go into the new features being introduced.

Installation of this feature pack is very straight forward, just download the installer and run the wizard.

Once installed, you will see the new features in your BizTalk installation.

One of the places where BizTalk was falling behind in this world where agile is becoming increasingly important was the ALM story. Of course, we had projects like the BizTalk Deployment Framework (of which I am a big fan) which helped with this, but being dependent on a third party for this proved to be cumbersome at times. Thankfully FP1 addresses this, and gives us the option to do continuous integration using VSTS! This allows us to automatically deploy new applications or update existing applications after checking in our project. After you have installed the feature pack, you can create a new BizTalk project, called BizTalk Server Application Project. If you don’t see it in the list, make sure to select the latest .NET framework version.

To be able to deploy the project, we will also need to add the Deploy BizTalk Application task to our VSTS. This will give us a new task we can use in our VSTS build definitions.

Add new deploy definition

You can check here for a full overview on how to set this up.

With FP1, we now have the option to log our message data to Application Insights. Application Insights is already supported or planned to be supported in various places within Azure like Web and API apps, Azure Functions and Logic Apps, and as of now we can use this platform from BizTalk as well, allowing us to have a consolidated tracking mechanism for our messages both from on-premise and in the cloud.

This is the same data we see in our tracking data, and can be set up very easily. To enable integration with Application Insights, open the settings of your BizTalk group, and check the Enable group-level analytics option. In the connection parameters, you have to enter the instrumentation key, which can be found in the portal for your Application Insights instance.

C:Userseldert.grootenboerAppDataLocalMicrosoftWindowsINetCacheContent.Word2017-04-26 13_13_04-Windows Server 2016 - BizTalk 2016 RTM on KRUKALA - Virtual Machine Connection.png

And finally enable analytics on your port.

You can check here for a full overview.

With this feature pack, BizTalk 2016 now also exposes REST API’s with which you can manage your BizTalk artifacts, like your ports, orchestrations, hosts, pipelines, as well as operational data like retrieving and working with messages, service instances and much more. This gives us the possibility to integrate our BizTalk management in any application we want, as well as easily script our BizTalk management tasks. And of course, the API’s are all documented with Swagger, allowing us to get started with this right away. You can read here on how to enable the REST API’s.

Visualize With PowerBI

And what is more, we can also use PowerBI to visualize the feed from these API’s. The BizTalk team provides an out of the box template for this, which you will find in X:Program Files (x86)Microsoft BizTalk Server 2016OperationalDataServiceBizTalkOperationalData.pbit which can then be loaded into PowerBI Desktop. This template will give you a starting point, on which you can then expand and build your own PowerBI dashboards, which you can also integrate in your monitoring solution like BizTalk360.

biztalk-power-bi-dashboard - BizTalk Server 2016 Feature Pack 1

More information can be found here.

When we wanted to schedule something on BizTalk’s receive locations, we were always very confined, with just being able to specify a start date, a stop date, and the start and stop time for the service window. With Feature Pack 1, this has now been made much more granular, allowing us to set much more granular schedules, allowing us to trigger every x days, or even on certain days of the week or month. Many of these features could already be found in the Scheduled Task Adapter by Sandro, which still provides some more advanced options, but it is good to see Microsoft looking at the popular extensions the community created and incorporating them into the product.

More information can be found here.

And finally, BizTalk can now use encrypted columns in SQL server, which allows us to protect sensitive data in SQL server.

More information can be found here.

JSON Encoder Type Bug

JSON Encoder Type Bug

The issue popped up at one or our projects, where we had to deliver a JSON file according to the specifications of an external party. The schema had multiple fields defined as decimal, but for some reason some of the decimals came out as strings. The difference is that the decimal value does not have quotes surrounding the actual value.
To recreate the issue, I created a very simple schema (which is specified below) and a send pipeline containing only the out of the box JSON Encoder.

I’ve chosen to base this scenario on receiving an XML file and sending a JSON file. For this I created a simple messaging-only solution with a file-based Receive Port and file-based Send Port, where the routing is done based on BTS.ReceivePortName. To test this setup I used the following test message.

This is where the issue shows itself. The JSON that is sent by BizTalk is not equal to the expected JSON output. See the comparison and the highlighted difference below.

This is very strange behavior, since both Level1/Field1 and Level1/Field2 are specified as a decimal, and yet Field1 is parsed as a string and Field2 is parsed as a decimal.
The important thing to note is that I have an element called “Field1” on multiple levels in the schema, the first has the type string, the second one has the type decimal.
What appears to be happening is that if you have multiple nodes on different levels in your schema the JSON Encoder always takes the type of the first occurrence of a node with the same name. In our case the first time ”Field1” occurs in our schema it is defined as a string and this is why in our output the second occurrence of the “Field1” node is incorrectly written as a string.
To prove this behavior I renamed the second occurrence of the “Field1” node to “Field3”, this time the output was as expected.

This obviously can be fixed very easily by renaming the fields. However I often find myself in the situation that the XSD cannot be changed as it is defined by an external party. It turns out that the out of the box JSON Encoder uses an old version of the Newtonsoft.Json library which I cannot find in the the Newtonsoft.Json respository on GitHub, so it probably is a Microsoft fork of the Newtonsoft.Json library.

This was all developed and tested on a BizTalk 2016 machine, but I suspect this bug has been present since the introduction of the out of the box JSON Encoder pipeline component with BizTalk 2013R2.

To solve this issue I had to write my own custom JSON Encoder pipeline component where I used the latest version of the Newtonsoft.Json library.

In fact, this issue has been raised to Microsoft via the BizTalk Server uservoice pages. You can find the topic here. If you agree, go there, and show your support by voting for this issue. 

BizTalk Server 2016 Feature Pack 1 is live

BizTalk Server 2016 Feature Pack 1 is live

fp1header

Enterprise integration is rapidly evolving, and we are pleased to announce the general availability of BizTalk Server 2016 Feature Pack 1. This release extends functionality in key areas such as Deployment, Analytics, and Runtime.

BizTalk Server 2016 Feature Pack 1 is available for Software Assurance customers running the BizTalk Server 2016 Developer and Enterprise editions or are running BizTalk Server in Azure under an Enterprise Agreement. Feature Pack 1 is production ready and provides only non-breaking features.

Deployment:

  • Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS)
  • New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support

Analytics:

  • Application Insights – Tap into the power of Azure for tracking valuable application performance and to better understand your workflow
  • Leverage operational data – View operational data from anywhere and with any device using Power BI

Runtime:

  • Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns
  • Advanced Scheduling – Set up advanced schedules using the new capabilities with Feature Pack 1

Feature Pack 1 builds on the investments we made to release BizTalk Server 2016 last December.

We encourage you to actively engage with us and propose new features through our BizTalk User Voice page
For more information select the feature you want to know more about below or go straight to our launch page and download the latest Feature Pack for BizTalk Server 2016

Our great Partner BizTalk360 has also written about this and the difference between a CU and a Feature Pack. Take a look here.