HO HO HO, yes, it is that magical time of the year again!… More than other years, this year Christmas is about spending time with close family, the small number possible and family members that you are in contact with every day to due this COVID 19 pandemic. It’s about creating happy memories that will last a lifetime, and we need it more than ever!
This was without a doubt a challenging year for everyone, a year not to forget but to learn and remember, and hoping to start 2021 with a blank page for us to write a better story.
To all my readers, friends, coworkers, customer, partners, to all Microsoft Integration and Azure Community (BizTalk Server, Logic Apps, API Management, Service Bus, and so on), MSFT Product Groups, all the Portuguese Communities, my MVP “family” and of course to my family, my sincere wishes for a Merry Christmas and a Happy New Year!
Thanks in advance for all the support and encouragement given throughout this year. I hope that the next will be a year filled with new distinct contributions. I promise that I will continue to share knowledge, hoping that they can help someone.
This year I had the privilege of receiving a surprise Christmas gift from good friends that intentionally or unintentionally (I believe it was intentional) combine two of my passions/hobbies:
Wine enthusiasm and the passion for exploring and taste new wines (you can find more in my Instagram account)
And my blog.
So I got this fantastic Blog (Black Label) 2016 Red Wine. Thanks for this amazing gift! No need to mention names. You know who you are
All the best for the coming year, and most importantly, stay safe!
When migrating your BizTalk Server environment or deploy to a new or different environment, there are many different resources or configurations that you need to take into considerations like:
local queue creations
cloud queue creations
folder creations
and so on.
One of the common resources that we use on our integration solutions is local folders, whether for archiving or routing messages. I do not mean that it is a good practice or not. That you should do it or not, this is not the goal, only that this is common to happen.
In a first analysis, we could think that the quickest and most effective solution would be to copy and paste the folder structure from one environment to another. Still, it may not be the best solution in many cases since it may contain thousands of documents, which are unnecessary to copy/migrate.
This blog post will address how we can easily recreate a folder structure in a different environment/server using PowerShell.
PowerShell script to recreate a local folder structure
With this PowerShell sample, we will be able to recreate an existing local folder structure on a different BizTalk Server environment.
The output of this PowerShell script will be a creation of a different PS script containing the instructions that you can use to recreate all the folder structures in different environments.
When migrating your BizTalk Server environment or deploy to a new or different environment, there are many different resources or configurations that you need to take into considerations like:
local queue creations
cloud queue creations
folder creations
and so on.
In this blog post, I will address how we can easily “export” a list of existing local private Message Queues (MSMQ) and recreate them, and set proper permissions in a different environment/server.
PowerShell script to extract a list of all private Message Queues (MSMQ) names
With this PowerShell sample, we will be able to extract a list of all private Message Queues (MSMQ) names to a CSV file to be used on a different script to deployed these resources on a different BizTalk Server environment.
set or update the URI (address) or part of the URI on a list of BizTalk Server Receive Locations deployed in your BizTalk Server environment.
PowerShell script to create private Message Queues (MSMQ)
With this PowerShell sample, we will be able to create local Private Message Queues (MSMQ) and set proper permissions on a different BizTalk Server environment.
Starting with BizTalk Server 2020 and newer, administrators can configure BizTalk Server to generate an audit trail for management operation on application artifacts, such as to send ports, receive ports, receive locations, orchestrations, and resources. Auditing of suspend/resume/terminate operations on service instances is also possible.
This feature may not be perfect yet, and it has space to grow in the future, but it is a good start, and I hope the BizTalk product team will improve these capabilities in the future.
How to configure the Audit Logs?
The auditing feature isn’t enabled by default. To enable auditing you need to:
Open BizTalk Server Administration console, right-click the BizTalk Group, and select Settings
On the Group panel, under the Tracking and Reporting section:
Enable the Audit management operations option.
And set a Maximum number of audit entries property according to your needs. By default, BizTalk stores 10000 most recent entries.
Select OK to save your changes.
Now every time you make the following operations:
On Ports (Receive or Send): Create, Update and Delete
On Service Instances: Suspend, Resume and Terminate
On Application resource: Add, Update and Remove
On Bindings: Import bindings files
One or more audit entries are logged. All these correlation entries have the same BatchId. The audit log has the following information:
Property
Description
ID
Id of type Guid, unique per entry. Example: 3bf539a3-4b59-4148-b589-d22c83f32d25
BatchId
Same for all audited operations performed in a single SQL transaction. Insightful in correlating user operations with lower level details Example: a4ffa64b-8064-4e54-9d99-2e7d60797100
UserPrincipal
User who performed the operation. Example: BTS2020LAB01Administrator
Machine
Machine name from which operation was performed. Example: BTS2020LAB01
ArtifactId
Unique id of the artifact. Example: 1
ParentArtifactId
If an artifact is child of another artifact, then this field will have artifact id of the parent. Example: 1
ArtifactType
Type of artifact on which operation was performed. Example: SendPort, ReceiveLocation, etc.
ArtifactName
User configured name of the artifact. Example: BatchControlMessageRecvLoc
OperationName
Action performed on the artifact. Example: Ports: Create/Update/DeleteService Instances: Suspend/Resume/TerminateApplication resources: Add/Update/RemoveBinding file: Import
Payload
Contains information about what is changed in JSON structure. Example: { “Disabled”: 0 }
CreatedDate
Timestamp when the operation was performed Example: 2020-11-27T09:21:48.55Z
How to view the audit logs?
The easier way to access the BizTalk Server Audit logs is to access the Audit Logs REST service using your browser by using the following URL:
GET http://localhost/BizTalkOperationalDataService/AuditLogs
This will provide a JSON response containing all the latest logs.
For a better user experience, you could use a tool like postman to interact with this service:
You can also define a date range to retrieve the desire logs. For that you should use the following query parameters:
fromDate: beginning date, for example, 2020-12-01T01:00:00
toDate: end date, for example, 2020-12-03T01:00:00
GET request example:
GET http://localhost/BizTalkOperationalDataService/AuditLogs?fromDate=2020-12-01T01:00:00&toDate=2020-12-03T01:00:00
Supported date formats are: yyyy-MM-dd or yyyy-MM-ddThh:mm:ss.
It was with great pleasure that I presented on December 7, 2020, another session in the Integration Monday series, this time about Logic Apps: Development Experiences. I had the pleasure to be accompanied as a co-presenter for the first time by my team member, Pedro Almeida.
Logic Apps: Development Experiences
How can I start developing Logic Apps? What are the different tools I can use? What are the advantages and drawbacks of each developer approach? What are the deployment options that I have? These are some of the questions that we will answer in this session, along with several tips that will improve your Logic Apps development experience.
I hope you enjoy it and find it an interesting session. Also, I advise you to visit and view the history of sessions that have taken place every Monday in the Integration User Group – Integration Monday series.
My other talks at Integration Monday – Integration User Group
Azure Logic Apps adapter is used by BizTalk Server to communicate with the Azure Logic Apps. This could also be possible using the HTTP adapter, but the Logic App adapter provides a better and straightforwardly experience.
You may be thinking that I’m out of my mind. Logic App Adapter was already available in BizTalk Server 2016. Indeed you are right. However, the Logic App adapter was an optional feature and a separate download in BizTalk Server 2016. You can know more about how to install it on this white paper I wrote some time ago: Step-by-Step Logic App Adapter Installation Guide. But it is now installed with the BizTalk Server 2020 default installation process, so you do not need to install it and configure it manually. Everything or almost everything will be already configured for you automatically.
The Logic App Adapter is one of the adapters that support two-way communications:
The Logic App Receive Adapter that is responsible for receiving messages from Logic Apps and delivering them to BizTalk is, in fact, a WCF Service that runs inside Internet Information Services (IIS).
Install and Configure an On-premises Data Gateway;
Click the Test Settings button to verify the application pool identity and pass the authentication and authorization tests.
Click the OK button to save the changes.
Open BizTalk Server Administration, expand BizTalk Server Administration à BizTalk Group > Applications, and expand our application.
Right-select Receive Ports, select New, and select One-way Receive Port.
In the Receive Port properties window, enter the following configurations:
Name: Enter a name for the receive port.
Select Receive Locations, and select New
In the Receive Location properties window, enter the following configurations:
Enter a Name for the receive location
For the Type, select LogicApp from the list, and select the Configure button.
In the General tab, configure the endpoint address for your logic app:
Address: Required. Enter the BizTalk ReceiveService IIS application URL. In our case: /LogicAppTestService/Service1.svc
Public Address: Required. This is the public full URL of the service. In our case: http:///LogicAppTestService/Service1.svc
In the Binding tab, you can configure any timeout and encoding-related properties of the underlying WCF-WebHttp binding. These properties are helpful when dealing with large messages.
In the Security tab, configure any security properties.
Click the OK button, to save your configurations.
For the Receive handler, select PassThruReceive from the list and select OK to save your configurations.
And finally, create our Logic App to send messages to BizTalk Server
Sign in to the Azure portal. Create a blank logic app.
After Logic App Designer opens, in the search box, enter Request as your filter, and from the triggers list, select the When a HTTP request is received trigger
Select + New step
In the search box, enter BizTalk Server as your filter
From the connector list, select the Send message action form the BizTalk Server connector
Once the action is added to your logic app, you need to setup the connections, perform the following actions:
Select the option Connect via on-premises data gateway and on the gateway properties, select the Subscription and the desired Connection Gateway.
On the Connection Name property, provide a proper name for your connector
On the BizTalk Server URL property, provide a public UTL for the Management Application on the BizTalk Server IIS
On the Authentication Type property, set Windows and provide a proper Username and Password to access the above service (BizTalkManagementService)
Select Create
Once you create the connector, the Send message action will appear on the Logic App designer
From the Receive Location list, select the receive location we just create above
On the Input Message, specify the Body token of the When a HTTP request is received trigger
The Logic App Send Adapter is responsible for sending messages from BizTalk Server to Logic Apps.
We need to first create a Logic App in our Azure Subscription has as a trigger the When a HTTP request is received present in the Request built-in Connector.
Note: The Logic App adapter doesn’t support the new Logic App (preview) that are hosted on Azure Function runtime.
We need to configure TLS 1.2 as the default security protocol on BizTalk Server.
And finally, create a BizTalk Server Application to send message to Logic App
Right-select Send Ports, select New, and select Static One-way Send Port
In the Send Port properties, enter the following:
Name for the send port. For example, enter POC_SEND_MSG_LOGIC_APP.
For the Type, select LogicApp from the list, and select the Configure button
In the General tab, configure the Callback URI of your logic app trigger by selecting Configure…
On the Logic App Details Page, select Sign-in to Azure and authenticate with an Azure account
After you authenticate, you can be able to access your Azure and select the Subscription, the Resource Group that contains the Logic Apps, and finally the Logic App and the Trigger
The Trigger will be always manual
Select OK to save your configurations.
In the Messages tab, set the content-type header as:
Content-Type:application/json
Setting up the BizTalk Server 2020 Logic App adapter
I told above that almost everything will be already configured. However, there is a critical bug in the default installation of the Logic App adapter that will affect the process of receiving messages from Logic Apps using the BizTalk Server Connector.
The Logic App Receive handler, or what we normally call the Logic App Receive adapter is by default configured to use the default In-Process Host, normally the BizTalkServerApplication, in this case as you saw in the picture bellow BizTalkServerReceiveHost.
If you leave this configuration, you will end up having errors when trying to activate a Receive Location that uses the Logic App adapter:
The receive location “Receive Location name” with URL “/iis-application-name/Service1.svc” is shutting down. Details
This happens assigned because the Receive handler is associated with the In-process Host and it should be bound to the Isolated Host.
To fix this bug we need to:
Remove the adapter from all assigned send ports and receive locations in my applications
Therefore, is important to do this immediately after the installation and configuration of your BizTalk Server environment. Otherwise, it will affect your existing application that uses the Logic App adapter to send messages to Logic Apps.
Until now, I usually have used the Logic App adapter to send messages to Azure Logic Apps and extend the BizTalk Server capabilities with the Azure Services. Yesterday, once I was trying the inverse capabilities, i.e., receiving a message from Logic App into BizTalk Server using the Logic App Adapter and, of course, the BizTalk Server Connector available on Logic App. I was surprised with the following error while I was trying to access the exposed service to receive messages from Logic App:
Receive location for address “/LogicAppTestServoce/Service1.svc” not found. (The BizTalk receive location may be disabled.)
This is a common error. It means that the Receive Location doesn’t exist or it is disabled. So I went to the BizTalk Server Administration Console and Enabled the Receive Location, but it automatically disabled again.
Once I check the Event Viewer for errors I found the 3 following errors:
The Messaging Engine encountered an error when creating the receive adapter “LogicApp”. The Assembly is: “Microsoft.BizTalk.Adapter.LogicApp.Runtime.LogicAppReceiver, Microsoft.BizTalk.Adapter.LogicApp.Runtime”. The error occurred because the component does not implement the mandatory interface “IBTTransportControl”.
The Messaging Engine failed to add a receive location “POC_LOGICAPP_TO_BIZTALK_LA” with URL “/LogicAppTestService/Service1.svc” to the adapter “LogicApp”. Reason: “80070057”.
The receive location “POC_LOGICAPP_TO_BIZTALK_LA” with URL “/LogicAppTestService/Service1.svc” is shutting down. Details:”The Messaging Engine failed while notifying an adapter of its configuration. “.
Cause
There is a critical bug in the default installation of the Logic App adapter that will affect the process of receiving messages from Logic Apps using the BizTalk Server Connector.
The Logic App Receive handler, or what we usually call the Logic App Receive adapter, is, by default, configured to use the default In-Process Host, normally the BizTalkServerApplication, in this case, as you saw in the picture bellow BizTalkServerReceiveHost.
Like what happens with the HTTP adapter, the Logic App Adapter is one of the adapters that support two-way communications. Still, unlike other adapters, this adapter has two characteristics that define it:
The Logic App Receive Adapter that is responsible for delivering messages to BizTalk is, in fact, a WCF Service that runs inside Internet Information Services (IIS).
And for that reason, it must be configured in IIS – it is not there out-of-the-box.
This means that when we create and configure a receive location that uses the Logic App adapter inside the BizTalk Server Administration Console, this receive location uses an application within IIS.
So, if you leave this Logic App Adapter default configuration, you will end up having the above errors when trying to activate a Receive Location. This happens assigned because the Receive handler is associated with the In-process Host and it should be bound to the Isolated Host.
Solution
To fix this bug we need to:
Remove the adapter from all assigned send ports and receive locations in my applications
Therefore, it is essential to do this immediately after the installation and configuration of your BizTalk Server environment. Otherwise, it will affect your existing application that uses the Logic App adapter to send messages to Logic Apps.
Last weekend I delivered a session about BizTalk Server 2020 in action on Global Integration Bootcamp 2020 Madrid, where I spoke about the new features available in the most recent version of the product.
Today I will start this series of blog posts about what’s new in BizTalk Server 2020 by beginning to speak about one of the most expected and log time requested features: XSLT 3.0 Support.
XSLT 3.0 represents a significant upgrade of the XSLT 1.0 (and even XSLT 2.0) standards to become a general-purpose transformation language for the most common data storage and messaging formats.
There are a lot of advantages to using XSLT 3.0 compare with the XSLT 1.0 or, in fact, XSLT 2.0, and the goal here is not to address every single of them, but we can highlight the following one:
Extended Function Set, Sequences, Arrays, and Maps: The initial function set for XSLT1.0 was the same as XPath 1.0 functions and was very limited. Minimal math support, no regular expression support, minimal string manipulation capabilities, no support for set (sequence) operations, no support for dates.
Grouping:
xsl:for-each-group: a set of items can be selected and arranged into groups based on specific criteria (for example, common values); then each group can be processed in turn
special XPath functions within for-each-group: current-grouping-key(), currentgroup()
Variants of <xsl:for-each-group>:
group-adjacent: adjacent items are allocated to the same group if they have common values for the grouping key
group-starting-with: whenever an item matches the pattern, a new group is started with this item
group-by: whenever an item matches the pattern, a new group is started after this item
Sorting:
xsl:sort: can be used inside of <xsl:for-each>, <xsl:for-each-group>, <xsl:applytemplates> and <xsl:perform-sort>. It defines the order in which the data is processed by the instruction.
Several subsequent sort keys can be defined
BizTalk’s default XSL transform engine implementation is based on .Net Framework XSLT Transformations. This support is limited to XSLT 1.0.
Starting with BizTalk Server 2020, users can choose Saxon:registered: 9 (Saxon 9 HE) as the XSLT transform engine. But most importantly, it will be possible to plug-in your own custom XSLT transform engine.
YYou can implement a custom XSLT transform engine by defining XSLT transform implementation derived from abstract class Microsoft.XLANGs.BaseTypes.ITransform2 in assembly Microsoft.XLANGs.BaseTypes.dll.
This makes it possible for BizTalk server maps to support newer versions of XSLT. Using Saxon 9 HE, we can readily use XSLT3.0.
Use XSLT Transform Engine property to specify the XSLT transform engine you wish to use:
This was one of the most expected features, but be careful, this can be a poisoned gift because Saxon 9 HE doesn’t support embedded scripting. As a result, functoids shipped as part of BizTalk may not function well. And for many cases, this can be a showstopper.
The advantages of using XSLT 3.0 with Saxon 9 HE:
Development: The XSLT file can be developed separately and hosted in a BizTalk map. It will simplify specific scenarios, especially scenarios that grouping and sorting are required. As we mentioned above, XSLT 3.0 provides an extended set of functions, which was very limited within XSLT 1.0.
Disadvantages when using XSLT 3.0 with Saxon 9 HD:
Level of effort: Not quite as intuitive, Functoids are easier to read visually on the map grid and therefore Requires “geeky” coding skills.
Overview: You lose the visual map representation. You can conjugate Saxon 9 HE with the BizTalk Mapper, but it is harder to do it.
BizTalk Mapper Editor limitations: except for a few built-in Functoids like Scripting Functoid (not all script types will work properly) or advance Functoids, BizTalk Functoids will not work while using the Saxon 9 HE XSLT Transformation Engine.
BizTalk Development Tool Integration: it is a brand new feature, and it seems that doesn’t have, at least for now, the same level of integration capabilities with the BizTalk Server development tools that we are usual to have and use if we compared with the default transformation engine
Following my last blog post and while I was trying to fix some issues regarding a BizTalk Server 2020 installation and configuration provided by my intern as is learning process. We end up solving some critical problems that allow us to install the platform’s most important components. Nevertheless, several components like BAM Tools, BAM Portal and BizTalk EDI/AS2 Runtime failed to configure:
Once, we inspect the logfile provide the the BizTalk Server Configurarion Wizard we found the following error message:
[2020-11-10 22:17:18:0674 Error Configuration Framework]Feature: [BAM Tools] Failed to configure with error message [Feature is skipped due to failed validation. Please go to Custom Configuration to fix the validation error.
Validation Error: <Exception Message=”Error validating the SSIS Catalog database. Please ensure SQL Server Integration Services is installed on the local machine and SSIS Catalog is created on target SQL Server.” Source=”BAMTools” HelpID=”idsErrorValidateSSISCatalogDatabase”><Exception Message=”SSIS Catalog (SSISDB) does not exist on server BTS2020LABAVMRG. Please create SSIS Catalog.” Source=”Microsoft.BizTalk.Bam.CfgExtHelper.ToolsHelper” HelpID=”error_SSISCatalogNotExists”/></Exception>]
[2020-11-10 22:17:18:0674 Error Configuration Framework]Feature: [BAM Portal] Failed to configure with error message [The Configuration of Feature ‘BAM Portal’ failed because the dependent Feature ‘BAM Tools’ was not configured. ]
[2020-11-10 22:17:18:0674 Error Configuration Framework]Feature: [BizTalk EDI/AS2 Runtime] Failed to configure with error message [Feature is skipped due to dependent feature (BAM Tools) failed to configure correctly.]
Three distinct errors but the last two are a sequence of the first one, all of them are related to the same cause.
Cause
SSISDB was automatically created during the BizTalk Server configuration process on previous versions if we enable BAM Portal. However, BAM Portal was deprecated in this new BizTalk Server version, and the configuration process no longer created the SSIS Catalog. Instead, we need to create the catalog manually.
If you don’t create the SSIS Catalog manually, you will not be able to configure BAM Portal. Nevertheless, you can always perform this operation whenever you think is required, and after that, configure BAM Portal.
Make sure you also configure SQL Server Database Mail feature if you wish to configure BAM Alerts on your BizTalk Server 2020 environment
Solution
The solution is very simple, we need to manually create the catalog by using the following instructions:
Open SQL Server Management Studio and connect to the SQL Server Database Engine.
In Object Explorer, expand the server node, right-click the Integration Services Catalogs node, and then click Create Catalog.
On the Create Catalog window, do the following configuration and click OK.
Select Enable CLR Integration option.
Enter a password to protect the encryption key.
And run the BizTalk Server Configuration again. At the end you will be able to configure all BizTalk Server components.
As the learning process’s path, all my interns or new team members, regardless of what technologies they will be using more, one of the first things they will do is create a BizTalk Server development environment from scratch. Yes, all my team members will have skills in all Integration technologies, BizTalk Server, Logic Apps, Service Bus, APIM, and the list goes on and on.
I don´t mind or care if you get it right the first time or not. Sometimes it is with failures that we learn best and more. And I find it curious that with these exercises, they will find problems that I would never imagine happening, so sometimes it is a learning process and continuous improvement for both or the team.
This is one of these cases. My recent team member, while he was trying to configure BizTalk Server using the BizTalk Server Configuration wizard, was getting a failure in all the components immediately:
As he did not understand why this problem was happening, he asked me for help. Once we analyze the log file generated by the configuration wizard, we saw this error:
[2020-11-10 22:08:14:0631 Error Configuration Framework]Feature: [Enterprise SSO] Failed to configure with error message [<Exception Message=”Failed to create the SQL database ‘SSODB’ on SQL Server ‘BTS2020LABAVMRG’ (with SSO Administrator account ‘SSO Administrators’).” Source=”SSO” HelpID=””><Exception Message=”(0xC0002A21) An error occurred while attempting to access the SSO database.
” Source=”SSO” HelpID=””><Exception Message=”An error occurred while attempting to access the SSO database. See the event log (on computer ‘BTS2020LABAVMRG’) for more details.
” Source=”SSO” HelpID=””/></Exception></Exception>]
Basically, this error message does not show any clue or cause as to why this problem is happening. Nevertheless, it will point to you a source to see more details about this issue: the event viewer.
Once we chech the event viewer we got a very clear error message:
Cannot create file ‘C:Program FilesMicrosoft SQL ServerMSSQL15.MSSQLSERVERMSSQLDATASSODB.mdf’ because it already exists. Change the file path or the file name, and retry the operation.
Cause
The reason for this error to happen is clear. The BizTalk Server configuration wizard is trying to create the database files on the default SQL Server folders, but they already exist.
This error happened because my team member initially didn’t install all the required SQL Server components correctly. Once he was trying to add new features to an existing instance, he accidentally created another SQL Server instance.
After that, he uninstalled all the SQL Server instances and appropriately installed all the components, at least for now, let’s assume so. Nevertheless, while he was trying to do a Basic Configuration, he got that “strange” error.
Thru the SQL Server Management Console, we couldn’t see these databases, not only SSO as the error description, but also with all other BizTalk Server databases. But once we saw that folder, we realize that the uninstalls did not delete BD files created before from the hard drive.
Solution
The solution is very simple, delete all the Unnecessary files from that folder, like:
SSO
SSO_log
BizTalkDTADb
BizTalkDTADb_log
BizTalkMgmtDb
BizTalkMgmtDb_log
BizTalkMsgBoxDb
BizTalkMsgBoxDb_log
BizTalkRuleEngineDb
BizTalkRuleEngineDb_log
And run the BizTalk Servr Configuration again. At the end you will be able to configure BizTalk Server.
Hey… but there still errors over there. Indeed, there are, but I will leave that to another post that will be published very soon. However, this blog post’s original error was solved, and we were able to configure the required and critical BizTalk Server components: SSO, Group, and BizTalk Runtime.
