End to End BizTalk Domain Setup in Windows Azure IaaS Scripts

Got a little over an hour?  Get a full featured multi-server BizTalk 2013 domain running in Windows Azure IaaS!  It is fully configured with Send, Receive, Processing, and Tracking hosts all ready to go.  Since it is scripted, it is repeatable so you can tear it down when you are done and create a brand new fresh environment the next time! 

Cost: For a MSDN user the cost is around $0.55 per hour for the compute time with Medium VMs!  Monthly charges may be incurred for storage, bandwidth, and transaction but in my experience these have be next to nothing.  The costs for non-MSDN users will be significantly higher as these are premium images in Azure.

As promised at the BizTalk Summit in London in March, here are the Windows Azure IaaS Full BizTalk Domain Setup scripts.  Of course these scripts can be used to create any type of domain, not just for BizTalk. And better yet, you only need to update as little as 2 variables file to be able to run them!

Download scripts: Microsoft Azure IaaS BizTalk Domain Setup PowerShell Scripts

Important Points to Note

1. These scripts use Premium IaaS Images.  If you are not using a subscription tied to a MSDN account you will occur significant changes.

2. Due to legal issues related to the Microsoft provided BizTalk Provisioning Tool, I am unable to allow people to download it.  The BizTalk 2013 Images on IaaS contain an older, none working version.  Because of this some additional manual steps are required to complete the auto BizTalk configuration.  I have made the auto configuration an optional step that is enabled by a Flag in the variables file.  The default is false.  I suggest everyone spend 20 minutes to get the files from a developer image using the instructions below until we have an updated IaaS image that includes the files or until we are able to download them from Microsoft.  See below information on “To enable full end to end auto configuration do the following (Note – must be a MSDN users)”

This is what the scripts do:

  • Creates a Virtual Network, Affinity Group, and Azure Storage account
  • Create a Domain Controller with the BizTalk Groups and Users created
  • Creates a SQL Server and assigns the correct permissions for the BizTalk users
  • Creates two BizTalk Servers
  • Optional – Downloads a bunch of helper files to configure the BizTalk Group
  • Optional – Installs Classic Shell (yes, I can not live without a Start button)
  • Optional – Installs BizTalk 2013 CU2
  • Downloads Remote Desktop connections to the newly created VMs

To run the scripts:

  • Review the prerequisites below.
  • Open variables.ps1 and review the variables.  Only the top 8 should need to be looked at or updated.  Of these, only 2 values must be changed.
    • $baseVMName = ‘<abcd1234>’                                 # UPDATE – Must be Globally Unique and 8 characters or less.
    • $subscriptionName = ‘<subscription name>’              # UPDATE – This is the name of your subscription from the publishing file
    • $setupDir = "C:\BizTalkGurus"                                   # OPTIONAL UPDATE – Used for files on the remote and local servers
    • $basePathToScripts = ‘<path to scripts>’                    # OPTIONAL UPDATE – This is used if you run the scripts through an IDE
    • $workStorgeAccount = ’<update for auto configure>’    # OPTIONAL UPDATE – Update with the storage location of the helper files for auto configuration
    • $installCU2 = ‘false’                          #OPTIONAL UPDATE – Must upload BizTalk CU2 exe to the root of the Storage Account above
    • $installClassicShell = ‘false’              #OPTIONAL UPDATE – Must upload classic shell version 4.1 exe to the root of the Storage Account above
    • $installBizTalkProv = ‘false’               #OPTIONAL UPDATE – Must upload 5 files from BizTalk 2013 Dev box to the root of the Storage Account above
  • Run Master.ps1 in an elevated PowerShell window.  If running in an IDE it is best to run the script from the command windows rather than hitting Run. 

The end result will look something like this:

What is included

Prerequisites

  • Tested with PowerShell late- May release.  PowerShell versions before March (sorry can’t keep the versions straight) will fail in the VM creation because static IP Addresses are assigned to the VM’s in the network. 
  • Ensure you have a working PowerShell installation before running.  This can be tested by running Get-AzureLocation to ensure it returns values.  Details on how to set this up can be found here.
  • Scripts must be ran elevated. 

Main PowerShell Scripts

  • Master.ps1 – This is the only file you need to run.  Ensure you run from an elevated PowerShell window.  Takes a bit over an hour for full configuration.
  • variables.ps1  – This is the only file you need to update.  You should only need to look at the first 8 parameters. 
  • 00_Create_Network.ps1 – This script creates the affinity group, storage account, and virtual network.
  • 01_Create_PDC.ps1 – This script create the domain controller by promoting a windows server using remote PowerShell. It also creates all the groups and uses needed for the BizTalk configuration.
  • 02_Create_Servers.ps1 – This script creates the SQL and BizTalk Servers using the Enterprise Edition of the Microsoft provided images. 
  • 03_Configure_Servers_RemoteProp.ps1 – This script uses remote PowerShell to complete additional configuration on the SQL and BizTalk Servers.  If flags are set, files will be downloaded to the servers to complete configuration.
  • 04_Configure_Servers_LaunchRDP.ps1 – This script will launch the Remote Desktop sessions needed to complete the auto configuration.
  • 05_RemoteDesktop – Downloads the remote desktop connect for each server.  This is stored inside a new folder called RemoteDesktop in the scripts folder.
  • 99_RemoveVM.ps1 – End-to-end clean up script.  Removes all items created based on the variables file.  Note – due to limitations in Virtual Networks only run this if you have no other virtual networks on your subscriptions.   If you do, you should comment out the virtual network section of this script.
  • SaveMoney_StopAll.ps1 – Scripts to stop and deallocate all the Azure Virtual Machines based on the values in your variables file.
  • SpendMoney_StartAll.ps1 – Scripts to start all the Azure Virtual Machines based on the values in your variables file.
  • Configs folder – NetworkConfig.xml – This is the XML Configuration used to create the virtual network in Azure.
  • Helper Files – Helper files are only needed if you plan to auto-configure BizTalk, auto install CU2, or auto install Classic Shell
    • multinodeconfigDemo_DomainHosts.xml – XML file used by the BizTalk Provisioning tool to auto configure the BizTalk group.  No changes are needed to this file.
    • rdp.exe – Use to auto launch a Remote Desktop session to complete the BizTalk configuration. 
    • RunBizTalkTask_Domain.xml – File used to create a Windows Scheduled Task.
    • RunLocalClient.ps1 – PowerShell script to be ran locally on the First BizTalk Server in the group. 
    • RunLocalClientServer2.ps1 – PowerShell script to be ran locally on all Other BizTalk Server in the group. 
    • StartPowershell.exe – Helper exe to enable Windows Scheduled Task to easily run a PowerShell script in elevated mode.
    • Optional – ClassicShellSetup_4_1.exe – You need to be download from http://www.classicshell.net/ and uploaded to your container.   Ensure the version names match what is in the 03_Configure_Servers_RemoteProp.ps1 file.
    • Optional – BizTalk 2013 CU2 – This needs to be downloaded yourself and uploaded to your own public Azure container.
    • Optional – BizTalk.Provisioning.files- This needs to do downloaded yourself and uploaded to your own public Azure container.

To enable full end to end auto configuration do the following (Note – must be a MSDN users):

  • Put all helper files on your own public Azure storage account container
  • Update $workStorgeAccount in the variables file to the full address of your storage container
  • To enable auto install BizTalk 2013 CU2
    • Download BizTalk 2013 CU2 exe and upload it the root of your storage account
    • Update $installCU2 in the variables file to “true”
  • To enable auto BizTalk configuration
    • Create a BizTalk 2013 Development Virtual Machine in Microsoft Azure (script here)
    • Copy the 5 files in the C:\BizTalk_Provisiong folder
    • Upload the 5 files to the root of your storage account

If you do the auto configuration (witch is highly recommended for anyone who plans to run the scripts more than once) your end result will look like this.

Known Issues and Limitations:

  • At present the auto configuration is only possible if you download and host the BizTalk Provisioning files yourself.  This will change in the future and I will release a new set of scripts once the process is cleaner.
  • Every once in a while Azure is really slow.  I have added some speed breaks in the scripts to better account for this.
  • Every once in a while the auto configuration fails.  The servers get joined to the group by no hosts are created.  At that point, they should just be created manually or remove and restart all the scripts from scratch.
  • If you have an existing virtual network associated with your subscription the 99_RemoveVM script will delete it or fail trying.  This is a limitation in PowerShell. 

These scripts are over a year in the making!  Ok, ok, do not laugh.  I did not spend a year straight on them but I have spent over 300 hours trying to get them fully working end to end.  I have create no less than 100 domains while testing.  I never have an issue with the IasS side of things until the most recent release of PowerShell so things seem to be a little unstable at the moment,  All the issue I have ran into have been related to setting up the auto configuration.

Enjoy and I look forward to community feedback!

Microsoft Azure IaaS Full BizTalk Domain Setup PowerShell Scripts

Microsoft Azure IaaS Full BizTalk Domain Setup PowerShell Scripts

Download this sample using the download button below.

Change Log

1.0 (06/02/2014) – Initial release.
1.1 (06/10/2014) – Better VNet support, added per VM size configuration, auto log off when configuration is complete, added note that base VM name must be all lower case.

 

Important Points to Note

1. These scripts use Premium IaaS Images.  If you are not using a subscription tied to a MSDN account you will occur significant changes.

2. Due to legal issues related to the Microsoft provided BizTalk Provisioning Tool, I am unable to allow people to download it.  The BizTalk 2013 Images on IaaS contain an older, none working version.  Because of this some additional manual steps are required to complete the auto BizTalk configuration.  I have made the auto configuration an optional step that is enabled by a Flag in the variables file.  The default is false.  I suggest everyone spend 20 minutes to get the files from a developer image using the instructions below until we have an updated IaaS image that includes the files or until we are able to download them from Microsoft.  See below information on “To enable full end to end auto configuration do the following (Note – must be a MSDN users)”

3. The $baseVMName variable must be 8 characters or less and all lower case.

4. If the scripts fails, ensure you run 99_RemoteAll before re-running the scripts. 

5.  Due to limitations with PowerShell’s support for Virtual Networks, the 99_RemoteAll script will remove all Virtual Networks that do not have resources attached to them.  Ensure this is what you want before you run the script.  If you have existing Virtual Networks in use, the 99_RemoveAll script will fail on the removal step and the Virtual Network will need to be removed manually.


This is what the scripts do:

  • Creates a Virtual Network, Affinity Group, and Azure Storage account
  • Create a Domain Controller with the BizTalk Groups and Users created
  • Creates a SQL Server and assigns the correct permissions for the BizTalk users
  • Creates two BizTalk Servers
  • Optional – Downloads a bunch of helper files to configure the BizTalk Group
  • Optional – Installs Classic Shell (yes, I can not live without a Start button)
  • Optional – Installs BizTalk 2013 CU2
  • Downloads Remote Desktop connections to the newly created VMs

 

To run the scripts:

  • Review the prerequisites below.
  • Open variables.ps1 and review the variables.  Only the top 8 should need to be looked at or updated.  Of these, only 2 values must be changed.
    • $baseVMName = ‘<abcd1234>’                                 # UPDATE – Must be Globally Unique and 8 characters or less.  All lower case.
    • $subscriptionName = ‘<subscription name>’              # UPDATE – This is the name of your subscription from the publishing file
    • $setupDir = “C:\BizTalkGurus”                                   # OPTIONAL UPDATE – Used for files on the remote and local servers
    • $basePathToScripts = ‘<path to scripts>’                    # OPTIONAL UPDATE – This is used if you run the scripts through an IDE
    • $workStorgeAccount = ‘<update for auto configure>’    # OPTIONAL UPDATE – Update with the storage location of the helper files for auto configuration
    • $installCU2 = ‘false’                          #OPTIONAL UPDATE – Must upload BizTalk CU2 exe to the root of the Storage Account above
    • $installClassicShell = ‘false’              #OPTIONAL UPDATE – Must upload classic shell version 4.1 exe to the root of the Storage Account above
    • $installBizTalkProv = ‘false’               #OPTIONAL UPDATE – Must upload 5 files from BizTalk 2013 Dev box to the root of the Storage Account above
  • Run Master.ps1 in an elevated PowerShell window.  If running in an IDE it is best to run the script from the command windows rather than hitting Run. 

The end result will look something like this:


What is included

Prerequisites

  • Tested with PowerShell late- May release.  PowerShell versions before March (sorry can’t keep the versions straight) will fail in the VM creation because static IP Addresses are assigned to the VM’s in the network. 
  • Ensure you have a working PowerShell installation before running.  This can be tested by running Get-AzureLocation to ensure it returns values.  Details on how to set this up can be found here.
  • Scripts must be ran elevated. 

Main PowerShell Scripts

  • Master.ps1 – This is the only file you need to run.  Ensure you run from an elevated PowerShell window.  Takes a bit over an hour for full configuration.
  • variables.ps1  – This is the only file you need to update.  You should only need to look at the first 8 parameters. 
  • 00_Create_Network.ps1 – This script creates the affinity group, storage account, and virtual network.
  • 01_Create_PDC.ps1 – This script create the domain controller by promoting a windows server using remote PowerShell. It also creates all the groups and uses needed for the BizTalk configuration.
  • 02_Create_Servers.ps1 – This script creates the SQL and BizTalk Servers using the Enterprise Edition of the Microsoft provided images. 
  • 03_Configure_Servers_RemoteProp.ps1 – This script uses remote PowerShell to complete additional configuration on the SQL and BizTalk Servers.  If flags are set, files will be downloaded to the servers to complete configuration.
  • 04_Configure_Servers_LaunchRDP.ps1 – This script will launch the Remote Desktop sessions needed to complete the auto configuration.
  • 05_RemoteDesktop – Downloads the remote desktop connect for each server.  This is stored inside a new folder called RemoteDesktop in the scripts folder.
  • 99_RemoveVM.ps1 – End-to-end clean up script.  Removes all items created based on the variables file.  Note – due to limitations in Virtual Networks only run this if you have no other virtual networks on your subscriptions.   If you do, you should comment out the virtual network section of this script.
  • SaveMoney_StopAll.ps1 – Scripts to stop and deallocate all the Azure Virtual Machines based on the values in your variables file.
  • SpendMoney_StartAll.ps1 – Scripts to start all the Azure Virtual Machines based on the values in your variables file.
  • Configs folder – NetworkConfig.xml – This is the XML Configuration used to create the virtual network in Azure.
  • Helper Files – Helper files are only needed if you plan to auto-configure BizTalk, auto install CU2, or auto install Classic Shell

    • multinodeconfigDemo_DomainHosts.xml – XML file used by the BizTalk Provisioning tool to auto configure the BizTalk group.  No changes are needed to this file.
    • rdp.exe – Use to auto launch a Remote Desktop session to complete the BizTalk configuration. 
    • RunBizTalkTask_Domain.xml – File used to create a Windows Scheduled Task.
    • RunLocalClient.ps1 – PowerShell script to be ran locally on the First BizTalk Server in the group. 
    • RunLocalClientServer2.ps1 – PowerShell script to be ran locally on all Other BizTalk Server in the group. 
    • StartPowershell.exe – Helper exe to enable Windows Scheduled Task to easily run a PowerShell script in elevated mode.
    • Optional – ClassicShellSetup_4_1.exe – You need to be download from http://www.classicshell.net/ and uploaded to your container.   Ensure the version names match what is in the 03_Configure_Servers_RemoteProp.ps1 file.
    • Optional – BizTalk 2013 CU2 – This needs to be downloaded yourself and uploaded to your own public Azure container.
    • Optional – BizTalk.Provisioning.files– This needs to do downloaded yourself and uploaded to your own public Azure container.


To enable full end to end auto configuration do the following (Note – must be a MSDN users):

  • Put all helper files on your own public Azure storage account container
  • Update $workStorgeAccount in the variables file to the full address of your storage container
  • To enable auto install BizTalk 2013 CU2

    • Download BizTalk 2013 CU2 exe and upload it the root of your storage account
    • Update $installCU2 in the variables file to “true”
  • To enable auto BizTalk configuration

    • Create a BizTalk 2013 Development Virtual Machine in Microsoft Azure (script here)
    • Copy the 5 files in the C:\BizTalk_Provisiong folder
    • Upload the 5 files to the root of your storage account


If you do the auto configuration (witch is highly recommended for anyone who plans to run the scripts more than once) your end result will look like this.

 

Known Issues and Limitations:

  • At present the auto configuration is only possible if you download and host the BizTalk Provisioning files yourself.  This will change in the future and I will release a new set of scripts once the process is cleaner.
  • Every once in a while Azure is really slow.  I have added some speed breaks in the scripts to better account for this.
  • Every once in a while the auto configuration fails.  The servers get joined to the group by no hosts are created.  At that point, they should just be created manually or remove and restart all the scripts from scratch.
  • If you have an existing virtual network associated with your subscription the 99_RemoveVM script will delete it or fail trying.  This is a limitation in PowerShell. 

 

Enjoy and I look forward to community feedback!

%ufeff

Sentinet – Service Virtualization Part 6 – BizTalk Server

In the previous post I have demonstrated how I can secure my internal BizTalk services application with X.509 certificates security (that might represent a hypothetical B2B application-to-application scenario). For that, I did not have to change anything in the way my internal BizTalk application is deployed or configured. I accomplished that by simply virtualizing the BizTalk service through a

ESB Toolkit Tip #7: Use Business Rules to make your solution reusable and dynamic

In classic BizTalk are Business Rules not often used in solutions because they are a bit undervalued and also out of the box you can only use them in Orchestrations. With the ESB Toolkit it’s almost the other way around. The ESB Toolkit is all about creating reusable components and making your solution dynamic so at runtime is decided what to do for a specific message type. To accomplish this you use a Resolver to resolve the itinerary, map or endpoint and Resolvers that use Business Rules make it really dynamic! Business Rules also make your components reusable because in your component you don’t need to implement logic anymore that’s specific for a particular message type.

The BizTalk ESB Toolkit includes several resolvers but only the BRE and BRI Resolver use Business Rules. They can be used in the following scenarios:

 

Specifying Itineraries

With the BRI Resolver you can select an itinerary based on the content or properties of the received message in BizTalk. This is very useful if you want to receive multiple message types in the same location and handle them by different itineraries. This can be for example a folder but also a table in a database or a web service with multiple actions.

 
 
 
 
 
 
 
 
 
 
 
 

 

 
 
 

  

Dynamic Routing

With the BRE Resolver you can dynamically route a message, based on the content or properties of the message. In this case you don’t create an itinerary for each message type but you create a generic itinerary that can handle multiple message types. With Business Rules you can decide to which location or endpoint a particular message has to be sent. Note that using the BRE Resolver for Dynamic Routing is not really suited for larger projects because in an OTAP street you have to call probably other web services in your development environment then in the production environment. In that case you have to create Business Rules Policies for each environment. For larger environments is the Sentinet Resolver more suited.

 
 
 
 
 
 
 

 

 
 
 
 
 
 

Dynamic Transformation

With the BRE Resolver you can also dynamically get a map based on the content or properties of the message . This is also very convenient when you want to create a generic itinerary for multiple message types.

 

 

 

 

 

 

 

Custom Solutions

There are also other scenarios where Business Rules can be quite useful like for example message validation. It’s not possible out of the box but you can modify and/or extend %u200bthe ESB Toolkit services and components to suit your specific requirements.

In the following example is the Itinerary Designer extended and an Orchestration Extender created which contains a Policy and a DocumentType property. These properties are used in a Validation Orchestration Service which validates the message by using the Business Rule Engine.

 

 

 

 

 

 

 

 

See Also

For more information on how to use Business Rules in the ESB Toolkit see:

TechEd NA Videos Now Online

TechEd NA Videos Now Online

I recently had the pleasure of speaking at Microsoft TechEd in Houston, TX, and the videos of those sessions are now online. A few thousand people have already watched them, but I thought it’d be good to share it here as well. The first one– Architecting Resilient (Cloud) Applications – went through a series of […]
Blog Post by: Richard Seroter

Why full NuGet support for BizTalk projects is important!

Why full NuGet support for BizTalk projects is important!

Let’s start with a summary for those who don’t feel like reading the full post.

Using NuGet to handle BizTalk dependencies for shared schemas, pipeline components and so on works fine today.

As .btproj files however aren’t supported by NuGet (as shown in this pull request) and are not in the current white list of allowed project types Package Restore will not work (issue closed as by design here).

Not having Package Restore of course is a problem as one now is forced to check in all packages as part of the solutions, something that in the end leads to bloated and messy solutions.

So please reach out to your Microsoft contacts and let’s get this fixed!

NuGet

As most people know NuGet is the package management solution from Microsoft for .NET. It started off as an initiative to further boost open source within the .NET community and NuGet packages uploaded to the NuGet.org platform are open and available directly within Visual Studio through the NuGet add-in. Currently there are well over 20 000 open packages for everyone to download and use.

Lately there has however been lots of discussions within the community to use NuGet as a package manager for internal shared resources as well (by Hanselman and others). Solutions like MyGet allows for private NuGet feeds – only available to those within your organization but still levering all the ease and control offered by NuGet.

Using NuGet for references has a number of of advantages:

  • Communication
    All available resources are directly visible in Visual Studio and when updates to a used library is a available a notification is shown. No more spam mails about changes and never read list of available libraries.

  • Versioning
    A NuGet package has it’s own versioning. This is useful as it isn’t always optimal to change the dll version, but by using the NuGet package version one can still indicate that something has changed.
    As you also reference a specific version of a NuGet package from your solution you always have full control of exactly what version you’re targeting and where to find the built and ready bits.

  • Efficiency
    When starting to work on a project with many references one first have to get the source code from source control for the references, build these (hopefully in the right version … hopefully you have your tags and labels in order …) until all the broken references are fix.
    With NuGet references this just works straight away and you can be sure you get the right version as the resource isn’t the latest from source control but the actual built dlls that’s part of the referenced NuGet package.

NuGet Feeds

As mentioned NuGet feeds can be public or private. A NuGet feed is basically a RSS feed with available resources and versions of these. A feed and a NuGet server can be a hosted web based solution or something as simple as a folder where you write your NuGet packages to. The NuGet documentation covers these options in depth.
The point being that creating your own private NuGet feed is very simple!

So if you haven’t already realized it by now – NuGet is not only a great tool to manage all public external dependencies but can add a a lot a value for internal references as well.

Couple of relevant NuGet features

  • NuGet Package Restore
    NuGet Package Restore enables NuGet to download the used referenced package from the package area. The goal is to avoid having to check in the actual references in source control as this will bloat the version control system and in the end the create a messy solution.

  • NuGet Specification (nuspec) metadata token replacements
    All packages are based on a nuspec file that dictates the version, package description and other meta information.
    NuGet has the capability to by using replacement tokens (such as $version$ for example) read some of this information from the AssemblyInfo files.
    This is far from a critical feature but nice to have and avoid having to repeat oneself and have the same information in a number of places.**

BizTalk and NuGet?

A typical BizTalk solution has a number of shared resources such as shared schemas, shared libraries and pipeline components. As the resources usually are shared between a number of project they often live in a separate development cycle. So when opening a BizTalk project with such resources it’s not only a lot of work getting the referenced code and building the references, there’s also this nagging feeling that it might not be in the right version and that the source might have changed since the first time the reference was added.

Another reference issue occurs when using a build server for a solution with references. As the solution has a dependency to the referenced project one has to make sure not only that the main solution is fetched to the build workarea by the build server, but also that all the referenced project are fetched from version control – and again, hopefully the latest version in the attended version …
This kind of works using TFS Build Service and common TFS Source Control. If however one’s using Git and have resources in separate repositories this becomes impossible as TFS Build Service currently only supports fetching a single repository per build definition to the build workarea …
(This issue does not apply for TeamCity that has a number of different options for dependency management)

All the these issues are actually solved when using NuGet references instead of traditional references as we can be sure we’re getting the packaged dlls as part of the NuGet package in the version that one referenced and not the latest checked in version.
A NuGet reference also makes things a bit easier when it comes to managing the workarea for the TFS Build Service as one only have to sure the NuGet package is available (either as checked in as part of the solution or by using Package Restore).

But …

NuGet doesn’t support BizTalk projects!

As disused here NuGet currently doesn’t support .btproj files. As BizTalk project files are are basically a standard .NET project file with some extra information a two line change in the NuGet whitelist is needed as in this pull request.

So the main issue it that by not having full support of .btproj files Package Restore won’t work and we’re for now force to check in all the NuGet packages as part of our solutions.
An other minor issue is that the token replacement feature also doesn’t work. I also think that if we could actually get full BizTalk support we’d see more BizTalk specific open shared packages for things like useful libraries and pipeline components.

Call for action: Reach out to the NuGet Team or other Microsoft connections and let’s get those two lines added to the white list!