by community-syndication | Dec 18, 2008 | BizTalk Community Blogs via Syndication
Article Source: http://geekswithblogs.net/michaelstephenson
Versioning is one of the more difficult aspects of a BizTalk project mainly because of the ability to deploy side by side versions of the components within a BizTalk application and the fact that often a BizTalk project would require long running processes making it more difficult to do deployments. There are a couple of useful resources available on this area:
-
-
In the MSDN documentation for BizTalk there is a
page which discusses versioning and the 2 main strategies for it. This is useful but from my experience I think it misses a couple of things and is wrong in relation to the BAM files
-
This
web cast includes a discussion around BizTalk versioning (mainly focusing on how to partition your BizTalk artefacts)
-
Nick Heppleson wrote an article about how you can use assembly redirection with BizTalk
-
There is also a new article on MSDN which provides some information similar to this article called Developing Integration Solutions using BizTalk Server 2006 and Team Foundation Server. Although again they explain how you could do versioning but don’t seem to walk through their approach.
While all of these articles are great there is a gap in that nothing really talks about how to implement a versioning approach in your development process. In the rest of this article I will discuss some of the different considerations and how we implemented dynamic versioning for our BizTalk projects at my current project. I will also provide links to a sample which shows this.
Considerations for Versioning with BizTalk
There are a number of different things you should consider for your versioning strategy which are discussed below.
Type of Versioning
There are two main types of versioning which are discussed in the MSDN article above:
In static versioning you would normally keep the Assembly Version Number the same all of the time and only increment the Assembly File Version with each build. This is the most common type of versioning approach I have seen used.
In dynamic versioning you would increment both the Assembly Version and Assembly File Version with each build. The most common way I have seen this implemented is to manually change these version numbers (this subsequently requires a number of changes to files within the project)
Deployment Pattern
The way you plan to deploy your application can have an affect on the versioning approach you might use.
- Replace application every time
Some BizTalk applications implement processes where there are clear windows where a deployment can be done. There are no significantly long running processes and it is easy to drain the system of in process messages. In this case when you deploy a new version of the application you would be likely to completely remove the existing BizTalk application and then add the new version.
This approach is well suited to either of the 2 types of versioning.
If you have a BizTalk project where you can not get a clear deployment window and you can not drain the system then your deployment approach needs to be able to support running multiple versions of processes at the same time. In this case you are essentially applying updates to your existing application and when the older instances have completed you may remove the old assemblies.
Latest Version for Development
One of the challenges is how within your code base you will implement these side by side versions and develop/test against them. When we deploy an official version we will take those assemblies and add them to an external libraries folder within our code base. When we build the code we will also deploy these older versions of the code and then run tests against this combined solution. One key thing we do is that we version the BizTalk project files in the code base to be version 999.999.999.999. When we do a local developer build we do not change the version numbers we only do this on our build server when the build is ran by Cruise Control. We need to ensure that the assemblies built from the code will always be the latest version when doing a developer build so we chose this approach.
What types of files do I need to care about for versioning?
There are a number of different file types you need to care about when applying versioning. In my opinion the above article on MSDN doesn’t cover everything it needs to so here are my thoughts below:
Binding Files
We already use an approach of tagging the binding files to support changing the configuration for different environments based on stuff I have discussed before in other blog posts. In a binding file any references to assemblies will typically use the strong name for that assembly. When we apply dynamic versioning in the development process we will need to update parts of the binding file in each build to ensure the binding file continues to be valid.
If this is not done correctly the we could get problems importing a binding file.
Web Service Description Files
If you publish any web services from your solution then you may choose to save the web service description files and then regenerate the web services as part of your build. We save these files and have MsBuild tasks to support regenerating these web services with each build for WCF/WSE2 and standard SOAP web services. When we apply version updates we need to amend the WebServiceDescription.xml file that is published before we do the build.
Map Files
BizTalk Map files may contain references to assemblies if you execute an external assembly using the scripting functoid. We need to ensure these maps are updated if the assembly version is changed. If we do not do this we may get errors while executing the map.
Pipeline Files
BizTalk Pipeline files may have references to external pipeline components or schemas or other assembly references in custom configuration. If we apply dynamic versioning we need to be aware of the impact on these files and make appropriate updates to ensure the pipeline still works when it is compiled.
BAM Files
If you use BAM on your project then your solution may contain Tracking Profiles or BAM Observation Models. The BAM Observation Model does not contain information relating to assemblies, but if you use any Tracking Profiles (.btt files) then they may contain references to assemblies which may need to change as you change the version number. In the MSDN documentation it indicates that .btt files are in binary format which Im not sure is right as the ones on our project are clearly XML files. Based on our experience you can just update these files to change the version numbers.
One side comment is that you can also use the Typed BAM API project on CodePlex which will mean you don’t have to worry about Tracking Profiles.
BRE Policies
If you manage your BRE policy as an XML file which you import and export then you may also need to update version numbers in it if you reference assemblies.
Configuration Files
Some configuration files in your solution may have assembly references in them. You may need to update the version numbers here too.
Our Versioning MsBuild Tasks
As you have read so far there are a number of different tasks which need to be done to implement versioning. We have developed the following list of MsBuild tasks which help us to do this in our MsBuild process. Note the code for these tasks is available in the accompanying sample.
BizTalkProjectAssemblyInfo
This task will look for any .btproj files within your solution directory (including sub directories) and then update the version attributes of the project file to take the value from Cruise Control
AssemblyVersion
This task will create a file called VersionNumber.cs which contains the C# attributes for versioning an assembly. All of the C# projects within our solution reference this file and pick up its version number when they are built.
ApplyVersionToAdhocFiles
This task is given a list of files which need to be inspected for versions to be updated. The task will use the assembly name regular expression, assembly public key token, assembly culture and assembly version regular expression to identify strong names within the file that need to be updated. Once a strong name is found the version number part of it is updated to the version passed into the task by Cruise Control.
Note that this task is also used to update any BRE policies which are stored as XML files in the solution.
ApplyVersionToWebServiceDescription
This task does the same as the apply version to adhoc files task however this task needs to force the file to be rewritten in a the right encoding for the web service publishing wizard to use.
ApplyVersionToBindingFiles
This task is given a list of paths to binding files which it needs to update. In the binding files it will look for strong names in the same way as the adhoc files task and update them. In binding files there are also some other formats in which assembly version numbers are stored so this task will also handle these to ensure that the binding file is correctly updated.
ApplyVersionToBizTalkArtefacts
This task will start at the root directory of the solution and look for all BizTalk artefacts which may require updating (Maps, Pipelines, Tracking Profiles). For each of these file types it will look for assembly references based on the strong name patterns provided and apply the updates to the version numbers which come from Cruise Control.
Walk Through
Setting up a new solution to use dynamic versioning
When we are starting with a new solution and we want to be able to implement the dynamic versioning straightaway there are a couple of standards we use.
C# Project Files
In our solution it is common to have C# projects to compliment our BizTalk projects. We take the approach of versioning these assemblies by adding an existing file to the project which includes the below code snippet (we usually call this file VersionNumber.cs and it is located in the root folder of the solution). This means that the assemblies are all versioned to the standard 999.999.999.999 during development
When our MsBuild process executes it will overwrite this AssemblyVersion.cs file and replace the version numbers in it to the version label taken from cruise control. We use our AssemblyVersion custom MsBuild task. There are a number of different tasks in the community and Microsoft.Sdc tasks which can do versioning of C# projects and using any of these should be sufficient.
BizTalk project version numbers
On all of the BizTalk projects within the solution we will amend the project properties (see pic below) to set the version attributes to the fixed 999.999.999.999. This will mean during a local developer machine build all of the BizTalk assemblies will be versioned as 999.999.999.999. When Cruise Control runs a build the extra build tasks will run and execute our BizTalkProjectAssemblyInfo custom MsBuild task. This will update the version numbers of all of the BizTalk project files within our solution to apply the version number from Cruise Control.
Other Artefacts
As we will have applied the 999.999.999.999 version number to our projects before we start adding artefacts to the projects then any files we add will correctly pick up these versions and we will not manually need to update them.
The Cruise Control Integration Bit
In the below picture you can see the Cruise Control configuration used for the Continuous Integration of this solution. In the tasks bit you can see we basically call 3 MsBuild scripts as follows:
- Apply versioning to the files in the solution
- Perform a debug/development build of the solution (including executing tests)
-
Perform a release/deployment build of the solution (including executing tests)
The bit we have added is the first step where we will call ccnet.targets MsBuild script. This file is in the solution and is where we will call the tasks to apply versioning.
Setting up the CCNet.targets File
The below picture shows the contents of the ccnet.targets file. This file will basically call the list of MsBuild tasks in the ApplyVersion target which will apply versioning to all of the appropriate files. Each of these tasks was discussed in detail earlier in this article.
In the property group at the top of this file you can see that we have defined some common attributes which each of the tasks uses to do its job. These properties are used as follows:
|
Property
|
Notes
|
|
AssemblyNamePattern
|
This is a pattern used to identify the name of assemblies. In our case the assemblies will usually be called something like Acme.Something.BizTalk.
We use the regular expression Acme.Something.BizTalk.?\w* which will find assemblies with the right names for us for example:
- Acme.Something.BizTalk
- Acme.Something.BizTalk.Orchestrations
- Acme.Something.BizTalk.Schemas
|
|
AssemblyCulture
|
All of our assemblies have the culture neutral so we use this as a static value in the script
|
|
AssemblyPublicKeyToken
|
All of our assemblies have the same strong name applied usually so we can just keep this as a static value of the public key
|
|
AssemblyDynamicVersionPattern
|
This property is a regular expression which you will use to identify version numbers that need to be changed. In our case because we use the fixed version 999.999.999.999 to indicate it needs to be dynamically changed we just use this value. Having this as a regular expression however allows some flexibility on how you might use this
|
As you can see from the order of calls to our tasks we version the files in the following order:
- BizTalk project file
- C# project file
- Adhoc Files
- Web Service Description Files
- Binding Files
- BizTalk Artefacts
The order here isn’t really that important as long as all of these activities are done before compiling the solution.
Converting an existing solution to use dynamic versioning
This walk through will explain how to configure an existing BizTalk solution to use dynamic versioning rather than static version approaches described above. One key point to reiterate here is that on a developers machine we always do builds as our fixed 999.999.999.999 version number and it is only on the continuous integration build server that we run the additional tasks to do full versioning as described below.
Reconfiguring parts of the solution
C# & BizTalk Project Files
We will need to update the BizTalk and C# projects to reference the 999.999.999.999 version number. We can do this the same as how is described in the section about setting up a new solution for dynamic versioning.
Updating Map Files
Maps are slightly strange. In the map if you reference an external assembly and execute it via the scripting functoid then you need to be aware that the strong name for executing assembly may need to change. In the MSDN documentation explaining this it indicates you need to change maps, but if you disassemble an assembly containing maps and the C# project was referenced as a project assembly then the compiler seems to change the xslt within the map for you. However as this is a little unclear we decided to change the maps anyway.
To do this we needed to open the map using the XML Editor in Visual Studio. We then would change the version numbers highlighted below as 1.0.0.0 (in the pic below) to 999.999.999.999 for all of the assemblies we know we are going to dynamically version.
We would do this for every map in the solution. When Cruise Control runs the build it will modify and strong names in .btm files which need to be versioned.
Pipeline files
Pipelines are similar to maps in that under the .btp file is XML which may contain assembly strong names (see picture). Again we would open the file using the Visual Studio XML Editor and change the version numbers from 1.0.0.0 to 999.999.999.999 for any references to assemblies we are going to dynamically version.
.
When the build is ran by Cruise Control our additional tasks will find all of the details we need to change and modify the .btp file before it is compiled.
Binding file template
In the solution we have a template binding file which contains tags which we use to configure the binding file for different environments. In a binding file there are lots of references to assemblies by their strong name. If we have any assemblies which we know are going to be dynamically configured then we will replace their version with the fixed 999.999.999.999. In the below picture you can see an example binding file template which shows the version numbers before they have been amended.
When Cruise Control runs our build then an additional step is ran which will execute our ApplyVersioningToBindingFiles task. This will modify the binding file correctly to replace the version numbers with the appropriate label from Cruise Control.
Configuration Files
In some of our solutions we include configuration files which may have assembly version numbers in which need to be dynamically versioned. We follow the same approach to make the version number 999.999.999.999 if it needs to be dynamically configured.
In the Cruise Control build a task called ApplyVersionToAdhocFiles is called passing in a list of these adhoc files.
Web service description Files
In some of our solutions where we publish web services from BizTalk we use a couple of MsBuild tasks in our build process which will support regenerating these web services each time the build is ran by using the WebServiceDescription.xml which is produced by the Web Service Generator Wizards. We have MsBuild tasks which support WSE2, SOAP and WCF.
When we use this technique we also need to apply updates to any version numbers which may need to change in the WebServiceDescription .xml. To do this we use the same approach of replacing the versions with 999.999.999.999 for the developer build and then Cruise Control will run the ApplyVersionToWebServiceDescription task to do the dynamic updates.
BAM Tracking Profiles
The BAM tracking profile is an XML file under the hood and we again replace the version number of any of our assembly strong names with 999.999.999.999.
When the Cruise Control build runs it will execute a task which will apply the Cruise Control version label in place of our version number and then when the build deploys BAM artefacts they will be already versioned correctly.
(For details of how we integrate BAM into our MsBuild process please refer to: http://geekswithblogs.net/michaelstephenson/archive/2008/06/15/122870.aspx)
BRE Policy Files
When we use BRE we usually take the approach of managing the policy as an artifact in the solution. We use custom MsBuild tasks to import and export the policy from BRE and store it in the solution as an XML file. If we reference any assemblies in the policy that need to be versioned then we will amend the xml that is exported from BRE and make the version numbers the usual 999.999.999.999. In terms of Cruise Control we treat the BRE exported XML file the same as an adhoc file which requires version updates. We then use the ApplyVersionToAdhocFile task to update this
If you would like to know more about how we implement the BRE features of our build process please refer to the following article: ???
!!!!!! vWhere is BRE post !!!!!
Check local build
Finally when we have done all of the above steps we will run a local developer build to see that the solution is fully versioned, compiles correctly and deploys. We also ensure that the build has ran our BizUnit tests and they are all successful.
The next stage is to get Cruise Control to so some extra stuff.
The Cruise Control Integration Bit
When modifying an existing solution to use dynamic versioning, the requirements for the ccnet.targets file and Cruise Control to implement the version updates is exactly the same as when you setup a brand new solution for dynamic versioning. For details please refer back to that section.
Summary
At this point you can see it is a fair about of work to modify an existing solution to use this versioning technique. I think the earlier you do it in your project life cycle the better. You can use things like Replace All to modify some of the version numbers manually when doing the above tasks but be careful when doing that as you may not want to change all version numbers.
Frequently Asked Questions
In relation to this topic the following questions have occasionally came up.
How do I do side by side versioning in development for a new major release?
The approach described in this article is entirely intended to support development of side by side versions of the same solution. Imagine you have version 1.1.0.0 of your application deployed and we know you want to now work on version 2.0.n.n. You know from your understanding of this topic that you will employ a deployment model which will mean you will have multiple versions of some of your assemblies deployed.
The way you could implement this is to add external Library folder within your solution store the version 1.1.0.0 versions of all of the deployed assemblies. You could also store the appropriate development binding files and any other artefacts to support this update. In your build script you might setup and deploy the existing application (v1.1.0.0) and then modify your build script to apply updates to the application by deploying some or all of your newly compiled code.
In development the newly compiled assemblies always have a very high version number they will always be the latest version and the versioning rules will apply correctly. You will probably also do things like modify binding files to be able to work with two versions of the assemblies. In the binding file remember that the way we do this only our assemblies which are versioned 999.999.999.999 are dynamically changed so if you add details of an older version of the assembly to the binding file we will not touch that instance of a string name as the version number within the string name will not be 999.999.999.999.
When your code is checked in and build on your build server Cruise Control would apply the correct 2.0.n.n (provided you have updated the major number in its config) version number and your build will allow you to have a correctly configured and working side by side versions of the application.
What’s the 999.999.999.999 thing all about?
In applying dynamic versioning to the solution we need to be able to ensure that in our development environment a developer can work on a solution which may have side by side versions. We have chosen to use 999.999.999.999 to indicate this is a developer machine only version of the assembly and it is given a high version number to ensure that when it is build it will always be the latest. Although the build tasks accept a regular expression for the version number which needs to be replaced we find it keeps things simpler by always using this fixed number.
Because we use source control we can not apply dynamic versions on a developers machine local build because the files we need to modify will be read only. A developer will always build version 999.999.999.999 of the solution. When the developer checks code in Cruise Control will replace all of these developer 999.999.999.999 versions with the build label from Cruise Control.
BizTalk 2009 Update
Following the release of BizTalk 2009 CTP I have had a brief look at how the new version would affect this versioning approach. In my opinion the approach stays very much the same with the exception that now BizTalk project files are MsBuild based and you can use the AssemblyInfo file to version the assembly rather than the old BizTalk project file. Based on this the change to our implementation would be to not use the old VersionBizTalkProjectFile task and just use the C# one for BizTalk projects as well.
Summary
As you can see from this article this isn’t a trivial undertaking. If you do it right from the start of the project then hopefully you will find things a lot easier.
Hopefully this article will plug the gap I identified at the start and there is now a detailed walk through of applying effective dynamic versioning to BizTalk projects which use continuous integration. If you would like to look at the code for the MsBuild tasks we used to help us implement this approach then please refer to the following location:
http://www.box.net/shared/7ebfbdjau0
(Note some companies block access to box.net so if you are unable to get the sample please email me via the blog)
by community-syndication | Dec 18, 2008 | BizTalk Community Blogs via Syndication
A while back I discussed the approach we use on some projects to configuration management and how we solve the problem of configuring binding and configuration (and any other) files for different environments. I had originally written the linked article on my blog. While it was fairly successful the main project I’ve been working on has a number of BizTalk projects within it and I felt that the configuration dictionary approach we were using was becoming a bit of a pain as the XML dictionaries were awkward to maintain and you couldn’t get a clear view of your configuration.
To help tackle this problem I wanted to create a tool to help with this, the aims of the tool were:
- Provide a nice simple to use UI which would allow me to manage configuration settings for multiple code bases on large projects
- Provide an easy way to integrate a build process into the use of this configuration store
- Try to enhance the previous approach by allowing you to import configuration as the XML files which people may already be using.
The tool is available on codeplex: http://www.codeplex.com/ConfigSettingsTool
Here are a few points to describe the model for the configuration data:
- The system will store information for multiple projects
- Each project can have multiple environments associated within
- Each project will have multiple solutions (code bases) associated with it
- Each solution will have a list of settings which can have default values
- For each solution within a project you can specify setting values for each environment, or inherit the default setting
In the rest of this article I will describe the use of the tool, and how we implemented it for our BizTalk projects. Note that nothing about this tool is BizTalk specific so you could easily use it for .net projects etc. Also in this article I have used screen shots from a real project but have had to censor some information from the screen shot.
Prerequisites
In order to run the tool the prerequisites are as follows:
- The database should be setup on a SQL Server 2005 database (ensure you configure backups for your database)
- The client application requires .net 3.5
- The MsBuild tasks need to run on a machine which has .net 3.5
Getting Started
To set up the application you should take the following steps:
Setting up the database
In the zip file you will download there is a folder containing the database script. You will first need to create a database (which I’m assuming you can do already) and then use the CreateDatabaseObjects.sql script to setup all of the database objects by running it in query analyser.
Setting up the application
In the zip file there is an application folder. In this folder the ConfigurationManagement.Administration.exe.config file contains settings for the application. You will need to edit the connection string to ensure it points to your new database.
Once you have done this and saved the file you can now run the application by double clicking the ConfigurationManagement.Administration.exe file.
Using the Application
Now you are setup if you open the application you can begin setting up your configuration. If you open the Tools %u00e0 Management menu you will see options for managing the different aspects described above.
Managing Projects
In the picture below you can see the screen for managing your projects. This is pretty simple, you can add/update/delete projects in the grid. Note any changes are not saved until you close the form.
I normally use 1 project per customer or major project of work. In this project you can configure multiple solutions and I normally relate each configuration solution to a visual studio solution.
Managing Environments
Within a project you can also configure multiple environments. As you can see in the below picture I have configured only a couple of environments. Examples of environments you might configure include (System Test, UAT, Production). The environments screen again is fairly simple in that it allows you to add/update and delete environments. Note again changes aren’t saved until you close the form.
Managing Solutions
Within the Project you can also define the details of multiple solutions. The solutions form allows you to add/update and delete solutions. The changes are not saved until you close the form. The below picture shows an example of the solutions within this project.
Manage the Solution Dictionary
Once your solution is configured you can now set up its dictionary. The below picture shows an example of this. Within the solution you will create a list of setting tags. These tags are things you will put in files to indicate a special key that you will replace. We typically use the format $(——-) to indicate a key. For each setting you can define a default value which will be used if no specific setting for that environment is configured.
In the below picture if we ran this solution dictionary over a file we would look for each setting and replace it with the environment specific or default value. If a setting key could not be found then we will move onto the next tag.
Again the form follows the usual format for adding/updating and deleting settings and as before the settings are only saved when the form is closed.
Manage Environment Settings
Now we come to managing the settings for each environment. We use the below form which allows you to choose a project and solution. This will then execute a cross tab query to create a grid listing the solutions configuration tags in the first column and then create additional columns for each environment as shown in the picture below.
The first column is read-only, but the other columns allow you to define a specific value for each environment. If you leave a cell empty then the default value is used instead.
This form is different to the others by the nature of what it does. You can not add of remove rows, only edit existing cells. Also when you make a change the details are saved when you finish editing the cell.
At this point you have now configured your solution and are well under way to effectively managing your configuration for your development effort.
Template Files to Configure
As mentioned previously you will typically have a template file which you will run the configuration dictionary over to produce the correctly configured file for a given environment. Some examples of files we typically configure on our BizTalk projects are:
- A template of BizTalk Binding Files
- A template of Web.config and policy cache files for published web services
- A template of BTSNTSVC.exe.config
In the below picture which is a snippet of a BizTalk Binding File Template you can see that I have circled in red the setting tags which are in the $(——) format (note that I have had to censor the tag named). We maintain this template as an artefact within the solution and then during our build process we will run the dictionary over this template file to produce the real correctly configured files for each environment.
Integrating the Build Tasks into Your Script
Along with the Configuration Management application you will also find a folder with a dll containing some MsBuild tasks. There are 3 tasks:
|
Task Name
|
Description
|
|
ConfigureFileBasedOnDictionary
|
This task will look up the configuration settings from the database and configure a template file.
|
|
ConfigureFileBasedOnDictionaryFile
|
This task will use the old dictionary file format to configure the template file. This will be discussed later.
|
|
ProduceSettingsReport
|
This task will produce an HTML report for the configuration. We use this task to deliver a report of the settings we want them to use to our deployment team.
|
The below picture shows the MsBuild UsingTask statements you would need to use to reference the Configuration Management build tasks. You should also see a ConfigurationManagement.tasks file in the folder containing the MsBuild tasks assembly. You can import this .tasks file into your build script using the Import MsBuild statement.
In the next picture you can see a snippet from our build script. In this case we are configuring two files (a web.config and a policycache file) based on information in the Configuration Management database. You need to tell the task the Project, Solution and Environment which you want information from and also the connection string to your database.
In the bottom part of the above snippet you can see how we use the ProduceSettingsReport task to produce the HTML report I discussed earlier. A sample of this report is below:
Working with the old Dictionary File Based Model
While the new application is very useful and we are able to access configuration directly from the database, some teams may still prefer to store the configuration as a dictionary file like in my original blog post. This has the benefit that you can easily keep the dictionary in a source control system and have a full version history for it. Based on this I wanted to allow teams to make this choice and if they still wanted to use the file based approach then could still take advantage of the tool anyway. This is described below.
Importing Configuration from File
You can begin by importing your dictionary file into the Configuration Management application. The below picture shows how you can choose the Project, Solution and path to the dictionary file. When you import this information any existing setting information is updated, and any new data is added.
Exporting Configuration to File
When you have edited your configuration you may wish to export it again to a file to update the file in your solution. Like with the import you can choose the Solution, Project and path then just export it.
Summary
Hopefully you will find this tool very useful and time saving. There are a couple of limitations and future enhancements you may wish to be aware of if you are considering using this.
-
At present there is no additional security other than if you can access the database. In future releases I plan to add the ability to configure reader and editor roles for each solution/environment/project so this can be locked down more. This will make the tool more usable for bigger projects when you want multiple teams to work on this.
-
At present I do not audit any changes made to the configuration. This would be a useful enhancement.
- Any changes made to configuration will not trigger a build in your Continuous Integration environment. This is one reason why a team may wish to combine using the tool with the old file based dictionary approach because when the updated dictionary file is checked in a build would be triggered. If you do scheduled nightly builds this may mitigate this.
