(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here. 
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

• The “core” NAnt functionality has now been separated from the piece that
  an individual BizTalk project must “own” and maintain.  This is a
  huge simplification, and allows for more rapid adoption — as well as the ability
  to deploy bug fixes & feature enhancements in a way that just wasn’t possible
  when stuff was intermingled.  (Peter Provost was right about
  that aspect of my first attempts.)  The NAnt script that a particular project
  must own can now be reduced to the following for a simple case (notice the ‘include’
  reference for the core):

  <project name="BizTalkSample" default="debugDeploy">
   <sysinfo/>
   <include buildfile="BizTalkDeploymentInclude.nant"/>  
   <-- Set following properties to true or false to 
include various pieces of a BizTalk deployment. -->
   <property name="includeSchemas"  value="true" />
   <property name="includeOrchestrations" value="true" />
   <property name="includeTransforms" value="true" />
   <property name="includePipelines" value="true" />
   <property name="includeComponents" value="true" />
   <property name="includePipelineComponents" value="false" />
   <property name="includeCustomFunctoids" value="false" />
   <property name="includeVocabAndRules" value="false" />
   <property name="includeVirtualDirectories" value="true" />
   <property name="includeMessagingBindings" value="true" />
   <property name="includeDeploymentTest" value="true" />
   <property name="includeCustomTarget" value="true" />
<project/>


• You now have the ability to supply a simple xml-driven Wizard-based UI when you are
  deploying after an MSI install.  You can use this UI to gather
  Windows identity information for virtual directories (for SOAP/HTTP receive ports),
  to indicate whether the scripts should deploy to the BizTalk management database (for
  working in BizTalk groups), and for many other purposes.  I call this “Install-Time
  Configuration” in the documentation below.  Here are some example Wizard
  screens:

  >

• You now have the ability to have an Operations-managed “spreadsheet-driven”
  mechanism for environment-relative configuration.  What do I mean?  I mean
  those aspects of your deployment that are particular to your development, QA, or production
  BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
  files!  Queue names, database names, file shares, FTP locations, etc.  This
  is information that you don’t want to manage with the Wizard UI above (there
  is too much of it) and you want to maintain “as a set” for each environment.
   The spreadsheet (pictured below) can start life with your developers, and ownership
  can gradually migrate to Operations.  This spreadsheet generates environment-specific
  “settings” files, and at the point you deploy, the values for the environment
  you are deploying to are automatically substituted into your binding file…Slick!

  (click)

• You now can use a highly templated (reusable) WiX based
  setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
  in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
  contents of an MSI.  One of the clear pieces of feedback I got on the last rev
  I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

• You no longer need to have your NAnt file enumerate your orchestration names or
  their deployment order – a new custom NAnt task eliminates the need for
  this.
• You no longer need to have your NAnt file enumerate how many receive ports/send
  ports/etc. you have – a new custom NAnt task eliminates the need for this.
• Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
  management database.
• Support for not following the naming recommendations I made in previous releases –
  you can have custom project names, assembly names, directory names – its just
  a little more work.
• Support for multiple assemblies of the various types (i.e. multiple orchestration
  assemblies, schema assemblies, etc.)
• Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
  above.
• Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
  policies and vocabularies.
• Inclusion of Loren Halvorsen’s XmlPreProcess tool
  for managing the environment-relative configuration discussed above.
• Support for creating IIS6 application pools (with specified Windows identity) for
  HTTP and SOAP-based receive locations, and for adding virtual directories to application
  pools.
• Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
• Support for selectively restarting multiple BizTalk host instances, for deploying
  custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here. 

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here. 
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

• The “core” NAnt functionality has now been separated from the piece that
  an individual BizTalk project must “own” and maintain.  This is a
  huge simplification, and allows for more rapid adoption — as well as the ability
  to deploy bug fixes & feature enhancements in a way that just wasn’t possible
  when stuff was intermingled.  (Peter Provost was right about
  that aspect of my first attempts.)  The NAnt script that a particular project
  must own can now be reduced to the following for a simple case (notice the ‘include’
  reference for the core):

  <project name="BizTalkSample" default="debugDeploy">
   <sysinfo/>
   <include buildfile="BizTalkDeploymentInclude.nant"/>  
   <-- Set following properties to true or false to 
include various pieces of a BizTalk deployment. -->
   <property name="includeSchemas"  value="true" />
   <property name="includeOrchestrations" value="true" />
   <property name="includeTransforms" value="true" />
   <property name="includePipelines" value="true" />
   <property name="includeComponents" value="true" />
   <property name="includePipelineComponents" value="false" />
   <property name="includeCustomFunctoids" value="false" />
   <property name="includeVocabAndRules" value="false" />
   <property name="includeVirtualDirectories" value="true" />
   <property name="includeMessagingBindings" value="true" />
   <property name="includeDeploymentTest" value="true" />
   <property name="includeCustomTarget" value="true" />
<project/>


• You now have the ability to supply a simple xml-driven Wizard-based UI when you are
  deploying after an MSI install.  You can use this UI to gather
  Windows identity information for virtual directories (for SOAP/HTTP receive ports),
  to indicate whether the scripts should deploy to the BizTalk management database (for
  working in BizTalk groups), and for many other purposes.  I call this “Install-Time
  Configuration” in the documentation below.  Here are some example Wizard
  screens:

  >

• You now have the ability to have an Operations-managed “spreadsheet-driven”
  mechanism for environment-relative configuration.  What do I mean?  I mean
  those aspects of your deployment that are particular to your development, QA, or production
  BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
  files!  Queue names, database names, file shares, FTP locations, etc.  This
  is information that you don’t want to manage with the Wizard UI above (there
  is too much of it) and you want to maintain “as a set” for each environment.
   The spreadsheet (pictured below) can start life with your developers, and ownership
  can gradually migrate to Operations.  This spreadsheet generates environment-specific
  “settings” files, and at the point you deploy, the values for the environment
  you are deploying to are automatically substituted into your binding file…Slick!

  (click)

• You now can use a highly templated (reusable) WiX based
  setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
  in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
  contents of an MSI.  One of the clear pieces of feedback I got on the last rev
  I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

• You no longer need to have your NAnt file enumerate your orchestration names or
  their deployment order – a new custom NAnt task eliminates the need for
  this.
• You no longer need to have your NAnt file enumerate how many receive ports/send
  ports/etc. you have – a new custom NAnt task eliminates the need for this.
• Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
  management database.
• Support for not following the naming recommendations I made in previous releases –
  you can have custom project names, assembly names, directory names – its just
  a little more work.
• Support for multiple assemblies of the various types (i.e. multiple orchestration
  assemblies, schema assemblies, etc.)
• Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
  above.
• Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
  policies and vocabularies.
• Inclusion of Loren Halvorsen’s XmlPreProcess tool
  for managing the environment-relative configuration discussed above.
• Support for creating IIS6 application pools (with specified Windows identity) for
  HTTP and SOAP-based receive locations, and for adding virtual directories to application
  pools.
• Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
• Support for selectively restarting multiple BizTalk host instances, for deploying
  custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here. 

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

Some time ago, I published a set of proposed
naming conventions for BizTalk 2004.  

In the interim, I’ve had a chance to see these used in a few different projects
and to review them with several people — and they have evolved & grown a fair
bit.  

So I thought I’d publish the new version – the Word version can be found here,
and an HTML version can be found here.
 (I’ve found it helpful to open the HTML version directly in VS.NET for
quick reference.)

Hopefully these will be of value to your project.  I’ve certainly found
that the documentation value to be had within Orchestrations and within the operational
tools can be hugely improved by following these conventions – especially
if you are using the UK
SDC BizTalk 2004 Documenter tool (you are, aren’t you?)  My comments
on the use of such documentation in
my old post still apply, I believe.

Leave comments with questions or suggestions…

I had the pleasure of presenting on BizTalk 2004 at the Heartland
Developer’s Conference 2004.  The whole conference was a lot of fun,
and as Kent
Tegels has noted, Joe Olsen did
a great job in organizing the logistics for this event – you wouldn’t
have known it was the first-of-a-kind (save for Joe’s mea culpa on caffeine…)

I had a chance to talk to Sam Gentile about
the work he has been doing at Addesso.  This was enlightening, since in my non-BizTalk-work-life
I do quite a bit mobile work in the field-force-automation space (using ruggedized
Pocket PC devices – great fun!)

As to the presentation itself – one of the topics I discussed was BizTalk 2004’s
scaling model, and the power of having being able to create multiple host instances
for a single logical host definition that contains your orchestrations.  This
gives you a “competing consumer” effect – multiple processes on
multiple servers, all pulling from a common work queue.  In my talk, I suggested
that this idea wasn’t new and related it back to a 1998
article in Enterprise Developer’s magazine.  In that article, an architect
from Merrill Lynch was discussing the scaling difficulties in using NT4/MTS –
the load balancing options weren’t pretty.  He proposed a pattern he dubbed
“Auctioning” where clients would submit work to an MSMQ queue, and multiple
servers (all running common server-side components) would pull work from that queue.
 The benefits he articulated relate quite closely to what you will find in the
parallel aspects of BizTalk 2004.

Check out this graphic from the article – it should look a little familiar to
BizTalk folks.

Below is my paraphrase of the article’s main points, recast with BizTalk terms
(and exactly relevant to BizTalk, I believe…)

• To load balance multiple servers, servers objects [orchestrations] should pull
  work from a global [per-host] work queue that clients place their requests in.
• Auctioning works well precisely because it relies on the message queue server [messagebox],
  not DCOM [or transport of choice], to route the request to the server object [orchestration].
• Server objects [orchestrations] compete for requests on the queue, so load balancing
  happens naturally; the server with the most available processing capability will automatically
  check the queue more often and service more requests. [unlike IP load balancing]
• If you add more server machines, the requests just get processed faster by more
  instances of the object [orchestration]
• If one server goes down, the remaining servers continue to process the requests.
  There’s no single point of failure.

You can find the whole presentation (which also discussed BizTalk compared with traditional
application servers) right here.

The isolation setting is fixed and is SERIALIZABLE. So beware that even your most basic receive locations that only execute those very simple SQL statements such as ‘select * from tablename’ can generate locks. BTW, this is a general misconception: usually people think that select-statements never lock resources. Nothing is less true of course.

 

While having a shared lock on a range of keys, in addition to the keys themselves having locked, no records can be inserted. 
Here’s a sample to demonstrate the effects of the SERIALIZABLE isolation setting:

 

Open your SQL Query analyzer
Open 2 seperate query windows to the local Pubs database
Copy & paste these samples statements:

 

Window A:

 

SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
Select * from authors where contract = 1
GO

 

Window B:

 

BEGIN TRAN
INSERT INTO [pubs].[dbo].[authors]([au_id],[au_lname], [au_fname],  [contract])
VALUES(‘666-66-6666′,’Grego’, ‘El’, 1)
GO

 

Now, if you try to execute both queries you will see that the second query will always be blocked by the first, irrespective of the order. You can immediately unblock the process by typing, selecting and executing ‘ROLLBACK TRAN’ in the blocking transaction’s window. Now repeat this test and replace ‘SERIALIZABLE’ with ‘READ COMMITTED’ and you will see that when you first start batch A, you can still execute Batch B simultaneously (which wasn’t so with the SERIALIZABLE level).

 

Here are the isolationlevel-enumerator values from system.data:

 

[Flags]
public enum IsolationLevel
{
      // Fields
      Chaos = 0x10,
      ReadCommitted = 0x1000,
      ReadUncommitted = 0x100,
      RepeatableRead = 0x10000,
      Serializable = 0x100000,
      Unspecified = -1
}

 

The BizTalk SQL adapter always uses 0x100000. You can check this by viewing the requested locks in Enterprise Manager:

 

 

Now, I believe the above will rarely be a problem in real-life. You should only expect performance problems when you have a lot of transactions and a lot of simultaneous lock requests for the same heavy resource while having a bad database-design (having no or having the wrong indexes). You should also know that lock waits are perfectly normal: a simple wait for a lock is different from a deadlock. The waiting process will get the lock anyway when the process that’s holding the lock completes.

 

 

Is there a solution?

SQL Server locking experts’ comments are very welcome…

I’ve just found out the hard way that GAC’ed publisher policy assemblies do not chain. When you have a policy.1.0, redirecting the binding of assembly v1.0.0.0 to assembly v1.1.0.0 and also having a policy.1.1, for redirecting the binding of assembly 1.1 to assembly 1.2, this doesn’t result in a binding to the assembly 1.2 when your app requests 1.0 (but rather to 1.1). If you want this kind of binding-behavior I guess you will have to use publisher policy versioning.

Why do I tell all this? Well, I’ve never read anything about this before so I thought it would be nice to mention here.

I also want to thank Alan Shi, who confirmed and explained this binding behavior to me. For more excellent information regarding the GAC and Fusion you should definitely visit his blog here.