A Really, Really Big Update for Deployment…

(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here. 
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

  • The “core” NAnt functionality has now been separated from the piece that
    an individual BizTalk project must “own” and maintain.  This is a
    huge simplification, and allows for more rapid adoption — as well as the ability
    to deploy bug fixes & feature enhancements in a way that just wasn’t possible
    when stuff was intermingled.  (Peter Provost was right about
    that aspect of my first attempts.)  The NAnt script that a particular project
    must own can now be reduced to the following for a simple case (notice the ‘include’
    reference for the core):

    <project name="BizTalkSample" default="debugDeploy">
       <sysinfo/>
       <include buildfile="BizTalkDeploymentInclude.nant"/>  
       <-- Set following properties to true or false to 
    include various pieces of a BizTalk deployment. -->
       <property name="includeSchemas"  value="true" />
       <property name="includeOrchestrations" value="true" />
       <property name="includeTransforms" value="true" />
       <property name="includePipelines" value="true" />
       <property name="includeComponents" value="true" />
       <property name="includePipelineComponents" value="false" />
       <property name="includeCustomFunctoids" value="false" />
       <property name="includeVocabAndRules" value="false" />
       <property name="includeVirtualDirectories" value="true" />
       <property name="includeMessagingBindings" value="true" />
       <property name="includeDeploymentTest" value="true" />
       <property name="includeCustomTarget" value="true" />
    <project/>

  • You now have the ability to supply a simple xml-driven Wizard-based UI when you are
    deploying after an MSI install.  You can use this UI to gather
    Windows identity information for virtual directories (for SOAP/HTTP receive ports),
    to indicate whether the scripts should deploy to the BizTalk management database (for
    working in BizTalk groups), and for many other purposes.  I call this “Install-Time
    Configuration” in the documentation below.  Here are some example Wizard
    screens:

    >

  • You now have the ability to have an Operations-managed “spreadsheet-driven”
    mechanism for environment-relative configuration.  What do I mean?  I mean
    those aspects of your deployment that are particular to your development, QA, or production
    BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
    files!  Queue names, database names, file shares, FTP locations, etc.  This
    is information that you don’t want to manage with the Wizard UI above (there
    is too much of it) and you want to maintain “as a set” for each environment.
     The spreadsheet (pictured below) can start life with your developers, and ownership
    can gradually migrate to Operations.  This spreadsheet generates environment-specific
    “settings” files, and at the point you deploy, the values for the environment
    you are deploying to are automatically substituted into your binding file…Slick!

    (click)

  • You now can use a highly templated (reusable) WiX based
    setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
    in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
    contents of an MSI.  One of the clear pieces of feedback I got on the last rev
    I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

  • You no longer need to have your NAnt file enumerate your orchestration names or
    their deployment order –
    a new custom NAnt task eliminates the need for
    this.
  • You no longer need to have your NAnt file enumerate how many receive ports/send
    ports/etc. you have –
    a new custom NAnt task eliminates the need for this.
  • Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
    management database.
  • Support for not following the naming recommendations I made in previous releases –
    you can have custom project names, assembly names, directory names – its just
    a little more work.
  • Support for multiple assemblies of the various types (i.e. multiple orchestration
    assemblies, schema assemblies, etc.)
  • Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
    above.
  • Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
    policies and vocabularies.
  • Inclusion of Loren Halvorsen’s XmlPreProcess tool
    for managing the environment-relative configuration discussed above.
  • Support for creating IIS6 application pools (with specified Windows identity) for
    HTTP and SOAP-based receive locations, and for adding virtual directories to application
    pools.
  • Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
  • Support for selectively restarting multiple BizTalk host instances, for deploying
    custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here. 

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

A Really, Really Big Update for Deployment…

(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here. 
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

  • The “core” NAnt functionality has now been separated from the piece that
    an individual BizTalk project must “own” and maintain.  This is a
    huge simplification, and allows for more rapid adoption — as well as the ability
    to deploy bug fixes & feature enhancements in a way that just wasn’t possible
    when stuff was intermingled.  (Peter Provost was right about
    that aspect of my first attempts.)  The NAnt script that a particular project
    must own can now be reduced to the following for a simple case (notice the ‘include’
    reference for the core):

    <project name="BizTalkSample" default="debugDeploy">
       <sysinfo/>
       <include buildfile="BizTalkDeploymentInclude.nant"/>  
       <-- Set following properties to true or false to 
    include various pieces of a BizTalk deployment. -->
       <property name="includeSchemas"  value="true" />
       <property name="includeOrchestrations" value="true" />
       <property name="includeTransforms" value="true" />
       <property name="includePipelines" value="true" />
       <property name="includeComponents" value="true" />
       <property name="includePipelineComponents" value="false" />
       <property name="includeCustomFunctoids" value="false" />
       <property name="includeVocabAndRules" value="false" />
       <property name="includeVirtualDirectories" value="true" />
       <property name="includeMessagingBindings" value="true" />
       <property name="includeDeploymentTest" value="true" />
       <property name="includeCustomTarget" value="true" />
    <project/>

  • You now have the ability to supply a simple xml-driven Wizard-based UI when you are
    deploying after an MSI install.  You can use this UI to gather
    Windows identity information for virtual directories (for SOAP/HTTP receive ports),
    to indicate whether the scripts should deploy to the BizTalk management database (for
    working in BizTalk groups), and for many other purposes.  I call this “Install-Time
    Configuration” in the documentation below.  Here are some example Wizard
    screens:

    >

  • You now have the ability to have an Operations-managed “spreadsheet-driven”
    mechanism for environment-relative configuration.  What do I mean?  I mean
    those aspects of your deployment that are particular to your development, QA, or production
    BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
    files!  Queue names, database names, file shares, FTP locations, etc.  This
    is information that you don’t want to manage with the Wizard UI above (there
    is too much of it) and you want to maintain “as a set” for each environment.
     The spreadsheet (pictured below) can start life with your developers, and ownership
    can gradually migrate to Operations.  This spreadsheet generates environment-specific
    “settings” files, and at the point you deploy, the values for the environment
    you are deploying to are automatically substituted into your binding file…Slick!

    (click)

  • You now can use a highly templated (reusable) WiX based
    setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
    in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
    contents of an MSI.  One of the clear pieces of feedback I got on the last rev
    I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

  • You no longer need to have your NAnt file enumerate your orchestration names or
    their deployment order –
    a new custom NAnt task eliminates the need for
    this.
  • You no longer need to have your NAnt file enumerate how many receive ports/send
    ports/etc. you have –
    a new custom NAnt task eliminates the need for this.
  • Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
    management database.
  • Support for not following the naming recommendations I made in previous releases –
    you can have custom project names, assembly names, directory names – its just
    a little more work.
  • Support for multiple assemblies of the various types (i.e. multiple orchestration
    assemblies, schema assemblies, etc.)
  • Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
    above.
  • Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
    policies and vocabularies.
  • Inclusion of Loren Halvorsen’s XmlPreProcess tool
    for managing the environment-relative configuration discussed above.
  • Support for creating IIS6 application pools (with specified Windows identity) for
    HTTP and SOAP-based receive locations, and for adding virtual directories to application
    pools.
  • Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
  • Support for selectively restarting multiple BizTalk host instances, for deploying
    custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here. 

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

Another round of BizTalk naming conventions

Some time ago, I published a set of proposed
naming conventions for BizTalk 2004.  

In the interim, I’ve had a chance to see these used in a few different projects
and to review them with several people — and they have evolved & grown a fair
bit.  

So I thought I’d publish the new version – the Word version can be found here,
and an HTML version can be found here.
 (I’ve found it helpful to open the HTML version directly in VS.NET for
quick reference.)

Hopefully these will be of value to your project.  I’ve certainly found
that the documentation value to be had within Orchestrations and within the operational
tools can be hugely improved by following these conventions – especially
if you are using the UK
SDC BizTalk 2004 Documenter tool (you are, aren’t you?)  My comments
on the use of such documentation in
my old post still apply, I believe.

Leave comments with questions or suggestions…

HDC 2004 Wrap-up

I had the pleasure of presenting on BizTalk 2004 at the Heartland
Developer’s Conference 2004
.  The whole conference was a lot of fun,
and as Kent
Tegels
has noted, Joe Olsen did
a great job in organizing the logistics for this event – you wouldn’t
have known it was the first-of-a-kind (save for Joe’s mea culpa on caffeine…)

I had a chance to talk to Sam Gentile about
the work he has been doing at Addesso.  This was enlightening, since in my non-BizTalk-work-life
I do quite a bit mobile work in the field-force-automation space (using ruggedized
Pocket PC devices – great fun!)

As to the presentation itself – one of the topics I discussed was BizTalk 2004’s
scaling model, and the power of having being able to create multiple host instances
for a single logical host definition that contains your orchestrations.  This
gives you a “competing consumer” effect – multiple processes on
multiple servers, all pulling from a common work queue.  In my talk, I suggested
that this idea wasn’t new and related it back to a 1998
article
in Enterprise Developer’s magazine.  In that article, an architect
from Merrill Lynch was discussing the scaling difficulties in using NT4/MTS –
the load balancing options weren’t pretty.  He proposed a pattern he dubbed
“Auctioning” where clients would submit work to an MSMQ queue, and multiple
servers (all running common server-side components) would pull work from that queue.
 The benefits he articulated relate quite closely to what you will find in the
parallel aspects of BizTalk 2004.

Check out this graphic from the article – it should look a little familiar to
BizTalk folks.

Below is my paraphrase of the article’s main points, recast with BizTalk terms
(and exactly relevant to BizTalk, I believe…)

  • To load balance multiple servers, servers objects [orchestrations] should pull
    work from a global [per-host] work queue that clients place their requests in.
  • Auctioning works well precisely because it relies on the message queue server [messagebox],
    not DCOM [or transport of choice], to route the request to the server object [orchestration].
  • Server objects [orchestrations] compete for requests on the queue, so load balancing
    happens naturally; the server with the most available processing capability will automatically
    check the queue more often and service more requests. [unlike IP load balancing]
  • If you add more server machines, the requests just get processed faster by more
    instances of the object [orchestration]
  • If one server goes down, the remaining servers continue to process the requests.
    There’s no single point of failure.

You can find the whole presentation (which also discussed BizTalk compared with traditional
application servers) right here.

Document splitting post-receive

An interesting scenario came up this afternoon. I receive a bunch of different documents containing address changes, which I map to my canonical “address changes” format in the receive port. I then want to split my canonical document using an envelope so that each address change record is out on its own.


As far as I can see, to do this I’m going to have to write the canonicalised document out to disk, then read it back in from disk and split it in the corresponding receive pipeline. This feels awfully inefficient, but luckily it’s only for a demo system.


I know that the messagebox allows for far more sophisticated subscriptions than are exposed through the User Interface, and obviously orchestrations can subscribe based just on document type, but does anyone know of an equivalent to “The Messagebox” as a receive location transport type?


On a side note, the lack of blogging recently is entirely down to being on holiday, and an addiction to Halo 2 on XBox Live ðŸ™‚ If anyone fancies a game, invite me over to your party – the gamertag is “shelfsider”. I’m varying between level 3 and 5 depending on the gamemode, so not too tough an opponent!


Update


Thanks to my colleague Charles and to Berneba (via comments) for the suggestion of using a loopback adapter. In this case it’s probably not going to work as I’m trying to split the document and it’s not clear what I’d receive back into the orchestration.


For the purposes of the demo system I’ll stick with taking a hop onto the disk, but Darren Jefford’s post about looping gives some good ideas – particularly using the little known xpath function.