A Really, Really Big Update for Deployment…

A Really, Really Big Update for Deployment…

(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

  • The “core” NAnt functionality has now been separated from the piece that
    an individual BizTalk project must “own” and maintain.  This is a
    huge simplification, and allows for more rapid adoption — as well as the ability
    to deploy bug fixes & feature enhancements in a way that just wasn’t possible
    when stuff was intermingled.  (Peter Provost was right about
    that aspect of my first attempts.)  The NAnt script that a particular project
    must own can now be reduced to the following for a simple case (notice the ‘include’
    reference for the core):

    <project name="BizTalkSample" default="debugDeploy">
       <sysinfo/>
       <include buildfile="BizTalkDeploymentInclude.nant"/>  
       <-- Set following properties to true or false to 
    include various pieces of a BizTalk deployment. -->
       <property name="includeSchemas"  value="true" />
       <property name="includeOrchestrations" value="true" />
       <property name="includeTransforms" value="true" />
       <property name="includePipelines" value="true" />
       <property name="includeComponents" value="true" />
       <property name="includePipelineComponents" value="false" />
       <property name="includeCustomFunctoids" value="false" />
       <property name="includeVocabAndRules" value="false" />
       <property name="includeVirtualDirectories" value="true" />
       <property name="includeMessagingBindings" value="true" />
       <property name="includeDeploymentTest" value="true" />
       <property name="includeCustomTarget" value="true" />
    <project/>

  • You now have the ability to supply a simple xml-driven Wizard-based UI when you are
    deploying after an MSI install.  You can use this UI to gather
    Windows identity information for virtual directories (for SOAP/HTTP receive ports),
    to indicate whether the scripts should deploy to the BizTalk management database (for
    working in BizTalk groups), and for many other purposes.  I call this “Install-Time
    Configuration” in the documentation below.  Here are some example Wizard
    screens:

    >

  • You now have the ability to have an Operations-managed “spreadsheet-driven”
    mechanism for environment-relative configuration.  What do I mean?  I mean
    those aspects of your deployment that are particular to your development, QA, or production
    BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
    files!  Queue names, database names, file shares, FTP locations, etc.  This
    is information that you don’t want to manage with the Wizard UI above (there
    is too much of it) and you want to maintain “as a set” for each environment.
     The spreadsheet (pictured below) can start life with your developers, and ownership
    can gradually migrate to Operations.  This spreadsheet generates environment-specific
    “settings” files, and at the point you deploy, the values for the environment
    you are deploying to are automatically substituted into your binding file…Slick!

    Image0051

    (click)

  • You now can use a highly templated (reusable) WiX based
    setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
    in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
    contents of an MSI.  One of the clear pieces of feedback I got on the last rev
    I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

  • You no longer need to have your NAnt file enumerate your orchestration names or
    their deployment order –
    a new custom NAnt task eliminates the need for
    this.
  • You no longer need to have your NAnt file enumerate how many receive ports/send
    ports/etc. you have –
    a new custom NAnt task eliminates the need for this.
  • Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
    management database.
  • Support for not following the naming recommendations I made in previous releases –
    you can have custom project names, assembly names, directory names – its just
    a little more work.
  • Support for multiple assemblies of the various types (i.e. multiple orchestration
    assemblies, schema assemblies, etc.)
  • Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
    above.
  • Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
    policies and vocabularies.
  • Inclusion of Loren Halvorsen’s XmlPreProcess tool
    for managing the environment-relative configuration discussed above.
  • Support for creating IIS6 application pools (with specified Windows identity) for
    HTTP and SOAP-based receive locations, and for adding virtual directories to application
    pools.
  • Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
  • Support for selectively restarting multiple BizTalk host instances, for deploying
    custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

A Really, Really Big Update for Deployment…

A Really, Really Big Update for Deployment…

(Update: See the latest on the Deployment Framework here.)

It has been about three months since my initial submission to the BizTalk 2004 developer
competition that I discussed here
The contents of that contest entry took a prize (woohoo!) but still left a lot to
be desired…

Since then, I’ve had a chance to make what I believe are major improvements
to the deployment story I’ve been discussing since last May.  In addition,
I’ve had a chance to see the practice being used across several large BizTalk
projects, and have done a lot of learning as a result.  (No guarantees or warranty
implied, but derivatives of the scripts/tools I discuss here have been in use for
some time by a very large BizTalk customer I work with.)

The most important changes you will see in this release are as follows:

  • The “core” NAnt functionality has now been separated from the piece that
    an individual BizTalk project must “own” and maintain.  This is a
    huge simplification, and allows for more rapid adoption — as well as the ability
    to deploy bug fixes & feature enhancements in a way that just wasn’t possible
    when stuff was intermingled.  (Peter Provost was right about
    that aspect of my first attempts.)  The NAnt script that a particular project
    must own can now be reduced to the following for a simple case (notice the ‘include’
    reference for the core):

    <project name="BizTalkSample" default="debugDeploy">
       <sysinfo/>
       <include buildfile="BizTalkDeploymentInclude.nant"/>  
       <-- Set following properties to true or false to 
    include various pieces of a BizTalk deployment. -->
       <property name="includeSchemas"  value="true" />
       <property name="includeOrchestrations" value="true" />
       <property name="includeTransforms" value="true" />
       <property name="includePipelines" value="true" />
       <property name="includeComponents" value="true" />
       <property name="includePipelineComponents" value="false" />
       <property name="includeCustomFunctoids" value="false" />
       <property name="includeVocabAndRules" value="false" />
       <property name="includeVirtualDirectories" value="true" />
       <property name="includeMessagingBindings" value="true" />
       <property name="includeDeploymentTest" value="true" />
       <property name="includeCustomTarget" value="true" />
    <project/>

  • You now have the ability to supply a simple xml-driven Wizard-based UI when you are
    deploying after an MSI install.  You can use this UI to gather
    Windows identity information for virtual directories (for SOAP/HTTP receive ports),
    to indicate whether the scripts should deploy to the BizTalk management database (for
    working in BizTalk groups), and for many other purposes.  I call this “Install-Time
    Configuration” in the documentation below.  Here are some example Wizard
    screens:

    >

  • You now have the ability to have an Operations-managed “spreadsheet-driven”
    mechanism for environment-relative configuration.  What do I mean?  I mean
    those aspects of your deployment that are particular to your development, QA, or production
    BizTalk environments.  Aspects that wind up appearing in your BizTalk binding
    files!  Queue names, database names, file shares, FTP locations, etc.  This
    is information that you don’t want to manage with the Wizard UI above (there
    is too much of it) and you want to maintain “as a set” for each environment.
     The spreadsheet (pictured below) can start life with your developers, and ownership
    can gradually migrate to Operations.  This spreadsheet generates environment-specific
    “settings” files, and at the point you deploy, the values for the environment
    you are deploying to are automatically substituted into your binding file…Slick!

    Image0051

    (click)

  • You now can use a highly templated (reusable) WiX based
    setup, rather than a Visual Studio Setup Project, to generate your MSI.  WiX,
    in a nutshell, is a set of tools and an Xml grammar that allow you to specify the
    contents of an MSI.  One of the clear pieces of feedback I got on the last rev
    I released is that reproducing a Visual Studio Setup Project was far too manual.

There are a large number of other changes that I’ll just enumerate quickly
here:

  • You no longer need to have your NAnt file enumerate your orchestration names or
    their deployment order –
    a new custom NAnt task eliminates the need for
    this.
  • You no longer need to have your NAnt file enumerate how many receive ports/send
    ports/etc. you have –
    a new custom NAnt task eliminates the need for this.
  • Support for BizTalk groups, where machines 2-n do not require deploying to the BizTalk
    management database.
  • Support for not following the naming recommendations I made in previous releases –
    you can have custom project names, assembly names, directory names – its just
    a little more work.
  • Support for multiple assemblies of the various types (i.e. multiple orchestration
    assemblies, schema assemblies, etc.)
  • Inclusion of a utility I call SetEnvUI.exe for creating the xml-based wizards described
    above.
  • Inclusion of a utility I call DeployBTRules.exe for deploying BizTalk Rule Engine
    policies and vocabularies.
  • Inclusion of Loren Halvorsen’s XmlPreProcess tool
    for managing the environment-relative configuration discussed above.
  • Support for creating IIS6 application pools (with specified Windows identity) for
    HTTP and SOAP-based receive locations, and for adding virtual directories to application
    pools.
  • Support for registering the btshttpreceive.dll ISAPI extension with IIS6.
  • Support for selectively restarting multiple BizTalk host instances, for deploying
    custom functoids, and for dealing with send port groups, and more…

There are now two downloads.  The first contains all of the core scripts and
utilities plus a sample BizTalk solution that uses them.  The second download
contains just the core scripts and utilities, and is designed to allow a BizTalk solution
to accept updates/bug-fixes/etc. over time (i.e. the zip can be expanded into your
directory structure on a developer workstation so you can test it out.) 

It should be noted, though, that some manual work will be required to “upgrade”
from the previous version I released in September – this release is quite different,
but I think you will find it is well worth the time.

Full download is here.

Core scripts and utilities only are here

GotDotNet Workspace is here.

The full documentation for this release (which includes more detail than this blog
entry…) is included in the zip files, but it can be viewed directly here (with
a diagram here.)

Enjoy!  And remember: if you didn’t make into a given environment (QA,
production, etc.) with a scripted deployment of some kind, you didn’t get there
at all.

(Not necessarily this stuff, but something automated, at any rate…)

Another round of BizTalk naming conventions

Another round of BizTalk naming conventions

Some time ago, I published a set of proposed
naming conventions
for BizTalk 2004.  

In the interim, I’ve had a chance to see these used in a few different projects
and to review them with several people — and they have evolved & grown a fair
bit.  

So I thought I’d publish the new version – the Word version can be found here,
and an HTML version can be found here.
 (I’ve found it helpful to open the HTML version directly in VS.NET for
quick reference.)

Hopefully these will be of value to your project.  I’ve certainly found
that the documentation value to be had within Orchestrations and within the operational
tools can be hugely improved by following these conventions – especially
if you are using the UK
SDC BizTalk 2004 Documenter
tool (you are, aren’t you?)  My comments
on the use of such documentation in
my old post
still apply, I believe.

Leave comments with questions or suggestions…

HDC 2004 Wrap-up

HDC 2004 Wrap-up

I had the pleasure of presenting on BizTalk 2004 at the Heartland
Developer’s Conference 2004
.  The whole conference was a lot of fun,
and as Kent
Tegels
has noted, Joe Olsen did
a great job in organizing the logistics for this event – you wouldn’t
have known it was the first-of-a-kind (save for Joe’s mea culpa on caffeine…)

I had a chance to talk to Sam Gentile about
the work he has been doing at Addesso.  This was enlightening, since in my non-BizTalk-work-life
I do quite a bit mobile work in the field-force-automation space (using ruggedized
Pocket PC devices – great fun!)

As to the presentation itself – one of the topics I discussed was BizTalk 2004’s
scaling model, and the power of having being able to create multiple host instances
for a single logical host definition that contains your orchestrations.  This
gives you a “competing consumer” effect – multiple processes on
multiple servers, all pulling from a common work queue.  In my talk, I suggested
that this idea wasn’t new and related it back to a 1998
article
in Enterprise Developer’s magazine.  In that article, an architect
from Merrill Lynch was discussing the scaling difficulties in using NT4/MTS –
the load balancing options weren’t pretty.  He proposed a pattern he dubbed
“Auctioning” where clients would submit work to an MSMQ queue, and multiple
servers (all running common server-side components) would pull work from that queue.
 The benefits he articulated relate quite closely to what you will find in the
parallel aspects of BizTalk 2004.

Check out this graphic from the article – it should look a little familiar to
BizTalk folks.

Below is my paraphrase of the article’s main points, recast with BizTalk terms
(and exactly relevant to BizTalk, I believe…)

  • To load balance multiple servers, servers objects [orchestrations] should pull
    work from a global [per-host] work queue that clients place their requests in.
  • Auctioning works well precisely because it relies on the message queue server [messagebox],
    not DCOM [or transport of choice], to route the request to the server object [orchestration].
  • Server objects [orchestrations] compete for requests on the queue, so load balancing
    happens naturally; the server with the most available processing capability will automatically
    check the queue more often and service more requests. [unlike IP load balancing]
  • If you add more server machines, the requests just get processed faster by more
    instances of the object [orchestration]
  • If one server goes down, the remaining servers continue to process the requests.
    There’s no single point of failure.

You can find the whole presentation (which also discussed BizTalk compared with traditional
application servers) right here.

Document splitting post-receive

Document splitting post-receive

An interesting scenario came up this afternoon. I receive a bunch of different documents containing address changes, which I map to my canonical “address changes” format in the receive port. I then want to split my canonical document using an envelope so that each address change record is out on its own.


As far as I can see, to do this I’m going to have to write the canonicalised document out to disk, then read it back in from disk and split it in the corresponding receive pipeline. This feels awfully inefficient, but luckily it’s only for a demo system.


I know that the messagebox allows for far more sophisticated subscriptions than are exposed through the User Interface, and obviously orchestrations can subscribe based just on document type, but does anyone know of an equivalent to “The Messagebox” as a receive location transport type?


On a side note, the lack of blogging recently is entirely down to being on holiday, and an addiction to Halo 2 on XBox Live 🙂 If anyone fancies a game, invite me over to your party – the gamertag is “shelfsider”. I’m varying between level 3 and 5 depending on the gamemode, so not too tough an opponent!


Update


Thanks to my colleague Charles and to Berneba (via comments) for the suggestion of using a loopback adapter. In this case it’s probably not going to work as I’m trying to split the document and it’s not clear what I’d receive back into the orchestration.


For the purposes of the demo system I’ll stick with taking a hop onto the disk, but Darren Jefford’s post about looping gives some good ideas – particularly using the little known xpath function.

The BizTalk SQL Server Adapter isolation level

The BizTalk SQL Server Adapter isolation level

The isolation setting is fixed and is SERIALIZABLE. So beware that even your most basic receive locations that only execute those very simple SQL statements such as ‘select * from tablename’ can generate locks. BTW, this is a general misconception: usually people think that select-statements never lock resources. Nothing is less true of course.

 

While having a shared lock on a range of keys, in addition to the keys themselves having locked, no records can be inserted. 
Here’s a sample to demonstrate the effects of the SERIALIZABLE isolation setting:

 

Open your SQL Query analyzer
Open 2 seperate query windows to the local Pubs database
Copy & paste these samples statements:

 

Window A:

 

SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
Select * from authors where contract = 1
GO

 

Window B:

 

BEGIN TRAN
INSERT INTO [pubs].[dbo].[authors]([au_id],[au_lname], [au_fname],  [contract])
VALUES(‘666-66-6666′,’Grego’, ‘El’, 1)
GO

 

Now, if you try to execute both queries you will see that the second query will always be blocked by the first, irrespective of the order. You can immediately unblock the process by typing, selecting and executing ‘ROLLBACK TRAN’ in the blocking transaction’s window. Now repeat this test and replace ‘SERIALIZABLE’ with ‘READ COMMITTED’ and you will see that when you first start batch A, you can still execute Batch B simultaneously (which wasn’t so with the SERIALIZABLE level).

 

Here are the isolationlevel-enumerator values from system.data:

 

[Flags]
public enum IsolationLevel
{
      // Fields
      Chaos = 0x10,
      ReadCommitted = 0x1000,
      ReadUncommitted = 0x100,
      RepeatableRead = 0x10000,
      Serializable = 0x100000,
      Unspecified = -1
}

 

The BizTalk SQL adapter always uses 0x100000. You can check this by viewing the requested locks in Enterprise Manager:

 


 

Now, I believe the above will rarely be a problem in real-life. You should only expect performance problems when you have a lot of transactions and a lot of simultaneous lock requests for the same heavy resource while having a bad database-design (having no or having the wrong indexes). You should also know that lock waits are perfectly normal: a simple wait for a lock is different from a deadlock. The waiting process will get the lock anyway when the process that’s holding the lock completes.
If you use SERIALIZABLE my best advice is to tune you sql statements for performance (also having correct db-design, normalization, the right indexes,…) in order to make your select statement execute as fast as possible .

 

**UPDATED**

 

Is there a solution?



  • If you try to add the ‘SET TRANSACTION ISOLATION LEVEL READ COMMITTED’ to a select-statement based receive location then you may have no issues at design time. But when you try to add a RL based on this SQL statement in the  BizTalk Explorer you will get a ‘The SQL statement must be either a select or an exec’ error.



  • You can add ‘With (Readcommitted)’ to the tables in the select, or data modification statement. ‘Select * from authors with (Readcommitted) where contract = 1’. This will override the default Serializeable isolation level, and keep the number of locked records to a minimum. Credits go to Dirk Gubbels from Microsoft…

SQL Server locking experts’ comments are very welcome…

Atomic Scope "Batch" property…

Atomic Scope "Batch" property…

Charles Young has a great post on transactions within orchestrations.  In it he mentions the Batch property of the ATOMIC scope.  I concur with him that the documentation on this is very light, as well as any information out in the communities.  Doing a little testing on my current project with Stephen Thomas (be sure to check out his blog), it has become apparent that the property acts just as Charles suspects.


Our particular scenario –


We had an orchestration processing a large sum of messages.  At one point in the orchestration, we were calling a component to insert the records into the database.  While that orchestration was processing the large batch of messages, we sent several very small batches of messages that quickly reached the same point in their instance of the orchestration (inserting the records through the component).  It turned out that the smaller messages would not complete processing thier messages until the large batch completed.


After we changed the property to be False (the proprety for some reason defaults to True, which seems a bit dangerous), redeployed and rean the test again, the much smaller messages completed ahead of the slower, larger batch of messages (our expected outcome originally).


So, keep in mind when using the Atomic transaction that this property defaults to True, and may have ill effects on your message processing.


Cheers!!

About publisher policy assembly chaining

About publisher policy assembly chaining

I’ve just found out the hard way that GAC’ed publisher policy assemblies do not chain. When you have a policy.1.0, redirecting the binding of assembly v1.0.0.0 to assembly v1.1.0.0 and also having a policy.1.1, for redirecting the binding of assembly 1.1 to assembly 1.2, this doesn’t result in a binding to the assembly 1.2 when your app requests 1.0 (but rather to 1.1). If you want this kind of binding-behavior I guess you will have to use publisher policy versioning.
Why do I tell all this? Well, I’ve never read anything about this before so I thought it would be nice to mention here.
I also want to thank Alan Shi, who confirmed and explained this binding behavior to me. For more excellent information regarding the GAC and Fusion you should definitely visit his blog here.