Running MSBuild scripts from Visual Studio

Running MSBuild scripts from Visual Studio

<br /> Gobbledygooks · Running MSBuild scripts from Visual Studio<br />

Jul 09, 2008

MSBuild2It seems like there more build script one writes, the more often one wants to run them and it’s always a bit annoying (and time consuming) having to leave Visual Studio and start MSBuild from the command line. Brennan Stehling has a cool solution to that problem here were he sets up MSBuild as an external tool and runs it.

One problem for us was the we had our solution files in one place on the file system and our build files in a totally different place. The solution was to add the build file for the current solution as a Solution Folder (as shown in the figure below) and then set MSBuild to use $(ItemDir) as its Initial Directory. That will kick of MSBuild from the directory that the current selected Solution Folder points to and in our case that’s were the XXX.Build.Article.proj file exists.

MSBuild

Looking forward to stackoverflow.com

Looking forward to stackoverflow.com

stackoverflow-logo-250I’m really looking forward to the release of stackoverflow.com – have you heard about it? If not it’s a collaboration between Joel Spolsky and Jeff Atwood. Joel and Jeff are of course the two blogger behind the popular Joel on Software and Coding Horror blogs.

Joel is also one of the founders of the FogCreek Software company. FogCreek has a couple of different products but the most known is probably FogBugz which is a really cool project management/bug tracking/wiki/support tool for software development and management. At first it might not look like a very impressing tool but have a look at this presentation by Joel and I think you’ll start thinking different (I know we did and we’re now using it for both planing and managing big integrations projects).

Anyway, if you missed the story behind stackoverflow have a look here and don’t miss the podcasts that Joel and Jeff record, some of them are really cool. I know I’ve learnt a lot by just listening in to their conversations.

We not have quite a few online BizTalk communities such as the Microsoft forums, Google Groups, BizTalk Gurus, facebook (search for BizTalk related groups) and at LinkedIn (a couple of open BizTalk related groups) – is there room for another one at stackoverflow? Do we need one?

Removing XML namespaces – revisit

Removing XML namespaces – revisit

I have a old post on removing XML namespace from outgoing messages using XSLT in a map on the send port. Removing XML namespace is usually a late requirement that shows up during integration tests with for example legacy systems that has problems reading XML and only finds XML namespaces messy and confusing and wants it removed.

The post recently got commented by Jeff Lynch (one of the codebetter.com bloggers) asking me why I just didn’t create a schema without any XML namespace in it to represent the outgoing schema (see figure below) and then map to that in the send port.

Say for example that we have incoming messages like the one below with namespaces.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">ns0:BlahRoot </span><span style="color: #FF0000; ">xmlns:ns0</span><span style="color: #0000FF; ">="http://Sample.BlahIncomingSchema"</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
  </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">Test Value</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">ns0:BlahRoot</span><span style="color: #0000FF; ">></span></div>

We’ve then defined a schema without namespace and map to that and get the following result.

<div><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahRoot</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
  </span><span style="color: #0000FF; "><</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">Test Value</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahNode</span><span style="color: #0000FF; ">></span><span style="color: #000000; ">
</span><span style="color: #0000FF; "></</span><span style="color: #800000; ">BlahRoot</span><span style="color: #0000FF; ">></span></div>

This method is of course much cleaner then any previous and it’s also more conceptually correct as the schema actually represents the contact between BizTalk and the receiving system (the contract is a message without namespace in it, not one with that we then remove).

The only problem with this kind of approach is that as BizTalk recognized the message type using a combination between root node and XML namespace we can’t have another schema with the BlahRoot root node without a defined XML namespace. Even if those two schemas would look totally different in structure and be two different message types BizTalk would be able to see the difference (to BizTalk they both be #BlahRoot message types).

Thanks Jeff for pointing this out to me.

Does BizTalk have man-boobs?

I’ve just finished watching this webcast from QCon 2008 with Martin Fowler and Jim Webber. It’s basically their view on how integration middleware is used today and how we plan and implement our SOA and ESB project.

Their main point is that we use much to bloated middleware (BizTalk is mentioned as an example here) and that we should have a more agile approach to implement SOA and ESBs. They’ve used all the agile ideas (testing, isolated testable functionality, deliver small and often, continuous builds etc, etc) and applied them to integration – fair enough. I totally agree that trying to convert your complete enterprise architecture into a SOA architecture is a guaranteed failure. This is also something we heard for a while now from others as well.

I do also agree that BizTalk is a huge platform and that it isn’t perfect in all aspects. IMHO it does however give us some important advantages compared to a custom coded message bus and services. I’ll try and list a few of them below.

  1. Fixed architecture
    We don’t have invent the wheel every time. BizTalk is a product with an architecture that one have to learn and use. There are times when this is a pain (did I hear low latency and BizTalk persistence points?) but it’s also a huge kick start to all projects once you learnt it. Once you figured out how you use the products you’ll actually have something up and running in no time.

Isn’t an early delivery that we can test something good? I’m sure I can deliver a BizTalk based integration faster that some can using custom code when starting from scratch.

  1. Drag-and-drop
    There is a learning curve to BizTalk and all it’s tools but once one gotten over this one can move really fast, even without a deep understanding of .NET and software development (there are of course both pros and cons to this). I’ve seen projects with 50+ integration processes (to me that’s a big, complex project) where we actually used people fresh out of school, spent two weeks to teach them basic BizTalk and had them deliver critical parts of the projects. I’d like to see that happen custom coded ESB project with thousands lines of code …

You probably get a nice design and implementation if you can hire 10 top developers and a couple of architects, but that isn’t always possible.

  1. Tools
    Does a custom code, lean approach, really scale in this scenario? Do you take the time to pause and build that management and configuration tool that you don’t get with a custom code project? I don’t say that we got the perfect view and control of our processes and messages in BizTalk but at least we got some control. At least I got the BizTalk Administration Console to let me see how my different application are doing, what messages and process etc that got suspended. At least I got the BAM framework where I can configure a tracking and monitoring in no time (usually …) etc, etc.

  2. It works
    Say your implementing a process that receives purchase orders and that these orders might contain orders for a couple of millions dollars. Do you want to be developer that tells you boss that you think you might have lost a message due to a exception in your custom code? Of course you have 90% test coverage and continuous integration but you never tested for this exception case … I don’t want to be that developer/architect.

I don’t say don’t test. I’m very pro testing and I really feel that agile is the right approach. I’m just saying I need something tested and safe to build this super critical solutions on. Something that I know works and that I can be really productive on and start solving business cases from minute one.

What do you think? Does BizTalk have man-bobs and is that only a bad thing? And does Martin Fowler really have leather pants on?

Speaking at KNUG – Karlstad .NET User Group

Speaking at KNUG – Karlstad .NET User Group

Yesterday I presented the MasterData Management using BizTalk 2006 R2 talk (I’ll soon have a post out with the presentation in English) I recently held at Developer Summit at the local .NET user group in Karlstad (KNUG).

KNUG is a new .NET user group that I actually helped start a couple of months ago. This meeting was the second meeting for the group. The meeting was attended by about 20 persons and we had two presentations on the agenda. Besides my own Thomas Heder showed the group some LINQ and how he and his colleagues uses LINQPad to develop and test there queries.

We also discussed future subjects, possible speakers and moving information on the group over to a Community Server driven site.

Does anyone have any experience on Community Server and how the feature set matches those need for running a user group (managing users, blogs, email lists, calendar etc)?

Using BAM for latency tracking in a BizTalk request response scenario

This post will try and explain how BAM tracking can be used in SOAP based request response scenario in BizTalk 2006. It important to notice that some of the issues discussed in the post are specific to the SOAP adapter and are non-issues if the scenario would for example use the the WCF adapter or similar.

Describing the scenario

In this case we have a SOAP receive port that receives a request and returns a response. The request is routed to a orchestration that calls three different send ports. These ports then sends new requests to back-end systems and returns responses (communication with back-ends systems are also SOAP based). The three responses are used to build up the final response that is then returned to original receive port as a final response.

Our goal is to track the duration between the request and response on each of the ports. The idea is also to find a solution and tracking model that doesn’t have to change if we add or remove ports or add similar processes to track.

Defining and deploying the tracking model

We’ll start by defining our tracking model in Excel. Our activity contains of the following items:

  • InterchangeId (data item)
    As we won’t correlate all the tracking point into one single row (that would break the goal of having one model for all processes, the model would then have to be specific for one process and it’s specific ports) the interchange id will then tell us what different rows belong together and that describes one process.

  • ReceivePortName (data item)
    The name of the receive port.

  • Request (milestone item)
    The time the request was either sent or received (depending on if we track a port that received/sent the request using a receive port or send port).

  • Response (milestone item)
    The time the response was either sent or received (depending on if we track a port that received/sent it’s response on a receive port or send port).

  • SendPortName (data item)
    The name of the send port.

After we described the model it’s time to export it to an XML representation and then to use the BM tool to deploy it and generate the BAM database infrastructure. You’ll find some nice info on this here.

Using the Tracking Profile Editor to bind the model

Next step is to bind the model to the implementation using the Tracking Profile Editor. The figure below shows the different properties that were used. Notice that none of the items was bound to the actual orchestration context. All properties are general properties that we track on the ports. This is important as that gives us the possibility to just add and remove ports to change the tracking.


The next figure shows how the tracking of the request milestone event actually happens on either the RP1 port or on any of the three different send ports! If we developed a new process using other ports we could just add it here, no new model required.

What about the continuation then?

Our final problem is that unless we somehow correlate our request tracking point with our receive tracking point the receive we’ll end up with each tracking point spread over several different rows. In the example below I’ve marked the request for the RP01 port and the response event on the same port.

The reason for this is of course that BAM doesn’t have a context for the two tracking points and doesn’t know that actually belongs together. This differs from tracking in a orchestration were we always are in a context (the context of the orchestration), it’s then easy for BAM to understand that we like to view all the tracking point as one row – when tracking on ports it’s different. Continuation helps us tell BAM that we like have a context and correlate these two points.

In our case ServiceID is the prefect candidate for correlating the two points. A request and a response will have the same service id. In an other situation we could just as well have used a value from inside the message (say for example an invoice id).

The result is one single row for the request response for each port. So in our case a complete process (a complete interchange) is shown on four rows (one row for each of the ports). In the example below the first rows shows us the complete duration (and the other tracking data) between the request response to the client. The other rows show the duration for the send ports communication with the back-ends systems.

This model might not be optimal in an other scenario where your process are more fixed and you can then create a tracking model that is more specific to you actual process. But this solution meets our design goal as we’re now able to just add and remove port using the tracking profiler to track new port in completely new processes without having to go back and change the tracking model.

>
> [![note](../assets/2008/04/windowslivewriterusingbamforlatencytrackinginabiztalkrequ-9c0bnote-thumb.gif)](../assets/2008/04/windowslivewriterusingbamforlatencytrackinginabiztalkrequ-9c0bnote-2.gif) NOTE: When configuring BAM to track a port the _[MessageType](http://msdn2.microsoft.com/en-us/library/aa561650.aspx)_ is actually promoted. This causes some problems in combination with the SOAP based ports that have been published using the Web Services Publishing Wizard. Saravana writes about this [here](http://www.digitaldeposit.net/blog/2007/08/soap-adapter-and-biztalk-web-publishing.html) and all his articles on this subject is a must read when working with SOAP ports. The problem however comes down to that the Web Services Publishing Wizard generates code that puts the wrong _DocumentSpecName _in the message context and that causes the _XmlDisassembler_ to fail (it tricks the _XmlDisassembler_ to look for a _MessageType_ that doesn’t exists).
>
>
>
> This usually isn’t a problem (unless you like to use a map on a port) but as BAM will force the port to promote the _MessageType_ based on the _DocumentSpecName _we’ll have to fix this. Saravana has two solutions to the problem and I find the one that replaces the _DocumentSpecName_ with a null value and lets the _XmlDisassembler_ find the _MessageType_ to work well.
>
>

Speaking at Developer Summit on Masterdata Management using BizTalk 2006

I recently spoke at the leading developer conference here in Sweden called Developer Summit. The talk was called Masterdata Management using BizTalk 2006. The slides can be found here.

I can really recommend Developer Summit as a conference. Everything is super well organized and having the opportunity to listen to celebrities like David Chappell, Jim Webber, Dan North and Christian Weyer in Sweden is really great!

This year I also liked the mix of presentation as some where more general presentation as for example Benjamin Ling who’s the director of platform at Facebook and who gave an insight to how the platform is managed and developed. I also really enjoyed (besides the obvious ones as for example Christian Weyer’s WCF talk) Frans Hanel’s talk on the history and architecture behind a major price comparison site called prisjakt.nu – very interesting and a great technical mix in otherwise Microsoft focused conference.

Writing BizTalk context properties to a message from a WCF service using behaviors

Writing BizTalk context properties to a message from a WCF service using behaviors

The new WCF adapter in BizTalk 2006 R2 offers a lot of new possibilities. One of those is to write data to the BizTalk Message context properties directly from an exposed WCF Service. A practical use of this technic could be to write the username from the Windows credentials of the calling client into the context of the BizTalk message. This could be useful as this information is encrypted in messages that are received via the WCF adapter and isn’t possible to read when inside BizTalk. I’ll try and demonstrate the technique in this post.

If you have used the SOAP adapter before you might know that all you had to do was to turn on Windows based security for the exposed SOAP service and the username was automatically promoted to the context of the incoming BizTalk message. That username could then be used for routing, tracking which user called the service or using the value in plain text when communicating further to other connected systems. However using the WCF adapter this is not true anymore – when using the new WCF Message Security model the username and password is encrypted in the message and once the message is received by BizTalk it’s to late to read it. Basically we have to read the username in the actual service and write it into our own context property (that doesn’t get encrypted).

One way of achieving this is to read the username in the service and then to add it to the WCF Message Headers. All WCF message headers will by default be written to a the BizTalk Message context property called InboundHeaders (in the http://schemas.microsoft.com/BizTalk/2006/1/Adapters/WCF-properties namespace). First we’ll create an EndpointBehavior that will use a MessageInspector to add the username to the message header. Finally we create BehaviorExtensionElement so we can use a WCF Custom Binding in BizTalk and configure it to add our new behavior.

Creating the new EndpointBehavior

To create the configurable behavior we’ll need the three classes we mentioned above.

  1. A class that implements the IDispatchMessageInspector interface to handle to reading and writing to the actual message.

  2. A class that implements the IEndpointBehavior interface to define what kind of endpoint we’re creating and what it should do.

  3. A class that implements the BehaviorExtensionElement abstract class to create the behavior and make it configurable.

    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.ServiceModel;
    using System.ServiceModel.Channels;
    using System.ServiceModel.Dispatcher;
    using System.ServiceModel.Description;
    using System.ServiceModel.Configuration;

    namespace CustomWCFProperties.Behavior
    {
    ///


    /// PromoteUserNameMessageInspector implements IDispatchMessageInspector and adds the name from the WindowsIdentity to a WCF header called WindowsUserName in the http://CustomWCFProperties.Schema namespace. BeforeSendReply only returns as we’re not interested in handling the response.
    ///
    public class PromoteUserNameMessageInspector : IDispatchMessageInspector
    {
    #region IDispatchMessageInspector Members

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">object</span><span style="color: #000000"> AfterReceiveRequest(</span><span style="color: #0000ff">ref</span><span style="color: #000000"> System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel, System.ServiceModel.InstanceContext instanceContext)
        {
            </span><span style="color: #0000ff">string</span><span style="color: #000000"> windowsUserName </span><span style="color: #000000">=</span><span style="color: #000000"> ServiceSecurityContext.Current.WindowsIdentity.Name;
            request.Headers.Add(MessageHeader.CreateHeader(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">, windowsUserName));
            </span><span style="color: #0000ff">return</span><span style="color: #000000"> </span><span style="color: #0000ff">null</span><span style="color: #000000">;
        }
    
        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> BeforeSendReply(</span><span style="color: #0000ff">ref</span><span style="color: #000000"> Message reply, </span><span style="color: #0000ff">object</span><span style="color: #000000"> correlationState)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }
    
        </span><span style="color: #0000ff">#endregion</span><span style="color: #000000">
    }
    
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"><summary></span><span style="color: #008000">
    </span><span style="color: #808080">///</span><span style="color: #008000"> PromoteUserNameBehavior implements IEndpointBehavior and adds a message inspector to the dispatch behavior. Doesn't use any binding parameters, doesn't validate any configuration etc and can't be used in a client (only in a service).
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"></summary></span><span style="color: #808080">
    

    public class PromoteUserNameBehavior : IEndpointBehavior
    {
    #region IEndpointBehavior Members

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> AddBindingParameters(ServiceEndpoint endpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }
    
        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> ApplyClientBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.ClientRuntime clientRuntime)
        {
            </span><span style="color: #0000ff">throw</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> Exception(</span><span style="color: #000000">"</span><span style="color: #000000">The method or operation is not implemented.</span><span style="color: #000000">"</span><span style="color: #000000">);
        }
    
        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> ApplyDispatchBehavior(ServiceEndpoint endpoint, System.ServiceModel.Dispatcher.EndpointDispatcher endpointDispatcher)
        {
            endpointDispatcher.DispatchRuntime.MessageInspectors.Add(</span><span style="color: #0000ff">new</span><span style="color: #000000"> PromoteUserNameMessageInspector());
        }
    
        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">void</span><span style="color: #000000"> Validate(ServiceEndpoint endpoint)
        {
            </span><span style="color: #0000ff">return</span><span style="color: #000000">;
        }
    
        </span><span style="color: #0000ff">#endregion</span><span style="color: #000000">
    }
    
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"><summary></span><span style="color: #008000">
    </span><span style="color: #808080">///</span><span style="color: #008000"> Defines the behavior.
    </span><span style="color: #808080">///</span><span style="color: #008000"> </span><span style="color: #808080"></summary></span><span style="color: #808080">
    

    public class PromoteUserNameBehaviorElement : BehaviorExtensionElement
    {
    protected override object CreateBehavior()
    {
    return new PromoteUserNameBehavior();
    }

        </span><span style="color: #0000ff">public</span><span style="color: #000000"> </span><span style="color: #0000ff">override</span><span style="color: #000000"> Type BehaviorType
        {
            </span><span style="color: #0000ff">get</span><span style="color: #000000"> { </span><span style="color: #0000ff">return</span><span style="color: #000000"> </span><span style="color: #0000ff">typeof</span><span style="color: #000000">(PromoteUserNameBehavior); }
        }
    }
    

    }

Finally we have to sign the assembly using a strong key and add it to the GAC.

Configure the machine.config

As we need BizTalk and the WCF adapter to pick up the need behavior and make it possible to configure our receive port we need to to add the behavior element to the machine.config.aspx). The easiest way of doing this is to use the new WCF Service Configuration Editor tool and point to the machine.config file.

After the dll been added and the machine.config file has been saved the the line below should have been added to the element (that is if you use the same strong name key as in the sample project I’ve linked here).

<div><span style="color: #0000ff"><</span><span style="color: #800000">add </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="addCustomWCFProperties"</span><span style="color: #ff0000"> type</span><span style="color: #0000ff">="CustomWCFProperties.Behavior.PromoteUserNameBehaviorElement, AddCustomWCFPropertiesBehavior, Version=1.0.0.0, Culture=neutral, PublicKeyToken=705e34637fdffc54"</span><span style="color: #ff0000"> </span><span style="color: #0000ff">/></span></div>

Create the BizTalk Receive Port and Receive Location

Next thing to do is to start the BizTalk WCF Service Publishing Wizard. Choose to publish a service endpoint and make sure you enable metadata and create a receive location. In this example we’ll next choose to “Publish schemas as WCF service” and then define our service by naming service operations and so on.

When you then browse to the URL you choose to publish your service to you’ll see the nice example of how to instance the service you just defined.

If we then send a request message to service (you’ll find a client as part of the attached solution here) and inspect the message and its context properties in BizTalk we’ll see that the username of the calling client is nowhere to be found.

Configure a WCF-Custom binding and adding a Endpoint Behavior

To add the username to the message context we’ll need to add our newly created behavior to our service. We’ll do this by switch the service over to use a WCF-Custom binding to enable configuration. We then need to add the URL in the address field, define the binding type to a wsHttpBinding and to add our addCustomeWCFProperties behavior to the list of endpoint behaviors.

>
> [![note](../assets/2008/04/windowslivewriterwritingbiztalkcontextpropertiestoamessag-72f4note-thumb.gif)](../assets/2008/04/windowslivewriterwritingbiztalkcontextpropertiestoamessag-72f4note-2.gif) NOTE: there is a limitation in the BizTalk WCF implementation in that you can’t create the WCF-Custom receive location that uses a HTTP in-process based binding (like the wsHttpBinding used in a WCF-Custom endpoint is) first and then use the WCF Publishing Wizard to only publish a metadata endpoint.
>
>

>
> [Richard Seroter writes about it here](http://www.topxml.com/code/cod-72_10213_biztalk-and-wcf-part-v-publishing-operations-patterns.aspx) and I found the same thing to be true.
>
>

>
> _”This error doesn’t have to do with mixing MEX endpoints and “regular” endpoints in the same IIS web site, but rather, creating MEX endpoints for in-process HTTP bindings seems to trigger this. **Note that an IIS-hosted MEX endpoint CAN be created for IIS-hosted HTTP endpoints, but not for in-process hosted HTTP endpoints.”**_
>
>

>
> If you however choose a different binding that Http or (as in this case) publishes the metadata first and then switches over to a custom binding you’re ok.
>
>

If we then post another message to the service and inspect the message we’ll see that the behavior actually added a header and that it’s part of our BizTalk context properties. The adapter is also smart enough to know that this header isn’t part of the original headers and therefore stores in it’s own field within the context properties (you’ll find as part of the InboundHeaders block as well).

One problem remains – the actual value of the user is nested inside a XML node and the property isn’t promoted.

Extract and promote the value

To extract and promote the value we use an old fashion pipeline component using the following code in the execute method (the complete project is part of the downloadable sample project).

<div><span style="color: #000000">        </span><span style="color: #0000ff">public</span><span style="color: #000000"> IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
        {
            StringReader reader </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> StringReader(inmsg.Context.Read(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">).ToString());

            </span><span style="color: #0000ff">if</span><span style="color: #000000"> (reader </span><span style="color: #000000">!=</span><span style="color: #000000"> </span><span style="color: #0000ff">null</span><span style="color: #000000">)
            {
                XPathDocument document </span><span style="color: #000000">=</span><span style="color: #000000"> </span><span style="color: #0000ff">new</span><span style="color: #000000"> XPathDocument(reader);
                XPathNavigator navigator </span><span style="color: #000000">=</span><span style="color: #000000"> document.CreateNavigator();
                </span><span style="color: #0000ff">string</span><span style="color: #000000"> value </span><span style="color: #000000">=</span><span style="color: #000000"> navigator.SelectSingleNode(</span><span style="color: #000000">"</span><span style="color: #000000">/</span><span style="color: #000000">"</span><span style="color: #000000">).Value;
                inmsg.Context.Promote(</span><span style="color: #000000">"</span><span style="color: #000000">WindowsUserName</span><span style="color: #000000">"</span><span style="color: #000000">, </span><span style="color: #000000">"</span><span style="color: #000000">http://CustomWCFProperties.Schema</span><span style="color: #000000">"</span><span style="color: #000000">, value);
            }

            </span><span style="color: #0000ff">return</span><span style="color: #000000"> inmsg;
        }</span></div>

All the component does is reading the XML node the value exists inside and then it reads the actual value. Finally it writes the value back and promotes it. To be able to promote the value we also have to have a Property Schema deployed with a corresponding property name and namespace (WindowsUser and http://CustomeWCFProperties.Schema in this case).

The end results looks something like this.

The username is extracted and promoted and available for example for tracking or to for example use in a routing scenario.

This technique could of course be used for all kinds of scenarios where you like to add information to the context properties and could potentially replace a lot of the classic scenarios for custom pipelines.

All kind of comments are of course more than welcome!

Download the sample solution here.

Random thoughts on SQL Server Adapter and T-SQL Stored Procedures for BizTalk 2006

Random thoughts on SQL Server Adapter and T-SQL Stored Procedures for BizTalk 2006

Writing Stored Procedures is an art of its own. As all of you know it’s very different from writing ordinary code and presents its own kind of problems and issues when it comes to performance, builds, version control, testing etc. This post will try and highlight a few points that I find important when it comes to Stored Procedures, and the especially in a BizTalk and integration related scenario.

Consider the simplified procedure below, I’ll use is an example for much of the discussion in the rest of the post. Basically the procedure inserts or updates (based on if the optional EmployeeID exists from before or not) employee information to the Employee table of a database named “TestDB”.

<span class="kwrd"></span>








<div><span style="color: #0000ff">Use</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">TestDB</span><span style="color: #ff0000">]</span><span style="color: #000000">
</span><span style="color: #0000ff">Go</span><span style="color: #000000"> 
</span><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">select</span><span style="color: #000000"> </span><span style="color: #808080">*</span><span style="color: #000000"> </span><span style="color: #0000ff">From</span><span style="color: #000000"> dbo.sysobjects </span><span style="color: #0000ff">Where</span><span style="color: #000000"> id </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">object_id</span><span style="color: #000000">(N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">AddEmployee</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">And</span><span style="color: #000000"> </span><span style="color: #ff00ff">OBJECTPROPERTY</span><span style="color: #000000">(id, N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">IsProcedure</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">) </span><span style="color: #0000ff">Drop</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployeeGo 
</span><span style="color: #0000ff">Create</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee    
    </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Int</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #808080">-</span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">,    
    </span><span style="color: #008000">@LastName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">),    
    </span><span style="color: #008000">@FirstName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">),    
    </span><span style="color: #008000">@MiddleName</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Varchar</span><span style="color: #000000">(</span><span style="color: #800000; font-weight: bold">30</span><span style="color: #000000">)
</span><span style="color: #0000ff">As</span><span style="color: #000000"> 
</span><span style="color: #0000ff">Begin</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Declare</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">int</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Declare</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">tinyint</span><span style="color: #000000">    
    </span><span style="color: #008080">--</span><span style="color: #008080"> Check based on the EmployeeId if the employee exists from before    </span><span style="color: #008080">
</span><span style="color: #000000">    </span><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Not</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">Select</span><span style="color: #000000"> EmployeeId </span><span style="color: #0000ff">From</span><span style="color: #000000"> Employee </span><span style="color: #0000ff">Where</span><span style="color: #000000"> EmployeeId </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">)    
        </span><span style="color: #0000ff">Begin</span><span style="color: #000000">        
            </span><span style="color: #008080">--</span><span style="color: #008080"> Inserts new employee        </span><span style="color: #008080">
</span><span style="color: #000000">            </span><span style="color: #0000ff">Insert</span><span style="color: #000000"> </span><span style="color: #0000ff">Into</span><span style="color: #000000"> Employee (LastName,FirstName,MiddleName)         
            </span><span style="color: #0000ff">Values</span><span style="color: #000000"> (</span><span style="color: #008000">@LastName</span><span style="color: #000000">, </span><span style="color: #008000">@FirstName</span><span style="color: #000000">, </span><span style="color: #008000">@MiddleName</span><span style="color: #000000">)        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">Scope_Identity</span><span style="color: #000000">()        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">    
        </span><span style="color: #0000ff">End</span><span style="color: #000000">    
    </span><span style="color: #0000ff">Else</span><span style="color: #000000">    
        </span><span style="color: #0000ff">Begin</span><span style="color: #000000">        
            </span><span style="color: #008080">--</span><span style="color: #008080"> Updates employee        </span><span style="color: #008080">
</span><span style="color: #000000">            </span><span style="color: #0000ff">Update</span><span style="color: #000000"> Employee </span><span style="color: #0000ff">Set</span><span style="color: #000000"> LastName </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@LastName</span><span style="color: #000000">,        
            FirstName </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@FirstName</span><span style="color: #000000"> </span><span style="color: #0000ff">WHERE</span><span style="color: #000000"> EmployeeId </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000">        
            </span><span style="color: #0000ff">Set</span><span style="color: #000000"> </span><span style="color: #008000">@currentAction</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">2</span><span style="color: #000000">    
        </span><span style="color: #0000ff">End</span><span style="color: #000000">      

    </span><span style="color: #0000ff">Select</span><span style="color: #000000"> </span><span style="color: #008000">@currentEmployeeId</span><span style="color: #000000"> </span><span style="color: #0000ff">As</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">@EmployeeId</span><span style="color: #ff0000">]</span><span style="color: #000000">, </span><span style="color: #008000">@currentAction</span><span style="color: #000000">     
    </span><span style="color: #0000ff">As</span><span style="color: #000000"> </span><span style="color: #ff0000">[</span><span style="color: #ff0000">@ProcedureAction</span><span style="color: #ff0000">]</span><span style="color: #000000"> </span><span style="color: #0000ff">For</span><span style="color: #000000"> Xml Path (</span><span style="color: #ff0000">'</span><span style="color: #ff0000">Response</span><span style="color: #ff0000">'</span><span style="color: #000000">)
</span><span style="color: #0000ff">End</span><span style="color: #000000">
</span><span style="color: #0000ff">Go</span></div>

Builds and version control

I always try to have the complete database version controlled. One solution to this that some people advocate is to have a central development database. I have however always found it to be a lot of hassle with overwriting each others changes and so on and always try to make it possible for each developer to build a local version of the database for development. It should also be possible to always check out the latest version and run it to build all the database objects – no matter what you have from before. This is of course a bit tricky when it comes to the actual tables as you might have data and so on (in my opinion it shouldn’t be the case during development but that’s another post discussing test data for the database etc).

When it however comes to Stored Procedure as in this post it’s easy – drop the old one if it exists and create the new one as in the example.

<div><span style="color: #0000ff">If</span><span style="color: #000000"> </span><span style="color: #808080">Exists</span><span style="color: #000000">(</span><span style="color: #0000ff">select</span><span style="color: #000000"> </span><span style="color: #808080">*</span><span style="color: #000000"> </span><span style="color: #0000ff">From</span><span style="color: #000000"> dbo.sysobjectsWhere id </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #ff00ff">object_id</span><span style="color: #000000">(N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">AddEmployee</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">And</span><span style="color: #000000"> </span><span style="color: #ff00ff">OBJECTPROPERTY</span><span style="color: #000000">(id, N</span><span style="color: #ff0000">'</span><span style="color: #ff0000">IsProcedure</span><span style="color: #ff0000">'</span><span style="color: #000000">) </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">) </span><span style="color: #0000ff">Drop</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee
</span><span style="color: #0000ff">Go</span><span style="color: #000000">
</span><span style="color: #0000ff">Create</span><span style="color: #000000"> </span><span style="color: #0000ff">Procedure</span><span style="color: #000000"> AddEmployee </span><span style="color: #008000">@EmployeeId</span><span style="color: #000000"> </span><span style="color: #000000; font-weight: bold">Int</span><span style="color: #000000"> </span><span style="color: #808080">=</span><span style="color: #000000"> </span><span style="color: #808080">-</span><span style="color: #800000; font-weight: bold">1</span><span style="color: #000000">, ...</span></div>

If found this article by Jeff Atwood over at Coding Horror (and the articles by K. Scott Allen it links to) very good on how to think about version control database objects.

Idempotent procedures

In complex integration solution it’s a common problem with multiple messages – sending systems might send the same message twice for a number of reasons and one will end up with a shaky system if that causes an error in the integration. I’ve also seen solutions to this problem where a check is done against the database (say from a BizTalk orchestration) and based on that result a decision is made to call an update or an insert procedure. In my opinion that just unnecessary complexity and database communication. I believe one should try and handle whether to update or insert inside the procedure as long as that’s possible and effective.

I in the AddEmployee Stored Procedure example might receive a EmployeeID as an input parameter and based on a check inside the procedure if the employee exists from before with that id a post is either updated or inserted.

Response message

I generally always want two things in return from a Store Procedure like the one in the AddEmployee example (In a select based procedure that would of course be very different as we like all the data in return).

  1. The id that been added or changed

  2. What action that’s been performed (insert or update)

What action that was performed is a good idea to return as it’s quite usual to use this in for example BAM tracking or other logging and so on (we might want know how many employees that was added versus updated for example).

Using the Xml Path syntax for generating the Xml response (thanks again Nick for a great post) from the procedure has made it even simpler to actually skip using the Add Generated Item option in BizTalk. All you have think about is to set name of the child node the request root node to the same name as the procedure (in this case AddEmployee). Also remember that the type should be set up as a Multi-Part Message Type in the orchestrations you use it from and that the Receive Pipeline you choose in the Send Port must use a Xml Disassembler component (XmlReceive pipeline will do fine unless you have your custom pipeline for some reason).

Error handling

Some people might find it strange that I don’t have any error handling in my Stored Procedures but in most cases an error in the procedure will cause an exception in the port communication with the database and that’s fine. If I’d like to handle that error I’ll handle that either in the orchestration or using other mechanisms in BizTalk like Failed Message Routing etc. If I can’t handle the error in the procedure I don’t see a reason to catch it.

Security

I’ve seen to many cases where developers actually used the user and password login option available on the database adapter and port. Even if the credentials are safe in the SSO database there is a risk you’ll end up in a mess with login data spread all over the BizTalk administration. One should always use try to use Windows NT Integrated Security!

Even more common than the above is to have a user that part of the administrators group or similar to hit the database – don’t! I usually try an set up one Host to run as in all the communication with a specific database (Could also be a Host for that database and that BizTalk Application so that different applications have their own Host). I’ll then give that specific user the least privileges needed in the specific database.

In the AddEmployee and TestDB example I’d create a new Group in the AD called for example “BizTalk TestDB Users”. I’d then create a User in the AD called “BizTalk TestDB User”. It’s based on this I’d then create a Host in BizTalk referencing the above created Group.

Finally we’ll create a Host Instance and make a Send Handler of it on the SQL Server Adapter.

Now we can set the least required privileges on the “BizTalk TestDB Users” in Sql Server as that the Group the User belongs that’ll hit the database. In this case this means granting the Group to the standard BizTalk required privileges for a Host besides granting it Execute right on the AddEmployee Stored Procedure.

Something I missed? How do you handle your database communication and security in your BizTalk implementations?

A loosely coupled Scatter-and-Gather implementation In BizTalk 2006

Why loose coupling?

There was a question in Microsoft BizTalk forums the other day about how one could implement a Scatter and Gather pattern in a more loosely coupled fashion.

Most examples on the implementation of this pattern in BizTalk use the Call Shape functionality available in BizTalk orchestrations. This however creates a hard coupling between the “Scatter orchestration” and it’s “partner orchestrations”. The downside of that is that when one adds or removes a partner the whole solution has to be recompiled and redeployed.

If one however could use the publish-subscribe architecture in the MessageBox to route messages between the “Gather orchestration” and it’s partners, it’d be possible to add partners without having to worry about the rest of the solution. This post shows and example on how to implement a solution like that.

The BizTalk process in steps

>
> [![Aa559774_note(ja-jp,MSDN_10)](../assets/2008/01/aa559774-noteja-jpmsdn-10-thumb.gif)](../assets/2008/01/aa559774-noteja-jpmsdn-10.gif) NOTE: Notice the difference between the **Partners**Request, the **Partners**Response, the **Partner**Request and the **Partner**Response messages. The names are unfortunately very similar.
>
>
>
> The **Partners**Request and **Partners**Response messages are used for communication between the Scatter and the Gather orchestrations. It’s also a **Partners**Request message that activates the process.
>
>
>
> **Partner**Response and **Partner**Request are used for communicating between the Scatter and the Gather orchestration and all the partner orchestrations.
>
>

  1. Request and scatter
    A PartnersRequest message is received. This message is an empty message and is only used for activating the process in this example scenario. The PartnersRequest message is consumed by the Scatter orchestration. The Scatter orchestration creates one PartnerRequest message. The orchestration also generates a unique key called a RequestID and start a correlation combining that that id and the PartnersRequest MessageType. Finally it post the PartnerRequest message to message box, writes the generated RequestID to the request messages context (RequestID is a MessageContextPropertyBase based context property) and dehydrates itself.

  2. Partners
    All the enlisted Partner orchestrations pick up the PartnerRequest message from the message box. These orchestration then communicates with their specific data source (could be a service, database, file or whatever), receives a response. Finally these orchestrations transform the response they received and creates a PartnerResponse message that’s posted to the message box. Notice that the RequestID that was generated by the Scatter orchestration is also part of the context of the newly created PartnerResponse message.

  3. Gather
    The PartnerResponse messages are routed to the Gather orchestration. This orchestration uses a Singleton pattern based on the RequestID which all PartnerResponse messages carried with them in their context. This means that it’ll receive all the PartnerResponse messages containing that same RequestID into the same orchestration instance (ergo all the Partners that were activated by the request message being sent from one Scatterer). For each message instance it receives it add it’s price to a total price variable. When the Gather orchestration has received all the PartnerResponse messages (the orchestration knows how many Partners responses it should expect from one Scatterer orchestration and we can timeout if we don’t get all expected with a timeframe) the total price we calculated is written to a PartnersResponse message.

  4. Response
    This message is routed back the Scatter orchestration by using the correlation it initialized in the start. It’s finally this orchestration that send the final outgoing message (a PartnersResponse message).

Example solution

An example of the implementation can be downloaded from here.

The solution contains five different schemas.

  1. PartnersResponse
    Used for initializing the process.

  2. PartnersRequest
    Send from the Gather orchestration to the Scatter orchestration. It’s also the final result and outgoing message from the process.

  3. PartnerRequest
    Picked up and activates all enlisted Partner orchestrations.

  4. PartnerRequest
    Send from the Partner orchestration containing the result from the Partner Service and send to the Gather orchestration.

  5. LooselyCoupledScatterGatherExampleProperties
    Property schema for storing the RequestID and to correlate all the PartnerResponses as well as the final PartnersResponse back to the Scatter orchestration.

Five orchestrations

  1. Scatterer orchestration
    The “main orchestration” that receives a request message from outside and “scatters” party requests to all the party orchestrations.

  2. Gatherer orchestration
    Gatherer orchestration that gathers all the responses from the partners and transforms these to a reply that is being routed back to the Scatterer and back out.

  3. Partner1, Partner2 and Partner3 orchestration
    Partner orchestrations that communicates to different services and receives price information.

Setting up and testing the example solution – it’s easy!

When the solution is built and deployed one needs to setup and bind two ports; one outgoing port and one incoming port (this could also be a Request-Response port by changing the port type in the Scatterer orchestration). That’s it!

Enlist and start everything by dropping a PartnersRequest test message (you’ll find one among the zipped files) in the incoming folder. A PartnersResposne message should then be published in the outgoing folder containing a calculated price from all the Partner orchestrations.

Test message and a binding file are part of the zipped solution.

What would be different in “real life” solution?

I’ve made some major simplifications in this example to make it easy for setting it up and test the concept. These would very different in a “real” solution.

Partner Services

The Partner orchestrations are very simple. They actually don’t communicate with outside world at all. All the do is setting a hard coded price and post a response. In a real solution these would not be part of the same solution as the Scatter and Gather orchestration (otherwise we would be force to redeploy when adding a Partner orchestration to the dll).

The Partner orchestrations would also communicate with some sort of outbound source like a web service or database for example. This would however complicate the setup therefore I’ve skipped that part in the example.

Managing partners

One of the benefits with a loosely coupled implementation is the possibility to add and remove Partner orchestrations without having redeploy the rest of the solution. Using this implementation the Gatherer orchestration needs to know how many Partner responses it should wait for before timing out. This requires that value being set in a config file or something similar. In this example the number of Partners are hard coded into the Gather orchestration (it’s set to 3 Partners) to simplify the setup.

Final thoughts

Knowing how to create loosely coupled solutions like this is good knowledge to have. It’s my own and others belief that this architecture makes it possible to create more robust and separated solutions that one can update without having to do a lot a work and disturb the current processes. It’s however not the best solution performancewise as it adds a lot of extra hits on the MessageBox database and generates more work for the MessageAgent.

There are also a few things to watch out for:

Eternal loops

It easy to end up in a situation where you’re subscribing to the same message as you posting to the MessageBox. That’ll create and endless loop and cause a lot disturbance before you’ll find it. Think through and document you subscriptions!

Correlations for promoting values

When doing a direct post in BizTalk most properties are not promoted. To force you properties to be promoted you’ll have to initialize a correlation on the property as you send it. I can’t say I like this. There should be a other way of saying that one wants it promoted.

A couple of other useful articles as we on the subject:

Download the example orchestration and let me know how you used it and what your solution looks like!