Executing pipelines from inside an orchestration: Introducing the LOOPBACK adapter

Let’s go over some of the out-of-the-box options for executing a pipeline from inside your orchestration.

 

It struck my mind first that the MSMQT adapter IS in fact the messagebox.

 


  1. MSMQT

 


  • Set up a schedule with 1 MSMQT send port and 1 MSMQT receive port sharing the same MSMQT queue name (for example ‘ loopback’ would make a lot of sense)
  • Create a correlation type and set based on the my MSMQT label property.
  • Assign the MSMQT message-label to a newly created GUID inside your orchestration.

 

Try to do a send and a receive and you will see that your pipelines will be executed. As a test I could successfully parse a message from inside a schedule.

 

Another very simple option is using the HTTP adapter, credits go 100% to Scott Colestock. Scott used this as just a temporary approach to get past this problem, and then switched to the loopback adapter below…

 


  1. HTTP

 


  • Use a solicit-response send port 
  • Use bogus HTTP page that simply echoes the content back.

 


using System;
using System.Web;
using System.IO;

public class LoopbackHandler : IHttpHandler
{
   public bool IsReusable
   { get { return true; } }
  
   public void ProcessRequest(HttpContext context)
   {
      using(StreamReader sr = new StreamReader(context.Request.InputStream, true))
      {
         StreamWriter sw = new StreamWriter(context.Response.OutputStream, sr.CurrentEncoding );

         sw.Write(sr.ReadToEnd());

         sw.Flush();
         sw.Close();
      }
   }
}


  • Call it loopback.ashx for example, use IIS to host the page (for example http://localhost/loopback/loopback.ashx).

 

This can be easily explained because it requires no knowledge of correlation types and correlation sets, since you are using request/response ports the whole way. If simplicity is the main goal then this solution is probably superior to the MSMQT one.

 


  1. Custom LOOPBACK Adapter.

 

I also had another idea. Why not make your own loopback adapter by writing a bogus solicit-response adapter that returns a response BizTalk IBaseMessage that is a copy of the original request message. Recreate the original individual message-parts and copy of the request stream to the response stream. It’s very simple – download my sample loopback adapter here.

 


  • It’s a custom coded VB.NET non-batched but non-blocking (async) adapter that really doesn’t do anything.
  • Uses limited memory-footprint, streams to disk. 
  • To install use the MSI package. Don’t forget to add the adapter by using BizTalk administration…
  • It auto generates the transmit location URI (in fact it’s a GUID). I also added a Boolean property to specify whether you want the original message and part properties to be copied on to the response-message and parts.

 

Here are some of the benefits:

 


  • Trigger pipeline execution from inside your orchestration. Just define your outbound and inbound pipelines and they will be executed.
  • Send pipeline errors can be catched by adding a deliveryfailure-exception handler. Also adapter exceptions can be catched there, but since this adapter really doesn’t do anything this will be very rare (at least I hope so).
  • Receive pipeline errors can be catched by adding a general SOAP-exception handler.
  • Execute mappings from inside your orchestration by defining a map on the inbound and outbound port. When you use the later, mapping failures will be catched by the SOAP-exception handler.
  • It’s a black-hole adapter if you want. It can be used to ignore the processing of certain incoming messages. Make a send-port, name it ‘VANISH’ and subscribe to a given message-type or receive-port name for example.
  • It’s very useful for doing in-order processing. You can for example receive messages from MSMQT through a pass-through pipeline (which is recommended BTW) and process messages in-order from start-to-end while doing exception-handled parsing/validation/mapping by implementing a serial convoy.

 

Here’s a sample project that demonstrates some of the benefits.

 



Remarks:

 


  • The code can definitely be improved. It has been written very quickly in the scope of a POC. Alternatively you could make a better one using the adapter base classes and the wizard. In fact I’m hoping the community will see the benefit of this loopback adapter and someone else will make a better one. I’m not (yet, ha!) a streaming and multi-threading expert.
  • Apparently it is advised to return a read-only forwarding stream to the receive-pipeline. At least until SP1 arrives. I just used the VirtualStream class from the BizTalk Streaming assembly.
  • Do not forget, and as usual: there’s no warranty at all. I haven’t tested this very thoroughly yet so…use the adapter at your own risk.

WSDL First and BizTalk

I’ve held off posting on the WSDL-first approach so far because I’ve been waiting for the tools to arrive to make my gripes go away. Well, I’m still waiting… and waiting…


In the BizTalk world, we already have a Service Oriented mindset. We communicate in terms of schema-defined documents; we offer up services not APIs. We want to be interoperable with as many people as possible. I don’t care whether you’re using Java, .NET or ARM assembler – if you can send me a soap message, I’ll talk to you. The trouble is that the .NET world approaches web services from a very object-y view point. Just pop a [WebMethod] in your asmx page and let ASP.NET create you some WSDL describing your objects. Easy, right, for people who want to knock up a quick web service in code?


The trouble is this promotes a very code-first approach. This really sucks from a BizTalk developer’s point of view. I already have my schemas. I know the shape of the messages I want to receive. Where’s the support for the Schema- and WSDL-first approach? The “publish orchestration” wizard seems ropey – a colleague today ended up with unusable WSDL just because he’d used an xs:import in his schema. Besides, I don’t want a “publish orchestration” wizard, I want a WSDL designer with “create BizTalk orchestration template” and “import BizTalk definitions” options.


How about BizTalk’s support for consuming web services? If I add a reference to a Web Service, I want the types defined in that WSDL added as first-class schemas in my BizTalk solution, not hidden under MyWebReference\Reference1.xsd. I want support for promoting properties out of the web reference – just remembering them if I refresh the web reference would be nice. Even worse, if I already have those schemas included in my solution please don’t create a duplicate copy of the schema – share the reference (.NET 2.0 has an enhancement to WSDL.exe that may support this, I believe).


For the record, I am very much in favour of WSDL first. I just think there’s an awful lot of pain to get it working in anything but the most simple cases. Let’s hope Microsoft’s new Domain Specific Languages designers lead to some WSDL designers pronto. XML Spy looks nice, but it’s damned expensive!


This hasn’t been a very structured post, for which I apologise, but I’d love to hear from anyone else who has found any silver bullets for BizTalk and WSDL-first…

More on deployment…

Ok, sorry for the short hiatus….very busy times on my current project.  Anyway, I previously mentioned some points around deployment.  It turns out that using BTS Installer doesn’t work well when utilizing roles and parties.  This doesn’t mean we have to scrap the BTSInstaller all together, we just have to do a little extra work in creating scripts that do the deployment as well as the undeployment, and then call those scripts from our MSI package.


 


In the zip file (get it here), I’ve included some sample scripts that may give you a head start in the deployment process.  Keep in mind that if you are NOT using roles and parties, BTSInstaller will get you about 80% of the way there, and you will not need all of the scripts.  You will still need some scripts to go ahead and enlist and start everything, and some work is needed to automagically undeploy everything using Add or Remove Programs.  With everything included in the zip file, you should be able to gleam something that will help.


 


A few key points –


 



  • Binding files – everything is driven by the binding file.  Make sure it is correct, or you will have issues.  One thing to watch out for is your binding file contains the PublicTokenKey of the assembly it is derived from.  If during multi-developer development these somehow get out of sync, when you deploy you won’t get any errors, but nothing gets deployed either.

  • Deleting parties – if you are implementing parties and roles, you will have to un-enlist the parties and delete them prior to un-deploying the assemblies that contain the orchestrations.  Included in the SDK are 2 sample projects that when compiled create one exe to un-enlist the parties, and one exe to delete the parties.  These are coded assuming that the SQL database where your management database resides is local to BizTalk.  This is most likely not the case, so you will have to edit this code to determine what server the database is on, and supply that in the connection string.  It’s quite simple using MSI, and if you need some code snippets just send me a comment.

  • Removing orchestrations – running the MSI to delete orchestrations (if using the default BTSInstaller) won’t remove your Orchestrations since they are in the running state.  Chances are if you’re testing they are even in the active state.  In the supplied StopOrchestrations script, there is a flag you can pass into the Unenlist method that will terminate any active instances.  Pay attention to this if this is not your intended actions.

  • Write to the event log – you will notice in my scripts I have added some code to shell out and use the LogEvent method to write errors to the Application Log.  With the nature of VBScript, if you don’t do this and run into errors, you will never figure out where to start looking.

 


Use the scripts at your own risk.  They are commented, so you should be able to read along and see what’s going on.  Remember, if you are not using roles, you won’t need half the stuff in the Install.bat file, and some of the stuff in Cleanup.bat, but read through those to see the order things are called, and you will need some of the other scripts regardless.


 


Cheers!!

Halo 2! :-)

Things I like to see in my MSN window #1:


“Can you get up to TVP? Halo 2 team are here and we’re playing.”


WOOHOO! Just had a 6 minute deathmatch round against the UK XBox team. We lost about 50-20, which is pretty shabby, but I did manage to score a double kill. Dual wielding SMGs is teh roxx0rs! I just about resisted temptation to hit campaign mode 😉


Roll on Thursday.

Parallel Convoys (Concurrent Receive) in BizTalk

In the past, I have tended to focus on sequential convoys.  Now, I decided to take a look at Concurrent Convoys.  First off, I was sure in older versions of the documentation they were called Parallel Convoys?  Maybe I’m mistaken or maybe the name changed over the past few months….  In any case, I will get with the times and call them Concurrent Convoys.  No matter what you call them, the power and usefulness of this type of convoy is the same.



Why use Concurrent Convoys?  Concurrent convoys allow you to receive multiple items in any order that all need to be processed by the same Orchestration before the next step in the process can happen. 



Setting up and using a Concurrent Convoy is rather straight forward.  All you have to do is drop a Parallel Shape inside your Orchestration.  Then, drop in your multiple Receive Shapes. If this is the first shape in your Orchestration then each Received must be set with Activate = True and all must initialize the same Correlation Set.  Otherwise, they must all follow the same Correlation Set that was initialized by a Send Shape.  That is all your need to do!



I have included a sample with two different Orchestrations.  One Orchestration shows a Concurrent Convoy with Active Receives and one without Active Receives.



The Orchestration with Activate Receives has two sets of sample files.  The set with ID 100 will produce 8 total records in the output file.  The set with ID 500 produces 6 records.  Note that since all three messages arrive on different ports, Receive Port mapping is possible to get all inbound documents into a common format (not shown in my sample). 



The Orchestration without Activate Receives uses the same set of sample files but it requires a Start Message to be received and a message to be sent out to initialize the correlation.  This will produce the same output of 8 and 6 records.  In this Orchestration, all the Receives uses the same Receive Port.



For information on how to run the samples, please see the Read Me file.


Make sure you keep an eye on the BTSSubscriptionViewer tool (located in the SDK).  Note the differences between the two Orchestrations.



DOWLOAD: Sample Concurrent Receive Convoys



Bonus:  I also show a simple say to merge all three messages together into a single output.  This uses a map inside the Orchestration. 

Parallel Convoy Message Processing

This sample shows how to use a simple Concurrent (Parallel) Convoys inside an Orchestration. For more information on Convoy, be sure to read the Convoy Deep Dive White Paper.

This sample also shows an example of multi-message mapping inside an Orchestration.

This sample should work for BizTalk 2004 and BizTalk 2006.

Get more information from the original blog post on this topic: https://www.biztalkgurus.com/biztalk_server/biztalk_blogs/b/biztalk/archive/2004/11/08/parallel-convoys-_2800_concurrent-receive_2900_-in-biztalk.aspx

BizTalk Code Behind

I was chatting to my colleague Charles Young earlier today and it reminded me of an idea that I had a while back. Wouldn’t it be nice if, instead of having expression shapes (and the awful expression editor window 😉 ), we instead had an ASP.NET-like “code behind” model allowing us to use any .NET language, and not have to worry about working within the restrictions of XLANG/s. Of course I appreciate the design principle that Orchestrations shouldn’t be littered with code and you should instead be calling out to .NET classes to do the real work, but the developer experience of ASP.NET code-behind development is just so appealing. Orchestrations are rather like ASP 1 for calling code in their current form.


That led me to thinking that effectively you’d have anonymous blocks of code (a la .NET 2.0) subscribing to certain events in the orchestration; following that thought, if you could create subscriptions for blocks of code, you’d be opening up a whole world of possibilities (or, maybe, pain). Then again, that’s not such a big step from the Aspect Oriented ethos – Aspect Oriented Biztalk anyone? 😉