Here is a 45 minute, step-by-step lab showing how to use the Flat File Disassembler, Delivery Notification, and a Sequential Convoy to map a large flat file document in BizTalk 2004. This lab will break up a large flat file into single messages, map each message, and write the ordered output to an output file.
This lab takes you through it in three Exercises. It is based on Microsoft’s hands-on-labs.
Download: Sequential Message Processing Hands On Lab
Setup is simple. Unzip the download to your c:\ drive and run the included binding file to create the send and receive ports used in the lab. Make sure the root folder named “Microsoft Hands-On-Labs” is located at c:\. The lab manual is located under \ILL010_Convoys. The exercises are located in the \Source\Exercises folder. The full solution is located in the \Source\Solution folder.
Although newbie’s to BizTalk can complete this lab, it covers advanced topics. If you are new to BizTalk, do not get discouraged if you find this lab difficult.
Many of these were released a few months ago in a beta form but now they are available on the Microsoft website.
They include the following:
BizTalk Server 2004: A Messaging Engine Overview
BizTalk Server 2004 and Web Services
Connecting to the Elemica Network with BizTalk Accelerator for RosettaNet 3.0
Developing Integration Solutions with BizTalk Server 2004
Implementing UAN on BizTalk Server 2004 to Integrate Siebel and SAP
Risk Scoring with BizTalk Server 2004 and the Business Rules Framework
Transactions Across BizTalk Server 2004
Working With BizTalk Adapter for SQL Server
BizTalk Server 2004 Convoy Deep Dive
They all look like great reading!
In addition, I have finally gotten around to getting my website up and running. It still needs A LOT of work, but at least it lists all my BizTalk samples, tools, and demos in one place. Check it out at https://www.biztalkgurus.com
NAcks or Negative Acknowledgments are small SOAP messages that exist in BizTalk when a messaging error occurs. This happens when adapters fail, pipelines fail, messages do not match subscriptions, ect.
These messages can be subscribed to and a business process can be written to react to all of these messages or specific error codes.
This sample Orchestration shows how simple it is to catch these messages. To subscribe to a NAck, just create a subscription to: BTS.AckType = NACK. It’s that easy!
Download: NAck Catching Sample
Setup: Download and extract the solution. Create a receive port that causes a routing failure. Drop a message and watch for the output in the Out folder.
For a routing failure NAck, the following properties are available in the message context and could be accessed inside the Orchestration:
With this information, logic could be included inside the Orchestration to extract / reprocess a message or log the error to an additional notification facility (other then the event log since it would already be logged there).
The Convoy Deep Dive white paper and sample code is now available on MSDN. You can read the paper on-line and download the sample code here.
For anyone who read the “beta” version of the paper, Scenario 2 went through a minor change. I now use an atomic scope to build the output message rather then a simple string concatenation. This eliminates messages collecting inside the message box in the “Delivered, not consumed” status.
It seems that the status of a message is not updated until a persistence point is reached or the Orchestration dehydrates. I would guess this is the case with any messages that are used inside an Orchestration and not just inside a looping-receive Convoy. But, I have not verified this.
Overall, this change provides a better solution since the Orchestration state is being saved when each new message arrives and can be easily be restated/recovered in the event of a system failure.
Microsoft has released 5 new White Papers today in a “Beta” mode. Full details on all 5 are available here.
They include the following:
– BizTalk Server 2004 Web Services (Brian Loesgen)
– BizTalk Server Internal Workings (Internal)
– SQL Adapter Deep Dive (Todd Uhl)
– Convoy Deep Dive (Stephen W. Thomas)
– Extending BizTalk Server with Host Integration Server (Jon Fancy)
They all look like great reading. I’m rather fond of the Convoy Deep Dive paper myself.
As always, feedback is welcome.
It is a common request. How can I debatch a message into smaller batches rather than single messages inside BizTalk 2004?
I have tried many times to get this accomplished natively inside the pipeline. One would think it would be as easy as setting the group property to 10 if you wanted a batch size of 10, but that does not seem to work. The only way I have been able to break up a message into smaller batches has been inside custom code.
I wrote a .net component that can be called using an atomic scope shape from inside an Orchestration. This component will return smaller batches of any size you specify inside the Orchestration and wrap the smaller batches inside a specified root node. This .net component uses the XmlTextReader to stream the message along and debatch it.
This .net component is called StreamHelper. It illustrates the following:
- Using XmlTextReader
- Passing an Xlang Message into a .net component
- Returning an Xlang Message from a .net component
- Setting context properties on a message returned from a .net component
Download: Debatch Streaming Sample
Setup: Download and extract the solution. Build and GAC the StreamHelper assembly then Build and Deploy the DebatchProperties and DebatchOrch assemblies. I have included a basic sample file to test with. For more detailed information on the sample see the comments throughout the solution.
A better way to do it?
I’m sure there is. I just put this together in a short amount of time. I would guess the most efficient place for this type of work would be inside a custom pipeline component. But, working with custom pipelines can be difficult.
How does this debatching approach perform compared to the others?
Performance was almost identical to that of the Xml Node List inside an atomic scope. In this sample, I used the XmlTextReader and passed Xlang messages in and out of the component to try to increase performance. That did not seem to accomplish anything. As expected, as the total number of messages inside the atomic scope increased the performance decreased.
XML Size (MB)
Msg Size (kb)
For more debatching information please see my past post:
Debatching Options and Performance Considerations in BizTalk 2004
I guess these came out several days ago. I did not hear about it until now.
Anyways, updated BizTalk 2004 Documentationisavailable here.
I have talked a lot in the past about Convoys. But, under the covers what really makes Convoy message processing different? What makes a Convoy a Convoy?
To sum it all up into one sentence: Convoys require database level message-to-Orchestration correlation independent of any running Orchestration instances.
Is this confusing? Yes. But trust me, more detailed information is coming soon on this topic. For now, I want to briefly go over the basic internals of how Convoys work in BizTalk Server 2004.
When you deploy an Orchestration:
The presence of a Convoy is detected at the time a BizTalk 2004 Orchestration is enlisted. At this point, an entry is inserted into the ConvoySets table inside the Message Box that lists the properties used in the Convoy Set. This is limited to only three properties because each property is listed in a column inside the table. The subscriptions that can activate this Convoy are marked with the uidConvoySetID for that Convoy Set.
When a message arrives:
Every time a new message arrives into BizTalk it is checked to see if it is part of a Convoy or if it should start a Convoy. If it should start a Convoy, an entry is inserted into the ConvoySetInstances table that routed messages to the correct instance. If it is part of an existing Convoy (i.e. it has the sample Convoy properties of an existing Convoy set), it is routed to the Service Instance listed inside the table. Otherwise, it just follows the normal subscription process.
Convoy Virtual Subscription:
This database level message routing creates a type of virtual subscription on top of the standard subscriptions inside the message box. This is set up internally using a series of tables.
I have put together a simple SQL query to view all Orchestrations that belong to a Convoy and what the Convoy virtual subscription properties are for those instances. I have not tested this code under all Convoy scenarios so please use caution when running the query and it may yield unexpected results. Note that for this viewer to return any data you must have running Orchestration instances that are part of a Convoy.
DOWNLOAD: Sample Convoy Virtual Subscription Viewer
CRITICAL: This application and included SQL will query critical BizTalk SQL Tables. This should not be run on production servers. Review all code prior to execution and use extreme caution when viewing any BizTalk related database tables. Use this code at your own risk.
This sample is also an excellent example of why I am a BizTalk Developer rather then a .net Developer :).
Here are some of my samples that use Convoys that can be used with this viewer.
DOWNLOAD: Concurrent Receive Convoys (post date 11/5/2004)
DOWNLOAD: Sequential Receive Convoys (post date 8/23/2004)