Sequential Flat File Message Processing Lab

Here is a 45 minute, step-by-step lab showing how to use the Flat File Disassembler, Delivery Notification, and a Sequential Convoy to map a large flat file document in BizTalk 2004.  This lab will break up a large flat file into single messages, map each message, and write the ordered output to an output file.



This lab takes you through it in three Exercises.  It is based on Microsoft’s hands-on-labs.



Download: Sequential Message Processing Hands On Lab



Setup is simple.  Unzip the download to your c:\ drive and run the included binding file to create the send and receive ports used in the lab.  Make sure the root folder named “Microsoft Hands-On-Labs” is located at c:\.  The lab manual is located under \ILL010_Convoys.  The exercises are located in the \Source\Exercises folder.  The full solution is located in the \Source\Solution folder.



Although newbie’s to BizTalk can complete this lab, it covers advanced topics.  If you are new to BizTalk, do not get discouraged if you find this lab difficult. 


New BizTalk Server White Papers on MSDN

Many of these were released a few months ago in a beta form but now they are available on the Microsoft website.    



They include the following:


BizTalk Server 2004: A Messaging Engine Overview



BizTalk Server 2004 and Web Services



Connecting to the Elemica Network with BizTalk Accelerator for RosettaNet 3.0



Developing Integration Solutions with BizTalk Server 2004



Implementing UAN on BizTalk Server 2004 to Integrate Siebel and SAP



Risk Scoring with BizTalk Server 2004 and the Business Rules Framework



Transactions Across BizTalk Server 2004



Working With BizTalk Adapter for SQL Server



BizTalk Server 2004 Convoy Deep Dive



They all look like great reading!



In addition, I have finally gotten around to getting my website up and running.  It still needs A LOT of work, but at least it lists all my BizTalk samples, tools, and demos in one place.  Check it out at https://www.biztalkgurus.com


 

NAck Message Processing Orchestration for BizTalk Server 2004

NAcks or Negative Acknowledgments are small SOAP messages that exist in BizTalk when a messaging error occurs.  This happens when adapters fail, pipelines fail, messages do not match subscriptions, ect.



These messages can be subscribed to and a business process can be written to react to all of these messages or specific error codes. 



This sample Orchestration shows how simple it is to catch these messages.  To subscribe to a NAck, just create a subscription to: BTS.AckType = NACK.  It’s that easy!



Download: NAck Catching Sample



Setup: Download and extract the solution.  Create a receive port that causes a routing failure.  Drop a message and watch for the output in the Out folder.



For a routing failure NAck, the following properties are available in the message context and could be accessed inside the Orchestration:


AckDescription


AckFailure Category


AckFailure Code


AckID


AckInboundTransportLocation


AckOwnerID


AckReceivePortID


AckReceivePortName


AckType



With this information, logic could be included inside the Orchestration to extract / reprocess a message or log the error to an additional notification facility (other then the event log since it would already be logged there).

Convoy Message Deep Dive White Paper on MSDN

The Convoy Deep Dive white paper and sample code is now available on MSDN.  You can read the paper on-line and download the sample code here.



For anyone who read the “beta” version of the paper, Scenario 2 went through a minor change.  I now use an atomic scope to build the output message rather then a simple string concatenation.  This eliminates messages collecting inside the message box in the “Delivered, not consumed” status. 



It seems that the status of a message is not updated until a persistence point is reached or the Orchestration dehydrates.  I would guess this is the case with any messages that are used inside an Orchestration and not just inside a looping-receive Convoy.  But, I have not verified this.



Overall, this change provides a better solution since the Orchestration state is being saved when each new message arrives and can be easily be restated/recovered in the event of a system failure.


 

New BizTalk Server 2004 White Papers

Microsoft has released 5 new White Papers today in a “Beta” mode.  Full details on all 5 are available here. 



They include the following:


– BizTalk  Server 2004 Web Services (Brian Loesgen)


– BizTalk Server Internal Workings (Internal)


– SQL Adapter Deep Dive (Todd Uhl)


– Convoy Deep Dive (Stephen W. Thomas)


– Extending BizTalk Server with Host Integration Server (Jon Fancy)



They all look like great reading.  I’m rather fond of the Convoy Deep Dive paper myself. 



As always, feedback is welcome.

Large Messages Can Cause Large Problems in BizTalk

Recently I have seen a lot of talk about large message support in BizTalk 2004.  The most interesting and depressing was on the BizTalk Core Engine WebLog.  It seems that large messages do not work as well in BizTalk 2004 as I would have hoped.



I have had success with some types of large messages using the file adapter and mapping.  I have gotten it to work with messages up to 500 MB but in a production environment this would not be a viable or supported solution. 



During some recent testing, I discovered the follow condition that rather surprised me.



Scenario


I was returning an unpredictable amount of data out of an Oracle database using .net code.  This data can be 1 MB or up to 250MB+ and needed to be processed in a single batch.  This data is then converted into Xml, cast to a message, and mapped with the results written to disk using a Send Shape.  Life was good until the extracted data returned was greater then 220 MB in size.


 


At this point, everything went downhill.  BizTalk Server began to have serious problems.  Receive functions were shutting down and I received the following Warning message in the Application Log:



A catastrophic failure occurred in the BizTalk service. The service will shutdown and auto-restart in 1 minute. If the database is still unavailable, this cycle will be repeated.


 


Error message: Exception of type System.OutOfMemoryException was thrown.


Error source:


 


BizTalk host name: BizTalkServerApplication Windows service name: BTSSvc{E93E7C4A-9E8B-40F6-AD7C-4F53B85F36C4}


 


For more information, see Help and Support Center at



All appears to be well and good since BizTalk seems to be handling the situation as expected… except for when the service actually starts again.  Since the Orchestration that caused the fatal error is still Active (the Orchestration itself did not through any errors), BizTalk starts to process the Orchestration from the last persistence point.  Thus, repeating the cycle in an endless loop.



Now, for the interesting behavior…  If this is running in a multi server BizTalk environment at the point the 1st BizTalk Server fails the 2nd one picks up; like I would expect.  I end up getting a completed Orchestration with no errors.  Super!  But, I only get a zero byte file as output.  So, looking inside HAT everything would look like it completed successfully with no errors.  Not good…



Overall, I found that an Orchestration seems to be able to handle about 440 MB of “stuff” before it crashes.  Just make sure you know what makes a copy of the message (like mapping).  Of course, processes with this amount of data would not be supported nor do I recommend actually doing it.  But, it has been fun to play around with and see what fails.



The moral of this story is to be very mindful of your message size.

Debatching Messages Into Smaller Batches In An Orchestration

It is a common request.  How can I debatch a message into smaller batches rather than single messages inside BizTalk 2004? 



I have tried many times to get this accomplished natively inside the pipeline.  One would think it would be as easy as setting the group property to 10 if you wanted a batch size of 10, but that does not seem to work.  The only way I have been able to break up a message into smaller batches has been inside custom code.



I wrote a .net component that can be called using an atomic scope shape from inside an Orchestration.  This component will return smaller batches of any size you specify inside the Orchestration and wrap the smaller batches inside a specified root node.  This .net component uses the XmlTextReader to stream the message along and debatch it. 



This .net component is called StreamHelper.  It illustrates the following:



  • Using XmlTextReader
  • Passing an Xlang Message into a .net component
  • Returning an Xlang Message from a .net component
  • Setting context properties on a message returned from a .net component


Download: Debatch Streaming Sample



Setup: Download and extract the solution.  Build and GAC the StreamHelper assembly then Build and Deploy the DebatchProperties and DebatchOrch assemblies.  I have included a basic sample file to test with.  For more detailed information on the sample see the comments throughout the solution.



A better way to do it? 


I’m sure there is.  I just put this together in a short amount of time.  I would guess the most efficient place for this type of work would be inside a custom pipeline component.  But, working with custom pipelines can be difficult.



How does this debatching approach perform compared to the others?


Performance was almost identical to that of the Xml Node List inside an atomic scope.  In this sample, I used the XmlTextReader and passed Xlang messages in and out of the component to try to increase performance.  That did not seem to accomplish anything.  As expected, as the total number of messages inside the atomic scope increased the performance decreased. 






























































































Type


XML Size (MB)


# Nodes


Batch Size


Time (Sec)


Msg/Sec


Msg Size (kb)


.net Stream


1.6


500


1


5


100.0


3.0


.net Stream


3.6


1100


1


18


61.1


3.0


.net Stream


7.2


2200


1


44


50.0


3.0


.net Stream


18.1


5500


1


224


24.6


3.0


.net Stream


128.6


38500


1



FAIL



.net Stream


1.6


500


10


1


500.0


3.0


.net Stream


3.6


1100


10


3


366.7


3.0


.net Stream


7.2


2200


10


9


244.4


3.0


.net Stream


18.1


5500


10


15


366.7


3.0


.net Stream


128.6


38500


10


159


242.1


3.0



For more debatching information please see my past post:
Debatching Options and Performance Considerations in BizTalk 2004


 

Community MSMQ Adapter for BizTalk 2004 Now Available


Yesterday Microsoft released a BizTalk 2004 Adapter for MSMQ v2.0.  This works with MSMQ 2.0 and 3.0. 



Here are the key features from the download site:



The BizTalk 2004 Adapter for MSMQ provides support for sending and receiving messages using MSMQ 2.0 and 3.0, and takes advantage of the new features in MSMQ 3.0. Major features of the adapter include support for:


• MSMQ 2.0 and 3.0
• Multi-threading option for faster performance
• Guaranteed delivery when using transactional queues
• Reading from remote non-transactional queues
• Dynamic send ports
• Support for messages up to 2 Gigabytes in size


 


Download this adapter here.


 

Convoy Subscription Viewer for BizTalk Server 2004

I have talked a lot in the past about Convoys.  But, under the covers what really makes Convoy message processing different?  What makes a Convoy a Convoy? 



To sum it all up into one sentence: Convoys require database level message-to-Orchestration correlation independent of any running Orchestration instances. 



Is this confusing?  Yes.  But trust me, more detailed information is coming soon on this topic.  For now, I want to briefly go over the basic internals of how Convoys work in BizTalk Server 2004.



When you deploy an Orchestration:


The presence of a Convoy is detected at the time a BizTalk 2004 Orchestration is enlisted.  At this point, an entry is inserted into the ConvoySets table inside the Message Box that lists the properties used in the Convoy Set.  This is limited to only three properties because each property is listed in a column inside the table.  The subscriptions that can activate this Convoy are marked with the uidConvoySetID for that Convoy Set.



When a message arrives:


Every time a new message arrives into BizTalk it is checked to see if it is part of a Convoy or if it should start a Convoy.  If it should start a Convoy, an entry is inserted into the ConvoySetInstances table that routed messages to the correct instance.  If it is part of an existing Convoy (i.e. it has the sample Convoy properties of an existing Convoy set), it is routed to the Service Instance listed inside the table.  Otherwise, it just follows the normal subscription process.



Convoy Virtual Subscription:


This database level message routing creates a type of virtual subscription on top of the standard subscriptions inside the message box.  This is set up internally using a series of tables. 



I have put together a simple SQL query to view all Orchestrations that belong to a Convoy and what the Convoy virtual subscription properties are for those instances.  I have not tested this code under all Convoy scenarios so please use caution when running the query and it may yield unexpected results.  Note that for this viewer to return any data you must have running Orchestration instances that are part of a Convoy. 



DOWNLOAD: Sample Convoy Virtual Subscription Viewer



CRITICAL: This application and included SQL will query critical BizTalk SQL Tables.  This should not be run on production servers.  Review all code prior to execution and use extreme caution when viewing any BizTalk related database tables.  Use this code at your own risk.



This sample is also an excellent example of why I am a BizTalk Developer rather then a .net Developer :).



Here are some of my samples that use Convoys that can be used with this viewer.


DOWNLOAD: Concurrent Receive Convoys (post date 11/5/2004)


DOWNLOAD: Sequential Receive Convoys (post date 8/23/2004)