It is a common request.  How can I debatch a message into smaller batches rather than single messages inside BizTalk 2004? 



I have tried many times to get this accomplished natively inside the pipeline.  One would think it would be as easy as setting the group property to 10 if you wanted a batch size of 10, but that does not seem to work.  The only way I have been able to break up a message into smaller batches has been inside custom code.



I wrote a .net component that can be called using an atomic scope shape from inside an Orchestration.  This component will return smaller batches of any size you specify inside the Orchestration and wrap the smaller batches inside a specified root node.  This .net component uses the XmlTextReader to stream the message along and debatch it. 



This .net component is called StreamHelper.  It illustrates the following:



  • Using XmlTextReader
  • Passing an Xlang Message into a .net component
  • Returning an Xlang Message from a .net component
  • Setting context properties on a message returned from a .net component


Download: Debatch Streaming Sample



Setup: Download and extract the solution.  Build and GAC the StreamHelper assembly then Build and Deploy the DebatchProperties and DebatchOrch assemblies.  I have included a basic sample file to test with.  For more detailed information on the sample see the comments throughout the solution.



A better way to do it? 


I’m sure there is.  I just put this together in a short amount of time.  I would guess the most efficient place for this type of work would be inside a custom pipeline component.  But, working with custom pipelines can be difficult.



How does this debatching approach perform compared to the others?


Performance was almost identical to that of the Xml Node List inside an atomic scope.  In this sample, I used the XmlTextReader and passed Xlang messages in and out of the component to try to increase performance.  That did not seem to accomplish anything.  As expected, as the total number of messages inside the atomic scope increased the performance decreased. 






























































































Type


XML Size (MB)


# Nodes


Batch Size


Time (Sec)


Msg/Sec


Msg Size (kb)


.net Stream


1.6


500


1


5


100.0


3.0


.net Stream


3.6


1100


1


18


61.1


3.0


.net Stream


7.2


2200


1


44


50.0


3.0


.net Stream


18.1


5500


1


224


24.6


3.0


.net Stream


128.6


38500


1



FAIL



.net Stream


1.6


500


10


1


500.0


3.0


.net Stream


3.6


1100


10


3


366.7


3.0


.net Stream


7.2


2200


10


9


244.4


3.0


.net Stream


18.1


5500


10


15


366.7


3.0


.net Stream


128.6


38500


10


159


242.1


3.0



For more debatching information please see my past post:
Debatching Options and Performance Considerations in BizTalk 2004