Home Page › Forums › BizTalk 2004 – BizTalk 2010 › Receive Pipeline – Very Large Flat Files
- This topic has 2 replies, 1 voice, and was last updated 9 years, 10 months ago by
community-content.
-
AuthorPosts
-
-
October 30, 2009 at 1:05 PM #23603
Need advice on best way to handle receiving very large flat files (~2 gigs each). Each is an interface with ~ million records-transactions. Some only need routing, but others will need initial transform then processing in BizTalk. Frequency is several files daily.
For those only needing routing, I’ve looked at Dunphy & Metwally’s Pro BizTalk 2006, Very Large File Pipeline component, that stores the file to disk with the path stored in the IBaseMessagePart.Data property. Is this still a good way to handle, or is there something better now?
For the interfaces requiring a transform, should the interface be read as a stream directly to the messagebox? Then subscribe to it with either the orchestration containing the map (not my 1st choice for the overhead); or a subscribing send port containing the map?
OR – will the receive port with a map have have a processing impact similar to the messagbox => subscribing orch or send port scenerio? Or is there another approach that should be considered?
-
October 30, 2009 at 1:47 PM #23604
Hi,
Please see the following links for more information on how to better approach the processing of large files in BizTalk:
http://technet.microsoft.com/en-us/library/aa560481(BTS.20).aspx
I hope these resources help.
Daniel.
-
November 1, 2009 at 8:56 PM #23610
Hi,
I would like to share my experience with large message handling. But I don’t know it will go with your requirements. Anyway hope this will help
We have to transfer 5000 files including 2 GB sizes. So the application was designed to support Canonical Data Model. For example the message properties like
XYZ Message
– GUID
– Receive Filename
– Payload
File adaptor receives the message and custom pipeline component will take the message, create the canonical message then check the message size, if the massage size is less than 1 MB (configured value) then attached the message in to canonical message part payload section. If not stream the message into file location & save with the Guid and keep the message payload as empty. Anytime if the process wants s to track, uses the GUID as unique key.
But this is good if we don’t need to process internal data in the message.
The other thing that we have faced is ‘out of memory exception’ issue because we load XML Document in to the memory. To get rid of that we write to the message into the temporary file and using XMLReader creates a new message.
The other issue that we have faced that use XMLDocument messages inside the orchestration. After reviewing the code by Microsoft, recommended to use XLANGMessage .
Best regards,
Nandika
-
-
-
AuthorPosts
- The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.