Receive Pipeline – Very Large Flat Files

Home Page › Forums › BizTalk 2004 – BizTalk 2010 › Receive Pipeline – Very Large Flat Files

Viewing 1 reply thread
  • Author
    Posts
    • #23603

      Need advice on best way to handle receiving very large flat files (~2 gigs each).  Each is an interface with ~ million records-transactions.  Some only need routing, but others will need initial transform then processing in BizTalk.  Frequency is several files daily.

      For those only needing routing, I’ve looked at Dunphy & Metwally’s Pro BizTalk 2006, Very Large File Pipeline component, that stores the file to disk with the path stored in the IBaseMessagePart.Data property.  Is this still a good way to handle, or is there something better now?

      For the interfaces requiring a transform, should the interface be read as a stream directly to the messagebox?  Then subscribe to it with either the orchestration containing the map (not my 1st choice for the overhead); or a subscribing send port containing the map?

      OR – will the receive port with a map have have a processing impact similar to the messagbox => subscribing orch or send port scenerio?  Or is there another approach that should be considered?

    • #23604
      • #23610

        Hi,

        I would like to share my experience with large message handling. But I don’t know it will go with your requirements. Anyway hope this will help

        We have to transfer 5000 files including 2 GB sizes. So the application was designed to support Canonical Data Model. For example the message properties like

        XYZ Message

           – GUID

          – Receive Filename

         – Payload

         File adaptor receives the message and custom pipeline component will take the message, create the canonical message then check the message size, if the massage size is less than 1 MB (configured value) then attached the message in to canonical message part payload section. If not stream the message into file location & save with the Guid and keep the message payload as empty. Anytime if the process wants s to track, uses the GUID as unique key.

        But this is good if we don’t need to process internal data in the message.

        The other thing that we have faced is ‘out of memory exception’ issue because we load XML Document in to the memory. To get rid of that we write to the message into the temporary file and using XMLReader  creates a new message.

        The other issue that we have faced that use XMLDocument messages inside the orchestration. After reviewing the code by Microsoft, recommended to use XLANGMessage .

         

        Best regards,

        Nandika

         

Viewing 1 reply thread
  • The forum ‘BizTalk 2004 – BizTalk 2010’ is closed to new topics and replies.